Not a setting: why we removed Ollama from our app

Return is a document analysis tool built for lawyers. The people it serves work under privilege, and the first question every one of them asks about an AI tool is the same: where does the document go?

There are two ways to answer that question. The first is “there’s a setting that keeps everything local, and we keep it set.” The second is “the engine analyzing your document has no code that could send it anywhere.” These sound similar. To a security auditor, they are different categories of claim. This post is about moving from the first answer to the second, and what it cost us.

What changed in Ollama 0.12

Until mid-2026, Return’s free tier ran on Ollama. It was a good choice for a long time: a well-maintained local inference server, a clean HTTP API on localhost, a model catalogue our users could draw from.

In September 2025, Ollama 0.12 introduced cloud models. The design is genuinely elegant. A model with a -cloud suffix behaves like any other model: you can list it, run it, point your existing tools at it. Your application keeps sending requests to localhost:11434 exactly as before. The daemon detects the suffix, attaches your account credentials, and proxies the request to Ollama’s datacenter infrastructure. From the client’s perspective, nothing changes.

To be fair about the details: cloud models are opt-in. They require signing in to an Ollama account, and they only run when someone explicitly selects one. Ollama later added a disable_ollama_cloud setting and an OLLAMA_NO_CLOUD environment variable after users asked for a way to enforce local-only operation. Nobody’s document gets uploaded by surprise. Ollama did nothing wrong here, and for plenty of products the hybrid model is exactly right.

Why it broke our architecture anyway

Our problem was narrower and more structural. Return’s core promise to lawyers is “Nothing leaked,” and we had been backing that promise with a specific, verifiable statement: run a packet capture while the free tier analyzes a document, and you will see traffic to 127.0.0.1 and nothing else.

After 0.12, that statement was still technically true from our process’s point of view. It just no longer meant what it used to. Traffic to localhost:11434 could now be the first hop rather than the destination. Whether inference stayed on the machine had become a function of things outside our binary: which model names existed in the daemon’s list, whether an account token was present, how the daemon was configured, and how that configuration surface might evolve in future releases.

A user on Ollama’s issue tracker put the general worry well when requesting the local-only toggle: even without malice, a misconfiguration or another piece of software could introduce a token or a cloud model without your knowledge. For an individual running Ollama on their own machine, the toggle answers that worry. For a product making a load-bearing confidentiality promise to law firms, it doesn’t, because the promise now rests on a flag we set rather than a property an auditor can verify. A GDPR Article 25 review asks what the system is capable of by design, not what its settings happen to say today. And a flag cedes control of our central claim to a third party’s release schedule. That’s not a criticism of Ollama’s choices. It’s an admission that our requirements had become unusually rigid.

From the decision record we wrote at the time: the inability to exfiltrate document content should be a property of the code we ship, not a setting.

What we did

We replaced Ollama with a bundled llama-server from llama.cpp, run as a sidecar process. llama-server has no code that talks to remote LLM providers. There is nothing to configure, because there is nothing to disable.

The sidecar binds to 127.0.0.1 on a port the operating system assigns at spawn: we bind a loopback socket to port zero, read the port it was given, release it, and pass that port to the server explicitly. Never a fixed well-known port, never 0.0.0.0. Chat and embeddings run as two separate sidecars, because a llama-server started in embeddings mode cannot also serve chat, and flipping one process between modes would add model-swap latency to every switch between chatting and searching. Both spawn lazily, so a cold app launch stays cheap.

And we deleted Ollama from the codebase entirely. No feature flag, no dormant OllamaEngine kept around just in case. The migration happened before launch, with no production installs to migrate, so the rollback path was git revert. Keeping two engine implementations behind a flag would have been permanent debt purchased as insurance against a scenario that version control already covers.

What it cost us

Honesty requires the other side of the ledger, because this trade was not free.

The biggest loss was model management. Ollama is a full application: it pulls models, stores them, lists them, loads them on demand. When we embedded it, all of that lived outside our app, which also meant our users had to install and operate a second program before Return’s free tier did anything. Removing Ollama meant a fresh install of Return now has no model at all, and the entire path from zero to working AI became ours to own. We ended up building a models view into Settings: a curated catalogue that ships inside the app and renders offline, plus a Hugging Face browser that reads GGUF metadata over HTTP range requests, so you can check a model’s size and parameters before committing to a multi-gigabyte download. That was real engineering time we hadn’t planned to spend. It also produced a better experience than what it replaced: nobody has to install a separate tool or open a terminal to get local AI running.

Model switching became a heavier operation on paper. Loading a different chat model now restarts the whole sidecar rather than swapping weights inside a running daemon, and we budgeted for a visible loading state to cover the gap. In practice the regression never showed up: load time is dominated by reading the weights from disk either way, and on the Apple Silicon hardware we develop on the restart is quick enough that we never built that loading state. Slower machines may feel it more, and if they do, the fix is obvious.

Memory went up. A second resident process for embeddings costs roughly 500 to 700 MB with our default model quantized, which lazy spawning mitigates but doesn’t eliminate.

And macOS notarization got meaningfully harder. Bundling a native binary with its Metal libraries means hardened-runtime signing with entitlements like allow-jit and disable-library-validation, and the first notarization run of new nested binaries took hours. We learned to smoke-test that pipeline on a minimal app well before release day.

What we got

The answer to the auditor’s question changed category. Return’s free tier is local because the bundled engine contains no path to a remote model provider. A packet capture during local inference shows loopback traffic and nothing else, and that will stay true regardless of what any upstream project ships next quarter.

Cloud AI still exists in Return. The paid tier can route requests to a frontier model, under a data processing agreement, through our own proxy. But that path is a separate, explicit choice with its own permission gate. It is never a default, never a fallback, and never something a configuration drift can switch on. Free means provably local. Paid means deliberately cloud. Two different products, instead of one product with a gradient of trust.

Ollama remains good software, and embedding it is the right call for most tools in this space. Our constraint is just unusual: when the premise of a product is that a specific thing cannot happen without the user’s explicit say-so, “cannot” has to live in the architecture. Anything that lives in a setting is a promise. Settings change.