Ollama, vLLM, or llama.cpp: picking a server you won't have to replace

The question always arrives in the same shape: “We’ve got the GPU. Which one do we run — Ollama, vLLM, or llama.cpp?”

Usually the person asking has read three blog posts that each declared a different winner, which is a reasonable outcome, because all three were probably right about their own situation. These projects are not competing to be the best inference server. They’re three different bets about how many people will be using the thing at once.

Once you see it that way the choice mostly makes itself.

They agree about more than you’d think

All three load a transformer, run attention over a KV cache, and stream tokens back over an OpenAI-compatible /v1/chat/completions endpoint. Point your app at any of them and it works. Ollama was built on llama.cpp and still shares much of its lineage. On one GPU, one user, one request at a time, with a comparable quantization, they will feel about the same — and llama.cpp or Ollama may well feel faster, because they carry less scheduling machinery.

That last point is worth sitting with, because it’s the source of most bad advice on this topic. Someone benchmarks single-stream generation on their laptop, finds vLLM unimpressive, and writes it up. The benchmark isn’t wrong. It’s just measuring the one regime where vLLM’s entire design contributes nothing.

The three diverge under load, and only under load.

What each one is actually optimizing

Ollama optimizes time-to-first-run. ollama run qwen3 and you have a model. It handles downloads, storage, model switching, and a background service that unloads idle models for you. Nothing else in this space respects your afternoon that much.

llama.cpp optimizes running anywhere. GGUF weights, a quantization ladder from Q2 to Q8, CPU inference, partial GPU offload, Metal, CUDA, ROCm, Vulkan. It is the only one of the three that runs a model bigger than your VRAM — you offload as many layers as fit and the rest execute on CPU. Slowly, but it runs, and “slow” beats “won’t load.”

vLLM optimizes tokens per second per dollar at concurrency. PagedAttention and continuous batching exist to keep a GPU saturated while many people talk to it. It wants the whole model resident in VRAM, in FP16 or a GPU-native quantization (AWQ, GPTQ, FP8), and it wants a real CUDA environment. In exchange it serves a crowd on hardware that would otherwise serve a handful.

The structural difference: how the KV cache gets divided

Sizing a GPU comes down to the KV cache — the weights are the easy part, and the cache is what actually runs out. The serving layer decides how efficiently you spend that budget, and this is where the three genuinely differ.

llama.cpp’s server takes a --parallel N flag that splits the context window into N slots. Ollama exposes the same idea through OLLAMA_NUM_PARALLEL. The division is static: give the server 32K of context and ask for 8 slots, and every slot gets 4K, whether or not anyone needs it. One user pasting a long document hits the wall at 4K while seven idle slots sit on context nobody is using.

vLLM allocates the cache in pages instead. A sequence takes blocks as it grows and returns them when it finishes, so one user can hold 20K while seven others use 1K each. Add continuous batching — new requests join the running batch at the next iteration rather than waiting for the current one to drain — and the practical ceiling on concurrent sessions moves by a large multiple on identical hardware.

That’s the whole argument. Not “vLLM is faster.” vLLM wastes less of a fixed KV budget when demand is uneven, and real demand is always uneven.

To put rough shape on it (measure your own — these move with model, quantization and context, and I’d distrust any table like this that didn’t say so):

RAG changes the calculation

If you’re serving retrieval-augmented answers rather than chat, the workload looks different in a way that matters here. A RAG request sends eight retrieved passages and a system prompt — several thousand input tokens — and generates a couple hundred back. It is prefill-heavy, and prefill is compute-bound and batches beautifully.

It’s also repetitive. Every request in a RAG system shares the same grounding instructions, and many share retrieved passages. vLLM’s prefix caching reuses the computed KV for a shared prefix instead of recomputing it per request, which on this workload is close to free throughput.

So the concurrency threshold where vLLM starts to matter arrives earlier for RAG than for chat. A ten-person internal document assistant is already in vLLM territory, even though ten people sounds small.

The switching cost nobody prices in

The standard advice — “start with Ollama, move to vLLM when you outgrow it” — is mostly right, and I give it myself. But it gets sold as a free move because the API is compatible, and the API was never the expensive part.

What actually costs you:

• The weights. GGUF doesn’t run on vLLM in any way you want in production. You re-acquire the model in AWQ, GPTQ or FP8. Different file, different quantization math, different quality profile.
• The quality baseline. Q4_K_M and AWQ 4-bit are not the same model. Outputs shift — usually slightly, occasionally on exactly the prompt your client cares about. If you have no evals you won’t find out; you’ll just get a report that “it got worse” with nothing to compare against.
• Sampling defaults. Each project ships different defaults for temperature, top-p, repeat penalty. Move a prompt tuned against one set to another and behavior changes for reasons that have nothing to do with the model.
• Operations. Ollama is a service that stays up. vLLM is a Python process with opinions about CUDA versions, a long startup, and pre-allocated GPU memory. Someone now owns that.

None of this is a reason to start with vLLM. It’s a reason to keep an eval set from day one, so the migration is a measurement instead of an argument. That part is free, and skipping it is the mistake I see most.

What I’d actually tell you to run

One caveat on Ollama that costs people more time than anything else on this page: check your context length. The default is small, and a long RAG prompt silently gets truncated rather than erroring. The model then answers from the fragment that survived, confidently, and the retrieval layer takes the blame for a limit that was set in the server config.

The short version

Pick by concurrency, not by benchmark. One user: any of them, choose on convenience. A handful of users on awkward hardware: llama.cpp. A team, a dedicated GPU, and a system people depend on: vLLM, and it isn’t close.

Then keep an eval set from the first week, because the only cheap migration is one you can measure. The server you started on is rarely the server you finish on — and that’s fine, as long as you can prove the move didn’t cost you anything.

The demo on this site runs the cloud path rather than a self-hosted server, but the retrieval shape is identical: eight passages, a grounding prompt, a citation on every claim. If you’re sizing a box to run that in-house, that’s a conversation worth having before you buy the GPU, not after.