How to work out what one GPU can actually run

The question I get most often, in some form: “We’ve got one card with N gigabytes. What can we run on it?”

Nobody can answer that from the model name alone, and the benchmark tables published alongside model releases won’t help — they’re measured on hardware you don’t have, at a batch size and context length you won’t use. But the arithmetic is simple enough to do on a napkin, and it will tell you the answer before you spend anything.

There are three consumers of VRAM. Most people only budget for the first one.

1. Weights

The easy part. Parameters × bytes per parameter:

4-bit isn’t exactly 0.5 bytes — quantization schemes keep some tensors at higher precision and carry scale factors — so budget 10–15% above the naive number.

Mixture-of-experts models break the intuition here. An MoE model advertises a large total parameter count but activates only a fraction per token. That helps your compute and throughput enormously. It does not help your VRAM: you still have to hold all the experts in memory, because you don’t know in advance which ones the next token needs. An MoE with a small active-parameter count is fast on modest hardware, not small on it.

2. KV cache — this is the one that gets you

Every token in the context window leaves behind cached keys and values for every layer. Roughly:

kv_bytes ≈ 2 × layers × kv_heads × head_dim × bytes_per_elem × seq_len × batch

The 2 is keys plus values. Two details matter enormously:

• kv_heads, not attention heads. Models using grouped-query attention share KV heads across query heads, often 4–8× fewer. This is the single biggest lever on cache size, and it varies wildly between models with identical parameter counts.
• It scales with seq_len × batch. Linearly, in both. Ten concurrent users at 32k context cost the same as one user at 320k.

This is why a model that loads fine sits there OOM-ing under load. The weights fit. The weights always fit. What didn’t fit was twelve people using it at once with long documents in the prompt — which, for a RAG system, is every single request.

Quantizing the KV cache to 8-bit halves it and is usually a much better trade than quantizing weights further.

3. Everything else

Activations, CUDA graphs, fragmentation, the serving framework’s own overhead. Leave 10–15% of the card free. Serving stacks that pre-allocate a KV pool will happily take everything you offer and then fail on the first long request.

Doing it in practice

Work backwards from the workload, not forwards from the model:

1. Decide your real context length. Not the model’s maximum — yours. For most RAG systems it’s 8k–32k, set by how many chunks you retrieve, not by ambition.
2. Decide your real concurrency. Peak simultaneous in-flight requests, which for an internal tool used by 50 people is usually closer to 5 than 50.
3. Compute the KV budget for those two numbers.
4. Whatever’s left, minus 15% headroom, is your weight budget. Now pick a model and a quantization that fits in it.

Teams that go in the other direction pick the largest model that loads, discover the cache has nowhere to live, and conclude they need a bigger card. Often they needed a smaller model and 8-bit KV.

Don’t forget the small models

A retrieval system isn’t just a chat model. An embedding model and a reranker have to live somewhere too — that’s typically a few GB more, and they’re in the hot path of every query. If they’re sharing the card with the generation model, budget them explicitly rather than discovering them at deploy time.

The short version

Take your card’s VRAM. Subtract 15% headroom. Subtract the KV cache for your actual context length times your actual concurrency. Subtract your embedding model and reranker. What remains is what you can spend on weights.

Run that calculation before you buy hardware, not after. It’s the difference between a deployment that gets quietly abandoned and one that people keep using.