The local-vs-cloud cost argument is usually wrong

Every few weeks someone shows me a spreadsheet proving that a GPU in the server room is cheaper than an AI API. The spreadsheet is usually wrong, and it’s usually wrong in the same direction.

I sell both. On-prem deployments and cloud builds. So let me argue against the expensive one first.

A worked example

Take a 20-person team doing document Q&A over an internal knowledge base. Reasonable usage assumptions:

• 40 questions per person per working day
• 20 working days per month
• Each question retrieves a handful of chunks: roughly 8,000 input tokens
• Each answer is roughly 400 output tokens

That’s 16,000 queries a month — 128M input tokens and 6.4M output tokens.

At mid-tier frontier-model pricing (call it $3 per million input, $15 per million output — check current rates, they move), that’s:

Now the local side. A workstation with a single 48GB card, built or bought sensibly, lands somewhere in the $6,000–$12,000 range. Electricity at a bursty 150W average is about 110 kWh, or roughly $15/month. Call the hardware $8,000.

Breakeven is around 17 months — and that’s before anyone’s time.

The line items that never make it into the spreadsheet

The $8,000 is the honest part of the local estimate. What gets left out:

• Someone has to own it. Model updates, driver upgrades, a serving stack that changes fast, an OOM at 4pm on a Friday. Even at a few hours a month, that’s real money.
• Idle capacity is paid for. Cloud bills for 16,000 queries. The GPU bills for 730 hours whether you use it or not. Most internal tools have brutal usage curves — a spike at 9am, dead by 5pm, nothing on weekends.
• You’re buying today’s model. Capex assumes the hardware stays adequate. If the model you actually want in 18 months doesn’t fit in 48GB, the breakeven math resets.
• Cloud prices fall. Every comparison I’ve run has gotten worse for on-prem between when I built the estimate and when the client decided.

Push those through and the honest breakeven for a team this size is usually two years or more. For a team of five, it may never arrive.

So why do I still deploy on-prem?

Because cost was never the real argument. The reasons that actually hold up:

The data legally cannot leave. Privileged case files. Patient records under a BAA your provider won’t sign. Client financials under a contract that predates the existence of AI APIs. This isn’t a preference you can optimize around — it’s a constraint, and it ends the discussion.

Predictability beats efficiency. A fixed monthly cost you can put in a budget is worth more to some finance teams than a variable cost that’s lower on average. This is not irrational; it’s how capex budgets work.

Volume changes the shape. Everything above assumes 16,000 queries a month. At 16,000 a day, the arithmetic inverts hard and on-prem wins outright. High-volume classification, extraction and embedding workloads are where local pays for itself quickly — not chat.

Latency and availability floors. No egress, no rate limits, no vendor incident page. For an interactive tool used constantly by people who bill by the hour, the tail latency matters more than the mean.

No vendor policy risk. Terms change. Models get deprecated. Your fine-tune target disappears. If your workflow has a five-year horizon, that’s a genuine risk to price in.

What I actually recommend

Start on cloud APIs unless a constraint forbids it. You’ll learn what the workflow really needs in weeks instead of quarters, for a few hundred dollars, and the application code barely changes when you move.

Then move the workload on-prem when one of the real reasons shows up — a compliance constraint, a volume curve, a latency floor. Not because a spreadsheet said $480 was more than $15.

The uncomfortable version: if the only argument for on-prem is cost, you’re probably buying a GPU to solve a problem you don’t have yet.