When self-hosting is the wrong answer

Two things send people towards private deployment. Either the data cannot leave the network, or the bill for calling a hosted API has stopped making sense.

Only the first is a decision that has already been made for you. If a contract or a regulator says the data stays inside, self-hosting is not a choice, and the only open question is which model fits the hardware you have. We worked with a US healthcare provider in that position. Sending patient records to a hosted endpoint was never on the table, so nobody spent an afternoon debating it.

The second reason is where the bad decisions get made.

The calculation most people run

It usually looks like this. Somebody exports a month of API spend, compares it against the cost of a machine that could run an open-weight model, and finds the machine wins comfortably.

On that comparison it nearly always does. The comparison is also wrong, because the machine was never the expensive part.

What the spreadsheet leaves out

Someone has to choose the model. That means testing candidates against your actual task rather than reading a leaderboard, and discarding the ones that score well but will not fit in the memory you have. A model that benchmarks beautifully and refuses to load is not a candidate.

Someone has to decide how far to quantise. Every step down in precision costs something, and where it costs you depends entirely on the task. A summarisation job tolerates compression that a structured-extraction job does not. The only way to find out is to test one variable at a time against your own documents and your own questions.

Someone has to serve it. Memory behaviour when several requests arrive together. What happens when demand exceeds capacity. Monitoring that notices the model degrading before a user reports it.

And someone has to keep doing that. Open-weight models improve every few months, so the model you deployed in March is not the one you would pick in September, and swapping it means running the benchmark again.

None of this is difficult. All of it is time, and time is the line item that never makes it onto the comparison.

“The data cannot leave” is worth testing

This gets stated more absolutely than it usually is.

Sometimes it is exactly as absolute as it sounds, and there is no route to a hosted API at any price. Often it is a policy written before data processing agreements were common, or a blanket reading of a rule that has an exception in it. Occasionally it is one person’s assumption that nobody has gone back to check.

It costs an hour to find out which of those you have. That hour is worth spending before anyone specifies hardware, because the answer changes the entire shape of the project.

When self-hosting clearly wins

When the data genuinely cannot leave. That one is settled before the conversation starts.

When volume is high and the task is narrow. An open-weight model on a well-specified job is often indistinguishable from a frontier model, and at sustained volume the economics stop being close.

When you need the output to be reproducible. A hosted model can change underneath you without notice, and for anything that has to produce the same result twice, that is a real risk rather than a theoretical one.

When it does not

If your volume is modest, a hosted API under a data processing agreement will cost a fraction of dedicated infrastructure and take an afternoon rather than a fortnight.

If your task needs broad reasoning, an open-weight model you can run will be behind, and no amount of engineering around it closes that gap.

If nobody on your team wants to own it, the system will drift. Somebody has to care about it in month eight, when the model everyone was excited about has been superseded twice.

What we say to clients

We say the second half of this out loud, including when it costs us the larger project. A private deployment that should have been an API call is a bad system, and it stays somebody’s problem long after the invoice is settled.

The question worth asking first is not whether you can run the model yourself. It is whether anything about your situation actually requires that you do.