Self-Hosted AI Models for privately operated model infrastructure.

Usually not, and the honest answer disappoints people who have already decided. Below roughly twenty thousand dollars a month in API spend, hosted APIs win on cost, quality and speed of delivery once you count engineering time properly. The break-even against a frontier API arrives at a few million tokens a day, and only if you keep GPU utilisation above about sixty percent. Against budget open-weight API providers charging well under a dollar per million tokens, self-hosting almost never wins on cost alone. Where it does win unconditionally is data residency, and that is a compliance decision rather than a financial one.

Three to five times the GPU rental figure that started the conversation, which is the number that catches teams out. On-demand H200-class capacity runs roughly three and a half to six dollars an hour depending on provider. On top of that sits engineering time — a production deployment typically consumes a fifth to a third of a senior engineer for monitoring, patching, model updates and incidents, which is several thousand dollars a month in salary. Then add idle capacity you pay for but do not use, redundancy if the service must stay up, and the depreciation or commitment cost of reserved hardware. Model the total, not the hourly rate.

For most narrow, well-defined tasks, yes. Current open-weight models sit within roughly five to ten percent of frontier hosted models on general benchmarks, and for classification, extraction, summarisation and routing that gap frequently has no practical effect on your output. The gap remains meaningful for complex multi-step reasoning, very long context, and reliable structured tool use. The only reliable way to know is to run both against your own evaluation set with your own inputs — public benchmarks predict product performance poorly, and a smaller model tuned to a narrow task often beats a larger general one.

It depends far more on model size and concurrency than on any general rule. A small quantised model serving a handful of concurrent users can run on a single mid-range GPU or even capable CPU hardware. A large model serving production traffic at low latency needs high-memory data-centre GPUs, and often several. The sizing exercise runs backwards from your requirements: model size, context length, concurrent requests and acceptable latency determine memory and throughput, which determine hardware. Quantisation frequently lets a smaller footprint carry the same workload, and should be measured rather than assumed.

Yes, and it is almost always the right sequence. Building against an API first means you find out whether the feature is useful before spending anything on infrastructure, and you accumulate real token consumption data to model the decision with. Keeping a provider boundary in your own code — one interface, swappable implementations — makes the later migration a configuration change rather than a rewrite. You also keep the API baseline to compare quality against, which is exactly what you need when evaluating whether the open model is genuinely good enough.

Open weights are not uniformly open, and this needs legal review rather than assumption. Some models carry genuinely permissive licences. Others impose usage thresholds above which separate terms apply, restrict specific fields of use, require attribution, or place conditions on training derivative models. Terms also change between model versions from the same publisher. Before a model reaches production, someone should read the actual licence for the specific version you intend to deploy and confirm it permits your commercial use — retrofitting that check after launch is considerably more awkward.

The model server should be treated as internal infrastructure, not a public endpoint — network isolation, authenticated access, rate limiting and request logging. Note also that self-hosting removes the safety filtering a hosted provider applies, so content filtering, prompt injection defences and output validation become your responsibility to build and test. In air-gapped environments there is an additional problem: automatic model and dependency updates are unavailable, so a documented manual patching procedure has to exist before launch rather than being improvised after a disclosure.

It means your data stays on infrastructure you control, which is the important part and is genuinely different from a contractual promise. It does not automatically mean compliant. Access control, logging, retention, encryption at rest and in transit, and how prompts and outputs are stored all remain your responsibility, and those are usually what an auditor actually examines. Self-hosting removes one category of risk — third-party processing — and hands you the rest. That is the right trade for regulated work, but it is a trade rather than a solution.

Bulk or sensitive traffic served by a local open-weight model, and complex or low-volume requests routed to a hosted frontier API, both behind a single interface in your application. It is common because it matches how workloads actually distribute: most requests are routine and cheap to serve locally, while a minority genuinely need frontier capability. It also means the routing split is an operational setting rather than an architectural commitment, so it can be adjusted as volumes, model quality and pricing change without touching the product.

By measuring cost per successful task rather than cost per token, before and after. A cheaper model that fails more often, triggers retries, or requires human correction is not cheaper in any way that matters. The comparison needs the same evaluation set run against both routes, plus operational metrics — latency percentiles, GPU utilisation, incident frequency and engineering hours spent — carried on the total. Teams that only track the infrastructure invoice frequently conclude self-hosting saved money while the real total went up.

An open-weight model — one whose parameters are published for download, such as Llama, Qwen, Mistral or DeepSeek — running on hardware you control rather than being called over a commercial API. You download the weights, serve them through an inference engine, and every request stays inside your network. The distinction from open source is worth knowing: open weights mean the trained parameters are downloadable, not necessarily that the training data or code are public, and the licences attached to them vary considerably in what commercial use they permit.

The smallest one that passes your evaluation, which is almost never the largest one available. Teams instinctively reach for the biggest model they can fit and then discover it is slow, expensive and no better at their actual task. Current open-weight families — Llama, Qwen, Mistral, DeepSeek and their smaller variants — all have strong options, and for classification, extraction, summarisation or routing a small model frequently matches a large one. Build an evaluation set from your real inputs and test candidates against it; a leaderboard position predicts very little about your workload.

Yes, and it is the sensible way to learn what the hardware requirements actually are. Ollama or LM Studio on a machine with a modern GPU will run useful mid-sized models comfortably, and quantised smaller models run acceptably on capable CPUs or Apple Silicon without any dedicated GPU. What a home setup will not give you is production characteristics — concurrent users, sustained throughput, uptime or latency guarantees. It is an excellent evaluation environment and a poor production one, and confusing the two is how projects get mis-scoped.

Text generation models across a wide size range, embedding models for search and retrieval, speech-to-text models for transcription, and image generation models. The practical limit is memory: model size in billions of parameters roughly translates to memory needed, and quantisation reduces that substantially at some quality cost. A modern consumer GPU handles small and mid-sized text models well. Larger models need data-centre hardware or aggressive quantisation. Multimodal models that handle images and text together have heavier requirements again.

The software is free — models, inference engines and interfaces all carry no licence cost for most uses. The hardware and electricity are not. On equipment you already own, running a model locally costs you power and time, which for personal use is effectively free. For anything serving other people, you are paying for GPU capacity whether you rent or buy, plus the engineering hours to keep it running. Free software on paid infrastructure is the accurate way to describe it.

AI hosting usually means a managed service that runs open-weight models for you — you choose a model, they handle the GPUs, scaling and uptime, and you pay per token or per hour. It sits between calling a frontier API and running your own infrastructure: you get model choice and often better pricing than frontier providers, without operating anything. It does not give you data residency in the way self-hosting does, since the compute is still someone else's, so it solves the cost question but not the compliance one.