We run production AI for multiple clients on a single 32GB GPU. No cluster, no cloud, one machine. People assume self-hosted AI means a rack of hardware and a data-centre bill. It does not. But it does mean being disciplined about video memory in a way cloud users never have to think about. Here is what the VRAM budget actually looks like.
The VRAM budget, line by line
- The main LLM: about 22GB. A 30-billion-parameter model at 4-bit quantisation. It stays loaded permanently, because reloading 22GB takes 30 to 40 seconds and you do not want that on the critical path.
- The embedding model: about 300MB. Runs alongside the LLM.
- The reranker: about 1.5GB. Also resident.
That is roughly 24GB committed. The remaining 8GB is KV cache, the model’s working memory while it generates, plus CUDA runtime headroom. It works. But it is tight, and the discipline is the point.
Two lessons nobody tells you
1. Quantisation quality is model-dependent
We tested one popular model at 4-bit that scored 30% lower on our evaluation suite than the same model at 8-bit. Some model families lose almost nothing when you quantise them. Others fall off a cliff. There is no shortcut here: benchmark both, on your actual task, before you commit.
2. Ollama’s keep_alive setting will ruin your day
Set it to anything other than permanent and Ollama evicts the model after a timeout. The next request then triggers that 30-to-40-second reload, and your users think the system has hung. Set it to -1 and forget about it. We lost hours debugging phantom timeouts before we traced it to this one setting.
The economics are the real story
Self-hosted AI in 2026 is viable on modest hardware for B2B workloads. A single 32GB GPU comfortably handles everything a team of ten to twenty users needs. The economics are compelling, a one-off hardware cost then close to nothing to run, but only if you are disciplined about VRAM in a way cloud users are insulated from.
This is the engineering behind our self-hosted systems, and it is why Cortex can run entirely on a client’s own hardware with no external AI calls. We are engineers first, and this is exactly the kind of constraint we enjoy designing around.
So I will turn it over: what is the tightest constraint you have had to design around in your own AI stack?