On-Prem LLM Inference Optimization: Throughput, Latency and VRAM with vLLM (2026 Guide)

Bottom line first: once the model is chosen, inference engineering decides everything

In August 2026, the enterprise AI battleground has shifted from “which model” to “how to run the model fast and cheap on our own servers” — the Model Migration playbook covers switching to open weights; this article covers how to deploy once you have switched.

The same Qwen3-32B: one team serves dozens of users on a single 4090; another stutters on four A100s. The gap is not the model — it is inference engineering. This guide walks through five things in production order: engine selection, VRAM and quantization, throughput optimization, latency optimization, and capacity sizing with monitoring.

---

1. Inference engine selection: pick the right tool first

Engine	Scenario	Throughput	Latency	Ecosystem	Verdict
vLLM	Production multi-user	Highest (PagedAttention + batching)	Good	Largest, official support	Production default
SGLang	Latency-sensitive / complex prefixes	High	Better (RadixAttention)	Newer, fewer resources	Specialist
llama.cpp/Ollama	Personal / dev / no-GPU machines	Low	OK	Good (GGUF ecosystem)	Experiments
TensorRT-LLM	Extreme latency tuning	High	Best	NVIDIA-locked	Heavy tuning

Rule of thumb: production multi-user → vLLM; multi-turn / agent workloads with heavy prefix reuse and TTFT sensitivity → try SGLang; personal dev and Mac → Ollama; squeezing extreme latency on NVIDIA-only → TensorRT-LLM.

vLLM is the default for three reasons: OpenAI-compatible API out of the box (zero-change migration on the business side), highest throughput via PagedAttention + Continuous Batching, and official support for Qwen/DeepSeek/Llama with the largest community. Having somewhere to ask when things break is the scarcest resource in production systems.

---

2. VRAM budgeting and quantization: get the math right first

The VRAM formula (put it in your deployment doc)

VRAM needed = model weights + KV Cache + runtime overhead (~15%)

Weights = parameters × bytes per parameter
  FP16/BF16: ~2GB per 1B parameters
  INT8/FP8:  ~1GB per 1B parameters
  INT4 (AWQ/GPTQ): ~0.5GB per 1B parameters

KV Cache (per request) ≈ 2 × layers × KV heads × head dim × bytes/token × context tokens

Heuristics: Qwen3-14B 4-bit ≈ 8-9GB weights, runs on a 4090 (24GB); Qwen3-32B 4-bit ≈ 16-18GB, needs ~48GB-class (two 4090s or one A100/H20); full-precision 32B ≈ 64GB weights, needs ~80GB-class. Add ~1-2GB KV Cache per concurrent session per 8K context — at 30 concurrency that is 30-60GB. The VRAM hog is often the KV Cache, not the weights.

Choosing a quantization

Quantization	Quality loss	VRAM	Speed	Best for
FP16/BF16	None	Baseline	Baseline	Ample VRAM, quality first
FP8	Minimal	Half	Fast	New GPUs (H20/4090 support well)
AWQ 4-bit	Small	-75%	Fast	Mainstream production choice
GPTQ 4-bit	Small	-75%	Fast	Same; AWQ slightly more stable
GGUF (llama.cpp)	Small	Flexible	Medium	Ollama / low-VRAM experiments

Production advice: start with AWQ 4-bit (quality loss under 2% on most commercial tasks in exchange for 75% less VRAM) and lock it in once your eval baseline passes; use FP8 or FP16 for quality-sensitive workloads (long-chain reasoning, code generation). After quantization you must re-test with your business eval set — some tasks are extremely quantization-sensitive, so never decide by leaderboard impressions (exactly what the evaluation series says: everything is judged by your eval).

Before touching hardware, run our LLM sizing calculator — enter model/quantization/concurrency/context and get VRAM, GPU options and ready-to-use vLLM launch flags.

---

3. Throughput optimization: make one GPU do the work of three

1. Continuous Batching — the throughput foundation

Traditional batching waits for a batch to fill before computing; long requests block short ones. Continuous Batching lets new requests join an in-flight batch at any time — short requests leave first, long ones grind on, and GPU utilization rises significantly. vLLM enables it by default — but when VRAM is tight, effective concurrency is capped by the KV Cache limit, showing up as queuing/OOM at high concurrency. Get the VRAM math right (Section 2) first, then batching has room to work.

2. PagedAttention — vLLM’s signature

KV Cache is managed in pages, like OS paging: memory is allocated only for what is actually used, cutting fragmentation and waste, letting far more concurrent requests fit on one card. This is the core of vLLM’s throughput lead. No configuration needed — but understand it: the KV Cache ceiling is your concurrency ceiling.

3. Prefix Caching — the 90% waste nobody notices

In multi-turn dialogue and RAG, system prompts and document prefixes are recomputed on every request. Prefix Caching stores the shared prefix’s KV and reuses it on hit. In RAG scenarios hit rates of 70-90% are common — that is 70-90% of input compute for free. Three config points:

• vLLM enables it by default (enable_prefix_caching=true); older versions need it turned on explicitly;
• Prompt structure “static first, dynamic last”: fixed content at the prefix, varying content after it, to maximize hits;
• Monitor Prefix Cache hit rate — low hit rate means the prompt structure needs work.

4. max_num_seqs and memory utilization — the two key knobs

• --gpu-memory-utilization: default 0.9. Lower it if the same GPU also runs an embedding or reranker, to avoid OOM;
• --max-num-seqs: caps effective concurrency. Too low wastes throughput; too high triggers KV eviction (frequent page swaps slow things down instead).

---

4. Latency optimization: users only have 500ms of patience

Align on metrics first (detailed definitions in Capacity Planning & Performance Testing)

Metric	Meaning	Healthy target
TTFT	Time to first token	P99 < 1s; < 500ms with streaming
TPOT	Time per output token	20-60ms (15-50 tokens/s)
Aggregate throughput	Whole-GPU tokens/s	Higher is better, hardware-bound

Watch P99, never the average — averages are pulled down by the majority of fast requests and hide the long tail, and users feel the worst request, not the typical one.

The three-move optimization play

1. Streaming (SSE): push tokens as they are generated; the user sees text the moment TTFT lands — perceived latency halves at zero cost;
2. Prefix hits: enable Prefix Caching for RAG/multi-turn; TTFT drops sharply on hit (Section 3);
3. Speculative Decoding: a small draft model proposes, the big model verifies; generation speed up 1.5-3×, best for 70B-class models that are latency-sensitive with spare compute. Supported inside vLLM — measure the gain before enabling; it is not worth it for every workload.

---

5. Capacity sizing: work backward from target QPS

Target QPS → concurrency = QPS × P95 response seconds × peak factor (usually 1.5-2)
concurrency × KV Cache per request → KV VRAM
weights VRAM + KV VRAM + 15% overhead → total VRAM → number of GPUs

Example: target 10 QPS, 3s average response, peak factor 2 → ~60 concurrency. 60 sessions × ~1.5GB per 8K context → ~90GB KV; Qwen3-32B 4-bit ≈ 18GB weights → ~120GB total → two A100 80G or three H20 96G-class. Do the math before buying GPUs — do not discover 20% utilization after purchase.

Golden rule: test before you buy. Validate capacity under real load (3-5× peak QPS recommended); see Capacity Planning & Performance Testing for the method.

---

Post-launch monitoring and the cost ledger

Six numbers to watch

1. P99 TTFT / TPOT — latency health;
2. Prefix Cache hit rate — is the prompt structure sound;
3. KV Cache utilization — is VRAM the concurrency bottleneck;
4. GPU utilization — is the expensive compute actually used;
5. Queue length / timeout rate — capacity warning signals;
6. Per-request cost (per model) — cost attribution continues after self-hosting; hook into the cost attribution dashboard.

Three pitfalls that sink deployments

1. Buying GPUs without load testing — sized by “how big is the model”, forgetting KV Cache concurrency needs; bottlenecked on day one;
2. Quantization without evaluation — shipped 4-bit directly, quality collapses silently until complaints explode. The same business eval set must be run before and after quantization;
3. Deploy without monitoring — “it is running, so it is fine”: KV eviction, GPU card drops, and collapsing prefix hit rates go unnoticed. Inference monitoring matters as much as business monitoring.

---

Deployment checklist (start today)

Step	What	Output	Time
1. VRAM math	Sizing calculator with model/quantization/concurrency/context	VRAM & GPU plan	0.5 day
2. Eval baseline	200+ real samples; score before/after quantization	Quantization pass/fail	2-3 days
3. Engine up	vLLM with gpu_memory_utilization / max_num_seqs	Callable inference service	1 day
4. Prefix caching	enable_prefix_caching + “static first, dynamic last” prompts	Hit rate > 50%	0.5 day
5. Load test	3-5× peak QPS; watch P99 and throughput	Concurrency capacity table	1-2 days
6. Monitoring	Six metrics into monitoring with alerts	Continuously observable	1 day

---

Further reading:

• Model Migration Playbook — why to migrate, when, and how: this article is the “how to deploy after migrating” second half
• LLM Model Selection & Routing — with mixed self-hosted + API deployment, how to make every request take the right model
• Enterprise AI ROI Estimation — before buying hardware, run the self-hosting vs API math
• Cloud Cost Optimization — broader compute bill governance
• Capacity Planning & Performance Testing — load testing methods and capacity models
• Fine-tuning vs RAG: How to Choose & Combine — fine-tuned artifacts are a new class of self-hosted workload: how post-training eval and VRAM budgeting connect
• Enterprise AI Compliance & Risk Management — data-sovereign deployment paths: private, local inference, federated learning — compliance is a hard constraint on deployment decisions

Model migration cuts the cost in half; inference engineering cuts the remaining half — and both steps hinge on the same precondition: eval baselines and capacity validation. Do the math first, then act.

We deliver the full on-prem LLM deployment chain: VRAM and hardware sizing, vLLM/SGLang deployment and tuning, quantization with eval baselines, load testing and capacity planning, and inference monitoring with alerting. If you are evaluating “how to deploy the open model we are switching to”, bring your model, concurrency target and GPU budget — we will give you a capacity and VRAM plan first, then talk implementation.