The vocabulary of serving AI in production
vLLM, PagedAttention, KV cache, speculative decoding, prompt caching. Every technique for making LLMs fast, cheap, and reliable in production.
Training gets the headlines, but inference is where the money is spent. A frontier model may cost tens of millions to train once — and then generate billions of inference requests, each costing compute. The techniques in this category — vLLM's PagedAttention, KV cache management, quantization, speculative decoding, prompt caching — are what make modern LLM APIs economically viable.
The 70 terms here are grouped into four sub-topics: inference engines (the software that serves models), optimization techniques (KV caching, batching, speculative decoding), model compression (quantization, distillation, pruning), and the operational vocabulary you'll encounter running LLMs in production.
The most important terms in Inference & Serving
Start here if you're new. These entries explain the foundational vocabulary in depth.
vLLM
The most popular open-source LLM inference engine. Introduced PagedAttention and now the standard for high-throughput self-hosted serving.
Read the full entry →OptimizationPagedAttention
vLLM's technique that manages KV cache like OS virtual memory pages. Dramatically improves memory efficiency and enables large batches.
Read the full entry →OptimizationKV Cache
Cached key-value pairs from attention computation. Reused across generation steps so the model doesn't recompute for tokens it has already processed.
Read the full entry →CompressionQuantization
Reducing numeric precision (fp16 → int8 → int4) to shrink model size and speed up inference. Modern methods (GPTQ, AWQ) achieve near-lossless int4 quantization.
Read the full entry →OptimizationSpeculative Decoding
A small "draft" model proposes tokens that the target model verifies in parallel. When verification succeeds, multiple tokens are accepted at once — often 2-3x speedup.
Read the full entry →OptimizationPrompt Caching
Caching the KV state of common prompt prefixes. Turns the second request for the same system prompt into near-instant response.
Read the full entry →All Inference & Serving terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
Inference engines
Optimization techniques
Model compression
Operations & serving
Related term categories
These categories connect naturally to Inference & Serving — many terms cross-reference between them.
Go deeper on Inference & Serving
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- How vLLM Works
PagedAttention and continuous batching explained.
- Understanding KV Cache
The optimization at the heart of LLM speed.
- LLM Inference Optimization Guide
End-to-end walkthrough.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- Groq
Custom LPU chips for fast inference.
- Cerebras
Wafer-scale inference services.
- Inference tools
vLLM, SGLang, Ollama, and more.
Questions about Inference & Serving
KV cache management. Serving LLMs is dominated by KV cache memory pressure — PagedAttention (vLLM), prompt caching (Anthropic/OpenAI), and grouped-query attention are all essentially KV cache optimizations.
Usually no. Providers (Anthropic, OpenAI, Together, Groq) have solved most of this. Self-hosting makes sense for latency-critical apps, privacy needs, or fine-tuned models. See our comparisons.
Modern INT4 quantization (GPTQ, AWQ) has minimal quality loss on most tasks — usually within 1-2% of full precision on standard benchmarks. Edge cases like reasoning and long-context can suffer more.
Roughly, yes. Different providers use different names. Anthropic calls it 'prompt caching,' Google calls it 'context caching.' Same idea: cache the KV state of a common prefix.
Throughput is total requests or tokens per second across all users. Latency is time for one user's request. Optimizations often improve one at the cost of the other — larger batches boost throughput but increase latency.