The vocabulary of training AI at scale
Adam, ZeRO, FSDP, gradient checkpointing, mixed precision, and every technique that makes it possible to train models with billions of parameters.
Training modern AI models is a massive engineering feat. A frontier model runs across thousands of GPUs for months, with orchestration, parallelism, and optimization tricks that make the difference between feasible and impossible. The vocabulary here is the toolkit of the teams actually doing this work.
These 70 terms are grouped into four sub-topics: optimization algorithms (the fundamentals like Adam), distributed training techniques (ZeRO, FSDP, tensor parallelism), memory and compute efficiency (gradient checkpointing, mixed precision, offloading), and hyperparameter and schedule vocabulary. For fine-tuning-specific terms, see Fine-tuning Methods; for inference, see Inference & Serving.
The most important terms in Training & Optimization
Start here if you're new. These entries explain the foundational vocabulary in depth.
Adam Optimizer
The dominant optimizer for deep learning. Adaptive learning rates per parameter, momentum, and bias correction. AdamW (with decoupled weight decay) is the modern default.
Read the full entry →DistributedZeRO
DeepSpeed's Zero Redundancy Optimizer — partitions optimizer states, gradients, and parameters across devices. Enables training massive models on GPU clusters.
Read the full entry →DistributedFSDP
PyTorch's Fully Sharded Data Parallel — the mainstream implementation of ZeRO-3-style parameter sharding.
Read the full entry →EfficiencyGradient Checkpointing
Trading compute for memory — recompute activations during the backward pass instead of storing them. Essential for training large models.
Read the full entry →EfficiencyMixed Precision Training
Using lower-precision numbers (fp16, bf16) for most operations while keeping higher precision where it matters. 2-3x speedup with minimal quality loss.
Read the full entry →HyperparametersLearning Rate Schedule
How the learning rate changes over training. Cosine decay with warmup is the modern standard for most large models.
Read the full entry →All Training & Optimization terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
Optimizers
Distributed training
Memory & compute efficiency
Hyperparameters & schedules
Related term categories
These categories connect naturally to Training & Optimization — many terms cross-reference between them.
Go deeper on Training & Optimization
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- Understanding Scaling Laws
Why bigger models are better.
- How Distributed Training Works
ZeRO, FSDP, and parallelism explained.
- Mixed Precision Training Guide
FP16, BF16, FP8 in practice.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- AI providers
The teams doing the training.
- Training frameworks
PyTorch, JAX, DeepSpeed, Megatron.
- Open-weight models
Where training details are public.
Questions about Training & Optimization
Not deeply. But knowing terms like 'context window' or 'quantization' (relevant to inference cost) requires understanding some training concepts. Read this category as a reference, not a required manual.
Because it just works. Adaptive learning rates make it robust to hyperparameters, and momentum handles messy loss landscapes. Newer optimizers (Lion, Sophia, Muon) show promise but haven't dethroned it universally.
FSDP is PyTorch's native implementation of ZeRO-3-style sharding. Same core technique — partition parameters, gradients, and optimizer states across devices. FSDP is more integrated with PyTorch; ZeRO comes with more DeepSpeed features.
For inference: yes, increasingly. For training: still emerging. NVIDIA H100/H200 and Blackwell have FP8 support and it saves memory and speeds up training, but numeric stability is more delicate than BF16.
Larger batches produce more stable gradient estimates, allowing larger step sizes. Linear scaling rule: LR ∝ batch size. Modern schedules add sophistication (warmup, layer-adaptive rates), but the intuition holds.