Mixture of Experts — scaling without cost
Sparse MoE, Switch Transformer, expert routing, load balancing. How mixture-of-experts models scale to trillions of parameters while keeping inference cost tied to active experts.
Mixture of Experts (MoE) is the technique that lets modern models have trillions of parameters while running as fast as a small dense model. The idea: at each layer, a router activates only a subset of specialized "expert" sub-networks per input. Total parameters scale, but per-token compute stays low. This is how GPT-5, Mixtral, DeepSeek, and other frontier models scale.
The 20 terms in this category are grouped into four sub-topics: MoE fundamentals, routing mechanisms (how the model chooses experts), training-specific challenges (load balancing, auxiliary losses), and notable implementations. For the broader picture, see our How Mixture of Experts Works tutorial.
The most important terms in Mixture of Experts
Start here if you're new. These entries explain the foundational vocabulary in depth.
Mixture of Experts (MoE)
A neural architecture where a router activates only a subset of "expert" sub-networks per input. Trillions of parameters total, but only billions are used per token.
Read the full entry →FoundationsSparse MoE
The dominant MoE variant — sparsity means most experts stay dormant on any given input. Distinct from soft MoE where all experts contribute weighted amounts.
Read the full entry →RoutingExpert Routing
The mechanism that decides which experts activate for each token. Usually a small learned linear layer that produces routing weights.
Read the full entry →TrainingLoad Balancing
Making sure experts get used roughly equally during training. Without it, a few experts dominate and the others go untrained.
Read the full entry →ImplementationsSwitch Transformer
Google's 2021 paper introducing top-1 expert routing at scale. Set the template for modern MoE.
Read the full entry →ImplementationsDeepSeekMoE
DeepSeek's MoE architecture with fine-grained expert segmentation and shared experts. Powers DeepSeek V3 and R1.
Read the full entry →All Mixture of Experts terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
MoE fundamentals
Routing mechanisms
Training challenges
Related term categories
These categories connect naturally to Mixture of Experts — many terms cross-reference between them.
Go deeper on Mixture of Experts
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- How Mixture of Experts Works
Routing, load balancing, and training explained.
- Why MoE Scales So Well
The math behind sparse activation.
- Training a MoE Model
The practical gotchas.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- DeepSeek V3
671B MoE, 37B active per token.
- Mixtral
Popular open-weight MoE family.
- GPT-5
Almost certainly MoE (details unpublished).
Questions about Mixture of Experts
Almost every open detail suggests yes. Mixtral, DeepSeek V3, Grok (variants), and most recent open-source flagships are MoE. Anthropic and OpenAI don't publish architecture, but strong hints suggest their frontier models are MoE too.
Compute-optimality. A 671B dense model would be intractable to train and serve. A 671B MoE with 37B active gets you the parameter count for capability with dense-model inference cost. It's the frontier of what's feasible.
Memory. To serve a 671B MoE, you need enough VRAM for all 671B parameters (loaded but not activated per token). MoE saves compute but not memory. This is why MoE serving requires beefy GPU clusters even for 'small active' models.
Each token is routed to 2 experts per MoE layer. Their outputs are combined (weighted sum). Top-1 (Switch Transformer) uses only 1 expert per token. Top-2 (Mixtral) uses 2. More experts = more compute but potentially better quality.
Somewhat, but not in the intuitive way. Experts don't cleanly split into 'the math expert' and 'the code expert' — they specialize in ways that emerge from training and are hard to interpret. Interpretability research is starting to unpack this.