Reinforcement learning — from Atari to LLMs
PPO, DQN, policy gradients, MCTS, and everything else — the RL vocabulary underlying RLHF, reasoning models, and agents.
Reinforcement Learning powered the AlphaGo era and now powers the reasoning-model era. Every time you interact with ChatGPT or Claude, you're using a model shaped by RLHF — reinforcement learning from human feedback. Every reasoning model (o-series, DeepSeek R1, Claude with extended thinking) uses RL to train chain-of-thought.
These 30 terms are grouped into four sub-topics: fundamental RL concepts (agent, environment, reward), classical algorithms (Q-learning, DQN, policy gradient), modern algorithms most relevant to LLM training (PPO, GRPO, DPO variants), and applications that shape the field today. For deeper context, see the fine-tuning category where RLHF and preference training live.
The most important terms in Reinforcement Learning
Start here if you're new. These entries explain the foundational vocabulary in depth.
Reinforcement Learning (RL)
A learning paradigm where an agent takes actions in an environment and receives rewards. Core problem: maximize cumulative reward through experience.
Read the full entry →AlgorithmsPolicy Gradient
A family of RL algorithms that directly optimize the policy (action distribution) via gradient ascent on expected reward. Foundation of PPO, GRPO, and REINFORCE.
Read the full entry →Modern algorithmsPPO
Proximal Policy Optimization — the workhorse of modern RL, especially in RLHF. Constrains updates to stay close to the old policy for stability.
Read the full entry →Modern algorithmsGRPO
Group Relative Policy Optimization — DeepSeek's PPO variant that skips the value function. Foundation of DeepSeek R1 reasoning training.
Read the full entry →RLHFReward Model
A model trained to predict human preferences, used to provide reward signal during policy optimization. Central component of RLHF.
Read the full entry →Search algorithmsMCTS
Monte Carlo Tree Search — the search algorithm behind AlphaGo, AlphaZero, and modern reasoning systems that explore action sequences.
Read the full entry →All Reinforcement Learning terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
RL fundamentals
Classical algorithms
Modern RL for AI
Related term categories
These categories connect naturally to Reinforcement Learning — many terms cross-reference between them.
Go deeper on Reinforcement Learning
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- How RLHF Works
PPO, reward models, walked through.
- Understanding Policy Gradients
The core RL math.
- How Reasoning Models Work
The RL behind o1, R1, Claude thinking.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- DeepSeek R1
Reasoning model trained with GRPO.
- DeepSeek R1 vs o-series
Two RL-trained reasoning families.
- All AI models
Every modern chat model is RL-shaped.
Questions about Reinforcement Learning
Not to use them. To reason about why they behave certain ways (sycophancy, refusal patterns, reasoning styles), knowing RLHF and DPO helps. To fine-tune with preferences, essential.
It's stable and forgiving with hyperparameters — critical for expensive LLM training runs. Newer variants (GRPO, DPO) reduce some PPO complexity but PPO remains the default in production RLHF.
Sort of. DPO removes the explicit reward model and RL loop, but it's still optimizing to produce preferred outputs — the underlying math is RL-adjacent. Practically it's simpler than PPO.
When a policy discovers a way to maximize the reward signal without actually doing what humans wanted. Famous example: an RL agent maximizing score by exploiting bugs instead of playing well. In RLHF: models generating overly long responses because the reward model favors length.
Roughly yes. They start from a strong base LLM, then use RL (often GRPO variants) to teach long chain-of-thought reasoning. The reward comes from correctness on verifiable tasks like math and coding.