Chain-of-Thought (CoT) prompting
At a glance
Definition
Chain-of-Thought (CoT) is a prompting technique that encourages a language model to produce intermediate reasoning steps before arriving at its final answer. In the original formulation by Wei et al. (2022), the researchers showed few-shot examples containing worked-out reasoning steps. But the technique quickly evolved into simpler forms — including the now-famous 'zero-shot CoT' from Kojima et al. (2022) where just appending Let's think step by step to a prompt produces reasoning behavior in a capable model.
The insight behind CoT is that LLMs generate one token at a time — meaning the tokens they've already produced condition all future tokens. When a model tries to answer 'What is 17 × 24?' directly, it must produce the correct answer in one shot with no intermediate scratchpad. When it reasons out loud — 17 × 24 = 17 × 20 + 17 × 4 = 340 + 68 = 408 — the model can use its own outputs as computational intermediates. This dramatically reduces error compounding.
CoT is now the foundation of an entire family of techniques. Tree of Thoughts generalizes to branching exploration. Self-consistency samples multiple CoT paths and takes the majority answer. ReAct interleaves reasoning with tool calls to enable agentic behavior. And modern reasoning models internalize the entire pattern — they're trained via reinforcement learning to produce long chains of reasoning tokens as part of their default behavior.
By 2026, CoT has moved from a prompting trick to a fundamental design pattern. Every major frontier model has been trained on chain-of-thought data. Reasoning models like DeepSeek R1, o-series, and Claude with extended thinking take this further, spending anywhere from seconds to minutes producing internal reasoning before their visible response. The prompting version — 'let's think step by step' — is still useful, but the more common pattern in 2026 is choosing a model that reasons by default.
Real-world example
The classic example that made CoT famous
In the original 2022 paper, Wei et al. showed a math word problem: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? Without CoT, GPT-3 (davinci) got this class of problem right about 18% of the time. With CoT prompting — showing a few examples with worked reasoning — accuracy jumped above 50%. Modern models are near-perfect on this specific example, but the technique still helps on harder problems.
# Zero-shot CoT: just add the magic phrase
prompt = """A juggler can juggle 16 balls. Half of the balls are golf balls,
and half of the golf balls are blue. How many blue golf balls are there?
Let's think step by step."""
# Model output:
# "The juggler has 16 balls total.
# Half are golf balls: 16 / 2 = 8 golf balls.
# Half of the golf balls are blue: 8 / 2 = 4 blue golf balls.
# Answer: 4"
# Few-shot CoT: show examples with reasoning
prompt = """Q: There are 3 cars in the parking lot. 2 more cars arrive.
How many cars are in the parking lot?
A: There are 3 cars originally. 2 more arrive. 3 + 2 = 5.
The answer is 5.
Q: Roger has 5 tennis balls. He buys 2 more cans with 3 each.
How many tennis balls now?
A: Roger starts with 5. He buys 2 cans of 3 balls each,
which is 2 * 3 = 6 more balls. 5 + 6 = 11.
The answer is 11.
Q: A juggler can juggle 16 balls. Half of the balls are golf balls,
and half of the golf balls are blue. How many blue golf balls are there?
A: """
# Model completes with reasoning steps, arrives at answerWhen you'll encounter this
- You're solving math or logic problems — the classic case where CoT dramatically improves accuracy.
- The task involves multi-step reasoning — breaking a complex query into parts and reasoning through each.
- You need to audit the model's reasoning — CoT outputs are inspectable; you can see (and correct) the model's chain of logic.
- You want to reduce hallucination — reasoning steps often expose where the model's logic breaks down.
- You're building an agent — CoT is the foundation of ReAct, planning, and other agentic patterns.
How it works
The next-token setup
LLMs generate one token at a time, and each generated token conditions all future tokens. Without intermediate steps, the model has to produce a complex answer in one uninterrupted pass.
Adding a reasoning scratchpad
By asking for reasoning first, the model produces intermediate tokens it can then attend to when generating the final answer. The intermediate tokens act as a computational scratchpad.
Few-shot demonstration
In the original formulation, prompts included 2-8 worked examples showing the reasoning format. The model learns the pattern from these demonstrations and applies it to the new query.
Zero-shot CoT
Kojima et al. (2022) showed that appending 'Let's think step by step' to a prompt is often enough to trigger CoT behavior in capable models. Simpler and often as effective as few-shot CoT.
Sampling and self-consistency
For hard problems, you can sample multiple CoT paths at higher temperature and take the majority answer. This is self-consistency and it further improves accuracy at the cost of more compute.
Modern reasoning models
Instead of prompting for CoT, reasoning models (o-series, DeepSeek R1, Claude with extended thinking) are trained via RL to reason internally. They produce hidden reasoning tokens before the visible response.
Cost trade-off
CoT generates more output tokens. On simple queries, this is wasted cost. On complex reasoning, the accuracy gain far outweighs extra tokens. Modern reasoning models manage this automatically by only reasoning when needed.
Common misconceptions
For simple factual queries ('capital of France') or well-known transformations, CoT wastes tokens without helping. CoT helps most on multi-step reasoning, math, and logic. Modern reasoning models decide automatically when to reason, avoiding this trade-off.
CoT is a prompting pattern that improves output quality. Whether the generated reasoning reflects internal 'thinking' or is post-hoc rationalization is an open research question. Interpretability studies show reasoning traces don't always match what the model would produce without them.
It often works surprisingly well on modern models, but few-shot CoT can still outperform zero-shot for specific formats, unusual domains, or where you need consistency in how the model reasons. Try both.
Related terms
Where you'll see this in practice
Learn more
- Read the deep-dive: How Chain-of-Thought Actually Works
- Take the Master Prompt Engineering path
- Compare Prompted CoT vs Reasoning Models
- Browse Prompt Engineering category
Sources & further reading
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al., 2022 (original CoT paper). Read →
- Large Language Models are Zero-Shot Reasoners — Kojima et al., 2022 (Zero-shot CoT). Read →
- Self-Consistency Improves Chain of Thought Reasoning — Wang et al., 2022. Read →
- Tree of Thoughts: Deliberate Problem Solving with LLMs — Yao et al., 2023. Read →
Frequently asked about Chain-of-Thought (CoT)
CoT effectiveness scales with model capability. Small models (under ~10B parameters historically) benefited less. Modern LLMs — even smaller ones — are typically trained on CoT data during fine-tuning, so most respond well. Very small local models may still struggle.
Less so. Reasoning models produce chain-of-thought internally and often output only the conclusion. Explicit CoT prompts may still help for tasks the model wasn't trained to reason through by default, but the improvement is smaller.
CoT produces one linear reasoning chain. Tree of Thoughts explores multiple reasoning branches, evaluating and pruning. ToT is more expensive but can solve problems where linear reasoning gets stuck. See our comparison.
For simple queries, no — it just adds cost and latency. For complex reasoning (math, logic, multi-step planning), yes. Reasoning models handle this automatically. If you're using a non-reasoning model, CoT prompting is a good default for hard tasks.
Debated. Reasoning traces improve output quality, but interpretability studies suggest they don't always match what the model would have computed without them. Treat them as a useful output that improves accuracy, not necessarily as ground truth about the model's internal reasoning.