Tokenization (and what a token really is)
At a glance
Definition
Tokenization is the process of converting text into a sequence of tokens — the discrete units that a language model actually processes. A tokenizer takes a string like 'The cat sat on the mat' and produces something like ['The', ' cat', ' sat', ' on', ' the', ' mat'] — where each item corresponds to an integer ID that the model uses internally.
The most common approach in modern LLMs is Byte-Pair Encoding (BPE) — introduced for machine translation by Sennrich et al. in 2016 and adopted by GPT-2 (2019). BPE starts with individual bytes as tokens, then iteratively merges the most frequent adjacent pairs. The result is a vocabulary (typically 50K-100K tokens) where common words are single tokens, rare words split into subword pieces, and any arbitrary text can be represented.
Different models use different tokenizers, and even the same base algorithm (BPE) produces different results depending on the vocabulary trained. Claude, GPT-4, GPT-5, Gemini, and Llama each have their own tokenizer. This is why the token count for the same text differs between models — and why comparing 'context window' numbers across providers requires care.
Tokenization is a subtle but critical piece of the LLM stack. It determines cost (all major APIs charge per token), context window math (a 200K context is 200K tokens, not characters), latency (more tokens = slower response), and even model behavior on edge cases. When a model gets the number of 'r's in 'strawberry' wrong, when it fails at character counting, when it produces gibberish on unusual text — the answer is often 'because of how it tokenizes.'
Real-world example
How the same text tokenizes differently across models
The sentence 'AI Terms Guide is the definitive AI reference.' tokenizes as roughly 10 tokens in GPT-4's cl100k tokenizer, 11 in Claude's tokenizer, and 10 in Llama's. The differences seem small but compound at scale — a document that's 100K tokens in one tokenizer might be 110-120K in another, changing what fits in a context window and what costs. Non-English text has even bigger differences: Chinese and Arabic text often tokenizes 2-3x more densely (more tokens per character) in older tokenizers, which is why some models cost significantly more for non-English content.
# Counting tokens with tiktoken (OpenAI's tokenizer)
import tiktoken
encoder = tiktoken.encoding_for_model("gpt-4")
text = "AI Terms Guide is the definitive AI reference."
tokens = encoder.encode(text)
print(f"Tokens: {tokens}")
# [1698, 27563, 13002, 374, 279, 45813, 15592, 5905, 13]
print(f"Count: {len(tokens)}")
# 9
# Decode individual tokens to see how they split
for tid in tokens:
print(f" {tid:>6} '{encoder.decode([tid])}'")
# 1698 'AI'
# 27563 ' Terms'
# 13002 ' Guide'
# 374 ' is'
# 279 ' the'
# 45813 ' definitive'
# 15592 ' AI'
# 5905 ' reference'
# 13 '.'
# For Claude, use Anthropic's tokenizer
from anthropic import Anthropic
client = Anthropic()
count = client.messages.count_tokens(
model="claude-opus-4-8",
messages=[{"role": "user", "content": text}]
).input_tokens
print(f"Claude tokens: {count}")When you'll encounter this
- You're calculating LLM costs — pricing is per million input/output tokens; you need to count.
- You're hitting context window limits — knowing what fits requires knowing token count, not character count.
- You're optimizing prompts — reducing tokens directly reduces cost and latency.
- You're debugging model behavior — many strange failures trace back to how the model tokenizes the input.
- You're working with non-English text — tokenization efficiency varies dramatically by language; costs and context math change accordingly.
How it works
Vocabulary training
The tokenizer is trained on a large text corpus. Different training corpora and merging rules produce different vocabularies.
BPE merging
Byte-Pair Encoding starts with individual bytes as tokens. At each step, it finds the most frequent adjacent pair and adds it as a new token. Repeat until vocabulary reaches target size (typically 50K-100K).
Encoding text
Given input text, the tokenizer greedily matches the longest sequences that appear in its vocabulary. Common words become single tokens; rare words split into subwords.
Special tokens
Beyond text tokens, tokenizers include special tokens for structure: <bos> (beginning of sequence), <eos>, role tokens for chat formatting, image or tool tokens in multimodal models.
Whitespace handling
Different tokenizers treat whitespace differently. Many prepend a space to tokens ('the' and ' the' are separate tokens). This affects tokenization of the same word in different positions.
Decoding
Given a sequence of token IDs, the tokenizer looks up each ID's string and concatenates. Well-designed tokenizers are lossless — encode-then-decode recovers the original text.
Impact on model
The tokenizer is baked into a model at training time. You can't swap it out. Every input goes through the same tokenizer; the model sees only token IDs, never raw characters.
Common misconceptions
Roughly true for common English words, but not universally. Common words are single tokens; rare words split into multiple subwords. 'tokenization' is one token in most modern tokenizers, but 'Kubernetization' might be four. Emoji, code, and non-English text tokenize very differently.
No. Roughly 1 token = 3-4 characters in English. Text with more common words is more compact in tokens; unusual text (URLs, code, non-English) is denser. Always count with the actual tokenizer, not characters, when it matters.
Not necessarily. Vocabulary size and model size are independent choices. GPT-2 (1.5B) uses ~50K vocab. Claude Opus 4.8 uses ~200K. Llama uses ~128K. Larger vocabularies reduce token count per document but require more embedding parameters.
Related terms
Where you'll see this in practice
Learn more
- Read the deep-dive: What is a Token
- Read: Understanding Context Windows
- Take the AI for Developers path
- Browse LLMs & Language Models category
Sources & further reading
- Neural Machine Translation of Rare Words with Subword Units — Sennrich et al., 2016 (BPE for NMT). Read →
- Language Models are Unsupervised Multitask Learners — Radford et al., 2019 (GPT-2 introduced byte-level BPE for LLMs). Read →
- SentencePiece: A simple and language independent subword tokenizer — Kudo & Richardson, 2018. Read →
- OpenAI Tokenizer Documentation — OpenAI tiktoken. Read →
Frequently asked about Tokenization
Roughly 3-4 characters in English, or about 0.75 words. But it varies significantly: common words are 1 token, rare words split into subwords, code and non-English can be much denser. Always count with the actual tokenizer when it matters.
Because their tokenizers are different. Claude, GPT, and Gemini use different vocabularies. The same document might be 10K tokens in one and 12K in another. This changes both cost and how much fits in a context window.
Older tokenizers were trained mostly on English text and produce dense (many-token) sequences for other languages. Modern tokenizers are more balanced, but Chinese, Arabic, Hindi, and other non-Latin scripts still often tokenize more densely than English. Cost per equivalent content is therefore higher.
Because LLMs don't see letters — they see tokens. 'strawberry' might tokenize as a single token or a few tokens, but never as individual letters. Asking 'how many r's are in strawberry' requires the model to know character-level information it never actually sees. Modern models are trained to handle this specific case, but the underlying issue is tokenization.
Only for token counting estimates. Actually running a model requires the tokenizer it was trained with — the model's parameters are tied to specific token IDs. You can't feed GPT-tokenized inputs to Claude or vice versa.