Transformer (neural network architecture)

Definition

A Transformer is a neural network architecture designed to process sequential data — text, code, audio, or images treated as sequences of patches — by using an attention mechanism to weigh the importance of different positions in the input. Unlike earlier architectures such as RNNs and LSTMs that process sequences one element at a time, Transformers process the entire input in parallel — dramatically speeding up training and enabling the scale of modern models.

The original Transformer, published in the paper Attention Is All You Need, was designed for machine translation. It used an encoder-decoder architecture with six layers each, roughly 65 million parameters, and outperformed the state of the art on English-German and English-French translation. What made the paper revolutionary was the claim in its title: attention, without recurrence or convolution, was sufficient to build a state-of-the-art model.

Since 2018, essentially all frontier language models — Claude, GPT, Gemini, Llama, Mistral, DeepSeek, and Qwen — are Transformer variants. Modern LLMs typically use a decoder-only variant with dozens or hundreds of layers and billions to trillions of parameters. Transformers have also been adapted for images (Vision Transformer), audio (Whisper), and multimodal understanding.

The three core innovations that make Transformers work are: (1) self-attention, which lets every position in a sequence look at every other position; (2) positional encoding, which adds order information since attention itself is order-invariant; and (3) parallelization, which lets the entire sequence be processed at once on GPUs. Combined, these turn what used to take days into hours — and what used to be impossible into achievable.

Real-world example

How a Transformer processes "The cat sat on the mat"

When you send a message to Claude or ChatGPT, the transformer inside first breaks your text into tokens, converts each token into a numeric vector (an embedding), and adds positional information so the model knows the order. Then, layer by layer, self-attention lets every token 'look at' every other token to build up a rich contextual understanding. By the final layer, the model has computed a probability distribution over what token should come next — which is how the response is generated, one token at a time.

# Simplified transformer block in PyTorch
import torch
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model=512, n_heads=8):
        super().__init__()
        self.attention = nn.MultiheadAttention(d_model, n_heads)
        self.norm1 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_model * 4),
            nn.GELU(),
            nn.Linear(d_model * 4, d_model),
        )
        self.norm2 = nn.LayerNorm(d_model)

    def forward(self, x):
        # 1. Self-attention with residual + norm
        attn_out, _ = self.attention(x, x, x)
        x = self.norm1(x + attn_out)
        # 2. Feed-forward with residual + norm
        x = self.norm2(x + self.ffn(x))
        return x

When you'll encounter this

• You're using any modern LLM — Claude, GPT-5, Gemini, Llama — you're using a Transformer under the hood.
• You're reading AI research papers — architecture papers assume you know how Transformers work.
• You're fine-tuning models — understanding attention and layer structure guides LoRA configuration.
• You're debugging performance — the quadratic cost of self-attention shapes context window limits and inference cost.
• You're designing prompts for long contexts — the 'lost in the middle' phenomenon is a consequence of how attention learns.

How it works

Common misconceptions

Related terms

Where you'll see this in practice

Learn more

• Read the deep-dive: How Transformers Work
• Start the AI for Developers learning path
• Compare Transformer vs Mamba architectures
• Browse the Deep Learning Architectures category