The architectures beyond transformers
Mamba, State Space Models, RWKV, Linear Attention, Retentive Networks. Every architecture challenging (or extending) the transformer.
The Transformer has dominated modern AI since 2017, but its quadratic scaling with sequence length has always motivated research into alternatives. In the last three years, State Space Models (SSMs) like Mamba have shown they can match transformer quality with linear scaling — potentially enabling much longer contexts and faster inference.
These 40 terms cover four sub-topics: State Space Models and their variants (Mamba, S4, S6), other alternative architectures (RWKV, Retentive Networks), hybrid approaches that combine SSMs with attention (Griffin, Jamba), and the linear/efficient attention research that motivates all of them. This is one of the most active research areas in AI — expect this category to evolve monthly.
The most important terms in Modern Architectures
Start here if you're new. These entries explain the foundational vocabulary in depth.
Mamba
A selective state space model that matches transformer quality on many tasks with linear scaling in sequence length. Introduced late 2023 by Albert Gu and Tri Dao.
Read the full entry →SSMsState Space Model (SSM)
A class of models that maintain a continuous state through time. Linear complexity in sequence length; the family Mamba belongs to.
Read the full entry →AlternativesRWKV
An RNN-transformer hybrid designed to have linear complexity like an RNN while training in parallel like a transformer. Popular in the open-source community.
Read the full entry →Efficient attentionLinear Attention
Reformulations of attention with linear rather than quadratic complexity. Compromise quality; foundations for many alternatives.
Read the full entry →HybridsJamba
AI21's hybrid model combining Mamba layers with attention. Demonstrates that hybrid designs can leverage strengths of both approaches.
Read the full entry →AlternativesRetentive Network (RetNet)
Microsoft's transformer alternative with linear complexity. Positioned as 'transformer + RNN' — parallel training, recurrent inference.
Read the full entry →All Modern Architectures terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
State Space Models
Alternative architectures
Hybrid architectures
Efficient attention research
Related term categories
These categories connect naturally to Modern Architectures — many terms cross-reference between them.
Go deeper on Modern Architectures
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- How Mamba Works
Selective SSMs walked through.
- State Space Models Explained
From S4 to Mamba, step by step.
- Linear Attention — The Full Guide
All the linear alternatives.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- AI models
Nearly all commercial models are transformers.
- Mistral AI
Ships Mamba-based Codestral variants.
- AI21
Ships Jamba hybrid models.
Questions about Modern Architectures
Unclear. Mamba matches transformers on many benchmarks with linear scaling — very appealing for long context. But transformers benefit from years of infrastructure investment, and hybrid architectures may capture the best of both.
SSMs are efficient but attention has better in-context learning for some tasks. Hybrids (Jamba, Griffin) put SSM layers where efficiency matters most and attention where quality matters — pragmatic engineering.
Yes but not widely. Jamba is deployed by AI21. Mamba variants appear in some open-weight models. Most commercial frontier models remain transformers — the ecosystem lag is real.
Linear attention reformulates the standard attention computation. SSMs are a different mathematical framework based on continuous state — they happen to have linear complexity. Different origins, similar efficiency gains.
If you're a researcher or building specialized systems, yes. If you're using APIs, follow the field but don't need deep understanding. The vocabulary appears in provider announcements and matters for informed decisions.