The quadratic bottleneck
Since "Attention is All You Need" in 2017, the Transformer defined modern AI. GPT, Claude, Gemini, LLaMA — all built on the same core: let every token attend to every other token. The problem: O(n²) cost in context length. Double the context, quadruple the compute. For 1 million token windows, this becomes prohibitive at scale.
In 2026, State Space Models (SSMs) — particularly the Mamba family — reached production at multiple AI labs. Not as replacements for the Transformer, but as complementary alternatives with O(n) cost — linear with context.
The Mamba family
Original Mamba (Gu & Dao, 2023) introduced the key innovation: selective SSM — the model parameters depend on the input, not fixed. This lets the model decide what to compress and what to retain in "state memory," instead of accumulating everything like the Transformer does in KV cache. Mamba-3 (March 2026) added complex-valued state tracking and multiple state channels in parallel. Production result: Mamba-3-Large performs at parity with mid-tier Transformers on standard benchmarks at 2-3x lower cost on long contexts.
RWKV — the shapeshifter
RWKV (Receptance Weighted Key Value) takes a different path: parallelizable Transformer training with RNN sequential inference. Result: constant memory footprint regardless of context length — unlike the Transformer whose KV cache grows linearly. Trade-off: for tasks requiring precise recall of specific tokens in long context, the Transformer still holds a structural advantage.
The 2026 trend: hybrids
The clearest 2026 conclusion: production models increasingly combine both. Attention layers for precise retrieval and complex reasoning; SSM layers for efficient long-context compression. The most relevant practical impact: Mamba-Transformer hybrid models are enabling sophisticated reasoning on consumer devices like smartphones — unthinkable with pure Transformers of the same capability level.
