Beyond the Transformer: the architectures that will redefine AI in the next 18 months

SSM, MoE, modular systems, and world models are challenging the dominant paradigm since 2017 — and publication data at NeurIPS suggests that the change is closer than the market imagines

The plateau that no one wants to admit

The scaling hypothesis was the driving force of the race between OpenAI, Google, and Anthropic in the last five years: larger models, trained on more data, with more compute, predictably produce more capable models. The Chinchilla Scaling Laws by Hoffmann et al. (2022) refined the ideal proportions between parameters and training tokens. GPT-4, Gemini 2.0, and Claude 4 were the products of this faith.

But the benchmark data tells a different story. The marginal improvement in MMLU (Massive Multitask Language Understanding) and GPQA Diamond is becoming statistically visible as a decreasing trend. The computational cost for each increment grows exponentially: GPT-4 cost an estimated US$ 100 million to train; next-generation models in the pure scaling paradigm may require US$ 1 billion or more.

This does not mean that the models have stopped improving — it means that the cost-benefit ratio of scaling is changing. And this change opens room for alternatives that were academically interesting but economically marginal when scaling still delivered increasing returns.

The three challengers: SSM, MoE, and neuromorphic computing

Three families of architectures are gaining simultaneous traction in frontier research labs, according to an analysis of publications on arXiv and NeurIPS 2025.

State Space Models (SSM): Mamba (Gu & Dao, 2023) proposes processing sequences with linear complexity instead of the quadratic complexity of transformers — a fundamental advantage for very long contexts. Early results show competitive performance with equivalently sized transformers in specific tasks. Mamba-2 and hybrid variants like Jamba (SSM + transformer) are being evaluated in production by multiple labs. The empirical question still open is whether SSMs maintain the advantage when scaling to the sizes of frontier models.

Mixture of Experts (MoE): Already in production. Mixtral 8×7B by Mistral AI and presumably GPT-4 use MoE. Instead of activating all parameters for each token, MoE activates only a subset of "experts", allowing models with enormous total capacity but the inference cost of a smaller model. DeepSeek-V3, launched in December 2025, demonstrated that a well-built MoE outperforms larger-scale models with a fraction of the training cost — a result that shook the sector and led to a repricing of shares of AI infrastructure companies.

Modular and cognitive systems: The analysis by Bessemer Venture Partners published in July 2026 identifies a transition from "monolithic models" to "modular cognitive systems" as the main AI infrastructure trend for the next 24 months. Instead of a model that does everything, the emerging architecture involves specialized modules: one generates, another verifies, another reasons, another plans, with memory and retrieval encompassing everything. This is closer to how biological information processing systems work than the classic transformer.

World models: the long-term bet

The most ambitious of the research frontiers is that of world models — systems that build internal representations of the functioning of the physical and causal world, not just statistical patterns in text. Yann LeCun, from Meta AI, has defended this approach as the path to genuine intelligence since 2022. The JEPA (Joint Embedding Predictive Architecture) is the most developed implementation of this vision.

The debate with advocates of large-scale LLMs is fundamentally empirical: if world models demonstrate reliable physical reasoning while LLMs continue to fail at intuitive physics, the center of gravity of research permanently shifts. If LLMs learn enough physics from data at scale, the argument for world models loses strength.

In 2026, the first results of hybrid systems — LLMs with world models as specialized reasoning modules — are starting to appear on arXiv. The verdict has not yet arrived.

What the publication data says

The analysis of submission trends to NeurIPS 2025 and ICML 2026 shows a 340% growth in papers on SSM and state space architectures compared to 2023, a 180% growth in papers on MoE and sparse models, and a 220% growth in papers on world models and causal representations. In contrast, papers on pure scaling laws grew only 12%.

The papers market is not the product market — there is an 18 to 36-month lag between a frontier discovery and its incorporation into production systems. But the signal is clear: the field is betting on architectural diversification, not on more scaling of the same.

What the Laboratory monitors

PrezencIA indexes papers from arXiv, NeurIPS, ICML, and ICLR with a focus on frontier research relevant to practical applications in 18–36 months. We do not cover every paper — we cover what moves the field. The Laboratory's editorial criteria prioritize replicable results, independent benchmarks, and evidence of transfer to real-world applications.

The central tension we monitor: promising architectures will need to overcome not only the technical benchmark, but the entire ecosystem of tools, hardware, and expertise built around the transformer. The history of computing is full of superior architectures that failed in adoption for ecosystem reasons. The transformer has proven resilient exactly because it is well understood, optimizable, and compatible with existing GPU infrastructure.