If you have spent any time around modern AI, you have probably been told that “transformers are everything” and “attention is all you need.” Every major frontier model—OpenAI’s GPT series (like ChatGPT), Anthropic’s Claude, Google’s Gemini, Meta’s Llama—sits on some flavor of transformer backbone.

But we’re running into the rough edges of that design. Self-attention, the core trick that made transformers so powerful, is also a scaling nightmare: it compares every token to every other token, which means computation and memory blow up quadratically as you increase context length. That is why 128K or 1M-token contexts feel cutting-edge and expensive, not cheap and boring.

So researchers have started asking a new question: what if attention is not all we need? Over the last couple of years, several serious contenders have emerged—state space models (SSMs) like Mamba, convolutional operators like Hyena, and attention-free RNN hybrids like RWKV—that try to keep what we like about transformers (quality, scalability) while dodging their worst bottlenecks.

This post will walk you through what is actually going on in this “post-transformer” space, without drowning you in math. You will see how these new architectures work at a high level, what problems they try to solve, and what that might mean for the tools you actually use, from ChatGPT to whatever comes next.

Why transformers are running into walls

Transformers earned their dominance because self-attention is incredibly flexible. For a sequence of length L, the model can, in principle, relate any token to any other token in a single layer. That global view is a huge part of why we got powerful general models like GPT-4 and Claude.

The trade-off is hidden in the math: self-attention scales as O(L²) with sequence length, because it builds an L×L matrix of token interactions. Stanford’s Hazy Research group (the folks behind Hyena and a lot of efficient attention work) highlight that this quadratic cost is the fundamental bottleneck blocking very long contexts in standard transformers.Hazy Research blog on Hyena

In practice, that quadratic scaling means:

  • Long context windows are expensive in both compute and memory.
  • Deployed models like ChatGPT, Claude, and Gemini must juggle quality vs context length vs latency.
  • Training truly massive context models (hundreds of thousands to millions of tokens) becomes painful.

So the “post-transformer” question is not just academic. If you want AI that can:

  • read whole codebases,
  • digest multi-day chat logs,
  • or handle entire video streams,

you need something closer to linear or near-linear scaling with sequence length, without sacrificing too much quality.

Enter state space models: Mamba and the S4 family

One of the most exciting lines of work is structured state space models (SSMs), especially the S4 family and Mamba.

Very loosely, an SSM treats the model as a dynamical system with a hidden “state” that evolves over time. Instead of directly comparing each token to every other token (attention), the model maintains a latent state that summarizes the past, and updates that state as new inputs arrive. Modern SSMs are designed so this process can be implemented efficiently and in parallel.

A recent survey on S4 models describes how this family evolved from the original S4 to faster, more memory-efficient variants, and how they underlie modern architectures like Mamba and Jamba for long-context modeling.Survey on S4 models and successors

What makes Mamba special?

Mamba is a specific architecture built around selective state space models. Instead of attention, each layer uses a state space update that can be computed in linear time with sequence length. IBM describes Mamba as a neural network architecture with a selective SSM core that can handle long sequences more efficiently than transformers.IBM overview of Mamba

High-level, Mamba gives you:

  • Linear-time sequence processing – scale to long contexts without blowing up compute.
  • Stateful sequence modeling – keeps a latent state that evolves over tokens, like a mathematically disciplined RNN.
  • Strong performance on language and vision – benchmarks show competitive or better results compared to transformers on several modalities, while using less memory and enabling longer sequences.Mamba architecture overview and benchmarks

You can think of it as: “What if we took the good parts of RNNs (constant-time step updates) and made them expressive and trainable enough to compete with transformers, but with modern hardware-friendly design?”

Because of those traits, SSM-based models are starting to show up in real systems, including hybrid architectures that mix Mamba blocks with transformer blocks to get “best of both worlds” behavior.

Hyena: convolutions instead of attention

Another line of attack is: what if we replaced attention with fast convolutions that can still capture long-range dependencies?

That’s the idea behind Hyena Hierarchy, a sequence operator introduced by researchers at Stanford. Instead of computing an L×L attention matrix, Hyena uses a series of implicit long convolutions plus gating to approximate the kind of global interactions attention provides, but with much better scaling.

Hyena’s complexity is subquadratic—roughly O(L log L) instead of O(L²)—because it uses FFT-based convolutions to handle long sequences efficiently.Illustrated Hyena explanation That might sound like alphabet soup, so in plain language:

  • Imagine scanning through a very long text with a wide “window” that can look far ahead and behind.
  • Instead of comparing every word to every other word individually, you use powerful filters (the convolutions) that can pick up patterns over long distances.
  • Gating lets the model modulate these filters based on the data, giving it some of the input-dependent flexibility of attention.

Hyena is appealing because:

  • It is a drop-in alternative for attention in many transformer-style stacks.
  • It enables much longer contexts at similar or lower compute cost.
  • It has been used in domains beyond language, such as financial text and biomedical imaging, to handle long sequences more efficiently.Hyena Hierarchy paper overview

Some experiments even combine Hyena and Mamba-like operators with transformers in the same model, choosing the most efficient operator for each layer depending on the task and context length.

RWKV: an RNN that behaves like a transformer

Then there is RWKV, a family of models that looks like a classic RNN on the outside but behaves a lot like a transformer on the inside.

RWKV is designed to be:

  • Attention-free – no QKV attention matrices.
  • Parallelizable during training – you can still batch and parallelize like transformers.
  • RNN-like at inference – process tokens one by one, with constant memory per step, which is ideal for long context generation and low-resource devices.

The RWKV authors show that with the right design, a pure RNN-style model can match transformer-level performance at scale, with models up to tens of billions of parameters trained on large language datasets.RWKV: Reinventing RNNs for the Transformer Era

Conceptually, RWKV is an example of a broader theme in post-transformer work:

  • Keep layered residual blocks, normalization, and scaling laws from transformers.
  • Replace the attention module with something cheaper (here, special recurrent updates with linear-time behavior).
  • Preserve training-time parallelism so we are not stuck with the old, slow RNN training regime.

You can think of RWKV as “a transformer that forgot how to do attention, but learned a more efficient habit instead.”

Are these really “after” transformers or just side quests?

So, are we actually moving beyond transformers, or just decorating them?

A recent survey on “What comes after transformers?” argues that we are seeing more of a broadening than a clean replacement.Selective survey on architectures beyond transformers Transformers still dominate foundation models today, but:

  • SSMs like Mamba, and convolutional operators like Hyena, are beginning to appear in competitive long-context and multimodal models.
  • Hybrid designs—transformer layers mixed with Mamba or other SSM layers—are increasingly common in research prototypes and some commercial models.
  • Architecture search is becoming more practical, so we can swap in different sequence operators depending on the hardware and task.

In other words, we are likely heading toward a world where:

  • Transformers are one component in a toolbox.
  • Long-context or streaming tasks lean on Mamba/SSM-style blocks.
  • Specific domains (e.g., audio, time series, certain vision tasks) lean on convolutional operators like Hyena.
  • Efficient deployment for edge devices might favor RWKV-like RNNs or SSMs with tiny memory footprints.

You probably will not see “pure Hyena-only” or “pure Mamba-only” everywhere, but you will see more models marketed as “Mamba-based,” “SSM-enhanced,” or “hybrid transformer–SSM” in the same way you now see “Mixture of Experts” or “multimodal” in model branding.

What this means for you right now

If you are just using ChatGPT, Claude, or Gemini, do you need to care about these new architectures today?

In the short term:

  • Your day-to-day UX improvements—longer context, faster responses, cheaper tokens—are increasingly driven by these efficiency gains under the hood.
  • Companies building the next generation of LLMs are actively experimenting with SSMs, Hyena-style operators, and hybrids to push context and reduce costs.

If you are building on top of AI (developer, researcher, or power user), this matters because:

  • Model choice will start to diversify. You will see more “Mamba-based” or “RWKV” models on hubs like Hugging Face, with different speed–quality–memory trade-offs.
  • Hardware utilization may change. Some of these architectures map differently to GPUs, TPUs, or custom accelerators, affecting how you scale and deploy models.
  • New failure modes and strengths will show up. For example, SSMs might shine on extremely long, structured sequences, but behave differently on short, noisy prompts.

How to keep up (and not drown)

The post-transformer space moves fast, but you do not have to read every paper. A practical way to stay ahead without getting lost:

  1. Watch a few anchor concepts:

    • “State space models” / “S4” / “Mamba”
    • “Hyena” / “subquadratic attention replacement”
    • “RWKV” / “attention-free RNN LLM”

  2. Track real-world adoption:

    • Which new open models on Hugging Face or popular inference engines are SSM-based or hybrids?
    • Are any major vendors (Anthropic, OpenAI, Google, Meta) openly acknowledging Mamba/SSM-style components in their stacks?

  3. Think in terms of capabilities:

    • When you see “Mamba-based” or “SSM-enhanced” in a model card, read it as: “likely better at long-context, streaming, or resource-limited deployment.”

Conclusion: attention is not dead, but it is sharing the stage

Self-attention unlocked the transformer era, but it is not the final word on sequence modeling. Mamba and other state space models, Hyena’s convolutional operators, and RWKV-style RNNs are all serious attempts to answer the question “what comes after attention” with architectures that scale better in time, memory, or both.

If you want to turn this into something useful today:

  1. When you are choosing or evaluating a model, start noting the sequence operator (transformer-only vs SSM vs hybrid) alongside the usual “parameter count” metric. It will increasingly matter for context length, latency, and cost.
  2. If you build AI products, experiment with an SSM- or RWKV-based model for long-context or streaming tasks (like log analysis, codebase QA, or real-time chat) and compare latency and memory use to a vanilla transformer.
  3. Set a recurring reminder—say, once a quarter—to skim a curated source (a survey, a guide, or a trusted blog) on state space models and post-transformer architectures, so you understand what is noise and what is genuinely changing the landscape.

You do not need to abandon transformers tomorrow. But the next time you hear “attention is all you need,” you will know that a growing chunk of the field politely disagrees—and is busy building what comes next.