Transformers have been the standard architecture for large language models, but the self-attention mechanism imposes quadratic complexity—making inference computationally expensive. As context length grows, so does the KV-cache, overwhelming GPU memory. And because attention operates over fixed context, models often lose track of global information, forcing workarounds like chunking and retrieval.

In December 2023, Albert Gu and Tri Dao introduced Mamba—a selective state-space model designed to improve throughput and efficiency without relying on attention. With input-dependent parameters and content-aware computation, Mamba offered a new framework for scalable sequence modeling.

This piece traces the architectural shifts shaping the rise of hybrid LLMs. Each entry marks a move away from the Transformer default—and a step toward what comes next.

Mamba hybrid LLMs timeline

Mamba (Gu & Dao, December 2023)

Paper
Authors: Albert Gu, Tri Dao

The original Mamba paper introduced a selective state-space model (SSM) that achieved linear-time inference and 5× throughput gains over Transformers. By making core parameters input-dependent, the model enabled selective, content-aware processing of sequences—without relying on attention or MLP blocks. The architecture combined efficiency with generalization, showing strong results across language, audio, and genomics. This opened the door for a new class of models—simple, fast, and expressive—ready to power foundation models beyond the Transformer era.

Jamba (AI21 Labs, March 2024)

Paper
Authors: Opher Lieber, Barak Lenz

Jamba introduced the first large-scale hybrid Transformer–Mamba–MoE language model. Its architecture interleaved attention and Mamba layers at a 1:7 ratio, with MoE layers added every two blocks—balancing long-context reasoning with efficient computation. The model supported context lengths up to 256K tokens while maintaining a compact memory footprint and high throughput. The team demonstrated that hybrid design choices—down to layer ordering and MoE sparsity—directly impact model quality and deployability. Jamba set a precedent for hybrid models built to scale, releasing both weights and architecture under Apache 2.0.

MambaVision (NVIDIA, July 2024)

Paper
Authors: Ali Hatamizadeh, Jan Kautz

MambaVision reimagined Mamba for computer vision, combining it with Transformer blocks in a hierarchical layout. Early layers used convolutional blocks for efficient feature extraction; later stages added self-attention to recover long-range spatial dependencies. The result was a Pareto-optimal backbone achieving state-of-the-art accuracy and image throughput on ImageNet-1K. The team showed that Mamba’s sequential strengths could complement ViT-style architectures without incurring latency costs. MambaVision stood out for its architectural pragmatism—adapting recurrence for spatial reasoning at scale.

Codestral Mamba (Mistral, July 2024)

Blog
Authors: Mistral AI team

Codestral Mamba marks Mistral’s first pure Mamba2-based code model, trained with guidance from Mamba’s original authors. Unlike Transformers, Mamba offers linear-time inference and the capacity to model sequences of theoretically infinite length—capabilities that suit large-context code workflows. The 7.3B parameter model supports up to 256K context tokens and is optimized for retrieval and code reasoning.

Jamba 1.5 (AI21 Labs, August 2024)

Paper
Authors: Barak Lenz and Opher Lieber

AI21 Labs scaled its hybrid Mamba–Transformer architecture to 398B total parameters with 94B active—marking the first large-scale deployment of such a model. Jamba 1.5 interleaves Mamba and attention layers across 72 layers, using grouped-query attention, low-rank adaptation, and 16 MoE experts for efficient routing. It supports 256K-token context and achieves top scores on long-context benchmarks, including state-of-the-art performance on NVIDIA’s RULER. Designed for real-world enterprise applications, Jamba 1.5 balances compute efficiency with generalization, showing that hybrid architectures can scale without compromising inference speed or accuracy.

Mamba-Llama (Together AI, August 2024)

Paper
Authors: Dmitriy Boylov, Ansh Radhakrishnan, Constantine Caramanis, Alexander M. Rush, Tri Dao

Together AI distilled LLaMA-3-8B-Instruct into a hybrid model by replacing 75% of its attention layers with Mamba blocks. The resulting architecture reduced inference latency by up to 5× while preserving chat performance across human evaluations and MMLU/ARC benchmarks. The team used exact weight mapping to retain instruction tuning and applied iterative distillation to recover quality after architectural modification. The model supports 128K context and is open-sourced under Apache 2.0. The main contribution of this work is showing that Transformer models can be “converted” into more efficient Mamba variants with relatively little compute.

Falcon Mamba (Technology Innovation Institute, July 2025)

Paper
Authors: Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, Hakim Hacid

Falcon Mamba 7B is a pure Mamba-based language model trained on 5.8 trillion tokens with 7.27B parameters. It uses a state-space model architecture without any attention layers and supports arbitrary sequence lengths during inference. The model outperforms LLaMA3.1-8B, Mistral 7B, Falcon2-11B, and several hybrid models on benchmarks such as MMLU, GSM8K, ARC, and MuSR. It maintains constant memory and throughput across long sequences.

Nemotron-H (NVIDIA, April 2025)

Paper
Authors: NVIDIA team

NVIDIA introduced Nemotron-H, a family of hybrid Mamba–Transformer models (8B, 47B, and 56B) optimized for inference-time scaling and speed. By replacing 92% of attention layers with Mamba2 blocks, Nemotron-H models deliver up to 3× faster throughput than similarly sized Transformers like LLaMA-3.1 and Qwen-2.5, while matching or exceeding accuracy on MMLU, GSM8K, HumanEval, and MATH. The 56B model was trained using an FP8-based recipe and compressed to 47B using MiniPuzzle, NVIDIA’s new distillation framework. All weights are open-sourced via Hugging Face and NeMo. Nemotron-H affirms that hybrid architectures can outperform traditional Transformers at scale—without compromising reasoning performance.

Bamba (IBM, April 2025) 

Blog
Authors: IBM Research (Lead: Raghu Ganti)

IBM released Bamba-9B, an open-source hybrid model combining the expressivity of Transformers with the efficiency of state-space models (SSMs). Built on Mamba2 and optimized for inference performance, Bamba achieved 2× throughput over comparable Transformer models by reducing KV-cache memory overhead. Trained on 3 trillion tokens and compressed via quantization from 18GB to 9GB, Bamba matched the accuracy of LLaMA-3.1-8B despite using 7× less data. The model integrates natively with vLLM and supports 32K-token inference—positioning it as a high-performance foundation for long-context workloads.

Hunyuan TurboS (Tencent, May 2025) 

Paper
Authors: Tencent Hunyuan team

Hunyuan-TurboS is a hybrid Transformer–Mamba2–MoE language model with 560b total parameters and 56b active during inference. The model architecture follows an interleaved Attention-Mamba-Feed-Forward pattern across 128 layers, combining grouped-query attention, linear-time Mamba2 blocks, and a 32-expert MoE configuration (2+1 experts active per token). It supports 256K-token context and is pre-trained on 16T tokens. Optimizations include adopting fp32 precision for Mamba states at the kernel level, which improves long-context generation quality. A chain-of-thought fusion mechanism enables dynamic selection between short and long reasoning strategies.

Phi-4-mini-flash-reasoning (Microsoft, July 2025) 

Blog
Authors: Weizhu Chen, Jianfeng Gao, Liliang Ren

Microsoft unveiled Phi-4-mini-flash-reasoning, a 3.8B-parameter open model built for low-latency reasoning in compute-constrained environments. It introduces a decoder-hybrid-decoder architecture—codenamed SambaY—that combines Mamba (SSM), sliding window attention, and Gated Memory Units (GMUs) across interleaved layers. The architecture preserves linear prefill complexity while achieving up to 10× higher throughput and 2–3× lower latency than its predecessor. Supporting 64K context and fine-tuned on synthetic math benchmarks, Phi-4-mini-flash-reasoning targets education and real-time agent use cases on edge devices.

Final Thought

What began as academic research is now influencing how foundation models are built. From Mamba to Jamba to Bamba, we’re seeing structured alternatives challenge the dominance of pure Transformers—not just in benchmarks, but in real-world deployment. In the next two years, hybrid and state-space architectures may become the new default. We’re proud to contribute to this next chapter in AI model design.