TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory

Hanxin Zhang

cs.LGOct 14, 2026

TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory

Authors: Hanxin Zhang

Organizations: Department of Neurology, University of Pittsburgh

Abstract

We introduce TurboMamba, a language-model architecture for faster inference, faster training, and better token prediction. A global Mamba-3 model runs once per completed token block. A small Transformer uses this global context and earlier tokens within the block to predict the next token. Three lightweight additions improve local predictions: direct access to the previous block's last token, a small correction to next-token scores, and a learned mixture with recent-token frequencies. We compare twelve small-scale implementations on public FineWeb-Edu, with comparable parameter counts and shared training-budget caps. The baselines include Transformer, Mamba-3, Jamba, and Zamba. The Jamba and Zamba versions use Mamba-3 in place of their original Mamba modules. TurboMamba achieves the lowest mean test prediction loss (negative log-likelihood, NLL) among all twelve models. Its mean NLL is 9.4% lower than that of Mamba-3, 7.3% lower than that of the Jamba version, and 18.3% lower than that of the Zamba version. End-to-end training throughput is 2.60x that of Mamba-3, 2.81x that of the Jamba version, and 4.82x that of the Zamba version. We measure end-to-end batch latency for a batch of 32 sequences, each with a 256-token prompt and 128 generated tokens. On this workload, TurboMamba achieves a 3.12x speedup over Mamba-3, a 3.53x speedup over the Jamba version, and a 6.21x speedup over the Zamba version.

Figures & tables

Architecture	Global computation	Local prediction	Added detail or routing
TurboMamba	Mamba-3 per fixed block	Small Transformer with global prefix	Boundary connection, residual head, count mixture
Jamba	Token-level Mamba and attention	Token-level hybrid layers	Top-2 experts in selected layers
Zamba	Token-level Mamba	Repeated shared attention branch	Original-embedding concatenation; shared attention/feedforward
BST	Token-level state-space model	Block-local self/cross-attention	State-space context in local attention
MBLM	Fixed multiscale hierarchy	Local decoder with global context vectors	Separate stage embeddings
H-Net	Learned dynamic chunks	Fine encoder and decoder around main model	Learned boundaries; smoothed chunk expansion

Table 1: Mechanism comparison. The rows describe architectural designs; the experiments use matched small-scale implementations. TurboMamba combines blockwise global computation with three additions to its local prediction path.

Figure 1: Training uses a known token sequence. The global scan is causal over

B

-token blocks, and its outputs are shifted by one block so each target block receives only earlier global context. Once these prefixes are available, the local causal decoders for different blocks can run in parallel. Every target uses earlier token inputs and counts of earlier tokens.

Figure 2: Generation predicts and commits one token at a time. The local prediction path runs at every token; each commit updates recent counts and the partial block. The global Mamba-3 state advances only after a complete

B

-token block, then the local cache resets. Sampling remains sequential. The final generated token is committed, including any completed-block global update.

Model	Parameters (M)	Test NLL $\downarrow$	Training-step throughput $\uparrow$	End-to-end training throughput $\uparrow$
Transformer	3.000	$5.290\pm 0.127$	$18.29\pm 2.86$	$17.50\pm 2.79$
Mamba-3 SISO	3.001	$5.455\pm 0.056$	$8.01\pm 0.08$	$6.83\pm 0.07$
BT	3.000	$5.201\pm 0.013$	$\mathbf{23.56}\pm 0.23$	$\mathbf{22.67}\pm 0.22$
BST (S4D)	3.000	$5.638\pm 0.010$	$10.06\pm 0.14$	$9.45\pm 0.14$
Cache Transformer	3.000	$5.001\pm 0.116$	$18.29\pm 2.86$	$17.50\pm 2.79$
Pointer Sentinel	3.016	$5.185\pm 0.098$	$16.14\pm 0.37$	$15.41\pm 0.38$

Table 2: Prediction quality, Training-step throughput, and End-to-end training throughput. Values are means

\pm

sample SD across three seeds. Both throughput metrics use thousands of scored target tokens/s; End-to-end training throughput covers the complete formal job. Bold numbers identify the best metric-column means.

Model	$n=1$	$n=4$	$n=8$	$n=16$	$n=32$
Transformer	$215.6$	$292.3$	$363.8$	$735.4$	$1457.1$
Mamba-3 SISO	$336.0$	$456.2$	$519.3$	$856.3$	$1595.5$
BT	$\mathbf{150.5}$	$204.8$	$\mathbf{234.8}$	$\mathbf{251.6}$	$\mathbf{269.9}$
BST (S4D)	$413.8$	$572.6$	$650.9$	$743.1$	$718.5$
Cache Transformer	$232.0$	$317.7$	$393.0$	$824.1$	$1503.5$
Pointer Sentinel	$247.9$	$326.9$	$424.8$	$803.5$	$1486.0$

Table 3: End-to-end batch latency (ms per batch, lower is better). Each of the

n

independent sequences has a 256-token prompt and 128 generated tokens. Values are means of three per-seed medians. Bold numbers identify the lowest mean in each batch column.

Figure 3: Test NLL and End-to-end training throughput (left); End-to-end batch latency for 256 prompt tokens and 128 generated tokens per sequence (right). Both panels include all twelve implementations. Each point or curve summarizes three formal seeds. Blue and rust highlight TurboMamba and Mamba-3. The other models, including the Plain block model, use gray lines and markers, with different line styles and marker shapes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	DEV pilots (s)	Formal training (s)	Total (s)
Transformer	$52.3$	$528.8$	$581.1$
Mamba-3 SISO	$50.7$	$513.3$	$564.0$
BT	$52.4$	$527.6$	$580.0$
BST (S4D)	$52.1$	$525.0$	$577.1$
Cache Transformer	$52.3$	$528.8$	$585.3$
Pointer Sentinel	$52.8$	$528.3$	$581.1$

Appendix

Table 4: Recorded training and model-selection costs (s). DEV is the two-pilot total; formal is the three-seed training total used for End-to-end training throughput. Cache total includes 4.250 s of adapter selection in addition to its full Transformer backbone training. Entries are rounded to one decimal.

Model	$n=1$	$n=4$	$n=8$	$n=16$	$n=32$
Transformer	$111.0$	$127.5$	$164.3$	$225.5$	$408.6$
Mamba-3 SISO	$173.9$	$200.6$	$262.1$	$380.3$	$774.7$
BT	$\mathbf{77.2}$	$\mathbf{89.4}$	$113.1$	$\mathbf{141.4}$	$\mathbf{119.5}$
BST (S4D)	$211.6$	$249.2$	$313.0$	$392.3$	$327.5$
Cache Transformer	$117.5$	$136.4$	$174.6$	$236.7$	$430.0$
Pointer Sentinel	$122.8$	$141.5$	$183.5$	$244.9$	$435.5$

Appendix

Table 5: End-to-end batch latency (ms per batch) for

n

independent sequences, each with a 128-token prompt and 64 generated tokens. Same aggregation and clock boundaries as Table 3 . Bold numbers identify the lowest mean in each batch column.

Comparator	Mean difference	95% interval
Transformer	$-0.349$	$[-0.422,-0.228]$
Mamba-3 SISO	$-0.515$	$[-0.544,-0.483]$
BT	$-0.261$	$[-0.281,-0.237]$
BST (S4D)	$-0.698$	$[-0.734,-0.659]$
Cache Transformer	$-0.061$	$[-0.130,+0.043]$
Pointer Sentinel	$-0.244$	$[-0.358,-0.130]$

Appendix

Table 6: Paired test NLL differences (nats per target token), TurboMamba minus comparator. Exploratory 95% intervals use paired seed/document resampling of the three-seed experiment.

Model	GPT-2 NLL/token	GPT-2 NLL/byte	Word 4-gram repeat (%)
Transformer	$7.254$	$1.927$	$\mathbf{0.000}$
Mamba-3 SISO	$6.307$	$1.971$	$0.030$
BT	$6.364$	$1.746$	$\mathbf{0.000}$
BST (S4D)	$6.366$	$1.878$	$\mathbf{0.000}$
Cache Transformer	$6.518$	$\mathbf{1.731}$	$0.968$
Pointer Sentinel	$\mathbf{6.049}$	$1.810$	$5.044$

Appendix

Table 7: Generated-text diagnostics, three-seed means. GPT-2 NLL uses nats per GPT-2 token or UTF-8 byte. It and word four-gram repetition are lower-is-better proxies; true-continuation prediction is in Table 2 .

Hybrid Transformer-State Space Model Architectures

See all 18 papers

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
Mamba-3: Improved Sequence Modeling using State Space Principles
Mar 16, 2026Aakash Lahoti, Kevin Y. Li, Berlin Chen +5Sequence Modeling Efficient Language Model Inference
Carnegie Mellon University · Princeton University · Together AI +1
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Jun 12, 2026NVIDIA, :, Aaron Blakeman +571Mixture-of-Experts Language Models Long-Context Language Model Inference
Allan

Apr 14, 2026J/K move · Enter open · S save

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

NVIDIA, :, Aakshita Chandiramani, Aaron Blakeman+543

Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models Language Model Pretraining Hybrid Mamba-Transformer Architectures LLM Inference Acceleration

Open paper AskPDF arXiv

We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.

Abstract

Hybrid Transformer-State Space Model Architectures

Explore similar work

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

Mamba-3: Improved Sequence Modeling using State Space Principles

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning