Hybrid Transformer-State Space Model Architectures

Latest papers

18 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory
Oct 14, 2026Hanxin ZhangHybrid Transformer-State Space Model Architectures Autoregressive Language Modeling
Department of Neurology, University of Pittsburgh
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter Transfer Hybrid Transformer-State Space Model Architectures
Seoul National University · SB Intuitions · LG AI Research +1
The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathHybrid Language Models Hybrid Transformer-State Space Model Architectures
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Tydra: An Efficient Hybrid Model for Tabular Data
Aug 21, 2026Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2Hybrid Transformer-State Space Model Architectures Tabular Foundation Models
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
Jun 15, 2026Kuzey Torlak, Hüseyin Arda Arslan, Anıl Dervişoğlu +2Long-Context Language Modeling Hybrid Transformer-State Space Model Architectures
Kadıköy Anadolu High School · Politecnico di Torino · Istanbul Technical University +2
Forget Attention: Importance-Aware Attention Is All You Need
Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax Attention Long-Context Language Modeling
Department of Computer Engineering Kangwon National University
Zamba2-VL Technical Report
May 29, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Efficient VLM Inference
Zyphra, San Francisco, CA
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenKV Caching Memory-Efficient Optimization
Codepawl Ho Chi Minh City, Vietnam
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6State Space Models Long-Context Language Model Inference
AWS Agentic AI
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation
Mar 23, 2026Duy D. Nguyen, Phat T. Tran-TruongHybrid Transformer-State Space Model Architectures 3D Medical Image Segmentation
Department of Software Engineering, Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), 268 Ly Thuong Kiet Street, Dien Hong Ward, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Linh Trung Ward, Ho Chi Minh City, Vietnam
Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
Mar 9, 2026John Cooper, Ilias Diakonikolas, Mingchen Ma +1Hybrid Transformer-State Space Model Architectures Sequence Modeling
Department of Computer Sciences, University of Wisconsin-Madison
InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
Mar 8, 2026Youjin Wang, Jiaqiao Zhao, Rong Fu +5Hybrid Mamba-Transformer Architectures Hybrid Transformer-State Space Model Architectures
Renmin University of China, Beijing, China · University of Macau, Macau SAR, China · Central South University, Changsha, China +1
DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization
Mar 5, 2026Xiaodong Zhu, Suting Wang, Yuanming Zheng +5Hybrid Transformer-State Space Model Architectures Temporal Video Forgery Localization
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1In-Context Retrieval Transformer
University of Edinburgh · Heriot-Watt University
Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model Architectures Text Summarization
Independent Researcher · University of Memphis

Oct 14, 2026J/K move · Enter open · S save

TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory

Hanxin Zhang

Department of Neurology, University of Pittsburgh

Hybrid Transformer-State Space Model Architectures Autoregressive Language Modeling Efficient Language Model Training LLM Inference Acceleration Hybrid Mamba-Transformer Architectures

Open paper AskPDF arXiv

We introduce TurboMamba, a language-model architecture for faster inference, faster training, and better token prediction. A global Mamba-3 model runs once per completed token block. A small Transformer uses this global context and earlier tokens within the block to predict the next token. Three lightweight additions improve local predictions: direct access to the previous block's last token, a small correction to next-token scores, and a learned mixture with recent-token frequencies. We compare twelve small-scale implementations on public FineWeb-Edu, with comparable parameter counts and shared training-budget caps. The baselines include Transformer, Mamba-3, Jamba, and Zamba. The Jamba and Zamba versions use Mamba-3 in place of their original Mamba modules. TurboMamba achieves the lowest mean test prediction loss (negative log-likelihood, NLL) among all twelve models. Its mean NLL is 9.4% lower than that of Mamba-3, 7.3% lower than that of the Jamba version, and 18.3% lower than that of the Zamba version. End-to-end training throughput is 2.60x that of Mamba-3, 2.81x that of the Jamba version, and 4.82x that of the Zamba version. We measure end-to-end batch latency for a batch of 32 sequences, each with a 256-token prompt and 128 generated tokens. On this workload, TurboMamba achieves a 3.12x speedup over Mamba-3, a 3.53x speedup over the Jamba version, and a 6.21x speedup over the Zamba version.