Hybrid Transformer-State Space Model Architectures

Follow this topic Ask about this topic

Latest papers

20 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Shallow Encoding Emits Deep Memories
Nov 4, 2026Mingyuan Ma, Yangmin Li, Harris Nover +4LLM Inference Acceleration Hybrid Transformer-State Space Model Architectures
NVIDIA
Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention
Oct 30, 2026Yannis Bendi-Ouis, Xavier HinautHybrid Transformer-State Space Model Architectures Memory-Augmented Language Models
Mnemosyne
TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory
Oct 14, 2026Hanxin ZhangHybrid Transformer-State Space Model Architectures Autoregressive Language Modeling
Department of Neurology, University of Pittsburgh
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter Transfer Hybrid Transformer-State Space Model Architectures
Seoul National University · SB Intuitions · LG AI Research +1
The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathKnowledge Retrieval in Language Models Hybrid Language Models
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Tydra: An Efficient Hybrid Model for Tabular Data
Aug 21, 2026Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2Hybrid Transformer-State Space Model Architectures Tabular Foundation Models
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
Jun 15, 2026Kuzey Torlak, Hüseyin Arda Arslan, Anıl Dervişoğlu +2Long-Context Language Modeling Hybrid Transformer-State Space Model Architectures
Kadıköy Anadolu High School · Politecnico di Torino · Istanbul Technical University +2
Forget Attention: Importance-Aware Attention Is All You Need
Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax Attention Long-Context Language Modeling
Department of Computer Engineering, Kangwon National University
Zamba2-VL Technical Report
May 29, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Efficient VLM Inference
Zyphra, San Francisco, CA
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenKV Caching Memory-Efficient Optimization
Codepawl Ho Chi Minh City, Vietnam
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6State Space Models Long-Context Language Model Inference
AWS Agentic AI
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation
Mar 23, 2026Duy D. Nguyen, Phat T. Tran-TruongHybrid Transformer-State Space Model Architectures 3D Medical Image Segmentation
Department of Software Engineering, Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), 268 Ly Thuong Kiet Street, Dien Hong Ward, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Linh Trung Ward, Ho Chi Minh City, Vietnam
Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
Mar 9, 2026John Cooper, Ilias Diakonikolas, Mingchen Ma +1Hybrid Transformer-State Space Model Architectures Sequence Modeling
Department of Computer Sciences, University of Wisconsin-Madison
InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
Mar 8, 2026Youjin Wang, Jiaqiao Zhao, Rong Fu +5Hybrid Mamba-Transformer Architectures Hybrid Transformer-State Space Model Architectures
Renmin University of China, Beijing, China · University of Macau, Macau SAR, China · Central South University, Changsha, China +1
DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization
Mar 5, 2026Xiaodong Zhu, Suting Wang, Yuanming Zheng +5Hybrid Transformer-State Space Model Architectures Temporal Video Forgery Localization
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1In-Context Retrieval Transformer
University of Edinburgh · Heriot-Watt University
Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model Architectures Text Summarization
Independent Researcher · University of Memphis

Nov 4, 2026J/K move · Enter open · S save

Shallow Encoding Emits Deep Memories

Mingyuan Ma, Yangmin Li, Harris Nover, Zheyu Fu+3

NVIDIA

LLM Inference Acceleration Hybrid Transformer-State Space Model Architectures Low-Rank Approximation

Open paper AskPDF arXiv

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers, so every layer keeps a memory of earlier tokens: a key-value cache or a fixed-size recurrent state. Serving them spends more compute and high-bandwidth memory on these memories than the output requires. The prefill runs every layer for every prompt token, and every decode step loads and stores whole recurrent states, which on Qwen3.8-Flash-Next take as many bytes per request as the attention cache of thousands of tokens. We find two properties of trained hybrids that locate this excess. First, writes are easier than reads: writing a deep layer's memory of a token needs little of the deep computation that predicting the next token needs. Trained copies of the deep layers' write paths, which we call emitters, therefore write a prompt's deep memories from the residual stream of a shallower anchor layer, and all prompt tokens but the last run as few as 63% of the layers. Second, recurrent states have a state sink: most of their energy lies in a rank-one term along a fixed direction of each head, and the rest is a low-rank working memory. SEED builds on both: it makes the prefill shallow in depth and the decode state shallow in representation. Each recurrent state is stored as its sink plus low-rank factors, which the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates extend exactly; we prove that the open-loop error of the periodic compressions back to low rank is bounded independently of their number. SEED keeps the original weights frozen, compresses no attention cache, and runs all layers for every generated token. On six checkpoints of five production hybrids of 30B to over 300B parameters, it is nearly lossless on eleven official exams. Served with SGLang on NVIDIA GB300 GPUs on an agentic-coding trace, it raises the per-GPU throughput of Qwen3.8-Flash-Next 1.2-1.4x.