Hybrid Mamba-Transformer Architectures

Latest papers

23 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
Oct 28, 2026NVIDIA, :, Aastha Jhunjhunwala +556LLM Agent Training Mixture-of-Experts Language Models
LayoutMamba: Efficient Document Layout Analysis with Hybrid Mamba-Transformer Networks
Oct 26, 2026Kehua Chen, Mingyi ChenDocument Layout Analysis Hybrid Mamba-Transformer Architectures
Hybrid Mamba-Transformer Network for Single-Shot 3D Reconstruction from Fringe Patterns
Oct 25, 2026Xinjie Zhou, XiaoLei Li3D Reconstruction Fringe Projection Profilometry
Aulin College, Northeast Forestry University, Harbin,150040, China · School of Mechanical Engineering, Hebei University of Technology, Tianjin, 300130, China
Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language Models Long-Context Language Modeling
IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher
TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory
Oct 14, 2026Hanxin ZhangHybrid Transformer-State Space Model Architectures Autoregressive Language Modeling
Department of Neurology, University of Pittsburgh
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter Transfer Hybrid Transformer-State Space Model Architectures
Seoul National University · SB Intuitions · LG AI Research +1
MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Model
Sep 22, 2026Adithi Shankar, Gopika Krishnan, Gloria Haro +2Hybrid Mamba-Transformer Architectures Audio Source Separation
Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain · Intelligent Multimodal Vision Analysis Group, Universitat Pompeu Fabra, Barcelona, Spain · Serra Húnter Fellow Programme, Universitat Pompeu Fabra, Barcelona, Spain
HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads
Aug 19, 2026Jiahao Lin, Alish Kanani, Sangwan Lee +2Hybrid Mamba-Transformer Architectures LLM Serving
University of Wisconsin–Madison · University of Ulsan
A Sovereign, Open-Source Foundation Model for German and English
Jul 10, 2026The Soofi-Team, :, Benedikt Droste +29Language Model Pretraining Multilingual Language Models
ellamind · Fraunhofer IAIS · Lamarr +8
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
Jul 2, 2026Pranshu Chaturvedi, Parth Shroff, Tarun Suresh +2Blockwise Diffusion Language Models Diffusion Language Models
Stanford University
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Jun 12, 2026NVIDIA, :, Aaron Blakeman +571Mixture-of-Experts Language Models Long-Context Language Model Inference
Allan
Forget Attention: Importance-Aware Attention Is All You Need
Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax Attention Long-Context Language Modeling
Department of Computer Engineering, Kangwon National University
Zamba2-VL Technical Report
May 29, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Efficient VLM Inference
Zyphra, San Francisco, CA
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6State Space Models Long-Context Language Model Inference
AWS Agentic AI
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
May 3, 2026Xing Lei, Jincheng Wang, Xuetao Zhang +1Hybrid Mamba-Transformer Architectures Offline RL
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Computer Science, University College London · School of Engineering, Westlake University
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit
Mar 22, 2026Khushiyant, Param ThakkarAudio Representation Learning Hybrid Mamba-Transformer Architectures
Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany · Department of Computer Engineering and Information Technology, Veermata Jijabai Technological Institute, Mumbai, India
InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
Mar 8, 2026Youjin Wang, Jiaqiao Zhao, Rong Fu +5Hybrid Mamba-Transformer Architectures Hybrid Transformer-State Space Model Architectures
Renmin University of China, Beijing, China · University of Macau, Macau SAR, China · Central South University, Changsha, China +1
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model Architectures Text Summarization
Independent Researcher · University of Memphis
VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification
Mar 1, 2026Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche +4Hybrid Mamba-Transformer Architectures Hyperspectral Image Classification
Institute of Applied Sciences and Intelligent Systems (ISASI), CNR, 73100 Lecce, Italy · Dept. of Engineering for Innovation, University of Salento, 73100 Lecce, Italy · VSC Laboratory, Department of Electronics and Automation, University of Biskra, Biskra, Algeria +4
Stateful Token Reduction for Long-Video Hybrid VLMs
Feb 27, 2026Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7Hybrid Mamba-Transformer Architectures Efficient VLM Inference
NVIDIA
HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
Sep 28, 2025Md Mozaharul Mottalib, Thao-Ly T. Phan, Rahmatollah BeheshtiRepresentation Learning Hybrid Mamba-Transformer Architectures
University of Delaware Newark, Delaware, USA · Thomas Jefferson University & Nemours Children’s Health Wilmington, Delaware, USA

Oct 28, 2026J/K move · Enter open · S save

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

NVIDIA, :, Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava+555

LLM Agent Training Mixture-of-Experts Language Models Multimodal Large Language Models LLM Inference Acceleration Hybrid Mamba-Transformer Architectures Software Engineering Agents

Open paper AskPDF arXiv

We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training. We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi- teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification. Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6x fewer active parameters. Post-training MTP boosting improves speculative- decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user.