Hybrid Language Models

Latest papers

18 papers4 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining
Nov 4, 2026João Eduardo Batista, Emil Vatai, Mohamed WahibLLM Inference Acceleration Long-Context Language Model Inference
RIKEN Center for Computational Science, Kobe, Hyogo, Japan.
Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models
Nov 1, 2026Wenqi Zeng, Chen Feng, YuAng ChenKnowledge Retrieval in Language Models Hybrid Language Models
The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University · The Chinese University of Hong Kong
Hybrid AI Patterns: Thirteen Ways to Couple Language Models with Rules, and How to Choose Among Them
Oct 29, 2026Siegfried Handschuh, Götz-Henrik Wiegand, Lorena RaichleHybrid Language Models Symbolic Reasoning in Language Models
Institute of Computer Science, University of St. Gallen (HSG)
Beyond the Sequentiality Bottleneck: A Hybrid Diffusion-Autoregressive Framework for Efficient Multimodal Reasoning
Oct 20, 2026Wei Cai, Jian Zhao, Yuchen Yuan +1Hybrid Language Models Parallel Reasoning in Language Models
Peking University · Institute of Artificial Intelligence (TeleAI), China Telecom
Competing Learning in Hybrid Architectures
Oct 19, 2026Xingyu Shen, Yingfa Chen, Ziqing Qiao +3Hybrid Language Models Hybrid Attention
Department of Science and Technology, Tsinghua University, Beijing, China
The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathKnowledge Retrieval in Language Models Hybrid Language Models
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Low-Bit Recurrent States in Hybrid Language Models
Sep 25, 2026Hongren Chen, Jiayang HeHybrid Language Models Recurrent-State Quantization
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models
Sep 20, 2026Zhiyuan MaSpeculative Decoding LLM Inference Acceleration
dQwen3.5: Hybrid-Attention Diffusion Language Models
Sep 17, 2026Anton Xue, Litu Rout, Aditya Akella +3Diffusion Language Model Decoding Diffusion Language Models
University of Texas at Austin
The Token Before the Value Is the Key
Sep 14, 2026Ke Cheng, Xin Xu, Zhichao Chen +7Language Modeling Circuits in Language Models
AMS, Tencent · CCSE lab, Beihang University · Data Platform, Tencent +2
LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
Sep 7, 2026Simon P. VillaniHybrid Language Models Language Modeling
An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
Aug 28, 2026Christos KoutsiarisHybrid Language Models Decoder-Only Language Models
Development Expert, SAP P&E, Cloud ERP, UX Foundation
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Parameter Sharing Transformer Attention
Carnegie Mellon University · Google Research
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?
Apr 23, 2026Shivam Rawat, Lucie Flek, Florian Mai +1CoT Reasoning LLM Memory
Lamarr Institute for Machine Learning and Artificial Intelligence · Rheinische Friedrich-Wilhelms-Universität Bonn
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Mar 23, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóHybrid Language Models Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain

Nov 4, 2026J/K move · Enter open · S save

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

João Eduardo Batista, Emil Vatai, Mohamed Wahib

RIKEN Center for Computational Science, Kobe, Hyogo, Japan.

LLM Inference Acceleration Long-Context Language Model Inference Hybrid Language Models Hybrid Attention Sparse Attention

Open paper AskPDF arXiv

The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long contexts. We accelerate these remaining layers without any training or weight changes using combined sparse attention: each layer attends to a sliding window of recent tokens, a pinned prefix, and a fixed budget of most relevant tokens in the context. The window size and budget are found on a per-layer basis using a multi-objective search algorithm. Across six hybrid models from three families, from 4B to 397B parameters, the selected configurations preserve their baseline model's accuracy on 21 long-context datasets. Decoding 64k/256k tokens becomes up to 1.26x/2.22x faster on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B.