Hybrid Language Models

Follow this topic Ask about this topic

Latest papers

14 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

Competing Learning in Hybrid Architectures
Oct 19, 2026Xingyu Shen, Yingfa Chen, Ziqing Qiao +3Hybrid Language Models Hybrid Attention
Department of Science and Technology, Tsinghua University, Beijing, China
The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathHybrid Language Models Hybrid Transformer-State Space Model Architectures
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Low-Bit Recurrent States in Hybrid Language Models
Sep 25, 2026Hongren Chen, Jiayang HeHybrid Language Models Language Modeling
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models
Sep 20, 2026Zhiyuan MaSpeculative Decoding LLM Inference Acceleration
dQwen3.5: Hybrid-Attention Diffusion Language Models
Sep 17, 2026Anton Xue, Litu Rout, Aditya Akella +3Diffusion Language Model Decoding Diffusion Language Models
University of Texas at Austin
The Token Before the Value Is the Key: How Hybrid Architectures Organize Induction Circuits
Sep 14, 2026Ke Cheng, Xin Xu, Yixiao Chen +6Language Modeling Circuits in Language Models
AMS, Tencent · Data Platform, Tencent · Independent Researcher +1
LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
Sep 7, 2026Simon P. VillaniHybrid Language Models Language Modeling
An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
Aug 28, 2026Christos KoutsiarisHybrid Language Models Decoder-Only Language Models
Development Expert, SAP P&E, Cloud ERP, UX Foundation
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Parameter Sharing Transformer Attention
Carnegie Mellon University · Google Research
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?
Apr 23, 2026Shivam Rawat, Lucie Flek, Florian Mai +1CoT Reasoning LLM Memory
Lamarr Institute for Machine Learning and Artificial Intelligence · Rheinische Friedrich-Wilhelms-Universität Bonn
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Mar 23, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóHybrid Language Models Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain

Oct 19, 2026J/K move · Enter open · S save

Competing Learning in Hybrid Architectures

Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao+2

Department of Science and Technology, Tsinghua University, Beijing, China

Hybrid Language Models Hybrid Attention In-Context Retrieval

Open paper AskPDF arXiv

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.