Speech Language Models

Follow this topic Ask about this topic

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan
Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
Jul 7, 2026Zhenyu Liu, Xuanyu Zhang, Yunxin Li +10Semantic Alignment Speech Language Models
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Center for Language, Intelligence and Machines, Shenzhen Loop Area Institute, Shenzhen, China · 4PCG, Tencent, Shenzhen, China +1
WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
Jul 7, 2026Sihang Nie, Jinxin Ji, Xiaofen Xing +4TTS Synthesis Speech Prosody
South China University of Technology · Huya Inc. · Tongji University +2
NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task
Jul 6, 2026Anand Kamble, Aniket TatheSpeech Language Models Multimodal Instruction Tuning
Florida State University · University of Illinois Urbana-Champaign
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
Jul 6, 2026Thomas Thebaud, Yuzhe Wang, Hao Zhang +5Audio-Language Model Evaluation Language Model Generation Evaluation
∗Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · †Amazon Inc., Seattle, WA, USA
LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering
Jul 2, 2026Nina Hosseini-Kivanani, Marco Matassoni, Alessio BruttiSpoken Language Understanding Synthetic Data Augmentation
Radio Télévisioun Lëtzebuerg (RTL) · University of Luxembourg · Fondazione Bruno Kessler (FBK)
GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech
Jul 2, 2026Antonis Asonitis, Francesco Verdini, Aref Farhadipour +4TTS Synthesis Zero-Shot TTS
AGIGO · ETH Zurich · 1AGIGO 2ETH Zurich +4
Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
Jul 2, 2026Congrui Du, Yang Zhang, Kaizhi Qian +1Language Model Pretraining Instruction Following
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
Jul 2, 2026Haoran Wang, Jinchuan Tian, Siddhant Arora +1Audio Understanding Speech Language Models
Carnegie Mellon University
Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
Jul 2, 2026Ruchao Fan, Yiming Wang, Rui Zhao +10Medical Speech Recognition Automatic Speech Recognition
Microsoft, USA
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Jul 1, 2026Siyi Wang, James Bailey, Ting DangLanguage Model Steering Conditional Flow Matching
The University of Melbourne, Australia · Monash University, Australia
FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates
Jun 30, 2026Jiaqi Li, Chaoren Wang, Xiaohai Tian +9Speech Processing Speech Language Models
1The Chinese University of Hong Kong, Shenzhen · 2ByteDance
Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation
Jun 29, 2026Yuxuan Hu, Heng Lu, Ruchao Fan +8TTS Synthesis Spoken Dialogue Systems
Microsoft, USA
VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
Jun 28, 2026Piyush Arora, Navlika Singh, Umberto Cappellazzo +2Automatic Speech Recognition Audio-Visual Speech Recognition
Imperial College London, UK · NatWest AI Research, UK
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
Jun 27, 2026Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso +9Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Idiap Research Institute, Martigny, Switzerland · Universite de Toulon, Aix Marseille Univ, LIS, CNRS, Toulon, France · EPFL, Lausanne, Switzerland +3
HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
Jun 26, 2026Artem Ploujnikov, Francesco Verdini, Samir Sadok +1Neural Audio Codecs Multimodal Token Compression
Mila, Quebec AI Institute, Canada · Concordia University, Canada · Sapienza University of Rome, Italy +1
RedVox: Safety and Fairness Gaps in Speech Models Across Languages
Jun 25, 2026Beatrice Savoldi, Sara Papi, Wafa Aissa +2Audio-Language Model Evaluation AI Safety Evaluation
Fondazione Bruno Kessler, Italy
FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following
Jun 25, 2026Zhihang Xie, Marco Gaido, Sara Papi +2Audio-Language Model Evaluation Speech Processing
Fondazione Bruno Kessler · University of Trento
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Jun 24, 2026Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu +2Audio-Language Model Evaluation Voice Agent Evaluation
New York University · NVIDIA · Carnegie Mellon University +1
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Jun 24, 2026Tomoya Mizumoto, Yusuke FujitaAudio Representation Learning Self-Supervised Speech Representation Learning
SB Intuitions
Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models
Jun 24, 2026Tomoya Mizumoto, Yusuke Fujita, Hao Shi +3Audio-Language Model Evaluation Speech Language Models
SB Intuitions, Tokyo, Japan
ZONOS2 Technical Report
Jun 23, 2026Gabriel Clark, Sofian Mejjoute, Mohamed Osman +2TTS Synthesis Streaming TTS Synthesis
Zyphra · San Francisco, CA
On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models
Jun 22, 2026Shunsuke Kando, Wataru Nakata, Shinnosuke Takamichi +1Speech Processing Speech Language Models
The University of Tokyo, Japan · Keio University, Japan
Interleaved Speech Language Models Latently Work In Text
Jun 21, 2026Talia Sternberg, Gallil Maimon, Yossi AdiLLM Interpretability Speech Language Models
The Hebrew University of Jerusalem
Post-Training Speech Enhancement Language Models with Perceptual Rewards
Jun 19, 2026Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis +1Multi-Objective Reinforcement Learning RL for Language Models
ETH Zürich, Switzerland
CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents
Jun 19, 2026Youngwon Choi, Hyeonyu Kim, Taeyoun Kwon +2Spoken Dialogue Systems Speech Language Models
Maum AI Inc. · Seoul National University · Korea Culture Technology Institute +1
AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstruction
Jun 14, 2026Pengfei Zhang, Hoang H Nguyen, Yutong Song +6RL for Language Models Speech Processing
University of California Irvine · University of Illinois Chicago · Kennesaw State University
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models
Jun 12, 2026Zachary Nicholas Houghton, Yu Zhou, Dan Pluth +2Memorization in Language Models Knowledge Representation
University of Oregon · Vail Systems, Inc
Adaptive Turn-Taking for Real-time Multi-Party Voice Agents
Jun 11, 2026Soumyajit Mitra, Prabhat Pandey, Abhinav Jain +2Spoken Dialogue Systems Speech Language Models
Amazon AGI · IIT Kharagpur, India

Jul 7, 2026J/K move · Enter open · S save

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

Ke-Han Lu, Keqi Deng, Ruchao Fan, Rui Zhao+1

Microsoft, USA · National Taiwan University, Taiwan

KV Caching Automatic Speech Recognition LLM Inference Acceleration KV-Cache Compression Speech Language Models

Open paper AskPDF arXiv

Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.