Speech Language Models

Follow this topic Ask about this topic

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment
Apr 13, 2026Kuang Wang, Lai Wei, Ping Lin +8Expressive Speech Synthesis Speech Language Models
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
Ti-Audio: The First Multi-Dialectal End-to-End Speech LLM for Tibetan
Apr 13, 2026Jialing Wang, Yue Zhao, Yuhao Zhang +5Speech Language Models Low-Resource Speech Recognition
Minzu University of China, Beijing, China · The Chinese University of Hong Kong, Shenzhen, China
Efficient Training for Cross-lingual Speech Language Models
Apr 13, 2026Yan Zhou, Qingkai Fang, Yun Hong +1Speech Language Models Cross-Modal Alignment
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models
Apr 11, 2026Chi-Yuan Hsiao, Ke-Han Lu, Yu-Kuan Fu +3Speech Language Models Full-Duplex Spoken Dialogue Systems
National Taiwan University · ASUS Open Cloud Infrastructure Software Center · NVIDIA AI Technology Center
Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
Apr 10, 2026Yizhong Geng, Yanliang Li, Jinghan Yang +4TTS Synthesis Speech Language Models
Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA +2
TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs
Apr 9, 2026Jing Peng, Chenghao Wang, Yi Yang +5Connectionist Temporal Classification Speech Language Models
Jiangsu Key Lab of Language Computing · AISpeech Ltd, Suzhou, China · Nanjing University, China
Raon-Speech Technical Report
Apr 8, 2026Beomsoo Kim, Changho Choi, Dohyun Kim +23Speech Language Models Speech Foundation Models
KRAFTON
Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
Apr 8, 2026Bajian Xiang, Tingwei Guo, Xuan Chen +1Speech Language Models Audio Token Compression
Beike Inc., Beijing, China
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Apr 7, 2026Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9Domain Adaptation for ASR Cross-Modal Alignment
Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland +2
In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads
Apr 7, 2026Charlotte Pouw, Hosein Mohebbi, Afra Alishahi +1In-Context Learning Speech Language Models
ILLC, University of Amsterdam · CSAI, Tilburg University
Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR
Apr 3, 2026Zhennan Lin, Shuai Wang, Zhaokai Sun +5Speaker-Attributed ASR Automatic Speech Recognition
School of Intelligence Science and Technology, Nanjing University · Shanghai Lingguang Zhaxian Technology
Speech LLMs are Contextual Reasoning Transcribers
Apr 1, 2026Keqi Deng, Ruchao Fan, Bo Ren +2Contextual ASR Automatic Speech Recognition
Microsoft Core AI, USA
Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling
Apr 1, 2026Kazuki Yano, Jun Suzuki, Shinji WatanabeSpeech Language Models Automatic Speech Recognition
Tohoku University, Japan · Carnegie Mellon University, USA
Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition
Mar 31, 2026Lukuan Dong, Ziwei Li, Saierdaer Yusuyin +2Multilingual ASR Speech Language Models
TasiTech Co., Ltd., China · Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, China · School of Computer Science and Technology, Xinjiang University, China
ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
Mar 30, 2026Anuj Diwan, Eunsol Choi, David HarwathAudio-Language Models Audio-Text Retrieval
Department of Computer Science, The University of Texas at Austin, USA · Computer Science and Data Science, New York University, USA
On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR
Mar 30, 2026Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi ShekharNeural Network Pruning Speech Language Models
School of Computer Science and Electronic Engineering, University of Essex
Beyond Acoustic Prefixes: Persistent Access to Serialized Acoustic Memory for LLM-Based Multi-Talker Speech Recognition
Mar 28, 2026Hao Shi, Yuan Gao, Xugang Lu +1Multi-Speaker ASR Memory-Augmented Language Models
Graduate School of Informatics, Kyoto University, Kyoto, Japan · National Institute of Information and Communications Technology, Japan
Abstract Compression of Conversational Audio Context for LLM-based ASR
Mar 27, 2026Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso +7Contextual ASR Automatic Speech Recognition
Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, U.S.A. +2
CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR
Mar 26, 2026Shangkun Huang, Huan Shen, Wei Zou +1Contextual ASR Automatic Speech Recognition
BRVoice Team, Bairong, Inc., China
From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs
Mar 25, 2026Xiaoyong Guo, Nanjie Li, Zijie Zeng +4Contextual ASR Automatic Speech Recognition
School of Computer Science and Technology, Xinjiang University, Urumqi, China · Timekettle · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Urumqi, China
Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics
Mar 24, 2026Naohiro Tawara, Samuele Cornell, Alexander Polok +3Conversational Speech Recognition Automatic Speech Recognition
NTT, Inc., Japan · CMU, USA · BUT, Czechia
Greater accessibility can amplify discrimination in generative AI
Mar 23, 2026Carolin Holtermann, Minh Duc Bui, Kaitlyn Zhou +3Social Bias in Language Models Gender Bias in Language Models
Trustworthy AI Lab, University of Hamburg · NALA Group, JGU Mainz · Cornell University +3
ERM-MinMaxGAP: Benchmarking and Mitigating Gender Bias in Multilingual Multimodal Speech-LLM Emotion Recognition
Mar 22, 2026Zi Haur Pang, Xiaoxue Gao, Tatsuya Kawahara +1Speech Emotion Recognition Multimodal Emotion Recognition
Kyoto University, Japan · Agency for Science, Technology, and Research (A*STAR), Singapore
LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
Mar 20, 2026Jianan Chen, Xiaoxue Gao, Tatsuya Kawahara +1ASR Evaluation Low-Resource Speech Recognition
Kyoto University, Kyoto, Japan · Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore
Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models
Mar 20, 2026Kyudan Jung, Jihwan Kim, Soyoon Kim +3Speech Processing Speech Language Models
KAIST AI · NAVER Cloud
From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction
Mar 19, 2026Shree Harsha Bokkahalli Satish, Maria Teleki, Christoph Minixhofer +3Intersectional Bias in Language Models Language Model Bias Evaluation
KTH Royal Institute of Technology Stockholm, Sweden · Texas A&M University College Station, USA · University of Edinburgh Edinburgh, UK
On Optimizing Multimodal Jailbreaks for Spoken Language Models
Mar 19, 2026Aravind Krishnan, Karolina Stańczak, Dietrich KlakowAdversarial Attacks LLM Jailbreak Attacks
Saarland University, Germany · DFKI GmbH, Saarbrücken, Germany · ETH AI Center, Switzerland
The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning
Mar 18, 2026Donghang Wu, Tianyu Zhang, Yuxin Li +4Speech Language Models Continuous Latent Reasoning
College of Computing and Data Science, Nanyang Technological University · Mila, Quebec Artificial Intelligence Institute, Université de Montréal
Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition
Mar 18, 2026Yuxiang Mei, Delai Qiu, Shengping Liu +2Multilingual ASR Speech Language Models
Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University, Shanghai, 200234, China. · Unisound AI Technology Co., Ltd., Beijing, China. · SHNU-Unisound Natural Human-Computer Interaction Lab, Shanghai Normal University.
On the Emotion Understanding of Synthesized Speech
Mar 17, 2026Yuan Ge, Haishu Zhao, Aokai Hao +10Speech Emotion Recognition Emotion Representation in Language Models
Northeastern University, China · Meituan · NiuTrans Research +1

Apr 13, 2026J/K move · Enter open · S save

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

Kuang Wang, Lai Wei, Ping Lin, Qibing Bai+7

The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University+2

Expressive Speech Synthesis Speech Language Models Text-to-Speech Controllable Speech Generation Emotional Speech Synthesis

Open paper AskPDF arXiv

Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.