Speech Language Models

Ask about this topic

Overview

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

CardsList

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

  1. Turning Speech Language Models into Multilingual Listeners

    Sep 27, 2026Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2Speech Language ModelsSpoken Language Understanding

    Department of Computer Science, Stanford University, United States · Cohere Labs, Canada · Cohere, Canada

  2. RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue

    Sep 25, 2026Sathvik Udupa, Naveen Kumar, Ryan FolmsbeeSpoken Dialogue SystemsMulti-Turn Dialogue Management

    Disney Research Imagineering, Los Angeles, USA

  3. Acoustic-to-Text KV Compression for Full-Duplex Speech Models

    Sep 25, 2026Yejin Lee, Seungbeom Kim, Yongha Lee +1KV-Cache CompressionSpeech Language Models

    Sungkyunkwan University, Republic of Korea

  4. Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance

    Sep 25, 2026Manato Yaguchi, Yotaro Kubo, Hikaru Asano +1Spoken Dialogue SystemsSpeech Language Models

    Sakana AI, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

  5. Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition

    Sep 24, 2026Bo Su, Yueru Yan, Thai LePrivacy-Preserving Speech ProcessingAutomatic Speech Recognition

    Indiana University, Bloomington, USA

  6. TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation

    Sep 24, 2026Yujie Guo, Hongjie Chen, Jian Kang +3Automatic Speech RecognitionOn-Policy Distillation

    College of Computer Science, Nankai University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd

  7. Controlling Backchannels in Streamable Full-duplex Models

    Sep 24, 2026Maike Züfle, Peter Polák, Sefik Emre Eskimez +3Spoken Dialogue SystemsSpeech Language Models

    Karlsruhe Institute of Technology, Germany · AppTek, Germany · Charles University, Czech Republic +2

  8. Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs

    Sep 24, 2026Siyu Yao, Du Q. Huynh, Lian Xu +1LLM PruningSpeech Language Models

    The University of Western Australia

  9. Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues

    Sep 24, 2026Wenxuan Wu, Shuhan Zhang, Shuai Wang +1Audio-Visual UnderstandingTarget Speaker Extraction

  10. AdaptDuplex: From Static to Adaptive Full-Duplex Spoken Dialogue

    Sep 24, 2026Zhiyang Zhou, Yingxin Shang, Zhou Wang +9Spoken Dialogue SystemsSpeech Language Models

    Zuoyebang Education Technology (Beijing) Co., Ltd.

  11. A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations

    Sep 23, 2026Matthew Sun, Vinay Kothapally, Meng Yu +4Spoken Dialogue SystemsSynthetic Data Generation

    Tencent Americas, Palo Alto, CA, USA · Tencent Americas, Bellevue, WA, USA · School of Electronic Information, Wuhan University, Wuhan, Hubei, China

  12. PTC-Bias: Phoneme-Level Temporal Competition for Bias Retrieval and Post-Decoding Correction in Speech LLMs

    Sep 23, 2026Zhiqi Ai, Han Cheng, Shiyi Mu +2Automatic Speech RecognitionSpeech Language Models

    Shanghai University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China

  13. The Second MLC-SLM Challenge: Multilingual Conversational Speech Diarization, Recognition, and Understanding

    Sep 23, 2026Bingshen Mu, Mingchen Shao, Zhennan Lin +7Spoken Language UnderstandingMultilingual Language Model Evaluation

    School of Intelligence Science and Technology, Nanjing University · College of Computing and Data Science, Nanyang Technological University, Singapore · Huawei Technologies, China +1

  14. Spoken Language Models that Think Aloud

    Sep 22, 2026Junyi Ao, Kainan Peng, Mingbo Ma +10Spoken Dialogue SystemsSpeech Language Models

    Meta Superintelligence Labs · The Chinese University of Hong Kong, Shenzhen · Work done at Meta

  15. Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models

    Sep 22, 2026Hongjin Song, Runwu Shi, Weiqiao Shan +4Curriculum LearningSpeech Language Models

    Beijing Institute of Technology, Zhuhai · Institute of Science Tokyo · Northeastern University +3

  16. Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking

    Sep 20, 2026Jordi Luque, Aleix Sant, Fernando LópezSpeech Language ModelsFederated Learning

    Telefónica Innovación Digital, Scientific Research

  17. Beyond Encoder Fusion: Multi-View Discrete Token Augmentation for LLM-Based ASR

    Sep 20, 2026Paul Moïse Gangbadja, Mickael Rouvier, Fabrice LefèvreMulti-View LearningAutomatic Speech Recognition

    LIA, Avignon University, France · EDL, France

  18. Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

    Sep 18, 2026Xize Cheng, Wenxu Jia, Chenyuhao Wen +4Long-Context Language Model EvaluationLong-Context Modeling

    Zhejiang University

  19. NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

    Sep 18, 2026Jagadeesh Balam, Travis Bartley, Edresson Casanova +52Tool-Augmented Language Model AgentsSpeech Language Models

    Wookie

  20. The Spoken Wikipedia Presentation Corpus

    Sep 18, 2026Thomas Ranzenberger, Steffen Freisinger, Tobias Bocklet +1Speech Corpus ConstructionAutomatic Speech Recognition

    Technische Hochschule Nürnberg Nürnberg, Germany

  21. From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition

    Sep 17, 2026Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4Speech Emotion RecognitionSpeech Language Models

    Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, USA & India +1

  22. Voice of Reason: Reinforcement Learning for Spoken Math

    Sep 16, 2026Timothée Weisselberger, Edouard Graves, Alexandre DéfossezNumerical Reasoning in Language ModelsRL for Language Model Reasoning

    Kyutai Paris, France · Kyutai, Gradium Paris, France

  23. Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMs

    Sep 16, 2026Abderrahmane Issam, Yusuf Can Semerci, Jan Scholtes +1Speech TranslationAutomatic Speech Recognition

    Department of Advanced Computing Sciences Maastricht University

  24. Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR

    Sep 16, 2026Xiuwen ZhengStreaming ASRSpeech Language Models

    University of Illinois Urbana-Champaign

  25. Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

    Sep 16, 2026Mohan Shi, Zilai Wang, Natarajan Balaji Shankar +3Domain Adaptation for ASRAdapter Tuning

    Department of Electrical and Computer Engineering University of California Los Angeles Los Angeles, USA

  26. Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation

    Sep 14, 2026Daxin Tan, Dehua Tao, Chengxi Deng +2On-Policy DistillationSpeech Language Models

    AI Lab, Leibniz Research Center, Huawei · The Chinese University of Hong Kong · City University of Hong Kong

  27. Quantifying the Generation Modality Gap in Speech-Text Language Models

    Sep 13, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Generation EvaluationSpeech Language Models

    TTI-Chicago · Stony Brook University

  28. Bridging Data, Reasoning, and Alignment: A Unified Framework for Context-Aware Instruction-Following TTS

    Sep 13, 2026Jingbin Hu, Luyu Wang, Wenjie Tian +7Text-to-SpeechSpeech Language Models

    Shenzhen Pimei Technology Co., Ltd., Guangdong, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China

  29. Grounded in Sound: Reinforcement Learning with a Frozen Acoustic Judge to Curb ASR Insertion Hallucinations

    Sep 13, 2026Tingzhen Xiong, Rilin Chen, Weiwei Li +2Automatic Speech RecognitionSpeech Language Models

    Tencent, Shenzhen, China

  30. Exploiting Speech LLM Representations for Multilingual and Cross-Lingual Parkinson's Disease Detection

    Sep 13, 2026Sarthak Giri, Zi Haur Pang, Tatsuya KawaharaParkinson's Disease Detection from SpeechParkinson's Disease

    Graduate School of Informatics, Kyoto University, Japan