Speech Language Models

Ask about this topic

Overview

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

CardsList

310 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

  1. Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

    Oct 26, 2026Yuchen Zhang, Haralambos Mouratidis, Ravi ShekharMultilingual ASRAutomatic Speech Recognition

    Institute for Analytics and Data Science, University of Essex · School of Computer Science and Electronic Engineering, University of Essex

  2. SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning

    Oct 26, 2026Haotian Tan, Hiroki Ouchi, Sakriani SaktiSimultaneous Speech TranslationSpeech Language Models

    Nara Institute of Science and Technology, Japan

  3. ASR-Protected Fusion and Reference-Anchored Policy Optimization for Joint ASR and Word-Level Speech Anti-Spoofing

    Oct 25, 2026Xueping Zhang, Xiaoxiao Miao, Tiantian Feng +1Speech ProcessingAudio Deepfake Detection

    Digital Innovation Research Center, Duke Kunshan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China

  4. Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

    Oct 23, 2026Xuan Shi, Chang Zeng, Tiantian Feng +5Speech ProcessingSpeech Language Models

    University of Southern California, USA · National Institute of Informatics, Japan · Massachusetts Institute of Technology, USA +1

  5. Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

    Oct 21, 2026Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi +2Audio-Language Model EvaluationSpeech Language Models

    †Institute of Advanced Intelligence and Computing, Agency for Science, Technology and Research, Singapore · ⋆Nanyang Technological University, Singapore

  6. SLM-SS: Token-Based Speech Separation with Speech Language Models

    Oct 20, 2026Chenda Li, Tianhua Li, Yiming Chen +4Speech SeparationSpeech Language Models

    Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China

  7. PhonoBridge: Error-Aware Phoneme-Mediated EMG-to-Text Reconstruction

    Oct 20, 2026Wen Qiu, Jiacheng Zhang, Xuanda Chen +2Automatic Speech RecognitionSynthetic Data Augmentation for Speech Recognition

    School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China

  8. Preference-Aware Pareto Steering for Multi-Task Speech LLMs

    Oct 19, 2026Liuyuan Jiang, Vishal Sunder, Samuel Thomas +4Multi-Task LearningSpeech Language Models

    Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA · IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA

  9. Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

    Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization GenerationSpeech Language Models

    Shanghai Jiao Tong University · Ant Group

  10. ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

    Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language ModelsSpeech Language Models

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research Asia, Hong Kong, China. · Microsoft Research, Redmond, WA, USA.

  11. PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models

    Oct 15, 2026Junjie Cao, Fengping Tian, Wenjie Zhou +5Mixture-of-Experts PruningMixture-of-Experts Models

    Alibaba, Hangzhou, China · University of Macau

  12. VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model

    Oct 15, 2026Yang XiaoTurn-Taking PredictionSpeech Language Models

    The University of Melbourne

  13. CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

    Oct 15, 2026Yingzhuo Deng, Chong Li, Bingxin Liu +6Spoken Dialogue SystemsSpeech Language Models

    Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research

  14. Zero-Shot Spoken Language Understanding with SLAM-ASR: What Do Monolingual ASR-Trained Linear Projectors Enable?

    Oct 14, 2026Edwin Arkel Rios, Sneh Sourabh, Jocelyn Zaruma +3Spoken Language UnderstandingSpeech Language Models

    National Yang Ming Chiao Tung University, Taiwan · Tranxform AI · Indian Institute of Technology Hyderabad, India +4

  15. PEDBench: Can Speech Models Detect Phonetic Errors?

    Oct 14, 2026Zhonghao Shi, Zackary Rackauckas, Jasmine Yip +9Speech Language Models

    Harvard University · University of Southern California · University of Michigan +1

  16. Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models

    Oct 14, 2026Yuxuan Chen, Yang Xiao, Jack Sun +3Speech Language ModelsFull-Duplex Spoken Dialogue Systems

    The Chinese University of Hong Kong, Shenzhen · The University of Melbourne · Lulula

  17. Repairing Speech Access After Text Updates with Frozen Affine Fields

    Oct 14, 2026Ming-Hao Hsu, Yuxiang Wang, Yingda Shen +1Multimodal Knowledge EditingSpeech Language Models

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China

  18. Estimating Surprisal from the Speech Signal

    Oct 13, 2026Zachary Hopton, Samuel Kiegeland, Francesco Ignazio Re +4Surprisal TheoryComputational Psycholinguistics

    University of Zurich · ETH Zürich · CHI-FRO +1

  19. Collapse, Repair, and Drift: Dissecting Fine Tuning Dynamics in Speech Language Models

    Oct 13, 2026Yichi Wang, Wangjin Zhou, Yizhou Zhang +4Supervised Fine-TuningNeural Network Training Dynamics

    Department of Intelligence Science and Technology, Kyoto University · Xiaowei, WXG, Tencent · Communication University of China

  20. Where Words Begin and End

    Oct 13, 2026Husein ZolkepliForced AlignmentAudio-Text Alignment

    Scicom (MSC) Berhad, Malaysia

  21. Full-duplex Spoken Language Modeling with Discrete Diffusion

    Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language ModelsFull-Duplex Spoken Dialogue Systems

    NVIDIA

  22. DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

    Oct 13, 2026Jiaheng Dong, Shihong Tan, Gongping Huang +3Spoken Dialogue SystemsSpeech Language Models

    The University of Melbourne · University of Birmingham · Wuhan University +1

  23. Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

    Oct 12, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Language ModelsLanguage Model Distillation

    TTI-Chicago · Stony Brook University

  24. D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

    Oct 12, 2026Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury +5Speech Language ModelsFull-Duplex Spoken Dialogue Systems

    MIT CSAIL · MIT-IBM Watson AI Lab

  25. What Matters in Preference Post-Training of Continuous Autoregressive Speech Language Models

    Oct 12, 2026Yingda Shen, Yao Qian, Dekun Chen +6Speech Language ModelsLanguage Model Post-Training

    The Chinese University of Hong Kong, Shenzhen · Microsoft Research

  26. ASSET: Adaptive Speech Segmentation via Reinforcement Learning

    Oct 9, 2026Cihan Xiao, Yuan Tseng, Matthew Wiesner +4Audio Token CompressionSpeech Processing

    Johns Hopkins University, USA · The University of Texas at Austin, USA · HLTCOE, Johns Hopkins University, USA +1

  27. Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?

    Oct 9, 2026Kshitij Ambilduke, Caroline Brun, Thibaut Thonet +2Social Bias in Language ModelsSpeech Language Models

    Université Paris-Saclay, France · NAVER LABS Europe, France

  28. Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining

    Oct 8, 2026Xinnian Zhao, Chia-Hua Wu, Pu Wang +1Speech Language ModelsLatent Space Alignment

    Department of Electrical Engineering (ESAT), KU Leuven, Leuven, Belgium · Institute of Information Science, Academia Sinica, Taiwan

  29. Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners

    Oct 7, 2026Xilin Jiang, Shun Zhang, Tejas Jayashankar +2Speech Language ModelsLLM Alignment

    Columbia University New York, NY, USA · Meta Superintelligence Labs Menlo Park, CA, USA

  30. Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech

    Oct 7, 2026Hounsu Kim, Joonyong Park, Yuki Saito +2Zero-Shot TTSAudio Diffusion Models

    KAIST, Republic of Korea · The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan