Speech Language Models

Ask about this topic

Overview

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

CardsList

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

  1. Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition

    Aug 11, 2026Gaopeng Xu, Zhenyu Wang, Zheng Xue +2Uncertainty QuantificationAutomatic Speech Recognition

    Qwen Business Unit of Alibaba, China

  2. Never Stop Speaking: a Denial-of-Service Attack on End-to-End Speech Language Models

    Aug 11, 2026Shuozhe Cheng, Kunlan Xiang, Mingxuan Li +3Adversarial Attacks on LLMsSpeech Language Models

  3. VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

    Aug 11, 2026Yejin Jeon, Marie Maltais, Virginia Ceccatelli +2Speech TranslationSpeech Language Models

    Mila - Quebec AI Institute · McGill University, Canada · Google DeepMind +1

  4. EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

    Aug 10, 2026Junyu Wang, Siyuan Zhang, Peiyuan Jiang +11Speech Language Models

    Tianjin Key Laboratory of Cognitive Computing and Application Tianjin University Tianjin, China · Tianjin University Tianjin, China · Fuzhou University Fuzhou, China +2

  5. VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

    Aug 9, 2026Wenxu Jia, Dongjie Fu, Xize Cheng +6Audio Token CompressionLLM Inference Acceleration

    1Zhejiang University · 2Meituan

  6. DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

    Aug 8, 2026Yi Shu, Tianyu Peng, Yingzhuo Deng +5Spoken Dialogue SystemsSpeech Language Models

    School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research +1

  7. Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors

    Aug 6, 2026Arya Labroo, Mengjie Qian, Kate KnillLanguage Model Bias EvaluationAutomated Evaluation

    University of Cambridge

  8. How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs

    Aug 6, 2026Christian Huber, Alexander WaibelASR EvaluationAutomatic Speech Recognition

    Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2

  9. Towards More Expressive Spoken LLMs: Fine-Grained Intent Benchmarking and Acoustic-Lexical Decoupled Policy Optimization

    Aug 4, 2026Xiang Lin, Tian-Hao Zhang, Chunfeng Wang +3RL for Language ModelsSpoken Dialogue Systems

    Li Auto Inc., Beijing, China · Tsinghua University, Beijing, China

  10. Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Models

    Aug 3, 2026Linkai Peng, Baorian NuchgedAudio-Language Model EvaluationSpeech Emotion Recognition

    Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT 06269, USA · Department of Linguistics, The University of Texas at Austin, Austin, TX 78712, USA

  11. Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

    Aug 1, 2026Chenlin Liu, Minghui Fang, Zhonghao Bi +3TTS SynthesisContrastive Decoding

    Harbin Institute of Technology, China · Zhejiang University, China

  12. ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition

    Jul 31, 2026Qingjian Lin, Yuxin Li, Haoyang Zhang +14Multi-Token PredictionAutomatic Speech Recognition

    StepFun · NTU · PKU +3

  13. Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO

    Jul 30, 2026Xilin Jiang, Riki Shimizu, Sukru Samet Dindar +3Spoken Dialogue SystemsSpeech Language Models

    Columbia University · University of Washington · University of Illinois Urbana-Champaign

  14. Latent-IM: Latent Interaction Management for Speech LLMs

    Jul 29, 2026Adar Avsian, Atahan Dokme, Tony Woo +1Language Model SteeringSpeech Language Models

    Georgia Institute of Technology

  15. SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies

    Jul 28, 2026Mohamed Nabih Ali, Daniele Falavigna, Alessio BruttiAutomatic Speech RecognitionSpeech Language Models

    Center of Augmented Intelligence Fondazione Bruno Kessler Trento, Italy

  16. Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis

    Jul 27, 2026Yifan Hu, Shuwei He, Rui Liu +1TTS SynthesisAudio-Driven Facial Animation

    Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China

  17. MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond

    Jul 24, 2026Lorenzo Concina, Seraphina Fong, Marco Matassoni +1Speech Language ModelsLow-Resource Speech Recognition

    Center for Augmented Intelligence, Fondazione Bruno Kessler, Trento, Italy · Department of Information Engineering and Computer Science, University of Trento, Italy

  18. Safeguards for Speech2Speech LLM-Assistants: A Case Study in Automotive Applications

    Jul 23, 2026Gregor Endler, Sebastian Kraus, Lukas StappenLanguage Model Safety EvaluationLLM Guardrails

    codemanufaktur GmbH, Germany · BMW Group, Germany

  19. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    Jul 22, 2026Pengchao Feng, Chao-Hong Tan, Qian Chen +3Efficient Language Model ReasoningSpeech Language Models

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai, China +2

  20. SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

    Jul 22, 2026Rongshen He, Xinyu Liang, Dekun Chen +3Speech TranslationSimultaneous Speech Translation

    1The Chinese University of Hong Kong, Shenzhen

  21. Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge

    Jul 19, 2026Aivo Olev, Tanel AlumäeLLM Hallucination MitigationText Summarization

    Tallinn University of Technology Tallinn, Estonia

  22. Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models

    Jul 18, 2026Ye Lu, Yihan Yan, Zhaoyang Zhang +4Adversarial AttacksPrivacy Leakage in Language Models

    School of Cyberspace Science, Harbin Institute of Technology, Harbin 150001, China

  23. Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approach

    Jul 14, 2026Junkun Chen, Jian Xue, Ming Tang +4Speech TranslationDecoder-Only Language Models

    Microsoft, USA

  24. An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge

    Jul 14, 2026Shuming Fang, Shuifei ZengASR EvaluationAutomatic Speech Recognition

    UGREEN AI Lab, Shenzhen, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China

  25. Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models

    Jul 11, 2026Yun-Shao Tsai, Chun-Wei Chen, Chee-En Yu +2Audio-Language Model EvaluationSpeech Language Models

    Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · Graduate Institute of Electrical Engineering National Taiwan University Taipei, Taiwan · Artificial Intelligence Center of Research Excellence National Taiwan University Taipei, Taiwan

  26. Efficiently Adapting Spoken Language Models for the Singaporean Context

    Jul 11, 2026Ng Jia Sheng JasonSpoken Language UnderstandingMulti-Task Learning

    Language AI R&D, xData Home Team Science & Technology Agency (HTX), Singapore

  27. Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

    Jul 10, 2026Xugang Lu, Peng Shen, Yu Tsao +1Automatic Speech RecognitionAudio-Visual Speech Recognition

    National Institute of Information and Communications Technology, Kyoto, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

  28. COALA: Robust Contextualized Speech-augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation

    Jul 9, 2026Jhih-Rong Guo, Bi-Cheng Yan, Tien-Hong Lo +1Automatic Speech RecognitionSpeech Language Models

    National Taiwan Normal University, Taiwan

  29. SAMPA: Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese

    Jul 8, 2026Rodrigo de Freitas Lima, Julio Cesar Galdino, Marcos Vinicius TrevisoSpeech ProsodySpeech Language Models

    ICMC University of São Paulo São Paulo, Brazil · Instituto Superior Técnico University of Lisbon Lisbon, Portugal

  30. Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs

    Jul 7, 2026Albert Zeyer, Ralf Schlüter, Hermann NeyGradient-Based AttributionSpeech Processing

    Machine Learning and Human Language Technology Group, RWTH Aachen University, Aachen, Germany · AppTek GmbH, Aachen, Germany