Speech Language Models

Ask about this topic

Overview

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

CardsList

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

  1. Attention-guided Evidence Grounding for Spoken Question Answering

    Mar 17, 2026Ke Yang, Bolin Chen, Yuejie Li +5Spoken QAMultimodal QA

    Ant Group Hangzhou, China

  2. Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning

    Mar 16, 2026Jingxiang Chen, Minseok Kim, Seong-Gyun Leem +13Speech Language ModelsEmotion Representation in Language Models

    Meta Reality Labs

  3. The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

    Mar 15, 2026Shree Harsha Bokkahalli Satish, Christoph Minixhofer, Maria Teleki +5Intersectional Bias in Language ModelsLanguage Model Bias Evaluation

    Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden · Centre for Speech Technology Research, University of Edinburgh, UK · Texas A&M University, USA

  4. Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs

    Mar 14, 2026Zijian Ling, Pingyi Hu, Xiuyong Gao +6Adversarial AttacksLLM Security

    School of Cyber Science and Engineering, Huazhong University of Science and Technology · State Key Laboratory of Internet Architecture, Tsinghua University · Hubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology +1

  5. Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization

    Mar 13, 2026Mengjie Zhao, Lianbo Liu, Yusuke Fujita +4Speech Language ModelsText Generation

    SB Intuitions, Tokyo, Japan

  6. TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

    Mar 12, 2026Liang-Hsuan Tseng, Hung-yi LeeSpeech Language ModelsAudio Tokenization

    Graduate Institute of Communication Engineering, National Taiwan University

  7. Can LLMs Help Localize Fake Words in Partially Fake Speech?

    Mar 11, 2026Lin Zhang, Thomas Thebaud, Zexin Cai +5Deepfake DetectionSpeech Language Models

    HLTCOE · CLSP · Xiaomi Corp., Beijing, China

  8. OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs

    Mar 11, 2026Yujie Liao, Xuelong Geng, Hongfei Xue +2Speech Foundation ModelsSpeech Language Models

  9. Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

    Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker EmbeddingSpeaker Verification

    Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

  10. Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognition

    Mar 11, 2026Yinfeng Xia, Jian Tang, Junfeng Hou +2Automatic Speech RecognitionStreaming ASR

    Qwen Applications Business Group, Alibaba, China · Tongyi AI Lab, Alibaba, China

  11. G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition

    Mar 11, 2026Jing Peng, Ziyi Chen, Haoyu Li +7Speaker-Attributed ASRAutomatic Speech Recognition

    Shanghai Jiao Tong University · Shenzhen Research Institute of Big Data · Nanjing University +2

  12. Do What I Say: A Spoken Prompt Dataset for Instruction-Following

    Mar 10, 2026Maike Züfle, Sara Papi, Fabian Retkowski +5Audio-Language Model EvaluationSpeech Language Models

    Karlsruhe Institute of Technology, Germany · Fondazione Bruno Kessler, Italy · ACC Cyfronet AGH, Poland +2

  13. SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models

    Mar 10, 2026Hsiao-Ying Huang, Cheng-Han Chiang, Hung-yi LeeSpeech Language ModelsEarly-Exit Neural Networks

    National Taiwan University, Taipei, Taiwan

  14. DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization

    Mar 10, 2026Jianing Yang, Yusuke Fujita, Yui SudoSpoken Dialogue SystemsFull-Duplex Spoken Dialogue Systems

    SB Intuitions Corp., Japan · The University of Tokyo, Japan

  15. VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

    Mar 9, 2026Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan +1Speech Emotion RecognitionSpeech Language Models

    Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA

  16. DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

    Mar 9, 2026Shangeth RajaaSpeech Language ModelsTurn-Taking Prediction

    Anyreach AI

  17. Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

    Mar 9, 2026Nikita Kuzmin, Tao Zhong, Jiajun Deng +6Privacy-Preserving Speech ProcessingSpeaker Anonymization

    Nanyang Technological University, Singapore · A*STAR, Singapore · Huawei Leibniz Research Center +2

  18. StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control

    Mar 8, 2026Haishu Zhao, Aokai Hao, Yuan Ge +3Controllable Speech GenerationSpeech Language Models

    NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

  19. Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

    Mar 7, 2026Shreyas Gopal, Donghang Wu, Ashutosh Anshul +5Speech Language ModelsLanguage Model Distillation

    College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore

  20. Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

    Mar 6, 2026Changsong Liu, Tianrui Wang, Ye Ni +2Long-Form TTS SynthesisTTS Synthesis

    Nanyang Technological University, Singapore · Tianjin University, China · Southeast University, China

  21. X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs

    Mar 6, 2026Di Cao, Dongjie Fu, Hai Yu +3Speech Language ModelsLLM Alignment

    Tencent Hunyuan · Zhejiang University

  22. WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation

    Mar 5, 2026Luca Della Libera, Cem Subakan, Mirco RavanelliSpeech Language ModelsAutoregressive Language Modeling

    Concordia University · Mila-Quebec AI Institute · Universit´e Laval

  23. Measuring the Redundancy of Decoder Layers in SpeechLLMs

    Mar 5, 2026Adel Moumen, Guangzhi Sun, Philip C WoodlandLLM PruningTransformer

    Department of Engineering, University of Cambridge, UK

  24. Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement

    Mar 4, 2026Fei Su, Cancan Li, Juan Liu +3Robust ASRAudio-Visual Speech Recognition

    School of Computer Science, Wuhan University, China · School of Artificial Intelligence, Wuhan University, China · AI Center, OPPO, China +2

  25. Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

    Mar 2, 2026Ming-Hao Hsu, Xiaohai Tian, Jun Zhang +3Speech Language ModelsAudio-Language Model Evaluation

    School of Data Science, The Chinese University of Hong Kong, Shenzhen · ByteDance

  26. Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment

    Feb 28, 2026Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini +1Speech Quality AssessmentSpeech Language Models

    Centre for Language Studies, Radboud University, The Netherlands

  27. Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion

    Feb 28, 2026Sen Zhang, Jianguo Wei, Wenhuan Lu +6Automatic Speech RecognitionSpeech Language Models

    College of Intelligence and Computing, Tianjin University, Tianjin, China · Banma Network Technology Co., Ltd.

  28. TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

    Feb 26, 2026Trung Dang, Sharath Rao, Ananya Gupta +6TTS SynthesisSpeech Processing

    Hume AI, USA · Dartmouth College, USA

  29. CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment

    Feb 23, 2026Hanwen Liu, Saierdaer Yusuyin, Hao Huang +1TTS SynthesisStreaming TTS Synthesis

    School of Computer Science and Technology, Xinjiang University, China · Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, China

  30. The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR→\rightarrowLLM Pipelines?

    Feb 19, 2026Jayadev BillaSpeech Language ModelsAudio-Language Model Evaluation

    San Jose CA, USA.