Overview

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

CardsList

327 papers36 papers in the last four weeks, down 40% on the four weeks before. 0.2% of all new papers.

  1. Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer

    Nov 11, 2026Zhisheng Zhang, Yifan Mi, Yixuan Zhou +6Transfer-Based Adversarial AttacksAdversarial Robustness of ASR

  2. On the information rate of phones, syllables and words

    Nov 10, 2026Herman KamperSubword TokenizationSpeech Language Models

  3. Omni-JEV: Text and Dual-Stream Speech Decisions

    Nov 10, 2026Shengzhou KongFull-Duplex Spoken Dialogue SystemsMulti-Turn Dialogue Management

  4. Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models

    Nov 4, 2026Yutong Bian, Yanglin Zhang, Yuxiang Wang +6Speech Language ModelsSpeculative Decoding

  5. LibriSpeech-Real-QA: A Benchmark for Streaming Question Answering

    Nov 4, 2026Zewen Zhan, Hui Lu, Xueyuan Chen +6Speech Language ModelsAudio QA

  6. The Right Source Is Not Enough: Understanding History Interference in Speech-LLM Memory

    Nov 3, 2026HongJin Song, Weiqiao Shan, Runwu Shi +11Conversational MemorySpeech Language Models

  7. Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark

    Nov 3, 2026Jiaming Zhou, Pei Zhang, Shiwan Zhao +3Audio-Language Model EvaluationSpeech Language Models

  8. Voice-Native LLMs Hear Frustration but Do Not Listen to It

    Nov 2, 2026Hanjun Luo, Yuhan Lu, Hanan SalamAudio-Language Model EvaluationSpeech Language Models

  9. ReFrame: Silence-Aware Compute Reclamation for Full-Duplex Speech Models

    Nov 1, 2026Zheng Zhou, Guanyi Chen, Jing Wei +3Full-Duplex Spoken Dialogue SystemsSpeech Language Models

  10. GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

    Oct 31, 2026Yosuke Higuchi, Tom Fizycki, Mahi Luthra +14Speech Language ModelsDiscrete Speech Representation Learning

  11. C3: Speech-Guided KV Retrieval for Long-Context SpeechLLM Inference

    Oct 30, 2026Dancheng Liu, Markus Müller, Rupak Vignesh Swaminathan +3Speech Language ModelsLong-Form Speech Recognition

  12. From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition

    Oct 30, 2026Qing Wang, Qihan Hu, Hang Lv +9Automatic Speech RecognitionASR Evaluation

  13. Developing a Speech Language Model for Basque

    Oct 27, 2026Andoni Sudupe, Inma Hernaez, Ander Arriandiaga +3Speech Language ModelsAudio-Language Models

  14. Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

    Oct 26, 2026Yuchen Zhang, Haralambos Mouratidis, Ravi ShekharMultilingual ASRAutomatic Speech Recognition

  15. ASR-Protected Fusion and Reference-Anchored Policy Optimization for Joint ASR and Word-Level Speech Anti-Spoofing

    Oct 25, 2026Xueping Zhang, Xiaoxiao Miao, Tiantian Feng +1Speech ProcessingAudio Deepfake Detection

  16. Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

    Oct 23, 2026Xuan Shi, Chang Zeng, Tiantian Feng +5Speech ProcessingSpeech Language Models

  17. Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

    Oct 21, 2026Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi +2Audio-Language Model EvaluationSpeech Language Models

  18. SLM-SS: Token-Based Speech Separation with Speech Language Models

    Oct 20, 2026Chenda Li, Tianhua Li, Yiming Chen +4Speech SeparationSpeech Language Models

  19. PhonoBridge: Error-Aware Phoneme-Mediated EMG-to-Text Reconstruction

    Oct 20, 2026Wen Qiu, Jiacheng Zhang, Xuanda Chen +2Automatic Speech RecognitionSynthetic Data Augmentation for Speech Recognition

  20. Preference-Aware Pareto Steering for Multi-Task Speech LLMs

    Oct 19, 2026Liuyuan Jiang, Vishal Sunder, Samuel Thomas +4Multi-Task LearningSpeech Language Models

  21. Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

    Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization GenerationSpeech Language Models

  22. ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

    Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language ModelsSpeech Language Models

  23. PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models

    Oct 15, 2026Junjie Cao, Fengping Tian, Wenjie Zhou +5Mixture-of-Experts PruningMixture-of-Experts Models

  24. CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

    Oct 15, 2026Yingzhuo Deng, Chong Li, Bingxin Liu +6Spoken Dialogue SystemsSpeech Language Models

  25. Zero-Shot Spoken Language Understanding with SLAM-ASR: What Do Monolingual ASR-Trained Linear Projectors Enable?

    Oct 14, 2026Edwin Arkel Rios, Sneh Sourabh, Jocelyn Zaruma +3Spoken Language UnderstandingSpeech Language Models

  26. PEDBench: Can Speech Models Detect Phonetic Errors?

    Oct 14, 2026Zhonghao Shi, Zackary Rackauckas, Jasmine Yip +9Speech Language Models