Speech Language Models

Follow this topic Ask about this topic

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs
Sep 11, 2026Kento NishiSpeech Language Models Causal Interventions in Language Models
Massachusetts Institute of Technology · Comcast Applied AI Research, Speech AI Team
Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning
Sep 11, 2026Hayato Futami, Hassan Shahmohammadi, Tushar Dhyani +5Speech Translation Speech Language Models
Sony Group Corporation · Sony Europe Limited
Nuha-Speech: Building General-Purpose Arabic Speech-LLMs
Sep 10, 2026Yingzhi Wang, Reem Alhazzani, Muhammad AlqurishiAudio-Language Model Evaluation Audio QA
Elm Company, KSA
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Sep 10, 2026Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed +1LLM Self-Correction Efficient Language Model Reasoning
The University of Texas at Austin · FAIR, Meta Superintelligence Labs
Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs
Sep 10, 2026Jordi Luque, Fernando López, Aleix SantPrivacy-Preserving Language Models Differentially Private Federated Learning
Scientific Research Telef´onica Innovaci´on Digital Barcelona, Spain · Telef´onica Innovaci´on Digital Universidad Aut´onoma de Madrid Madrid, Spain · Scientific Research Telef´onica Innovaci´on Digital Universitat Polit`ecnica de Catalunya Barcelona, Spain
The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge
Sep 10, 2026Jordi Luque, Lorenzo Concina, Marco Matassoni +2Multilingual Language Model Evaluation Multilingual QA
Telefónica Innovación Digital, Scientific Group · Fondazione Bruno Kessler · Consiglio Nazionale delle Ricerche
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Sep 10, 2026Haojun Zhang, Yi Zou, Min Chen +7LLM Compression Language Model Distillation
XPeng Inc.
SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ
Sep 10, 2026Huy Hoang Le, Long-Bao Nguyen, Minh Tri DaoAudio-Language Model Evaluation Spoken Language Understanding
CAKE by VPBank, Vietnam
Xiaomi-CocktailASR-1 Technical Report
Sep 10, 2026Yiru Zhang, Hang Su, Lichun Fan +10Automatic Speech Recognition Speech Language Models
Xiaoai Plus, ASR team Xiaomi Inc., China
LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection
Sep 9, 2026Xiao Wei, Yuqin Lin, Yaru Cao +6Speech-Based Dementia Detection Speech Prosody
Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · College of Computer and Data Science, Fuzhou University, Fuzhou, China +1
StreamAlign: Streaming Text-Aligned Speech Tokenization
Sep 9, 2026Kang-wook Kim, Jinyoung Park, Jinsoo Kim +3Audio Tokenization Speech Language Models
Seoul National University · University of California, Berkeley · KRAFTON +1
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sep 8, 2026Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu +7TTS Synthesis Streaming TTS Synthesis
MediaTek Research · Internship at MediaTek Research · National Taiwan University +2
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
Sep 8, 2026Fritz Cremer, Jonathan CremerTTS Synthesis Streaming TTS Synthesis
TamilEOT: A Dataset and Model for Semantic End-of-Turn Detection in Tamil Telephone Speech
Sep 4, 2026Santhoshkumar VTurn-Taking Prediction Conversational Agents
What Did I Just Say? Self-Listening for Full-Duplex Speech Models
Sep 4, 2026Xuanning Zhou, Junyi Ao, Xiaotong Liu +3Speech Language Models Full-Duplex Spoken Dialogue Systems
Shenzhen Loop Area Institute, China · The Chinese University of Hong Kong, Shenzhen, China
Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue
Sep 3, 2026Yihang Li, Chenhui ChuSpoken Dialogue Systems Speech Language Models
Kyoto University
When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models
Aug 31, 2026Joonyong Park, Jerry LiReward Modeling RL for Language Models
Spellbrush
Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions
Aug 31, 2026Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang +3Automatic Speech Recognition Speech Language Models
Carnegie Mellon University · NVIDIA · Work done at NVIDIA +1
Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
Aug 31, 2026Chanhee Cho, Junhyuk Choi, Bugeun KimAudio Token Compression Automatic Speech Recognition
Department of Artificial Intelligence Chung-Ang University, Seoul, Republic of Korea
HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Aug 29, 2026Dongwook Lee, Sangkwon Park, Eunwoo Song +6Speech Language Models Audio-Language Model Evaluation
IPAI, Seoul National University (SNU) · Department of ECE, SNU · Department of EE, Yonsei University +3
SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs
Aug 27, 2026Mengzhe GengAudio-Language Model Evaluation Hallucination Detection
National Research Council Canada
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
Aug 26, 2026Zhifei Xie, Jiaqi Lang, Ze An +7Conversational Memory Long-Term Conversational Memory
Nanyang Technological University · Open Interaction Lab · National University of Singapore +2
Relative Time Intervals Representation for Word-level Timestamping with Masked Training
Aug 25, 2026Quanwei Tang, Zhiyu Tang, Xu Li +3Speech Language Models Automatic Speech Recognition
Soochow University · University of Queenland · AISpeech Ltd +1
Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
Aug 24, 2026Hyeonyu Kim, Hwayeon Kim, Youngwon Choi +2Speech Language Models Spoken Language Understanding
Maum AI Inc. · KAIST · Atmanity Inc.
DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios
Aug 24, 2026Bingshen Mu, Xian Shi, Xiong Wang +6Multi-Speaker ASR Automatic Speech Recognition
Independent Researcher
WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
Aug 24, 2026Yiming Yao, Chenyang Lyu, Xuanfan Ni +4KV-Cache Offloading KV-Cache Compression
School of Informatics, Xiamen University · Alibaba Group
Grounded Decoding for Autoregressive Speech Enhancement via Adaptive Code-Space Grounding and Local LLM Refinement
Aug 21, 2026Hao Shi, Yuan Gao, Zhaoheng Ni +4Speech Enhancement Speech Language Models
Kyoto University · Meta · Brno University of Technology +3
Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding
Aug 21, 2026Haoyue Liu, Zhichao Wang, Ye Chen +2Emotion Recognition in Conversations Speech Language Models
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · University of California, Berkeley +1
Towards Quantifying Benchmark Optimization in ASR Models
Aug 20, 2026Theo Lebryk, David Ayllon, Alice Baird +3ASR Evaluation Speech Language Models
Hume AI Research
CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model
Aug 13, 2026Nhan Phan, Ilona Lähteenmäki, Anna von Zansen +4Automated Evaluation Speech Language Models
Department of Information and Communications Engineering, Aalto University, Espoo, Finland · Department of Education, University of Helsinki, Helsinki, Finland · Programmable Autonomous Systems Division, Walton Institute, Waterford, Ireland

Sep 11, 2026J/K move · Enter open · S save

Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs

Kento Nishi

Massachusetts Institute of Technology · Comcast Applied AI Research, Speech AI Team

Speech Language Models Causal Interventions in Language Models Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Speech-to-speech LLMs like Moshi, and its derivative PersonaPlex, can listen and speak concurrently through full-duplex generation. However, they can begin speaking inappropriately during prolonged user silence: under digital-zero input, Moshi and PersonaPlex initiate speech in 30% and 27.5% of five-minute continuations, respectively. What causes this spurious speech? We investigate two hypotheses: either repeated sampling selects speech despite persistently low onset probabilities, or self-conditioning on nonspeech outputs causes an abrupt spike in onset probability. We find that, at every observed onset, speech probability spikes by over nine orders of magnitude in one 80-ms frame, supporting the latter hypothesis. Then, to suppress these onsets without blocking genuine responses, we ask a causal counterfactual question: is the model responding to user speech, or would its next-token distribution remain similar if the preceding user input were muted? Accordingly, we suppress onsets whose distributions change little under this intervention. Under realistic microphone noise, our method suppresses spurious onsets, while preserving genuine responses: one-sided 95% lower confidence bounds are 98.68% and 98.82% for Moshi, and 96.90% and 99.25% for PersonaPlex. Our inference-time method runs in real-time without retraining, with 95th-percentile decision time below 61 ms, within the 80-ms frame budget. Our code is available at https://github.com/KentoNishi/icassp27-spurious-onsets.