Speech Language Models

Follow this topic Ask about this topic

Momentum

25 papers in the last four weeks, up 400% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 102

All topics

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs
Oct 6, 2026Hemant Yadav, Sunayana Sitaram, Roger Zimmermann +1Speech Language Models
IIIT Delhi, India · Microsoft Research, India · National University of Singapore
HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Oct 6, 2026Kyudan Jung, Hyunsin Park, Yoonhyung Lee +5Full-Duplex Speech Models Speech Language Models
Qualcomm AI Research · KAIST AI
AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models
Oct 1, 2026Yuxiang Wang, Kunyu Feng, Yuancheng Wang +12Speech Language Models Efficient Latent Reasoning
The Chinese University of Hong Kong, Shenzhen · Tencent Hunyuan · Tsinghua University +2
Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenization
Oct 1, 2026Jeeyoung Yun, Seohwan Yun, Sungwoong KimNeural Speech Codecs Audio Tokenizers
Korea University, Seoul, Republic of Korea
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
Sep 30, 2026Hezhao Zhang, Thomas HainEmotion Recognition Speech Language Models
School of Computer Science, University of Sheffield Sheffield, United Kingdom
Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue
Sep 29, 2026Shengbo Cai, Yuxiang Wang, Jingran Xie +5Speech Language Models Paralinguistic Cues
Tsinghua University · Tencent Hunyuan · The Chinese University of Hong Kong, Shenzhen
SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale
Sep 28, 2026Zhaoyi An, Sihan Tan, Youngbae Hwang +2Gloss-Free Sign Language Translation Sign Language Translation
Institute of Science Tokyo · Chungbuk National University
DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation
Sep 27, 2026Yayue Deng, Dingdong Wang, Yuxuan Hu +7Speech Translation Speech Language Models
The Chinese University of Hong Kong, China · Microsoft Corporation · Work done during an internship at Microsoft
Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring
Sep 27, 2026Gabriele CinàSpeech Language Models
Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends
Sep 27, 2026Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha +1Full-Duplex Speech Models Speech Language Models
Mindlogic
TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation
Sep 24, 2026Yujie Guo, Hongjie Chen, Jian Kang +3Speech Language Models Automatic Speech Recognition
College of Computer Science, Nankai University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd
agentic-ger: terminology recovery in long-form speech using global context
Sep 24, 2026Yanqiao Zhu, Wupeng Wang, Zhifu Gao +2Speech Language Models Automatic Speech Recognition
X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Alibaba Token Foundry
Controlling Backchannels in Streamable Full-duplex Models
Sep 24, 2026Maike Züfle, Peter Polák, Sefik Emre Eskimez +3Full-Duplex Speech Models Speech Language Models
Karlsruhe Institute of Technology, Germany · AppTek, Germany · Charles University, Czech Republic +2
Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs
Sep 24, 2026Siyu Yao, Du Q. Huynh, Lian Xu +1Speech Language Models Speaker
The University of Western Australia
Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translation
Sep 22, 2026Yanghe Dong, Wanting Huang, Weiran WangSpeech Translation Speech Language Models
Independent Researcher · Department of Computer Science, University of Iowa, USA
Spoken Language Models that Think Aloud
Sep 22, 2026Junyi Ao, Kainan Peng, Mingbo Ma +10Speech Language Models Reasoning Skills
Meta Superintelligence Labs · The Chinese University of Hong Kong, Shenzhen · Work done at Meta
Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models
Sep 22, 2026Hongjin Song, Runwu Shi, Weiqiao Shan +4Speech Language Models Batch Normalization
Beijing Institute of Technology, Zhuhai · Institute of Science Tokyo · Northeastern University +3
Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition
Sep 17, 2026Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4Emotion Recognition Speech Language Models
Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, USA & India +1
Full-Duplex Speech Models Take the Floor When Asked, Not When Needed
Sep 17, 2026Linkai Peng, Baorian Nuchged, Kaiqi Fu +1Full-Duplex Speech Models Utterances
University of Connecticut · The University of Texas at Austin · X Square Robot +1
Voice of Reason: Reinforcement Learning for Spoken Math
Sep 16, 2026Timothée Weisselberger, Edouard Graves, Alexandre DéfossezSpeech Language Models Mathematical Reasoning Benchmarks
Kyutai Paris, France · Kyutai, Gradium Paris, France
Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMs
Sep 16, 2026Abderrahmane Issam, Yusuf Can Semerci, Jan Scholtes +1Speech Language Models Acoustic Latent Space
Department of Advanced Computing Sciences Maastricht University
Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation
Sep 14, 2026Daxin Tan, Dehua Tao, Chengxi Deng +2Speech Language Models Autoregressive Generation
AI Lab, Leibniz Research Center, Huawei · The Chinese University of Hong Kong · City University of Hong Kong
ER-EDF: A Psychology-Grounded Emotion Regulation Framework for Speech Empathetic Dialogue Generation in Large Audio-Language Models
Sep 14, 2026Hongyu Jin, Wenda Zhang, Runqiu Fei +3Empathy Emotional Regulation
Quantifying the Generation Modality Gap in Speech-Text Language Models
Sep 13, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Language Models
TTI-Chicago · Stony Brook University
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Sep 12, 2026Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed +1LLM Reasoning Strategies Speech Language Models
The University of Texas at Austin · FAIR, Meta Superintelligence Labs
Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs
Sep 11, 2026Kento NishiSpeech Language Models Full-Duplex Speech Models
Massachusetts Institute of Technology · Comcast Applied AI Research, Speech AI Team
$S^3$ -Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants
Sep 9, 2026Heyang Liu, Jiayi Huang, Wenyang Xiao +8Speech Language Models Full-Duplex Voice Agents
Shanghai Jiao Tong University · Ant Group · Tongji University
Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue
Sep 3, 2026Yihang Li, Chenhui ChuFull-Duplex Speech Models Turn-Taking
Kyoto University
When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models
Aug 31, 2026Joonyong Park, Jerry LiSpeech Language Models Post-Training
Spellbrush
Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecasting
Aug 26, 2026Angela Yifei Yuan, Christine De Kock, Christopher LeckieUtterances Speech Language Models
The University of Melbourne
Never Stop Speaking: a Denial-of-Service Attack on End-to-End Speech Language Models
Aug 11, 2026Shuozhe Cheng, Kunlan Xiang, Mingxuan Li +3Speech Language Models Attacker Large Language Model
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Aug 10, 2026Junyu Wang, Siyuan Zhang, Peiyuan Jiang +11Emotion Recognition Speech Language Models
Tianjin Key Laboratory of Cognitive Computing and Application Tianjin University Tianjin, China · Tianjin University Tianjin, China · Fuzhou University Fuzhou, China +2
Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models
Aug 5, 2026Yuezhang Peng, Yuxin Liu, Changfeng Gao +4Speech Language Models Large Audio Language Models
Shanghai Jiao Tong University Shanghai, China · Token Foundry, Alibaba Group Hangzhou, China · Shanghai Jiao Tong University, Shanghai Innovation Institute Shanghai, China
Towards More Expressive Spoken LLMs: Fine-Grained Intent Benchmarking and Acoustic-Lexical Decoupled Policy Optimization
Aug 4, 2026Xiang Lin, Tian-Hao Zhang, Chunfeng Wang +3Paralinguistic Cues Speech Language Models
Li Auto Inc., Beijing, China · Tsinghua University, Beijing, China
Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Models
Aug 3, 2026Linkai Peng, Baorian NuchgedSpeech Language Models Readout
Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT 06269, USA · Department of Linguistics, The University of Texas at Austin, Austin, TX 78712, USA
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Jul 30, 2026Xilin Jiang, Riki Shimizu, Sukru Samet Dindar +3Speech Language Models Repair-Based Group-Relative Policy Optimization
Columbia University · University of Washington · University of Illinois Urbana-Champaign
Latent-IM: Latent Interaction Management for Speech LLMs
Jul 29, 2026Adar Avsian, Atahan Dokme, Tony Woo +1Dialogue Speech Language Models
Georgia Institute of Technology
SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies
Jul 28, 2026Mohamed Nabih Ali, Daniele Falavigna, Alessio BruttiSpeech Language Models Speech Encoder
Center of Augmented Intelligence Fondazione Bruno Kessler Trento, Italy
Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating
Jul 25, 2026Yuriko Kikuchi, Takato Hayashi, Ryusei Kimura +3Speech Language Models
Japan Advanced Institute of Science and Technology · Nomi, Ishikawa, Japan · NTT, Inc. +1
MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond
Jul 24, 2026Lorenzo Concina, Seraphina Fong, Marco Matassoni +1Multilingual Automatic Speech Recognition Speech Translation
Center for Augmented Intelligence, Fondazione Bruno Kessler, Trento, Italy · Department of Information Engineering and Computer Science, University of Trento, Italy
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
Jul 22, 2026Pengchao Feng, Chao-Hong Tan, Qian Chen +3Speech Language Models LLM Reasoning Strategies
Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai, China +2
SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
Jul 22, 2026Rongshen He, Xinyu Liang, Dekun Chen +3Speech Translation Speech Language Models
1The Chinese University of Hong Kong, Shenzhen
Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models
Jul 11, 2026Yun-Shao Tsai, Chun-Wei Chen, Chee-En Yu +2Speech Language Models Cross-Modal
Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · Graduate Institute of Electrical Engineering National Taiwan University Taipei, Taiwan · Artificial Intelligence Center of Research Excellence National Taiwan University Taipei, Taiwan
Efficiently Adapting Spoken Language Models for the Singaporean Context
Jul 11, 2026Ng Jia Sheng JasonSpeech Language Models Thai
Language AI R&D, xData Home Team Science & Technology Agency (HTX), Singapore
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2Speech Language Models Autoregressive Decoding
Microsoft, USA · National Taiwan University, Taiwan
Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
Jul 2, 2026Congrui Du, Yang Zhang, Kaizhi Qian +1Speech Language Models Instruction Tuning
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
Jul 2, 2026Haoran Wang, Jinchuan Tian, Siddhant Arora +1Modern Generative Audio Models Speech Language Models
Carnegie Mellon University
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Jul 1, 2026Siyi Wang, James Bailey, Ting DangSpeech Language Models Emotion Recognition
The University of Melbourne, Australia · Monash University, Australia
LOPA: Enhancing Spoken Language Assessment via Latent Ordinal Prototype Alignment
Jun 30, 2026Hong-Yun Lin, Fu-An Chao, Bi-Cheng Yan +1Speech Language Models Speech Encoder
National Taiwan Normal University
FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates
Jun 30, 2026Jiaqi Li, Chaoren Wang, Xiaohai Tian +9Speech Language Models Neural Speech Codecs
1The Chinese University of Hong Kong, Shenzhen · 2ByteDance
wav2VOT: Automatic estimation of voice onset time, closure duration, and burst realisation with wav2vec2
Jun 27, 2026James Tanner, Morgan Sonderegger, Jane Stuart-Smith +2Wav2Vec Grapheme-To-Phoneme
University of Glasgow, United Kingdom · McGill University, Canada · University of Oregon, United States +1
FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following
Jun 25, 2026Zhihang Xie, Marco Gaido, Sara Papi +2Speech Language Models Speaker
Fondazione Bruno Kessler · University of Trento
SFL-MTSC: Leveraging Semantic Frame-Level Multi-Task Self-Consistency for Robust Multi-Intent Spoken Language Understanding
Jun 24, 2026Po-Yen Chen, Berlin ChenSpeech Language Models Motion-Language Alignment
National Taiwan Normal University, Taiwan
Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models
Jun 24, 2026Tomoya Mizumoto, Yusuke Fujita, Hao Shi +3Dialects Speech Language Models
SB Intuitions, Tokyo, Japan
Selective Capability Unlearning in End-to-End Spoken Language Understanding
Jun 23, 2026Akanksha Singh, Vinod Kumar KurmiSpeech Language Models Suppression
Indian Institute of Science Education and Research Bhopal, India
On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models
Jun 22, 2026Shunsuke Kando, Wataru Nakata, Shinnosuke Takamichi +1Speech Language Models Discrete Speech Representations
The University of Tokyo, Japan · Keio University, Japan
Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity
Jun 22, 2026Vyom Agarwal, Mokshda Gangrade, Siddharth Pal +1End-To-End Automatic Speech Recognition Models Speech Language Models
Computer, Mathematical, and Natural Sciences, University of Maryland, College Park, MD, USA · RTX BBN Technologies, Columbia, MD, USA · Electrical and Computer Engineering, University of Maryland, College Park, MD, USA
Interleaved Speech Language Models Latently Work In Text
Jun 21, 2026Talia Sternberg, Gallil Maimon, Yossi AdiSpeech Language Models Large Language Model Pretraining
The Hebrew University of Jerusalem
XAI-Grounded Explanation Generation for Speech Deepfake Detection with Training-Free Multimodal Large Language Models
Jun 15, 2026Yupei Li, Qiyang Sun, Xiaoliang Wu +3Speech Language Models Explainable Artificial Intelligence
Imperial College London, United Kingdom · University of Southampton, United Kingdom · MBZUAI, United Arab Emirates +2
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengFull-Duplex Speech Models Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Hemant Yadav, Sunayana Sitaram, Roger Zimmermann, Rajiv Ratn Shah

IIIT Delhi, India · Microsoft Research, India · National University of Singapore

Speech Language Models

Open paper AskPDF arXiv

Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech. It computes distance-based CTC loss over the frozen LLM embedding matrix and uses greedy CTC labels to derive geometrically and temporally aligned speech embeddings respectively as an input to the LLM. Trained solely on 960 hours of LibriSpeech ASR data, DirectSpeech2LLM outperforms the cascaded system on ASR (seen task) and generalizes zero-shot to speech translation and emotion recognition (two unseen tasks), closely matching the cascaded system upper bound on these two new instructions despite seeing neither during training. We also find that geometric alignment strength plays a smaller role than previously assumed, as our modified CTC loss is shown to provide sufficient implicit geometric grounding without requiring an explicit regression loss. Results are consistent across two LLM families and scale with both more training data and model capacity.

Momentum

Related topics

Latest papers 102