cs.SDNov 16, 2026

Learning from Distribution: Rethinking One-Hot Supervision for ASR and QA in Speech Language Models

Authors: Yujie Guo, Hongjie Chen, Jiaming Zhou, Jian Kang, Jie Li, Yongxiang Li, Yong Qin

Organizations: College of Computer Science, Nankai University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd

Abstract

Speech language models (SLMs) exhibit strong capabilities in automatic speech recognition (ASR) and general-purpose speech understanding, yet ASR-specific training often degrades spoken question answering (QA) performance. We revisit this conflict from the perspective of task-specific input--output mappings and one-hot supervision. Our findings reveal pronounced task-asymmetric effects: ASR-only training severely limits QA capability, whereas QA-only training retains reasonable ASR performance. We hypothesize that the relatively deterministic mapping of ASR, combined with single-reference supervision, biases models toward transcription-specific behavior. Motivated by this perspective, we advocate broadening ASR learning objectives while preserving recognition accuracy. We propose Learning from Distribution (LFD), which aligns speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Extensive experiments demonstrate that LFD improves both ASR and QA performance, exhibits superior data efficiency, and remains effective under full-parameter fine-tuning. CoT-ASR further supports the broader applicability of our proposed supervision principle.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation

    Sep 24, 2026Yujie Guo, Hongjie Chen, Jian Kang +3Automatic Speech RecognitionOn-Policy Distillation

  2. Turning Speech Language Models into Multilingual Listeners

    Sep 27, 2026Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2Speech Language ModelsSpoken Language Understanding

  3. Efficiently Adapting Spoken Language Models for the Singaporean Context

    Jul 11, 2026Ng Jia Sheng JasonSpoken Language UnderstandingMulti-Task Learning