Speech language models (SLMs) exhibit strong capabilities in automatic speech recognition (ASR) and general-purpose speech understanding, yet ASR-specific training often degrades spoken question answering (QA) performance. We revisit this conflict from the perspective of task-specific input--output mappings and one-hot supervision. Our findings reveal pronounced task-asymmetric effects: ASR-only training severely limits QA capability, whereas QA-only training retains reasonable ASR performance. We hypothesize that the relatively deterministic mapping of ASR, combined with single-reference supervision, biases models toward transcription-specific behavior. Motivated by this perspective, we advocate broadening ASR learning objectives while preserving recognition accuracy. We propose Learning from Distribution (LFD), which aligns speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Extensive experiments demonstrate that LFD improves both ASR and QA performance, exhibits superior data efficiency, and remains effective under full-parameter fine-tuning. CoT-ASR further supports the broader applicability of our proposed supervision principle.
Figures & tables
Figure 1: Illustration of ASR’s approximately one-to-one mapping and QA’s one-to-many mapping with multiple valid responses.
Figure 2: Overview of LFD. The text LLM teacher generates responses and token distributions, which the Speech LLM student learns via forward KL divergence with a tail-aware top- k approximation.
System
Training Strategy
Training Data
ASR(Librispeech ↓ )
QA(TELEVAL ↑ )
clean
other
Overall
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
A1
CE
ASR
4.83
8.96
6.89
1.67
2.87
2.37
2.44
A2
LFD
4.51
8.63
6.57
65.33
31.78
39.94
40.20
B1
CE
QA
6.35
10.11
8.23
72.00
36.08
42.05
43.35
B2
LFD
5.18
9.23
7.20
73.33
36.56
42.57
43.93
Table 1: ASR and QA performance under CE and LFD supervision with ASR-only and QA-only training data. ASR performance is measured by WER(%), while QA performance is evaluated by ACC(%) on TELEVAL.
Training Data
Training Strategy
Training Hours
2k
4k
6k
8k
ASR Overall WER (%) ↓
ASR
CE
113.89
9.73
7.00
6.89
LFD
9.79
7.32
6.83
6.57
QA
CE
123.91
11.82
7.89
8.23
LFD
13.68
7.71
7.75
7.20
Table 2: Scaling results of CE and LFD under ASR-only and QA-only training settings with different amounts of training data.
Figure 3: Effect of distribution granularity (top- k ) on ASR and QA performance under ASR-only and QA-only training settings. Dashed gray lines indicate the corresponding CE baselines.
System
Training Strategy
Training Data
Performance
Entropy
ASR ↓
QA ↑
ASR
QA
A1
CE
ASR
6.89
2.44
0.25
0.25
B1
QA
8.23
43.35
0.17
0.39
A2
LFD
ASR
6.57
40.20
0.20
0.48
B2
QA
7.20
43.93
0.15
0.60
Table 3: ASR and QA performance and token-level predictive entropy under CE and LFD supervision.
System
Training Data
Training Strategy
ASR(Librispeech ↓ )
QA(TELEVAL ↑ )
clean
other
Overall
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
C0(init)
ASR
LFD
4.45
8.62
6.53
72.33
32.50
39.01
40.49
C1
CE
3.61
7.43
5.52
0.33
2.39
1.91
1.89
C2
LFD
3.70
7.46
5.57
72.00
32.86
42.78
42.93
D0(init)
QA
LFD
5.24
9.27
7.25
73.33
36.68
42.11
43.67
D1
CE
4.79
8.73
6.76
74.00
36.92
43.86
44.91
Table 4: ASR and QA performance and token-level predictive entropy under full-parameter fine-tuning with CE and LFD supervision. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%). C0 and D0 denote the corresponding adapter-only LFD initialization models.
Model
Fine-Tuning Data
ASR WER (%) ↓
QA ACC (%) ↑
Base Model
–
7.57
42.86
CoT-ASR
ASR-only
7.38
42.54
CoT-QA
QA-only
7.42
42.67
Table 5: ASR and QA performance of CoT-based supervision. The base model is obtained through adapter-only alignment with mixed CoT-ASR and CoT-QA data. CoT-ASR and CoT-QA are then separately full-parameter fine-tuned from the same initialization.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Type
ASR
QA
test-clean
test-other
LlamaQA-en
TriviaQA-en
WebQ-en
Qwen3-4B
LLM
0.04
0.04
0.58
0.67
0.58
LFD-ASR
SLM
0.10
0.17
0.59
0.72
0.62
LFD-QA
SLM
0.09
0.13
0.54
0.66
0.56
Appendix
Table 6: Token-level predictive entropy of text LLMs and SLMs on ASR and QA benchmarks. Lower values indicate more concentrated predictive distributions. LFD-ASR denotes the model obtained through full-parameter fine-tuning in Section 5.5 .
Training Data
Training Strategy
Data Duration
ASR
QA (TELEVAL ↑ )
LibriSpeech-test ↓
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
clean
other
Overall
ASR
CE
2000
113.17
114.62
113.89
0.52
0.96
0.10
0.52
4000
7.31
12.16
9.73
2.33
2.75
2.32
2.44
6000
4.83
9.17
7.00
2.67
2.51
3.25
2.99
8000
4.83
8.96
6.89
1.67
2.87
2.37
2.44
Appendix
Table 7: Detailed scaling results of CE and LFD under ASR-only and QA-only training settings with 2,000 to 8,000 hours of training data. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%) on TELEVAL. Lower WER and higher accuracy indicate better performance.
Speech Language Models (SLMs) inherit strong instruction-following capabilities from pretrained language models, yet ASR specialization can substantially degrade them. To address this ASR--QA trade-off, we propose Task-Specific On-Policy Distillation (TS-OPD), which leverages models before and after ASR specialization as complementary QA and ASR teachers. The student generates separate task-conditioned trajectories for ASR and QA, each supervised only by its corresponding teacher, thereby reducing direct competition between the two supervision signals. Experiments on basic ASR, contextual ASR, and QA demonstrate that TS-OPD improves recognition while preserving QA capability. Moreover, TS-OPD remains robust across different balancing coefficients and continues to benefit from increased distillation data.
Yujie Guo, Hongjie Chen, Jian Kang +3
College of Computer Science, Nankai University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd
Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.
Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2
Department of Computer Science, Stanford University, United States · Cohere Labs, Canada · Cohere, Canada
Spoken language models (SLMs) unify speech perception and reasoning, but adapting them to sensitive domains is underexplored, especially when the original training data is inaccessible and the use case demands multilingual, spoken-query interaction. We adapt an open-source SLM to the Singaporean Home Team context across five speech tasks in Singapore's four official languages, combining LoRA fine-tuning, a surrogate text-QA dataset that guards against catastrophic forgetting, and a multi-task objective that adapts the CoBa reweighting scheme to speech. We also build HTD-multilingual-QA, a 504,853 sample multilingual QA dataset in text and spoken form. The resulting HT-Moonstone (5B) matches or outperforms SLMs up to 7x its size on most tasks, attains the best accent and gender recognition among all models evaluated, and loses under 2% of its original speech QA ability.
Ng Jia Sheng Jason
Language AI R&D, xData Home Team Science & Technology Agency (HTX), Singapore