Speech language models (SLMs) exhibit strong capabilities in automatic speech recognition (ASR) and general-purpose speech understanding, yet ASR-specific training often degrades spoken question answering (QA) performance. We revisit this conflict from the perspective of task-specific input--output mappings and one-hot supervision. Our findings reveal pronounced task-asymmetric effects: ASR-only training severely limits QA capability, whereas QA-only training retains reasonable ASR performance. We hypothesize that the relatively deterministic mapping of ASR, combined with single-reference supervision, biases models toward transcription-specific behavior. Motivated by this perspective, we advocate broadening ASR learning objectives while preserving recognition accuracy. We propose Learning from Distribution (LFD), which aligns speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Extensive experiments demonstrate that LFD improves both ASR and QA performance, exhibits superior data efficiency, and remains effective under full-parameter fine-tuning. CoT-ASR further supports the broader applicability of our proposed supervision principle.
Figures & tables
Figure 1: Illustration of ASR’s approximately one-to-one mapping and QA’s one-to-many mapping with multiple valid responses.
Figure 2: Overview of LFD. The text LLM teacher generates responses and token distributions, which the Speech LLM student learns via forward KL divergence with a tail-aware top- k approximation.
System
Training Strategy
Training Data
ASR(Librispeech ↓ )
QA(TELEVAL ↑ )
clean
other
Overall
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
A1
CE
ASR
4.83
8.96
6.89
1.67
2.87
2.37
2.44
A2
LFD
4.51
8.63
6.57
65.33
31.78
39.94
40.20
B1
CE
QA
6.35
10.11
8.23
72.00
36.08
42.05
43.35
B2
LFD
5.18
9.23
7.20
73.33
36.56
42.57
43.93
Table 1: ASR and QA performance under CE and LFD supervision with ASR-only and QA-only training data. ASR performance is measured by WER(%), while QA performance is evaluated by ACC(%) on TELEVAL.
Training Data
Training Strategy
Training Hours
2k
4k
6k
8k
ASR Overall WER (%) ↓
ASR
CE
113.89
9.73
7.00
6.89
LFD
9.79
7.32
6.83
6.57
QA
CE
123.91
11.82
7.89
8.23
LFD
13.68
7.71
7.75
7.20
Table 2: Scaling results of CE and LFD under ASR-only and QA-only training settings with different amounts of training data.
Figure 3: Effect of distribution granularity (top- k ) on ASR and QA performance under ASR-only and QA-only training settings. Dashed gray lines indicate the corresponding CE baselines.
System
Training Strategy
Training Data
Performance
Entropy
ASR ↓
QA ↑
ASR
QA
A1
CE
ASR
6.89
2.44
0.25
0.25
B1
QA
8.23
43.35
0.17
0.39
A2
LFD
ASR
6.57
40.20
0.20
0.48
B2
QA
7.20
43.93
0.15
0.60
Table 3: ASR and QA performance and token-level predictive entropy under CE and LFD supervision.
System
Training Data
Training Strategy
ASR(Librispeech ↓ )
QA(TELEVAL ↑ )
clean
other
Overall
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
C0(init)
ASR
LFD
4.45
8.62
6.53
72.33
32.50
39.01
40.49
C1
CE
3.61
7.43
5.52
0.33
2.39
1.91
1.89
C2
LFD
3.70
7.46
5.57
72.00
32.86
42.78
42.93
D0(init)
QA
LFD
5.24
9.27
7.25
73.33
36.68
42.11
43.67
D1
CE
4.79
8.73
6.76
74.00
36.92
43.86
44.91
Table 4: ASR and QA performance and token-level predictive entropy under full-parameter fine-tuning with CE and LFD supervision. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%). C0 and D0 denote the corresponding adapter-only LFD initialization models.
Model
Fine-Tuning Data
ASR WER (%) ↓
QA ACC (%) ↑
Base Model
–
7.57
42.86
CoT-ASR
ASR-only
7.38
42.54
CoT-QA
QA-only
7.42
42.67
Table 5: ASR and QA performance of CoT-based supervision. The base model is obtained through adapter-only alignment with mixed CoT-ASR and CoT-QA data. CoT-ASR and CoT-QA are then separately full-parameter fine-tuned from the same initialization.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Type
ASR
QA
test-clean
test-other
LlamaQA-en
TriviaQA-en
WebQ-en
Qwen3-4B
LLM
0.04
0.04
0.58
0.67
0.58
LFD-ASR
SLM
0.10
0.17
0.59
0.72
0.62
LFD-QA
SLM
0.09
0.13
0.54
0.66
0.56
Appendix
Table 6: Token-level predictive entropy of text LLMs and SLMs on ASR and QA benchmarks. Lower values indicate more concentrated predictive distributions. LFD-ASR denotes the model obtained through full-parameter fine-tuning in Section 5.5 .
Training Data
Training Strategy
Data Duration
ASR
QA (TELEVAL ↑ )
LibriSpeech-test ↓
LlamaQA-en
TriviaQA-en
WebQ-en
Overall
clean
other
Overall
ASR
CE
2000
113.17
114.62
113.89
0.52
0.96
0.10
0.52
4000
7.31
12.16
9.73
2.33
2.75
2.32
2.44
6000
4.83
9.17
7.00
2.67
2.51
3.25
2.99
8000
4.83
8.96
6.89
1.67
2.87
2.37
2.44
Appendix
Table 7: Detailed scaling results of CE and LFD under ASR-only and QA-only training settings with 2,000 to 8,000 hours of training data. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%) on TELEVAL. Lower WER and higher accuracy indicate better performance.