Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a dominant paradigm. Although recent LLM-based ASR models have shown promising performance on public benchmarks, it remains challenging to balance recognition quality with latency and overhead, while hallucinations further limit real-world deployment. In this study, we revisit LLM-based ASR from an entropy allocation perspective and introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM. To remedy entropy-allocation inefficiencies in prevailing approaches, we propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and hallucination robustness. Specifically, we redesign the pretraining strategy to alleviate the speech-text modality gap, and further introduce an iterative asynchronous SFT stage between alignment and joint SFT to preserve functional decoupling and constrain encoder representation drift. Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters, while also effectively mitigating hallucinations through our decoupling-oriented design.
Figures & tables
Figure 1: The shift in encoder-side metrics before and after joint training. We hold the encoder architecture fixed within each group to examine the impact of joint training on encoder representations.
Figure 2: Comparison of our multi-stage training design with the traditional training pipeline.
Figure 3: Comparison of the three metrics along our training trajectory and the FireRedASR-AED → FireRedASR-LLM transition, sharing the same encoder architecture. “Our Training Trajectory” corresponds to the full standard “phoneme-level pretraining → IA-SFT → joint SFT” pipeline.
Fun-ASR-nano
GLM-ASR-nano
Qwen3-ASR-1.7B
FireRedASR-LLM
Step-Audio2-mini
Qwen3-Omni-Inst
Ours
Model Size
0.8B
1.5B
2.0B
8B+
8B+
30B-A3B
2.3B
Mandarin
AISHELL-1 dev ∣ test
1.59 ∣ 1.81
2.40 ∣ 2.41
1.40 ∣ 1.51
0.71 ∣ 0.73
0.76 ∣ 0.81
0.86 ∣ 0.92
0.45 ∣ 0.59
AISHELL-2-ios dev ∣ test
2.62 ∣ 2.73
3.21 ∣ 3.45
2.41 ∣ 2.60
2.08 ∣ 2.12
2.24 ∣ 2.29
2.11 ∣ 2.31
2.32 ∣ 2.45
AISHELL-2021- Eval A ∣ C ∣ D
4.75 ∣ 4.29 ∣ 2.33
7.25 ∣ 9.48 ∣ 3.40
4.22 ∣ 3.51 ∣ 1.82
12.61 ∣ 4.06 ∣ 7.38
4.54 ∣ 3.69 ∣ 2.34
5.19 ∣ 3.34 ∣ 1.66
3.45 ∣ 1.71 ∣ 2.53
Chinese Dialect
Table 1: Comparison with advanced LLM-ASR and LALM baselines on public benchmarks.
Model
Mandarin
Dialect
English
Code-switch
Fun-ASR-nano
0.018%
0.217%
0.014%
0.397%
GLM-ASR-nano
0.030%
0.201%
0.014%
0.315%
Qwen3-ASR-1.7B
0.018%
0.120%
0.014%
0.345%
FireRedASR-LLM
0.053%
0.228%
0.014%
0.324%
Step-Audio2-mini
0.020%
0.194%
0.014%
1.255%
Qwen3-Omni-Inst
0.013%
0.370%
0.007%
1.778%
Table 2: Hallucination rates on scenario benchmarks.
Figure 4: Linear CKA scores between layer-wise representations and ground-truth text embeddings, averaged over 1,000 AISHELL/LibriSpeech utterances; “Adap.” denotes the post-adaptor embedding.
Configuration
Mandarin
Dialect
English
Code-switch
Our full pipeline
1.93
10.42
3.34
8.39
– Joint SFT
2.18
12.84
4.22
10.15
– IA-SFT
2.08
11.47
3.79
9.11
– Encoder iter. in IA-SFT
1.95
10.87
3.40
8.57
Table 3: Ablation study on post-training strategies. “– Encoder iter. in IA-SFT” denotes removing encoder hot-swapping and asynchronous parallel during IA-SFT, reducing it to a standard encoder-frozen stage with only the adaptor and LLM trained.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Language(s)
Domain
Hours
YODAS-Granary [ 29 ]
English
Variety
120K
Emilia [ 23 ]
English / Mandarin
Variety
189K
MLS [ 39 ]
English
Audiobook
45K
MSR-86K [ 32 ]
English
YouTube
10K
Common-Voice-v15 [ 3 ]
English / Mandarin
Read
3K
GigaSpeech [ 8 ]
English
Variety
10K
Appendix
Table 4: An overview of the Mandarin and English speech corpora used across all training stages.
Figure 5: Trajectory of CKA scores during pretraining. It reports the average CKA between the encoders and the corresponding reference checkpoint.
Pretraining Step
Trigger
Action Details
0.50 M
–
\small1⃝Pretraining begins. The reference checkpoint is initialized at the 0.50 M pretraining step.
1.01 M
CKA<0.975
\small2⃝Pretraining continues. The encoder snapshot updates the reference checkpoint and initializes the encoder for post-training. The alignment stage then begins and runs for 1.3 M steps in total. Upon its completion, IA-SFT begins and runs for 1.0 M steps, while pretraining continues asynchronously.
1.32 M
CKA<0.975
\small3⃝Pretraining continues. The encoder snapshot updates both the reference checkpoint and the encoder for IA-SFT, which then runs for another 1.0 M steps.
2.00 M (end)
–
\small4⃝Pretraining ends. The encoder snapshot updates the encoder for IA-SFT, which then runs for another 2.0 M steps.
–
–
\small5⃝IA-SFT ends , followed by joint-SFT, which runs for 2.00 M steps in total.
Appendix
Table 5: Detailed training procedures, including the encoder update schedule, CKA thresholds, and stage-wise training steps. Here, the step counts for pretraining, alignment, and IA-SFT are measured independently in their respective training processes.