Speech LLM post-training increasingly relies on efficient cross-modal alignment and robust low-resource adaptation, yet collecting large-scale audio-text pairs remains costly. Text-only alignment methods such as TASU reduce this burden by simulating CTC posteriors from transcripts, but they provide limited control over uncertainty and error rate, making curriculum design largely heuristic. We propose TASU2, a controllable CTC simulation framework that simulates CTC posterior distributions under a specified WER range, producing text-derived supervision that better matches the acoustic decoding interface. This enables principled post-training curricula that smoothly vary supervision difficulty without TTS. Across multiple source-to-target adaptation settings, TASU2 improves in-domain and out-of-domain recognition over TASU, and consistently outperforms strong baselines including text-only fine-tuning and TTS-based augmentation, while mitigating source-domain performance degradation.
Figures & tables
Figure 1: An Overview of TASU2.
Method
CE ↓
KL ↓
Acc ↑
ProbDiff ↓
TASU-style baseline
2.5149
2.3372
0.8220
0.0813
TASU2 (AR simulator)
1.2296
1.0497
0.8782
0.0615
Table 1: Posterior similarity metrics (lower is better except Acc).
Figure 2: WER controllability under WER-bin conditioning.
System
Train Data
Libri
Slide
TED
Text
(Audio, Text)
clean/other
TASU
Libri
–
4.57 / 9.90
24.07
19.36
TASU
Libri+Slide
–
4.21 / 10.31
18.70
13.23
SLAM-CTC
–
Libri
3.13 / 8.59
18.59
14.61
TASU2 ⚫
Libri
–
4.63 / 9.82
16.41
14.02
TASU2 ❍
Libri
–
3.41 / 8.15
17.67
14.49
Table 2: Alignment and generalization of simulated CTC training. All models are trained on LibriSpeech and evaluated on LibriSpeech, SlideSpeech, and TED-LIUM; WER (%) ↓ is reported. TASU2 conditions : ⚫ None (no WER conditioning); ❍ WER-binned (coarse 3-level WER bins; we use bin 1 here).
Model
Train audio duration(h)
LibriSpeech ↓ clean/other (WER ↓ )
CoVoST2 ↑ En2Zh (BLEU ↑ )
TASU
0
6.47 / 10.35
33.35
TASU2
0
3.68 / 8.25
33.08
SLAM
1.8k
3.30 / 7.24
37.34
Step-Audio
> 1000k
2.36 / 6.32
–
Qwen2.5-Omni
> 1000k
2.37 / 4.21
41.40
Table 3: TASU2 performance under multi-task evaluation. TASU2 uses WER-conditioned simulation with discretized WER bins; here we use only bin 1 for conditioning.
System
Train data
LibriSpeech
Medical
Stage 1
Stage 2
clean/other
test
SLAM-CTC
Audio
–
2.43 / 6.07
17.55
Raw text [ 11 ]
2.56 / 6.62
13.62
TTS audio
2.72 / 6.80
12.79
Raw audio
2.70 / 6.76
12.35
TASU2 ❍
Text Sim-CTC
–
2.94 / 7.16
15.34
Table 4: Two-stage domain adaptation (source → target). All systems are first trained on the source domain (LibriSpeech) and then adapted to the target domain (Medical). We report WER% ↓ . LoRA is used for all systems for better performance.
School of Computer Science and Technology, Xinjiang University, China · Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, China