Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
Figures & tables
Figure 1: CTC vs. attention-based decoding (greedy). Left: AVSR word error rate (WER) of USR on in- (LRS3) and out-of-distribution (VoxCeleb2, automatically transcribed) samples. CTC decoding is notably more robust to domain shift and long sequences, while autoregressive attention-based decoding performs best in-distribution. Right: Decoding speed on a H200 GPU. CTC is ∼ 40 × faster than autoregressive decoding. While teacher forcing could significantly speed up attention-based decoding, it typically relies on ground-truth tokens unavailable during pseudo-labelling.
Figure 2: Pseudo-labelling in USR and USR 2.0 In USR (left), the teacher generates CTC and attention-based pseudo-labels (PLs) from unmasked audiovisual inputs: CTC PLs are generated in parallel, while attention-based PLs require autoregressive decoding. The student predicts each target type independently, leading to decoupled supervision. USR 2.0 introduces two modes for tighter integration. In CTC-driven mode (centre), attention-based PLs are generated by feeding collapsed CTC PLs into the decoder via teacher forcing, avoiding autoregression. The student decoder predicts both types of PLs. In AR mode (right), the teacher operates autoregressively as in USR, and the student’s CTC branch predicts both CTC and attention-based PLs. USR 2.0 alternates between modes at each iteration: CTC-driven mode improves efficiency and robustness, while AR mode mitigates exposure bias. Note: B refers to batch size, and typically, UCTC,UAR≪L (not to scale in the figure).
Figure 3: Robustness to long utterances. (a) Greedy decoding: USR 2.0 maintains robustness to longer input lengths, significantly outperforming other models. (b) Beam search ( beam size=30 , joint CTC-attention decoding) improves USR robustness but still lags behind USR 2.0. (c) Increasing beam size reduces the WER gap between USR and USR 2.0, but at a significant computational and memory cost. The size of the markers corresponds to the relative memory cost with batched beam search.
Method
All Samples
Samples with >100 Frames
10 dB
5 dB
0 dB
-5 dB
Avg
10 dB
5 dB
0 dB
-5 dB
Avg
ASR
AV-HuBERT
8.4
18.9
54.2
94.6
44.0
6.5
16.2
49.3
97.2
42.3
BRAVEn
7.2
16.8
50.1
99.2
43.3
6.4
13.8
49.2
108.4
44.5
USR
5.8
14.3
48.5
104.4
43.3
4.9
12.6
48.1
111.7
44.3
USR 2.0
5.2
13.4
44.0
94.4
39.3
3.8
10.6
42.8
98.3
38.9
Table 1: Comparison of WER (%) across different SNR levels (dB) for ASR and AVSR on the LRS3 test set (beam size 30). None of the methods saw noise-augmented samples during training.
Table 2: In-distribution comparisons with self-supervised methods. LRS3 WER (%) results for low- (left) and high-resource (right) labelled data settings. Best results in bold . We specify model size and pre-training dataset for each setting in bold . “Shared params” denotes whether a unified model is used. * Trained on LRS2+LRS3 (labelled) and English-only VoxCeleb2+AVSpeech (unlabelled).
Method
LibriS
WildVSR
AVS
AV-HuBERT
29.1
82.4
26.0
BRAVEn
38.4
81.2
44.6
USR
25.3
80.0
34.7
USR 2.0
15.4
73.7
25.0
Table 3: Performance on OOD datasets : LibriSpeech (LibriS), WildVSR, and AVSpeech (AVS). We report WER (%) under greedy decoding.
Table 7
Figure 5: VSR WER vs. training time for the low-resource setting.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Base
Base+
Large
Huge
Total parameters (M)
86
171
503
953
Transformer blocks (enc / dec)
12 / 6
12 / 6
24 / 9
36 / 9
Attention dim
512
768
1024
1280
MLP size
2048
3072
4096
5120
Heads
8
12
16
16
Appendix
Table 5: Model hyperparameters. Parameter count includes encoder, decoder, and feature extractors.
Hyperparameter
LRS3 (Base)
LRS3+VoxCeleb2
All (Huge)
Learning rate
3e-3
2e-3
7e-4
Weight decay
0.04
0.04
0.04
(β1,β2)
(0.9,0.98)
(0.9,0.98)
(0.9,0.98)
Drop path rate
0.1
0.1 (Base), 0.2 (Large)
0.3
Frames per GPU (lab./unlab.)
600 / 4400
300 / 3200
–
Frames per GPU (high-resource)
–
1200 / 3200
700 / 2400
Appendix
Table 6: Training hyperparameters for different pre-training datasets and model sizes. “All” refers to the combination of LRS2, LRS3, VoxCeleb2, and AVSpeech. Additional hyperparameters are provided in Appendix A.5 .
Method
Labelled hours
Unlabelled hours
Lang. model
Shared params
WER (%)
V
A
AV
Supervised *
V2P
3,886
-
✗
✗
55.1
-
-
RNN-T
31,000
-
✗
✓
33.6
4.8
4.5
VTP
2,676
-
✓
✗
30.7
-
-
Llama-AVSR †
680,000
-
✓
✗
24.0
0.8
0.8
Appendix
Table 7: Comparisons with the state-of-the-art on LRS3. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training. † Uses Whisper as its encoder, trained on ∼ 680k hours of transcribed speech data ( Radford et al., 2023 ) .
Method
Labelled hours
Unlabelled hours
Lang. model
Shared params
WER (%)
V
A
AV
Supervised *
CM-seq2seq
380
-
✓
✗
37.9
3.9
3.7
CM-aux
1,459
-
✓
✗
25.5
-
-
VTP
698
-
✓
✗
28.9
-
-
VTP
2,676
-
✓
✗
22.6
-
-
Appendix
Table 8: Comparisons with the state-of-the-art on LRS2. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training. † Uses Whisper as its encoder, trained on ∼ 680k hours of transcribed speech data ( Radford et al., 2023 ) .
Method
Labelled hours
Unlabelled hours
Lang. model
Shared params
WER (%)
Supervised *
CM-seq2seq
1,459
-
✓
✗
58.4
VTP
698
-
✓
✗
75.6
VTP
2,676
-
✓
✗
68.7
Auto-AVSR
661
-
✓
✗
62.3
Auto-AVSR
1,759
-
✓
✗
49.3
Appendix
Table 9: Comparisons with the state-of-the-art on WildVSR. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training.
Table 10: Ablation results. We report WER (%) for VSR (V), ASR (A), and AVSR (AV) on in-distribution test data (LRS3) and AVSR on out-of-distribution data (sampled, automatically transcribed AVSpeech). Default settings for USR 2.0 are highlighted in gray .
Figure 6: WER vs. input length on VoxCeleb2 (automatically transcribed using Whisper). Models are trained in the low-resource LRS3 setup, where labelled utterances are no longer than around 150 video frames. We evaluate CTC-only and attention-only greedy decoding for BRAVEn, USR, and USR 2.0.
(a) Omission Failures
GT
because with them we can break this cycle of suffering without them the cycle will continue I’ll leave you with the word that we use at the end of every yoga class which also illustrates this fundamental truth I really believe in that we are all one
USR
because with them we can break this cycle of every yoga class which also illustrates this fundamental truth I really believe in is that we are all one
USR 2.0
because with them we can break this cycle of suffering without them the cycle will continue I’ll leave you with the word that we use at the end of every yoga class which also illustrates this fundamental truth I really believe in is that we are all one
GT
so the evidence is there and it’s going to be designed to interpret it correctly it will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it’s a complicated problem but as it learns more about our objectives it will become more and more
USR
so it’s a complicated problem but as it learns more about our objectives it will become more and more
USR 2.0
so the evidence is there it’s going to be designed to interpret it correctly it will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it’s a complicated problem but as it learns more about our objectives it will become more and more
Appendix
Table 11: Qualitative analysis of USR vs. USR 2.0 on long, out-of-distribution utterances. We show examples where USR either omits substantial segments (top) or degenerates into repetitions (bottom), both of which USR 2.0 resolves. Segments omitted by USR but correctly transcribed by USR 2.0 are highlighted in green ; repetition errors by USR are highlighted in red .
Figure 7: Examples of AR inconsistencies in decoder outputs under CTC-driven teacher forcing. In each case, the decoder is conditioned on CTC-derived prefixes, sometimes leading to repeated or malformed outputs at the sequence level, even though each token prediction is plausible given its input. These artefacts do not affect training, as the student learns to match the teacher’s outputs under the same input.
Figure 8: Validation curves for USR and USR 2.0. Top row shows decoder accuracy curves; bottom row shows CTC validation loss. Each curve shows mean ± std over three seeds.
Method
Epochs
V
A
AV
USR
50
37.6±0.7
3.4±0.1
3.2±0.1
USR 2.0
50
36.2±0.5
3.0±0.1
2.9±0.1
USR
75
36.0
3.2
3.0
USR 2.0
75
36.1
3.0
2.9
Appendix
Table 13: Training Duration Comparison. We report LRS3 low-resource WER (%) results for 50- and 75-epoch training. Mean and standard deviation over three random seeds are reported for the 50-epoch runs.
GT
We push and pull and poke and prod things
V
We push and pull and poke and break things
A
We push and pull and poke and prod things
AV
We push and pull and poke and prod things
GT
You’re more cautious
V
You’re more conscious
A
You’re more cautious
Appendix
Table 14: Representative failure cases for USR 2.0 on the LRS3 test set . We show outputs from the Huge model. Correct fragments are shown in green ; errors are in red .
Speech LLM post-training increasingly relies on efficient cross-modal alignment and robust low-resource adaptation, yet collecting large-scale audio-text pairs remains costly. Text-only alignment methods such as TASU reduce this burden by simulating CTC posteriors from transcripts, but they provide limited control over uncertainty and error rate, making curriculum design largely heuristic. We propose TASU2, a controllable CTC simulation framework that simulates CTC posterior distributions under a specified WER range, producing text-derived supervision that better matches the acoustic decoding interface. This enables principled post-training curricula that smoothly vary supervision difficulty without TTS. Across multiple source-to-target adaptation settings, TASU2 improves in-domain and out-of-domain recognition over TASU, and consistently outperforms strong baselines including text-only fine-tuning and TTS-based augmentation, while mitigating source-domain performance degradation.
Jing Peng, Chenghao Wang, Yi Yang +5
Jiangsu Key Lab of Language Computing · AISpeech Ltd, Suzhou, China · Nanjing University, China
Streaming automatic speech recognition (ASR) underperforms on domain-shifted target audio, where labeled in-domain data is costly to prepare while unlabeled audio is abundant. We present StreamHear, a semi-supervised pipeline that adapts a pretrained streaming student by fine-tuning an offline transducer teacher on the labeled training set, generating pseudo-labels on the unlabeled portion, and fine-tuning the student on the mixture. We further introduce a prior-regularized dynamic-programming realignment step that fixes chunk-level word placement using an ASR-hypothesis anchor. Across four datasets spanning financial calls, prepared read speech, and phone-quality dialogue, StreamHear consistently outperforms supervised student fine-tuning and narrows the gap to the offline teacher.
BEST-RQ is a simple and effective self-supervised training method for speech representation learning that performs well on automatic speech recognition (ASR) tasks. It generates pseudolabels using a fixed online quantization scheme, which simplifies training but provides weaker supervision than HuBERT-style models that iteratively refine pseudo-labels. In this work, we improve online pseudo-label generation while preserving simplicity. We propose three modifications: replacing the quantizer's linear projection with Principal Component Analysis (PCA), updating the codebook via iterative codebook refinement, and introducing an additional codebook updated via codebook distillation. We pre-train on the LibriSpeech 960-hour dataset and fine-tune using 100 hours of supervised LibriSpeech data. With all three modifications enabled, we achieve a 12% relative reduction in word error rate (WER) on the LibriSpeech test-other set, improving from 10.1% to 8.8%.
Jingjing Xu, Zijian Yang, Mohammad Zeineldeen +3
Machine Learning and Human Language Technology Group, RWTH Aachen University, Germany · Apptek GmbH, Aachen, Germany