cs.CVFeb 22, 2026
SavePay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
Organizations: NatWest AI Research · Imperial College London
Abstract
Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
Figures & tables
Figure 1: CTC vs. attention-based decoding (greedy). Left: AVSR word error rate (WER) of USR on in- (LRS3) and out-of-distribution (VoxCeleb2, automatically transcribed) samples. CTC decoding is notably more robust to domain shift and long sequences, while autoregressive attention-based decoding performs best in-distribution. Right: Decoding speed on a H200 GPU. CTC is 40 faster than autoregressive decoding. While teacher forcing could significantly speed up attention-based decoding, it typically relies on ground-truth tokens unavailable during pseudo-labelling.
Figure 2: Pseudo-labelling in USR and USR 2.0 In USR (left), the teacher generates CTC and attention-based pseudo-labels (PLs) from unmasked audiovisual inputs: CTC PLs are generated in parallel, while attention-based PLs require autoregressive decoding. The student predicts each target type independently, leading to decoupled supervision. USR 2.0 introduces two modes for tighter integration. In CTC-driven mode (centre), attention-based PLs are generated by feeding collapsed CTC PLs into the decoder via teacher forcing, avoiding autoregression. The student decoder predicts both types of PLs. In AR mode (right), the teacher operates autoregressively as in USR, and the student’s CTC branch predicts both CTC and attention-based PLs. USR 2.0 alternates between modes at each iteration: CTC-driven mode improves efficiency and robustness, while AR mode mitigates exposure bias. Note: refers to batch size, and typically, (not to scale in the figure).
Figure 3: Robustness to long utterances. (a) Greedy decoding: USR 2.0 maintains robustness to longer input lengths, significantly outperforming other models. (b) Beam search ( , joint CTC-attention decoding) improves USR robustness but still lags behind USR 2.0. (c) Increasing beam size reduces the WER gap between USR and USR 2.0, but at a significant computational and memory cost. The size of the markers corresponds to the relative memory cost with batched beam search.
| Method | All Samples | Samples with Frames | ||||||||
| 10 dB | 5 dB | 0 dB | -5 dB | Avg | 10 dB | 5 dB | 0 dB | -5 dB | Avg | |
| ASR | ||||||||||
| AV-HuBERT | 8.4 | 18.9 | 54.2 | 94.6 | 44.0 | 6.5 | 16.2 | 49.3 | 97.2 | 42.3 |
| BRAVEn | 7.2 | 16.8 | 50.1 | 99.2 | 43.3 | 6.4 | 13.8 | 49.2 | 108.4 | 44.5 |
| USR | 5.8 | 14.3 | 48.5 | 104.4 | 43.3 | 4.9 | 12.6 | 48.1 | 111.7 | 44.3 |
| USR 2.0 | 5.2 | 13.4 | 44.0 | 94.4 | 39.3 | 3.8 | 10.6 | 42.8 | 98.3 | 38.9 |
Table 1: Comparison of WER (%) across different SNR levels (dB) for ASR and AVSR on the LRS3 test set (beam size 30). None of the methods saw noise-augmented samples during training.
Table 2: In-distribution comparisons with self-supervised methods. LRS3 WER (%) results for low- (left) and high-resource (right) labelled data settings. Best results in bold . We specify model size and pre-training dataset for each setting in bold . “Shared params” denotes whether a unified model is used. * Trained on LRS2+LRS3 (labelled) and English-only VoxCeleb2+AVSpeech (unlabelled).
| Method | LibriS | WildVSR | AVS |
| AV-HuBERT | 29.1 | 82.4 | 26.0 |
| BRAVEn | 38.4 | 81.2 | 44.6 |
| USR | 25.3 | 80.0 | 34.7 |
| USR 2.0 | 15.4 | 73.7 | 25.0 |
Table 3: Performance on OOD datasets : LibriSpeech (LibriS), WildVSR, and AVSpeech (AVS). We report WER (%) under greedy decoding.
Figure 5: VSR WER vs. training time for the low-resource setting.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Base | Base+ | Large | Huge | |
| Total parameters (M) | 86 | 171 | 503 | 953 |
| Transformer blocks (enc / dec) | 12 / 6 | 12 / 6 | 24 / 9 | 36 / 9 |
| Attention dim | 512 | 768 | 1024 | 1280 |
| MLP size | 2048 | 3072 | 4096 | 5120 |
| Heads | 8 | 12 | 16 | 16 |
Table 5: Model hyperparameters. Parameter count includes encoder, decoder, and feature extractors.
| Hyperparameter | LRS3 (Base) | LRS3+VoxCeleb2 | All (Huge) |
| Learning rate | 3e-3 | 2e-3 | 7e-4 |
| Weight decay | 0.04 | 0.04 | 0.04 |
| Drop path rate | 0.1 | 0.1 (Base), 0.2 (Large) | 0.3 |
| Frames per GPU (lab./unlab.) | 600 / 4400 | 300 / 3200 | – |
| Frames per GPU (high-resource) | – | 1200 / 3200 | 700 / 2400 |
Table 6: Training hyperparameters for different pre-training datasets and model sizes. “All” refers to the combination of LRS2, LRS3, VoxCeleb2, and AVSpeech. Additional hyperparameters are provided in Appendix A.5 .
| Method | Labelled hours | Unlabelled hours | Lang. model | Shared params | WER (%) | ||
| V | A | AV | |||||
| Supervised * | |||||||
| V2P | 3,886 | - | ✗ | ✗ | 55.1 | - | - |
| RNN-T | 31,000 | - | ✗ | ✓ | 33.6 | 4.8 | 4.5 |
| VTP | 2,676 | - | ✓ | ✗ | 30.7 | - | - |
| Llama-AVSR † | 680,000 | - | ✓ | ✗ | 24.0 | 0.8 | 0.8 |
Table 7: Comparisons with the state-of-the-art on LRS3. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training. † Uses Whisper as its encoder, trained on 680k hours of transcribed speech data ( Radford et al., 2023 ) .
| Method | Labelled hours | Unlabelled hours | Lang. model | Shared params | WER (%) | ||
| V | A | AV | |||||
| Supervised * | |||||||
| CM-seq2seq | 380 | - | ✓ | ✗ | 37.9 | 3.9 | 3.7 |
| CM-aux | 1,459 | - | ✓ | ✗ | 25.5 | - | - |
| VTP | 698 | - | ✓ | ✗ | 28.9 | - | - |
| VTP | 2,676 | - | ✓ | ✗ | 22.6 | - | - |
Table 8: Comparisons with the state-of-the-art on LRS2. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training. † Uses Whisper as its encoder, trained on 680k hours of transcribed speech data ( Radford et al., 2023 ) .
| Method | Labelled hours | Unlabelled hours | Lang. model | Shared params | WER (%) |
| Supervised * | |||||
| CM-seq2seq | 1,459 | - | ✓ | ✗ | 58.4 |
| VTP | 698 | - | ✓ | ✗ | 75.6 |
| VTP | 2,676 | - | ✓ | ✗ | 68.7 |
| Auto-AVSR | 661 | - | ✓ | ✗ | 62.3 |
| Auto-AVSR | 1,759 | - | ✓ | ✗ | 49.3 |
Table 9: Comparisons with the state-of-the-art on WildVSR. * Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. “Shared params” denotes whether a unified model is used at inference. “ST” denotes offline self-training.
Table 10: Ablation results. We report WER (%) for VSR (V), ASR (A), and AVSR (AV) on in-distribution test data (LRS3) and AVSR on out-of-distribution data (sampled, automatically transcribed AVSpeech). Default settings for USR 2.0 are highlighted in gray .
Figure 6: WER vs. input length on VoxCeleb2 (automatically transcribed using Whisper). Models are trained in the low-resource LRS3 setup, where labelled utterances are no longer than around 150 video frames. We evaluate CTC-only and attention-only greedy decoding for BRAVEn, USR, and USR 2.0.
| (a) Omission Failures | |
| GT | because with them we can break this cycle of suffering without them the cycle will continue I’ll leave you with the word that we use at the end of every yoga class which also illustrates this fundamental truth I really believe in that we are all one |
| USR | because with them we can break this cycle of every yoga class which also illustrates this fundamental truth I really believe in is that we are all one |
| USR 2.0 | because with them we can break this cycle of suffering without them the cycle will continue I’ll leave you with the word that we use at the end of every yoga class which also illustrates this fundamental truth I really believe in is that we are all one |
| GT | so the evidence is there and it’s going to be designed to interpret it correctly it will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it’s a complicated problem but as it learns more about our objectives it will become more and more |
| USR | so it’s a complicated problem but as it learns more about our objectives it will become more and more |
| USR 2.0 | so the evidence is there it’s going to be designed to interpret it correctly it will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it’s a complicated problem but as it learns more about our objectives it will become more and more |
Table 11: Qualitative analysis of USR vs. USR 2.0 on long, out-of-distribution utterances. We show examples where USR either omits substantial segments (top) or degenerates into repetitions (bottom), both of which USR 2.0 resolves. Segments omitted by USR but correctly transcribed by USR 2.0 are highlighted in green ; repetition errors by USR are highlighted in red .
Figure 7: Examples of AR inconsistencies in decoder outputs under CTC-driven teacher forcing. In each case, the decoder is conditioned on CTC-derived prefixes, sometimes leading to repeated or malformed outputs at the sequence level, even though each token prediction is plausible given its input. These artefacts do not affect training, as the student learns to match the teacher’s outputs under the same input.
Figure 8: Validation curves for USR and USR 2.0. Top row shows decoder accuracy curves; bottom row shows CTC validation loss. Each curve shows mean ± std over three seeds.
| Method | Epochs | V | A | AV |
| USR | 50 | |||
| USR 2.0 | 50 | |||
| USR | 75 | 36.0 | 3.2 | 3.0 |
| USR 2.0 | 75 | 36.1 | 3.0 | 2.9 |
Table 13: Training Duration Comparison. We report LRS3 low-resource WER (%) results for 50- and 75-epoch training. Mean and standard deviation over three random seeds are reported for the 50-epoch runs.
| GT | We push and pull and poke and prod things |
| V | We push and pull and poke and break things |
| A | We push and pull and poke and prod things |
| AV | We push and pull and poke and prod things |
| GT | You’re more cautious |
| V | You’re more conscious |
| A | You’re more cautious |
Table 14: Representative failure cases for USR 2.0 on the LRS3 test set . We show outputs from the Huge model. Correct fragments are shown in green ; errors are in red .