\toprule Method Encoder Criterion Training data (h) WER (%) ↓ \downarrow ↓ \cmidrule (lr)4-5 \cmidrule (lr)6-9 Labeled Unlabeled 0 ∘ \mathbf{0^{\circ}} 0 ∘ 5 ∘ \mathbf{5^{\circ}} 5 ∘ 10 ∘ \mathbf{10^{\circ}} 1 0 ∘ 15 ∘ \mathbf{15^{\circ}} 1 5 ∘ \midrule Supervised CM-seq2seq Ma et al. (2021) Conformer CTC+S2S 433 – 43.3 64.6 70.8 83.9 XMD Afouras et al. (2020) CNN CTC 433 – 68.8 74.1 78.6 82.0 MSR Xu et al. (2020) RNN S2S 433 – 57.8 66.4 72.5 77.6 Auto-AVSR Ma et al. (2023) Conformer CTC+S2S 818 – 33.0 51.5 64.8 73.0 \midrule Self-supervised CMA Kim et al. (2024) Transformer S2S 433 1,759 32.5 40.4 50.5 73.8 RAVen Haliassos et al. (2023) Transformer CTC+S2S 433 1,759 27.8 35.9 49.1 52.3 AV-HuBERT-B Shi et al. (2022) Transformer S2S 433 1,759 34.8 45.7 66.2 72.1 AV-HuBERT-L Shi et al. (2022) Transformer S2S 433 1,759 28.2 36.3 43.6 50.5 AV-HuBERT-L (VA-433h) Transformer S2S 433 433 44.6 63.7 71.5 82.4 AV-HuBERT-L (VA-1759h) Transformer S2S 433 1,759 28.6 40.2 46.6 52.7 \midrule Our Method MuViRL-L (VA-433h) Transformer S2S 433 433 42.2 61.4 68.1 71.4 MuViRL-L (VA-1759h) Transformer S2S 433 1,759 27.0 34.3 40.8 46.9 \bottomrule Table 2: Multimodal representation evaluation. Here, MuViRL denotes the configuration trained without decision or depth weighting. Values are means ± \pm ± standard deviations over three pretraining seeds. \toprule Configuration LRS3 OuluVS2 \cmidrule (lr)2-4 \cmidrule (lr)5-7 Prediction accuracy (%) ↑ \uparrow ↑ Cross-view Recall (%) ↑ \uparrow ↑ Cross-modal Recall (%) ↑ \uparrow ↑ Prediction accuracy (%) ↑ \uparrow ↑ Cross-view Recall (%) ↑ \uparrow ↑ Cross-modal Recall (%) ↑ \uparrow ↑ \midrule Baseline 3.85 ± 0.03 3.85\pm 0.03 3.85 ± 0.03 64.59 ± 0.27 64.59\pm 0.27 64.59 ± 0.27 53.56 ± 0.23 53.56\pm 0.23 53.56 ± 0.23 29.17 ± 0.41 29.17\pm 0.41 29.17 ± 0.41 83.40 ± 0.35 83.40\pm 0.35 83.40 ± 0.35 64.02 ± 0.60 64.02\pm 0.60 64.02 ± 0.60 Random view 3.93 ± 0.06 \mathbf{3.93}\pm 0.06 3.93 ± 0.06 65.32 ± 0.34 65.32\pm 0.34 65.32 ± 0.34 53.93 ± 0.28 53.93\pm 0.28 53.93 ± 0.28 32.84 ± 0.48 32.84\pm 0.48 32.84 ± 0.48 85.68 ± 0.51 85.68\pm 0.51 85.68 ± 0.51 67.23 ± 1.08 67.23\pm 1.08 67.23 ± 1.08 MuViRL 3.91 ± 0.05 3.91\pm 0.05 3.91 ± 0.05 67.64 ± 0.06 \mathbf{67.64}\pm 0.06 67.64 ± 0.06 55.32 ± 0.09 \mathbf{55.32}\pm 0.09 55.32 ± 0.09 35.09 ± 0.33 \mathbf{35.09}\pm 0.33 35.09 ± 0.33 88.66 ± 0.22 \mathbf{88.66}\pm 0.22 88.66 ± 0.22 68.60 ± 0.67 \mathbf{68.60}\pm 0.67 68.60 ± 0.67 \bottomrule With synthesized yaw offsets up to ± 15 ∘ \pm 15^{\circ} ± 1 5 ∘ , MuViRL-L pretrained on VA-1759h achieves 46.9% WER, outperforming CMA (73.8%) and RAVen (52.3%). Using the same Large encoder and VA-1759h training data, it achieves 27.0% WER on the original test set and 46.9% with the ± 15 ∘ \pm 15^{\circ} ± 1 5 ∘ range, compared with 28.6% and 52.7% for AV-HuBERT-L. The relative improvement is greater with synthesized views than on the original test set, averaging 10.9% across the four settings. Improvements are also obtained with VA-433h, where MuViRL-L achieves 71.4% WER with the ± 15 ∘ \pm 15^{\circ} ± 1 5 ∘ range, compared with 82.4% for AV-HuBERT-L. The improvements under both pretraining settings, with encoder capacity and training data matched, indicate robustness to viewpoint variation. 4.4 Evaluation of Multimodal View-Invariant Representations
We use AV-HuBERT as the baseline to assess view invariance and generalization. The qualitative and quantitative comparisons cover recognition on real multiview data, spatial responses, feature distributions, and temporal correspondence.
Qualitative evaluation. We adopt a spatiotemporal occlusion sensitivity method similar to receptive-field visualization approaches Zeiler and Fergus (2014) ; Zhou et al. (2015) to identify regions contributing to prediction. As shown in Fig. 4 , the baseline’s spatial responses vary across views, with additional prediction errors. In the illustrated sequence, MuViRL maintains sensitivity around the mouth opening and lip contours and makes fewer word prediction errors. These localized responses and predictions indicate preserved articulatory evidence across views.
Figure 4: Qualitative evaluation of view-invariant representations. We overlay sensitivity heatmaps on an LRS3 test sequence for (a) the original view and (b) a synthesized view with yaw + 20 ∘ +20^{\circ} + 2 0 ∘ , showing the contribution of different mouth regions to prediction. Colors range from blue (low sensitivity) to red (high sensitivity), according to the loss change Δ L \Delta\mathcal{L} Δ L under local occlusion. Word predictions are aligned below the frames. Red words mark transcript mismatches, and [miss] denotes a deletion. Quantitative evaluation. Fig. 5 presents results on real and synthesized multiview data. After OuluVS2 fine-tuning from matched VA-1759h pretraining and VA-433h recognition checkpoints, MuViRL-L achieves 3.65% overall WER on OuluVS2, compared with 4.47% for the baseline, an 18.3% relative improvement. WER is also lower at 60 ∘ 60^{\circ} 6 0 ∘ and 90 ∘ 90^{\circ} 9 0 ∘ , and the overall improvement remains with VA-30h fine-tuning. Direct transfer to TCD-TIMIT uses all 5,782 paired utterances from 59 volunteers with identical preprocessing and decoding. Overall WER is 79.85% and 59.78% with 30 h and 433 h fine-tuning, compared with 81.48% and 61.06% for the baseline. Lower WER at both viewpoints without additional training on the transfer corpus indicates generalization to real multiview data. Evaluation on synthesized data covers different angular offsets and proportions of synthesized test clips. With 433 h fine-tuning, both Base and Large configurations outperform the baseline in most settings. The recognition gains extend across different angular ranges and proportions of synthesized clips. Figure 5: Quantitative evaluation on multiview data. B-30h/B-433h denote AV-HuBERT Base pretrained on 433 h and fine-tuned on 30 h/433 h, respectively. L-30h/L-433h denote AV-HuBERT Large pretrained on 1,759 h and fine-tuned on 30 h/433 h, respectively. MuViRL uses the corresponding view augmented data. (a,c) Angular Offset: performance as the range of synthesized yaw offsets expands. (b,d) Random Proportions: performance with varying proportions of synthesized test clips. (e,f) Results on TCD-TIMIT and OuluVS2, respectively. Representation structure analysis. We assess whether representations remain consistent within multimodal units and across synchronized views while preserving temporal distinctions. Table 2 presents prediction and retrieval results, and Fig. 6 shows feature structure. Experiments on both datasets use the Base configurations. Baseline continues prediction on the original view, Random view samples one view per update, and MuViRL uses MMP, Adaptive Weighting, Agg2S, and Temporal RSA without decision or depth weighting.
Figure 6: Multimodal representation structure. (a) Unit consistency and (b) view consistency on OuluVS2. Points below the diagonal indicate higher similarity with MuViRL. (c) Correspondence between audio and synthesized visual frames on LRS3. The bright diagonal indicates matching timestamps. (d) Mean similarity at different frame offsets for visual and audiovisual pairs from the same sequence. Peaks at zero offset indicate temporal alignment. Prediction accuracy is the proportion of valid masked positions with a correctly predicted unit. Both retrieval metrics Lu et al. (2023) ; Araujo et al. (2025) report Recall@1. For each candidate view, cosine similarity ranks its valid frames within the same utterance. Cross-view Recall uses original visual queries, and Cross-modal Recall uses audio queries. A query is correct when the highest-ranked frame has the same timestamp. As shown in Table 2 , Cross-view Recall improves from 64.59% to 67.64% on LRS3 and from 83.40% to 88.66% on OuluVS2. OuluVS2 Cross-modal Recall increases from 64.02% to 68.60%. The gains in both recalls indicate more accurate temporal matching across viewpoints and modalities.
Unit consistency averages cosine similarity between distinct normalized features sharing a teacher label. View consistency averages it over synchronized feature pairs within each utterance. MuViRL improves similarity for 436 of 447 units and 353 of 360 test utterances in Fig. 6 (a,b). Across three seeds, the respective means increase from 0.6379 to 0.6997 and from 0.8514 to 0.9202. Unit centers also become more separated, with squared distance increasing from 1.0403 to 1.1300. For the same LRS3 sequence, the bright diagonal and peaks at zero offset in Fig. 6 (c,d) indicate temporal alignment. Offsets are synthesized visual frame indices minus original visual or audio frame indices. Supplementary Table S2 reports prediction and retrieval results for component comparisons.
4.5 Evaluation of Calibrated View Aggregation
To assess how Calibrated View Aggregation supports multimodal representation learning, we evaluate aggregate prediction and recognition from individual views. Table 3 presents aggregation results with the encoder and Adaptive Weighting fixed, and Fig. 7 shows weight allocation and prediction.
Decision model adaptation analysis. The aggregation comparison uses scores from a decision model fine-tuned with four quality grades on ORL, Yale, Notting Hill, and MEAD. Training uses blur and occlusion, while evaluation on pixelation tests generalization. Actors and identities are disjoint across adaptation, calibration, and evaluation. Supplementary Table S1 presents grading accuracy across the four datasets. Aggregation evaluation uses 120 OuluVS2 test utterances covering all twelve test speakers. Distortion is applied to two oblique views at one of three levels.
Figure 7: View aggregation analysis. (a) Weight gains for views without added distortion. Boxes show quartiles and medians, and diamonds mark means. (b) Changes in similarity and accuracy per utterance relative to aggregation without reliability weighting. Similarity on the horizontal axis is the paired change, scaled by 10 3 10^{3} 1 0 3 . Points in the upper right indicate improved stability and prediction. \toprule Configuration Adaptation Prediction accuracy (%) ↑ \uparrow ↑ Similarity ↑ \uparrow ↑ \cmidrule (lr)3-5 Original inputs Blur/ occlusion Pixelation \midrule w/o reliability weighting – 37.29 36.23 37.05 0.9702 Laplacian – 37.32 36.73 37.05 0.9723 Pixel similarity – 37.35 36.73 37.11 0.9854 \midrule Decision model × \times × 37.44 36.82 36.87 0.9918 Decision model ✓ \checkmark ✓ 38.22 37.87 36.93 0.9926 \bottomrule Table 3: Aggregation evaluation with different input configurations. Original inputs are evaluated without added distortion. Adaptation means fine-tuning with four quality grades. w/o reliability means q i , t = 1 q_{i,t}=1 q i , t = 1 . Similarity means cosine agreement with the reference aggregate computed from original inputs, with larger values indicating greater stability. View aggregation analysis. As shown in Table 3 , aggregation with the adapted decision model achieves 37.87% accuracy under blur and occlusion, compared with 36.82% without adaptation and 36.23% without reliability weighting. It also gives the highest accuracy on original inputs (38.22%) and similarity to the reference aggregate (0.9926). The reference uses inputs without added distortion and uniform reliability. Table 3 pools frame contributions across distortion conditions within each seed, then averages over three seeds. Fig. 7 shows weight allocation and paired changes per utterance, averaged over three distortion types, three severity levels, and three seeds. Under unseen pixelation, pixel similarity gives the highest accuracy (37.11%), compared with 36.93% using adapted decisions. The adapted decision scores improve aggregate prediction under blur and occlusion.
Evaluation of decision scores. Supplementary Table S3 evaluates teacher supervision, ranking, and temperature calibration under matched data, initialization, and training budgets. Ranking improves pairwise accuracy from 52.90% to 54.02%, while calibration lowers expected calibration error from 0.68% to 0.49%. We compare aggregation using uniform reliability, ranking scores, and calibrated scores on the same OuluVS2 split. Base encoder initialization, view sampling, training budgets, Agg2S, and uniformly weighted Temporal RSA remain fixed. Across three seeds, mean WER is 17.52% with uniform reliability, 17.10% with ranking scores, and 16.48% with calibrated scores. Calibrated scores also yield the highest aggregate accuracy (37.56%). These results show that calibrated decision scores improve aggregate prediction and reduce recognition errors.
4.6 Evaluation of Aggregation-to-Single Distillation To evaluate Agg2S, we compare prediction objectives and aggregation methods in Table 4.6 . Both comparisons reuse the LRS3 split and Base training settings from Section 4.4 , with uniform reliability and Temporal RSA disabled. Adaptive Weighting is used for the aggregation branch in the objective comparison. The aggregation comparison retains MMP and Agg2S. Table 4: Evaluation of Aggregation-to-Single Distillation (Agg2S). L s i n g l e \mathcal{L}_{\mathrm{single}} L single and L a g g \mathcal{L}_{\mathrm{agg}} L agg mean target prediction from one view and aggregated views, respectively. \toprule Configuration Cross-view Recall (%) ↑ \uparrow ↑ WER (%) ↓ \downarrow ↓ \midrule Prediction objective L s i n g l e \mathcal{L}_{\mathrm{single}} L single 65.32 69.50 L a g g \mathcal{L}_{\mathrm{agg}} L agg 64.42 68.84 MMP 65.25 69.50 MMP + Agg2S 67.63 66.21 \midrule Aggregation method Mean 66.18 69.47 Learned 67.21 69.55 Cross-view Attention 67.32 69.43 Adaptive Weighting 67.63 66.21 \bottomrule Prediction objective comparison. MMP with Agg2S achieves 67.63% Cross-view Recall and 66.21% WER, compared with 65.25% and 69.50% for MMP alone. The OuluVS2 comparison reuses the representation experiment’s split and training settings, with uniform reliability and Temporal RSA disabled. Increasing the Agg2S coefficient from zero to 0.1 raises mean Cross-view Recall from 86.73% to 86.91% over three seeds (Supplementary Table S2). The retrieval gains on both datasets and lower LRS3 WER support transferring aggregated predictions to individual views. Aggregation method comparison. Mean assigns equal weights to all views. Learned estimates weights from individual views, while Cross-view Attention models interactions across views. Adaptive Weighting achieves the highest recall and lowest WER. Cross-view Recall increases from 66.18% with Mean to 67.63% with Adaptive Weighting. With the distillation objective fixed, Adaptive Weighting improves temporal correspondence and reduces recognition errors. 4.7 Ablation Study We conduct an ablation study of MuViRL using the LRS3 dataset. As shown in Table 4.7 , all configurations use the same data, initialization, and training settings. Baseline uses MMP and Adaptive Weighting with uniform aggregation reliability. Decision and Depth apply utility u i , b u_{i,b} u i , b and geometry g i , b g_{i,b} g i , b to RSA weights. Table 5: Ablation studies of MuViRL on LRS3. Decision and Depth mean decision and geometric weighting of Temporal RSA, respectively. \toprule Agg2S RSA Decision Depth Cross-view Recall (%) ↑ \uparrow ↑ Cross-modal Recall (%) ↑ \uparrow ↑ WER (%) ↓ \downarrow ↓ \midrule Baseline 65.25 53.93 69.50 \midrule ✓ \checkmark ✓ – – – 67.63 55.12 66.21 – ✓ \checkmark ✓ – – 66.22 54.30 68.54 ✓ \checkmark ✓ ✓ \checkmark ✓ – – 67.64 55.32 65.87 ✓ \checkmark ✓ ✓ \checkmark ✓ ✓ \checkmark ✓ – 68.17 56.03 65.07 ✓ \checkmark ✓ ✓ \checkmark ✓ – ✓ \checkmark ✓ 67.84 55.35 65.95 ✓ \checkmark ✓ ✓ \checkmark ✓ ✓ \checkmark ✓ ✓ \checkmark ✓ 68.69 56.40 64.75 \bottomrule Agg2S. As shown in Table 4.7 , Agg2S improves Cross-view Recall from 65.25% to 67.63% and Cross-modal Recall from 53.93% to 55.12%. WER falls from 69.50% to 66.21%. With Temporal RSA retained, Agg2S further improves WER from 68.54% to 65.87%. This improvement indicates that prediction distillation provides supervision beyond temporal feature alignment. Reliability-Aware Real–Synthetic Alignment. Temporal RSA alone achieves 66.22% Cross-view Recall, 54.30% Cross-modal Recall, and 68.54% WER, improving all three metrics over Baseline. When combined with Agg2S, WER improves from 66.21% to 65.87%, with recalls of 67.64% and 55.32%. The joint window and relation objective improves recognition when used with distillation. With decision weighting of Temporal RSA, WER falls from 65.87% to 65.07%, while both recalls increase to 68.17% and 56.03%. Depth weighting alone achieves 67.84% and 55.35% recall but gives 65.95% WER. Combining decision and depth weighting yields 68.69% and 56.40% recall and 64.75% WER. Weighting alignment by both predictive usefulness and geometric consistency improves correspondence across views and modalities. Compared with Baseline, Agg2S with weighted Temporal RSA increases Cross-view Recall from 65.25% to 68.69% and Cross-modal Recall from 53.93% to 56.40%. WER decreases from 69.50% to 64.75%. With masked prediction supervision held fixed, these gains support the combined use of Agg2S and reliability weighted temporal alignment for learning consistent and discriminative representations. 5 Conclusion In this paper, we proposed MuViRL, a self-supervised framework for multimodal view-invariant visual speech representation learning. Synchronized 3DGS synthesis provides corresponding views for learning from shared audiovisual targets. Calibrated View Aggregation uses calibrated multimodal decisions and Adaptive Weighting to select informative visual features. Agg2S transfers aggregated predictions to individual views, and Temporal RSA preserves temporal correspondence between original and synthesized representations. Experiments show improved multimodal representation consistency and visual speech recognition under view variation, including generalization to real multiview data. Ablation studies support the contributions of Agg2S and reliability weighted alignment. Declarations of Conflict of Interest The authors declare that they have no financial or nonfinancial conflicts of interest related to this work.