Multimodal representation learning combines complementary signals to capture shared semantic information. For dynamic scenes, these representations need to remain consistent across views while preserving distinctions between temporal states. In visual speech representation learning, view variation changes the visibility and reliability of articulation cues, making it difficult to learn such representations from synchronized audio and video. To address this challenge, we propose MuViRL, a self-supervised framework for multimodal view-invariant visual speech representation learning via calibrated view aggregation. The framework combines synchronized 3D Gaussian view synthesis with multimodal self-supervision to learn consistent representations from real and synthesized data. A multimodal decision model learns relative view utility from teacher prediction losses, and its calibrated scores guide the aggregation of complementary information. Aggregation-to-Single Distillation transfers the aggregated predictions to a single branch. Real-synthetic alignment matches synchronized windows across views while preserving their temporal relations. Experiments on real and synthesized multiview data demonstrate that MuViRL improves multimodal representation consistency and visual speech recognition performance across views.
Figures & tables
Figure 1: Overview of the proposed framework. (a) Target extraction uses a frozen audiovisual teacher to generate shared unit targets Y . (b) Multimodal self-supervised pretraining uses Calibrated View Aggregation to combine visual features according to reliability qi,t and learned coefficients si,t . Aggregation-to-Single Distillation transfers aggregated predictions to the single branch, while MMP and Temporal RSA provide target supervision and temporal alignment. (c) Fine-tuning optimizes the pretrained encoder and a sequence decoder with transcript supervision.
Figure 2: Region-Focused 3DGS View Synthesis pipeline. (a) Pose-guided clip selection identifies clips with complementary yaw and pitch coverage from head pose trajectories. (b) Region-focused 3DGS fitting uses original motion and restored appearance for reconstruction, with errors averaged over the mouth box, lip rim, and inner mouth regions. (c) Synchronized view synthesis renders the same facial state through virtual cameras at each timestamp. Frame correspondence, lip motion, and rendering artifacts are then checked.
Figure 3: Calibrated View Aggregation. (a) Decision model adaptation uses synchronized lip grids and a language rubric to predict articulation grades, supervised by teacher loss ordering and ordinal grades. (b) Weighted aggregation converts grade logits ℓk into calibrated utility ui,b , then combines it with depth correspondence to obtain reliability qi,t . Here, ui,b abbreviates ui,b(τcal) , and E[k]/3 denotes the normalized expected grade. Reliability and learned coefficients si,t determine weights αi,t for aggregating view features into Hagg .
Table 1: Overall performance on LRS3. The 0∘ column uses original test data. Other columns add synthesized views up to the indicated yaw magnitude in both directions, with zero pitch offset. VA denotes pretraining with synthesized multiview data.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Audio-Visual Speech Recognition (AVSR) enhances speech recognition robustness by leveraging visual cues, while real-world scenarios remain challenging due to viewpoint variation, audio distortion, and visual occlusion, which degrade modality quality and increase audio-visual asynchrony. In this paper, we propose a novel Modality-aware Multi-view Self-supervised representation framework for robust Audio-Visual Speech Recognition (M2S-AVSR). First, we introduce a multi-view representation learning encoder to learn view-invariant visual speech representations. Next, we employ a modality-aware module that explicitly models modality quality and cross-modal synchrony to perform fine-grained modality-aware fusion, enabling fine-grained visual information injection during decoding. In addition, we release AISHELL8-RealScene, a public multi-scenario, multi-view conversational audio-visual dataset recorded in real-world environments, and establish a speech recognition benchmark on it. Experiments on English and Mandarin benchmarks demonstrate the effectiveness of the proposed method under challenging conditions. On LRS3, M2S-AVSR achieves up to 29.4% relative improvement under viewpoint perturbation and visual degradation settings. Our method also achieves new state-of-the-art performance on the MISP2021-AVSR test set. On AISHELL8-RealScene, it achieves the best result in outdoor scenes. The proposed method and dataset provide useful support for future research on robust speech and multimodal tasks under realistic conditions.
Fei Su, Cancan Li, Ming Li +1
School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China
Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead to improvements in performance and may even be critical in certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention a few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do so, we propose a Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.
Linge Wang, Yingying Chen, Bingke Zhu +2
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China