cs.CVNov 12, 2026

MuViRL: Multimodal View-Invariant Visual Speech Representation Learning via Calibrated View Aggregation

Authors: Fei Su, Kai Qiu, Juan Liu

Organizations: School of Computer Science, Wuhan University, Wuhan, China. · School of Artificial Intelligence, Wuhan University, Wuhan, China. · Xiaomi EV Technology Co., Ltd., Beijing, China.

Abstract

Multimodal representation learning combines complementary signals to capture shared semantic information. For dynamic scenes, these representations need to remain consistent across views while preserving distinctions between temporal states. In visual speech representation learning, view variation changes the visibility and reliability of articulation cues, making it difficult to learn such representations from synchronized audio and video. To address this challenge, we propose MuViRL, a self-supervised framework for multimodal view-invariant visual speech representation learning via calibrated view aggregation. The framework combines synchronized 3D Gaussian view synthesis with multimodal self-supervision to learn consistent representations from real and synthesized data. A multimodal decision model learns relative view utility from teacher prediction losses, and its calibrated scores guide the aggregation of complementary information. Aggregation-to-Single Distillation transfers the aggregated predictions to a single branch. Real-synthetic alignment matches synchronized windows across views while preserving their temporal relations. Experiments on real and synthesized multiview data demonstrate that MuViRL improves multimodal representation consistency and visual speech recognition performance across views.

Figures & tables

Explore similar work

CardsList
  1. M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

    Jun 4, 2026Fei Su, Cancan Li, Ming Li +1Multimodal RobustnessAudio-Visual Representation Learning

  2. Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

    Apr 9, 2026Linge Wang, Yingying Chen, Bingke Zhu +2Audio-Visual Representation LearningMultimodal Pretraining