Audio-Visual Representation Learning

Overview

Audio-visual representation learning studies how models can encode sound and video so that information in one modality helps interpret the other. Common approaches include contrastive alignment, masked or predictive self-supervision, cross-modal distillation, and attention-based fusion, with settings ranging from speech recognition and sound-event localization to video retrieval and spatial grounding. Evaluation spans benchmarks such as AudioSet, VGGSound, AVE, AVSBench, and LRS3, as well as tests with noisy, missing, mismatched, or unseen inputs. A central tension is how to learn broad shared semantics without losing the fine-grained timing and spatial detail needed for reliable predictions.

Latest papers

CardsList

34 papers3 papers in the last four weeks, down 62% on the four weeks before. 0.0% of all new papers.

  1. Learning Spatially Aware Audio-Visual Features from Videos with Mono Audio

    Nov 14, 2026Mengyu Yang, You Zhang, Vishnu Raj +4Audio-Visual Representation LearningMultimodal Spatial Reasoning

  2. Attribution-Guided Shared-Private Decoupling for Noise-Reduced Audio-Visual Representation Learning

    Nov 14, 2026Linge Wang, Yingying Chen, Bingke Zhu +4Audio-Visual Representation LearningShared Representation Learning

  3. MuViRL: Multimodal View-Invariant Visual Speech Representation Learning via Calibrated View Aggregation

    Nov 12, 2026Fei Su, Kai Qiu, Juan LiuAudio-Visual Representation LearningMulti-View Consistency

  4. CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

    Oct 18, 2026Jaehong Myung, Seungho Eum, Taegon Kim +2Audio-Driven Facial AnimationAudio-Visual Representation Learning

  5. Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

    Oct 14, 2026Jing-Xuan Zhang, Genshun Wan, Jianqing Gao +3Audio-Visual Representation LearningCross-Modal Knowledge Distillation

  6. PVSync: A Unified Lip-Sync Expert for Timing and Articulation

    Oct 6, 2026Kevin Stephen, Varun Menon, Timo Mertens +1Lip SynchronizationAudio-Visual Synchronization

  7. GIVE-KWS: Gated Injection of Visual Evidence for Noise-Robust Query-by-Example Keyword Spotting

    Oct 5, 2026Ming-Hsiang Hu, Kuan-Tang Huang, Hung-Shin Lee +1Keyword SpottingAudio-Visual Understanding

  8. SyncRA: Learning Temporal Correspondence in Omni-Modal Models

    Sep 28, 2026Zelong Xu, Yan Li, Wenhe Hu +1Audio-Visual Representation LearningAudio-Visual Synchronization

  9. Commanding at the Speed of Sound: Audio-Centric Referring Expression Segmentation for Aerial Scenes

    Sep 28, 2026Ziyun Chen, Yijun Wang, Chuanyi Zhang +5Referring Expression SegmentationAudio-Visual Understanding

  10. Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection

    Sep 13, 2026Willem Berner, Julio Cesar Cavalcanti, Kalle Åström +1Turn-Taking PredictionAudio-Visual Understanding

  11. A Hybrid Mamba for Audio-Visual Navigation

    Jul 14, 2026Yi Wang, Yinfeng YuMambaEmbodied Navigation

  12. Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking

    Jul 7, 2026Manning Gao, Tingyi Liu, Leheng Zhang +3Representation LearningLearning to Rank

  13. C3C^3ASD: Multi-Level Consistency-Driven Representation Learning

    Jul 3, 2026Jin Hong, Jisoo Park, Junseok KwonMultimodal RobustnessAudio-Visual Representation Learning

  14. Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

    Jun 22, 2026Hugo Malard, Michel Olvera, Sanjeel Parekh +3Multimodal GroundingAudio-Visual Understanding

  15. Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio

    Jun 9, 2026Avi Gupta, Nilotpal Sinha, Vishnu Raj +4Class-Incremental LearningAudio-Visual Representation Learning

  16. M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

    Jun 4, 2026Fei Su, Cancan Li, Ming Li +1Multimodal RobustnessAudio-Visual Representation Learning

  17. Towards multi-modal forgery representation learning for AI-generated video detection and localization

    May 8, 2026Dat Le, Khoa Nguyen, Xin Wang +1Audio-Visual Representation LearningDeepfake Detection

  18. Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning

    May 3, 2026Xinmeng Xu, Haoran Xie, S. Joe Qin +3Audio-Visual UnderstandingAudio-Visual Representation Learning

  19. OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

    May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal LearningVideo Understanding

  20. Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

    Apr 9, 2026Linge Wang, Yingying Chen, Bingke Zhu +2Audio-Visual Representation LearningMultimodal Pretraining

  21. AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection

    Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual UnderstandingAudio-Visual Representation Learning