Overview

Audio-visual alignment studies how to match audio and video in time and meaning, such as linking speech to lip movements, sounds to visible sources, and music to edits. Methods include contrastive learning, cross-attention, optimal transport, and temporal alignment losses for recognition and localization, alongside diffusion and flow-matching models for synchronized dubbing and video-to-audio generation. Researchers evaluate these capabilities on tasks and benchmarks such as LRS3 speech recognition, audio-visual segmentation, and video-to-audio generation. Models are improving on defined tasks, but still struggle with fine-grained synchronization, conflicting cues, and reliable evaluation; stronger visual conditioning can also improve alignment while reducing control over the intended audio.

Latest papers

CardsList

91 papers4 papers in the last four weeks, down 82% on the four weeks before. 0.0% of all new papers.

  1. Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation

    Nov 17, 2026Jihoo Jung, Ji-Hoon Kim, Suyeon Lee +2Audio-Video GenerationAudio-Visual Alignment

  2. Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

    Nov 6, 2026Junbo Zhang, Hang Su, Zhaofan Li +2Audio-Visual UnderstandingMultimodal Semantic Segmentation

  3. CAST-Bench: Tracing Every Sound to Its Source in Multi-Shot Audio-Video Generation

    Nov 2, 2026Yucong Liu, Qi Tian, Shuyuan Tu +3Audio-Video GenerationText-to-Audio-Video Generation Evaluation

  4. Multimodal Concert Video Editing: System Design and Exploratory Evaluation

    Oct 26, 2026Lennart Schiweck, Adrian Kempf, Michelle Retig +3Video EditingAudio-Visual Alignment

  5. AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment

    Oct 19, 2026Mingyang Wu, Kaituo Feng, Bohao Li +3Dense Video CaptioningAudio-Visual Understanding

  6. VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

    Oct 18, 2026Kaidi Wang, Wenhao Guan, Peijie Chen +4Audio-Visual Alignment

  7. JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

    Oct 15, 2026Junseok Ahn, Jaehyun Park, Jeongsoo Choi +2Audio-Video GenerationConversational Agents

  8. EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

    Oct 15, 2026Jiapeng Shi, Qiuxia Hou, Jiaqi Leng +5Video CaptioningAudio-Visual Understanding

  9. Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation

    Oct 11, 2026Chen Zhao, Lehan Shen, Qilong Huang +5Audio-Video GenerationVideo Generation

  10. X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

    Oct 9, 2026Saumyaranjan Mohanty, Aravind Reddy, Konda Reddy MopuriDataset DistillationAudio-Visual Alignment

  11. PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation

    Oct 9, 2026Yabiao Wang, Jiafu Wu, Yicheng Xu +7Video GenerationCo-Speech Gesture Generation

  12. Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

    Oct 8, 2026Shivanshu Shekhar, Raghavendra Addanki, Tong Zhang +1Independent Metropolis-HastingsDiffusion Model Alignment

  13. AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidance

    Oct 8, 2026Junyu Deng, Jiale Cao, Mengtian Li +5Audio-Visual AlignmentCross-Modal Retrieval

  14. Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

    Oct 8, 2026Jia Li, Yinfeng YuAudio-Visual NavigationAudio-Visual Alignment

  15. SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

    Oct 7, 2026Yu Chen, Ruihang Liu, Yangguang Xu +5Audio-Visual UnderstandingSpatial Reasoning Benchmarks

  16. Ariadne's Thread of LipSync: Unraveling Forgeries via Inconsistency between Lip Motions and Head Poses

    Oct 6, 2026Tianyi She, Jiawei Liu, Weifeng Liu +3Digital ForensicsDeepfake Detection

  17. UltraDub: Towards Authentic Dubbing by Unifying Visually-Steered Flow Learning and Trajectory Guidance

    Oct 5, 2026Gaoxiang Cong, Liang Li, Jianwei Wen +3Audio-Video GenerationAudio-Visual Alignment

  18. Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

    Sep 30, 2026Rui Liu, Bhavin Jawade, Haoqi Li +4Audio-Visual SynchronizationAudio-Visual Alignment

  19. Soundwich: Video Generation with Layered and Controllable Audio

    Sep 30, 2026Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek +2Audio-Video GenerationAudio-Visual Synchronization

  20. Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

    Sep 30, 2026Hanmo Chen, Chengcheng Liu, Tianxiao Chen +7Audio-Video GenerationAudio-Visual Alignment

  21. Enabling Immersive Audio-Visual Experience from Any Video

    Sep 28, 2026Zitong Lan, Mutian Tong, Jiatao Gu +1Audio-Video GenerationAudio-Visual Alignment

  22. Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy

    Sep 28, 2026Abhinav Sharma, Sai Karthik Navuluru, Wang Wei +21Audio-Video GenerationMultimodal Generation

  23. SyncRA: Learning Temporal Correspondence in Omni-Modal Models

    Sep 28, 2026Zelong Xu, Yan Li, Wenhe Hu +1Audio-Visual Representation LearningAudio-Visual Synchronization

  24. Uncovering Ordinal-Matching Bias in Audio-Visual LLMs

    Sep 28, 2026Jihoo Jung, Youngjoon Jang, Hyebin Cho +2Audio-Visual UnderstandingMultimodal Large Language Models

  25. Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs

    Sep 25, 2026Jihoo Jung, Youngjoon Jang, Joon Son ChungMultimodal Large Language ModelsAudio-Visual Understanding

  26. AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation

    Sep 24, 2026Zhiyu Xu, Weilong Yan, Yufei Shi +4Audio-Video GenerationAudio-Visual Alignment

  27. The Attention Triangle in Audio-Video Models

    Sep 3, 2026Sagi Polaczek, Noa Kraicer, Gal Metzer +4Audio-Video GenerationAttention Control in Diffusion Models