Overview

The field builds models that generate or edit sound and moving images together, or use one modality to create the other. Common approaches include diffusion transformers and flow-matching models, with text, reference media, event timelines, or motion cues guiding generation; autoregressive versions support streaming and longer clips. Applications range from video soundtracks and music videos to talking avatars, with controls for identity, timing, sound sources, and scene structure. A central challenge is that perceptual quality and basic synchronization do not guarantee semantic or physical correctness, especially across long scenes, so evaluation increasingly tests human judgments, fine-grained control, and physical grounding.

Latest papers

CardsList

166 papers8 papers in the last four weeks, down 67% on the four weeks before. 0.1% of all new papers.

  1. Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation

    Nov 17, 2026Jihoo Jung, Ji-Hoon Kim, Suyeon Lee +2Audio-Video GenerationAudio-Visual Alignment

  2. Découpage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation

    Nov 13, 2026Yifan Wang, Hao Ouyang, Qiuyu Wang +9Audio-Video GenerationMulti-Shot Video Generation

  3. LongCat-Video-Avatar 1.6 Technical Report

    Nov 13, 2026Meituan LongCat Team, Xunliang Cai, Yiqi Chen +8Audio-Video GenerationAudio-Driven Facial Animation

  4. LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

    Nov 4, 2026Yusheng Dai, Vladimir Iashin, Kangdi Wang +6Audio-Video GenerationAudio Diffusion Models

  5. TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation

    Nov 2, 2026Qijun Gan, Chenwei Zhang, Xinyang Li +3Audio-Video GenerationStreaming Video Generation

  6. OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation

    Nov 2, 2026Zihao Fan, Xin Lu, Jie Huang +3Audio-Video GenerationOn-Policy Distillation

  7. CAST-Bench: Tracing Every Sound to Its Source in Multi-Shot Audio-Video Generation

    Nov 2, 2026Yucong Liu, Qi Tian, Shuyuan Tu +3Audio-Video GenerationText-to-Audio-Video Generation Evaluation

  8. CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

    Oct 22, 2026Lifan Jiang, Yibo Zhao, Senlin cheng +6Audio-Video GenerationDiffusion Model Caching

  9. Semantic-Critical Regions for Video-to-Audio Generation

    Oct 16, 2026Yun Fang, Wei Guo, Zhuonan Liang +2Audio GenerationAudio-Video Generation

  10. JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

    Oct 15, 2026Junseok Ahn, Jaehyun Park, Jeongsoo Choi +2Audio-Video GenerationConversational Agents

  11. AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

    Oct 15, 2026Yuruo Wu, Kaihang Pan, Kai Shen +5Video Generation EvaluationAudio-Video Generation

  12. RIGen: Beyond Action Following with Reciprocal Interaction in Generative Gameplay

    Oct 14, 2026Mengfei Li, Yi Zhang, Xingtong Ge +8Real-Time Interactive Video GenerationInteractive World Models

  13. Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation

    Oct 13, 2026Nikolay Semenov, Artem Riabinin, Sergey Bassenko +8Audio-Video GenerationAudio-Driven Facial Animation

  14. SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

    Oct 12, 2026Kaisi Guan, Ying Ba, Wenjing Wang +4Audio-Visual SynchronizationAudio-Video Generation

  15. Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation

    Oct 11, 2026Chen Zhao, Lehan Shen, Qilong Huang +5Audio-Video GenerationVideo Generation

  16. SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

    Oct 8, 2026Aviad Dahan, Rajaei Khatib, Yonatan Bitton +3Audio-Video GenerationAudio Source Separation

  17. WorldSonus: Bringing Sound to Worlds

    Oct 6, 2026Pengjun Fang, Jingyi Fa, Kam Man Wu +9Audio-Video GenerationAudio Diffusion Models

  18. UltraDub: Towards Authentic Dubbing by Unifying Visually-Steered Flow Learning and Trajectory Guidance

    Oct 5, 2026Gaoxiang Cong, Liang Li, Jianwei Wen +3Audio-Video GenerationAudio-Visual Alignment

  19. Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

    Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video GenerationHigh-Resolution Video Generation

  20. Soundwich: Video Generation with Layered and Controllable Audio

    Sep 30, 2026Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek +2Audio-Video GenerationAudio-Visual Synchronization

  21. PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation

    Sep 30, 2026Tiernon Riesenmy, You Zhang, Gautam Bhattacharya +1Text-to-Audio GenerationAudio-Video Generation

  22. Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

    Sep 30, 2026Hanmo Chen, Chengcheng Liu, Tianxiao Chen +7Audio-Video GenerationAudio-Visual Alignment

  23. Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

    Sep 29, 2026Xingtong Ge, Yutong Wang, Lunjie Zhu +7Streaming Video GenerationVideo Diffusion Models

  24. AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes

    Sep 29, 2026Weihan Xu, Kan Jen Cheng, Koichi Saito +8Audio-Video GenerationAudio Editing

  25. Enabling Immersive Audio-Visual Experience from Any Video

    Sep 28, 2026Zitong Lan, Mutian Tong, Jiatao Gu +1Audio-Video GenerationAudio-Visual Alignment

  26. ORAV: Benchmarking Audio-Video Generation from Multimodal Contexts

    Sep 28, 2026Jiacheng Hua, Xiaokun Feng, Jiaqi Hua +3Audio-Video GenerationText-to-Audio-Video Generation Evaluation