Audio-Visual Alignment

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 57

All topics

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision
Jun 12, 2026Shiyao Wang, Xijuan Zeng, Hui Wang +4Audio-Video Generation Audio-Visual Synchronization
Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China · Kling Team, Kuaishou Technology, Beijing, China
Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization
Jun 5, 2026Zhe Yang, Ruyi Zhang, Hongtao Chen +4Audio-Visual Understanding Heterogeneous GNNs
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China
Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation
Jun 2, 2026Ye Tao, Lupeng Liu, Xuenan Xu +6Audio-Video Generation Unified Multimodal Models
School of Intelligence Science and Technology, Nanjing University · Video Rebirth · Shanghai Jiao Tong University +2
Inference-Time Scaling for Joint Audio-Video Generation
Jun 2, 2026Jaemin Jung, Kyeongha Rho, Inkyu Shin +1Audio-Video Generation Inference-Time Scaling
Korea Advanced Institute of Science and Technology · Luma AI
JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
Jun 1, 2026Jiashuo Yu, Yao Yao, Boyu Chen +1Audio-Video Generation Audio-Visual Alignment
Jen Music AI
SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
May 31, 2026Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1Audio-Visual Understanding Multimodal Large Language Models
KAIST
Benchmarking Single-Factor Physical Video-to-Audio Generation
May 28, 2026Tingle Li, Siddharth Gururani, Kevin J. Shih +6Audio-Video Generation Audio-Visual Alignment
UC Berkeley · NVIDIA · University of Washington
Native Audio-Visual Alignment for Generation
May 28, 2026Longbin Ji, Guan Wang, Xuan Wei +6Audio-Video Generation Audio-Visual Synchronization
ERNIE Team, Baidu Inc.
BEAT: Rhythm-Elastic Alignment for Agentic Music-guided Movie Trailer Generation
May 26, 2026Yutong Wang, Yunke Wang, Xinyuan Chen +1Audio-Video Generation Video Generation
The University of Sydney · Shanghai AI Laboratory
Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
May 24, 2026Shuyuan Tu, Qi Tian, Zihan Yang +9Cross-Modal Learning Audio-Video Generation
Fudan University · Tencent Hunyuan
SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing
May 24, 2026Sen Liang, Cong Wang, Fengbin Guan +6Video Editing Audio-Visual Synchronization
University of Science and Technology of China · 2Tencent Hunyuan
AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
May 23, 2026Jialiang Yang, Bin Xia, Ruihang Chu +6Audio-Video Generation Audio-Visual Alignment
Tsinghua University · The Chinese University of Hong Kong
MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation
May 19, 2026Yujie Wei, Yujin Han, Zhekai Chen +20Audio-Video Generation Audio-Visual Alignment
Fudan University · The University of Hong Kong · Tongyi Lab, Alibaba Group +2
When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
May 12, 2026Guohui Zhang, XiaoXiao Ma, Jie Huang +9Audio-Video Generation Audio-Visual Synchronization
University of Science and Technology of China · JD Explore Academy · Peking University
LightAVSeg: Lightweight Audio-Visual Segmentation
May 9, 2026Qing Zhong, Guodong Ding, Lingqiao Liu +3Audio-Visual Understanding Audio-Visual Alignment
College of Informatics, Huazhong Agricultural University, Wuhan, China · School of Computing, National University of Singapore, Singapore · School of Computer Science, Adelaide University, Australia +2
Audio-Visual Intelligence in Large Foundation Models
May 5, 2026You Qin, Kai Liu, Shengqiong Wu +12Audio-Visual Understanding Audio-Video Generation
National University of Singapore · University of Oxford · University of Toronto +5
MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video
May 1, 2026Kazuya Tateishi, Akira Takahashi, Atsuo Hiroe +3Audio-Video Generation Sound Event Detection
Sony Group Corporation, Japan · Sony AI, USA
Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
Apr 26, 2026Chunyu Li, Jiaye Li, Ruiqiao Mei +4Audio-Video Generation Video Diffusion Models
Shanghai Innovation Institute · Fudan University · University of Science and Technology of China +2
BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios
Apr 24, 2026Advait Tilak, Jiwon Choi, Nazifa Mouli +1Text-to-Video Generation Audio-Visual Alignment
Department of Computer Science, Iowa State University, Ames, Iowa, USA
Hierarchical Codec Diffusion for Video-to-Speech Generation
Apr 17, 2026Jiaxin Ye, Gaoxiang Cong, Chenhui Wang +4Audio-Video Generation Diffusion Transformer
Fudan University · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +1
ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
Apr 16, 2026Jianxuan Yang, Xinyue Guo, Zhi Cheng +10Audio-Video Generation Multimodal Generation
MiLM Plus, Xiaomi Inc. · Wuhan University
AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection
Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual Understanding Audio-Visual Representation Learning
Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK
Conditional Flow Matching for Visually-Guided Acoustic Highlighting
Feb 3, 2026Hugo Malard, Gael Le Lan, Daniel Wong +3Conditional Flow Matching Audio-Visual Alignment
LTCI, Télécom Paris, Institut Polytechnique de Paris · Meta
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
Nov 23, 2025Kaidi Wang, Yi He, Wenhao Guan +9TTS Synthesis Audio-Visual Synchronization
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation
Date pendingYuchen Sun, Qian Yang, Jun Wang +4Audio-Video Generation Audio-Visual Synchronization
Meituan Beijing, China

Jun 12, 2026, cs.SDJ/K move · Enter open · S save

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

Shiyao Wang, Xijuan Zeng, Hui Wang, Shiwan Zhao+3

Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China · Kling Team, Kuaishou Technology, Beijing, China

Audio-Video Generation Audio-Visual Synchronization Sound Effects Generation Audio-Visual Alignment Audio Generation

Open paper AskPDF arXiv

We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.

Momentum

Related topics

Latest papers 57