Audio-Video Generation

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 101

All topics

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
Jun 1, 2026Jiashuo Yu, Yao Yao, Boyu Chen +1Audio-Video Generation Audio-Visual Alignment
Jen Music AI
Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs
Jun 1, 2026Sicheng Xu, Yu Deng, Shoukang Hu +5Variational Autoencoders Audio-Video Generation
1Microsoft Research · 2Microsoft AI
Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer
May 29, 2026Ke Lei, Yu Zhang, Changhao Pan +4Text-to-Audio Generation Audio-Video Generation
1Zhejiang University, China · 2ByteDance
Benchmarking Single-Factor Physical Video-to-Audio Generation
May 28, 2026Tingle Li, Siddharth Gururani, Kevin J. Shih +6Audio-Video Generation Audio-Visual Alignment
UC Berkeley · NVIDIA · University of Washington
Native Audio-Visual Alignment for Generation
May 28, 2026Longbin Ji, Guan Wang, Xuan Wei +6Audio-Video Generation Audio-Visual Synchronization
ERNIE Team, Baidu Inc.
CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation
May 28, 2026Xuangeng Chu, Yuan Gan, Ziteng Cui +4Audio-Video Generation Talking Head Generation
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP
MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation
May 27, 2026Haitian Li, Yanghao Zhou, Heyan Huang +15Audio-Video Generation Multimodal Model Evaluation
Shanghai University · Tsinghua University · Beijing Institute of Technology +7
BEAT: Rhythm-Elastic Alignment for Agentic Music-guided Movie Trailer Generation
May 26, 2026Yutong Wang, Yunke Wang, Xinyuan Chen +1Audio-Video Generation Video Generation
The University of Sydney · Shanghai AI Laboratory
LongCat-Video-Avatar 1.5 Technical Report
May 26, 2026Meituan LongCat Team, Xunliang Cai, Meng Cheng +10Audio-Video Generation Long Video Generation
Meituan LongCat Team
LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV
May 25, 2026Tengfei Liu, Yang Shi, Xuanyu Zhu +17Audio-Video Generation Long-Horizon Video Generation
Peking University · Kling Team · Nanjing University +6
StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
May 25, 2026Linrui Tian, Qi Wang, Bang ZhangAudio-Video Generation Long-Horizon Video Generation
Tongyi Lab, Alibaba Group
Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
May 24, 2026Shuyuan Tu, Qi Tian, Zihan Yang +9Cross-Modal Learning Audio-Video Generation
Fudan University · Tencent Hunyuan
AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
May 23, 2026Jialiang Yang, Bin Xia, Ruihang Chu +6Audio-Video Generation Audio-Visual Alignment
Tsinghua University · The Chinese University of Hong Kong
MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation
May 19, 2026Yujie Wei, Yujin Han, Zhekai Chen +20Audio-Video Generation Audio-Visual Alignment
Fudan University · The University of Hong Kong · Tongyi Lab, Alibaba Group +2
WavFlow: Audio Generation in Waveform Space
May 18, 2026Feiyan Zhou, Luyuan Wang, Shoufa Chen +6Text-to-Audio Generation Audio-Video Generation
Meta AI · Northeastern University
InstructAV2AV: Instruction-Guided Audio-Video Joint Editing
May 18, 2026Haojie Zheng, Yixin Yang, Siqi Yang +2Audio-Video Generation Video Diffusion Models
Beijing Academy of Artificial Intelligence, Beijing, China · Peking University, Beijing, China
CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation
May 18, 2026Gyubin Lee, Junwon Lee, Juhan NamText-to-Audio Generation Audio-Video Generation
Kim Jaechul Graduate School of AI, KAIST · 2Graduate School of Cultural Technology, KAIST
Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation
May 17, 2026Yuheng Chen, Qingdong He, Teng Hu +4Audio-Video Generation Audio-Visual Synchronization
Shanghai Jiao Tong University · University of Electronic Science and Technology of China · Zhejiang University
Sound Sparks Motion: Audio and Text Tuning for Video Editing
May 14, 2026AmirHossein Naghi Razlighi, Aryan Mikaeili, Ali Mahdavi-Amiri +2Audio-Video Generation Inference-Time Optimization
University of Cyprus, Cyprus · Simon Fraser University, Canada · Tel Aviv University, Israel +1
OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
May 12, 2026Guohui Zhang, XiaoXiao Ma, Jie Huang +9Audio-Video Generation Audio-Visual Synchronization
University of Science and Technology of China · JD Explore Academy · Peking University
SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning
May 12, 2026Xin Cheng, Xihua Wang, Ying Ba +5Audio-Video Generation Direct Preference Optimization
Renmin University of China · Westlake University
AllocMV: Optimal Resource Allocation for Music Video Generation via Structured Persistent State
May 11, 2026Huimin Wang, Leilei Ouyang, Chang Xia +3Audio-Video Generation Long-Horizon Video Generation
College of Computer Science, Sichuan University.
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
May 9, 2026Shihao Cheng, Jiaxu Zhang, Quanyue Song +6Audio-Video Generation Audio-Visual Synchronization
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China · ByteDance, China · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China +2
Do Joint Audio-Video Generation Models Understand Physics?
May 8, 2026Zijun Cui, Xiulong Liu, Hao Fang +8Physical Consistency in Video Generation Audio-Video Generation
University of Texas at Dallas · University of Washington · University of California, Los Angeles
Audio-Visual Intelligence in Large Foundation Models
May 5, 2026You Qin, Kai Liu, Shengqiong Wu +12Audio-Visual Understanding Audio-Video Generation
National University of Singapore · University of Oxford · University of Toronto +5
MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video
May 1, 2026Kazuya Tateishi, Akira Takahashi, Atsuo Hiroe +3Audio-Video Generation Sound Event Detection
Sony Group Corporation, Japan · Sony AI, USA
Generate Your Talking Avatar from Video Reference
Apr 30, 2026Zujin Guo, Zhenhui Ye, Yi Ren +4Video-Reference Video Generation Audio-Video Generation
Nanyang Technological University · HeyGen Research · University of Melbourne
Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation
Apr 28, 2026Yupeng Zhou, Lianghua Huang, Zhifan Wu +7Audio-Video Generation Long-Horizon Video Generation
1VCIP, School of Computer Science, Nankai University · 2Tongyi Lab · 2Tongyi Lab,3Peking University +1
Talking Slide Avatars: Open-Source Multimodal Communication Approach for Teaching
Apr 26, 2026Xinxing WuAudio-Video Generation Educational Technology
School of Mathematics and Computer Science, Kentucky State University, Frankfort, Kentucky, United States
Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
Apr 26, 2026Chunyu Li, Jiaye Li, Ruiqiao Mei +4Audio-Video Generation Video Diffusion Models
Shanghai Innovation Institute · Fudan University · University of Science and Technology of China +2

Jun 1, 2026, cs.SDJ/K move · Enter open · S save

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang

Jen Music AI

Audio-Video Generation Audio-Visual Alignment Audio Generation

Open paper AskPDF arXiv

We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions. Existing AI music systems are mainly designed for short, isolated clips and lack mechanisms to ensure narrative continuity. We present JenBridge, a modular and interpretable framework for adaptive long-form video soundtracking that ensures both high-fidelity audio generation and transition naturalness. The core architecture is a Transformer-based generative model trained with a flow-matching objective, following a two-stage paradigm: pretraining on large-scale text-audio corpora to establish robust musical priors, then adapting to the video domain with dual text-visual conditioning for precise cross-modal alignment. Crucially, to achieve long-form coherence across diverse scene changes, JenBridge incorporates a novel adaptive transition mechanism. This system features a versatile toolkit of transition styles, including a generative transition method, and uniquely employs a Large Language Model (LLM) Agent that acts as a director to select the most appropriate transition for each narrative shift intelligently. To rigorously assess this task, we propose the LVS Benchmark, a new benchmark that includes a curated dataset and novel evaluation metrics focusing on holistic and transition-aware assessment. Extensive experiments on the proposed benchmark demonstrate that JenBridge significantly outperforms existing methods in both objective and subjective metrics, particularly in terms of transition naturalness and overall narrative coherence. JenBridge represents a significant step towards fully automated, professional-quality video soundtracking.

Momentum

Related topics

Latest papers 101