Video-to-Music Generation

Follow this topic Ask about this topic

Latest papers

10 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Self-Representation Alignment in Diffusion Models for Video Background Music Generation
Oct 11, 2026Jincheng Zhang, György Fazekas, Charalampos SaitisVideo-to-Music Generation Diffusion Models
Centre for Digital Music, Queen Mary University of London, UK
Video-to-Music Generation for Gameplay Videos
Sep 25, 2026Felipe Marra, Lucas N. FerreiraAudio-Video Generation Music Generation
Universidade Federal de Viçosa Departamento de Informática Viçosa, Minas Gerais, Brazil · Universidade Federal de Minas Gerais Departamento de Ciência da Computação Belo Horizonte, Minas Gerais, Brazil
MVWeaver: A Hierarchical Music Video Generation Agent with a Learned Song-to-Visual Bridge
Sep 6, 2026Sifei Li, Minyan Luo, Xu Li +7Video-to-Music Generation Audio-Video Generation
Institute of Automation, Chinese Academy of Sciences, China, School of Artificial Intelligence, University of Chinese Academy of Sciences, China, and KlingAI Research, China · Institute of Automation, Chinese Academy of Sciences, China and School of Artificial Intelligence, University of Chinese Academy of Sciences, China · KlingAI Research, China +3
VIBE: Video Instruction-aligned Background music gEneration
Aug 31, 2026Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj +5Controllable Music Generation Music Generation
University of Maryland, College Park, USA · Dolby Laboratories, USA
Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections
Aug 12, 2026Haven Kim, Zachary Novack, Julian McAuley +1Audio-Video Generation Music Generation
University of California San Diego · University of Michigan
Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation
Jun 2, 2026Ye Tao, Lupeng Liu, Xuenan Xu +6Audio-Video Generation Unified Multimodal Models
School of Intelligence Science and Technology, Nanjing University · Video Rebirth · Shanghai Jiao Tong University +2
JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
Jun 1, 2026Jiashuo Yu, Yao Yao, Boyu Chen +1Audio-Video Generation Audio-Visual Alignment
Jen Music AI
Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation
Apr 19, 2026Vaibhavi Lokegaonkar, Aryan Vijay Bhosale, Vishnu Raj +5Controllable Music Generation Audio Diffusion Models
University of Maryland College Park, USA · Dolby Laboratories, USA · NVIDIA, USA
V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation
Mar 11, 2026Yan-Bo Lin, Jonah Casebeer, Long Mai +3Video-to-Music Generation Audio-Visual Synchronization
UNC Chapel Hill · Adobe Research
Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment
Feb 19, 2026Ivan Rinaldi, Matteo Mendula, Nicola Fanelli +4Video-to-Music Generation Music Generation
University of Bari Aldo Moro, Bari, Italy · Catalonia’s Telecommunications Technology Centre, Barcelona, Spain · University of Picardie Jules Verne, Amiens, France +1

Oct 11, 2026J/K move · Enter open · S save

Self-Representation Alignment in Diffusion Models for Video Background Music Generation

Jincheng Zhang, György Fazekas, Charalampos Saitis

Centre for Digital Music, Queen Mary University of London, UK

Video-to-Music Generation Diffusion Models Audio Diffusion Models Symbolic Music Generation

Open paper AskPDF arXiv

Diffusion models have been widely used for various generative tasks. However, recent studies have demonstrated that the internal representations learned in the diffusion models' denoising process are insufficiently meaningful, which can hinder the generation performance. To address this, we propose a novel diffusion U-Net architecture enhanced by explicitly aligning the weaker representation from the earlier layer conditioned on higher noise with the better representation from the later layer conditioned on lower noise during training, which we use for the challenging multimodal task of video background music generation. Our diffusion U-Net Self-Representation Alignment (U-SRA) employs an alignment loss to effectively guide our diffusion model to learn good internal representations, thereby enabling it to produce higher-quality symbolic music conditioned on video features. We employ a cross-attention mechanism to synchronize music with video segments. Experiments on a benchmark video-music dataset demonstrate that the proposed U-SRA approach for video background music generation surpasses the state-of-the-art across key evaluation metrics, including video-music alignment and overall music quality.