cs.CVMar 11, 2026

V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation

Authors: Yan-Bo Lin, Jonah Casebeer, Long Mai, Aniruddha Mahapatra, Gedas Bertasius, Nicholas J. Bryan

Organizations: UNC Chapel Hill · Adobe Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generating music that temporally aligns with video events is challenging for existing text-to-music models, which lack fine-grained temporal control. We introduce V2M-ZERO, a video-to-music generation approach that generates time-aligned music with disentangled time synchronization and semantic control (e.g., genre, mood) from video while requiring zero video-music pairs at training time. Our method is motivated by a key observation: temporal synchronization requires matching when and how much change occurs, not what changes. While musical and visual events differ semantically, they exhibit shared temporal structure that can be captured independently within each modality. We capture this structure through event curves computed from intra-modal similarity using pretrained music and video encoders. By measuring temporal change within each modality independently, these curves provide comparable representations across modalities. This enables a simple training strategy: fine-tune a text-to-music model on music-event curves, then substitute video-event curves at inference without cross-modal training or paired data. Across OES-Pub, MovieGenBench-Music, and AIST++, V2M-ZERO achieves state-of-the-art performance without any paired music-video data, surpassing the strongest prior baselines per metric with 5-9% higher audio quality, 13-15% better semantic alignment, 21-52% improved temporal synchronization, and 28% higher beat alignment on dance videos. We find similar results via a large crowd-source subjective listening test. Our results validate that temporal alignment through within-modality features is not only effective for video-to-music generation but also leads to better performance than paired cross-modal supervision. Furthermore, our approach enables independent controls for timing and music style (e.g., genre, mood) for more controllable generation.

Figures & tables

Figure 1: Why zero-pair video-to-music generation is possible. V2M-Zero is trained with music-event curves and, at inference, replaces them with video-event curves. This transfer is enabled by the shared temporal structure of event curves across music and video.
Figure 2: Method overview. Top : During training, V2M-Zero learns a rectified-flow diffusion process conditioned on text prompts and a music-event curve derived from intra-music similarity. Bottom : At inference, music conditioning is swapped with a video-event curve based on framewise similarity, enabling zero-pair, time-synchronized video-to-music generation. For semantic alignment, a text prompt is predicted from video and speech via Vibe [ 27 ] , which first summarizes video content and recommends a matching music text prompt.
   OES-Pub.MovieGenBench-Music
 Audio QualityHigh-Level AlignmentLow-Level MatchingAudio QualityHigh-Level AlignmentLow-Level Matching
MethodV–M PairsFAD* ↓\downarrowCLAP ↑\uparrowSCH ↑\uparrowFAD ↓\downarrowCLAP ↑\uparrowSCH ↑\uparrow
M 2 UGen [ 56 ]✔36.7h6.670.070.355.840.020.24
GVMGen [ 118 ]✔147h6.250.180.353.960.060.48
MTCV2M [ 101 ]✔147h5.440.200.374.020.160.29
VidMuse [ 92 ]✔18000h10.40.160.402.980.040.47
Table 1: Comparison with state of the art. We evaluate all methods [ 56 , 118 , 92 , 91 , 101 , 110 ] on OES-Pub [ 37 ] and MovieGenBench-Music [ 72 ] in audio quality (FAD), high-level semantic alignment (CLAP), and our proposed temporal metric (SCH). V2M-Zero outperforms all competing methods across all datasets and metrics. Note, OES-Pub reference data includes non-musical sounds, so we use SongDescriber [ 61 ] for the reference distribution for FAD results.
Table 2: Human evaluation. Pairwise win rates of our model against each baseline (1403 ratings, Bonferroni-corrected tt -tests, p<0.0083p<0.0083 ). † above chance but not significant after correction. Scene cut subset comprises 67% of all votes.
Figure 3: Smoothing kernel size sweep on OES-Pub with MusicFM and DINOv2: audio quality (FAD*) improves with larger kernels; synchronization (SCH) peaks at intermediate sizes. We hyperparameter tune the kernel size and apply a standard (31) across all other datasets and encoders.
Table 3: Generalization across video types and model implementations. ( a ) Eval. on [ 95 , 45 ] using beat consistency (BCS, BHS, F1) and temporal deviation (TD). ( b ) V2M-Zero on a public text-to-music model evaluated on OES-Pub.
Table 4: (a) Encoder selection for event curves. Impact of music (training) and visual (inference) encoders on OES-Pub. MusicFM + DINOv2 achieves the best balance. (b) LLM selection for music prompts. Impact of LLM choice for music prompt generation on OES-Pub. V2M-Zero uses Vibe [ 27 ] based on Gemma-4B [ 89 ] .
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

MethodFAD ↓\downarrowCLAP ↑\uparrowSCH ↑\uparrow
Text-only3.630.230.35
V2M-Zero4.950.230.61
    
Appendix
Table 5: Comparison with a text-only baseline. We compare V2M-Zero against the text-to-music model used for our finetuning, without event-curve conditioning, on OES-Pub.
MethodFAD* ↓\downarrowCLAP ↑\uparrowSCH ↑\uparrow
HunyuanVideo-Foley [ 78 ]15.020.120.36
V2M-Zero4.950.230.61
Appendix
Table 6: Comparison with a large-scale open-source model. We compare V2M-Zero with HunyuanVideo-Foley [ 78 ] on OES-Pub.
 OES-PubMovieGenBench-MusicHuman Eval
MethodMM+VM-VM∣V\textbf{M}\mid\textbf{V}MM+VM-VM∣V\textbf{M}\mid\textbf{V}Temporal Align.
M 2 UGen [ 56 ]6.296.9618.261.331.961.9924.761.52Ours wins ∗
GVMGen [ 118 ]3.454.0410.100.481.311.3420.091.02Ours wins ∗
MTCV2M [ 101 ]4.334.5310.900.921.531.7521.030.98Ours wins ∗
VidMuse [ 92 ]3.253.8612.400.794.274.2940.894.02Ours wins
AudioX [ 91 ]4.134.631.751.931.521.5419.701.18Ours wins ∗
Appendix
Table 7: Event-curve Fréchet distance comparison. M evaluates generated vs. ground-truth music event curves. M+V compares concatenated music–video event curves. M-V compares the distribution of generated music and ground-truth video. M ∣\mid V evaluates conditional music event distributions given video. We additionally report human preference for temporal alignment. ∗ denotes statistical significance.
Video-event curve designFAD* ↓\downarrowCLAP ↑\uparrowSCH ↑\uparrow
No video-event conditioning4.860.180.28
60 frames (2 s)3.760.150.29
15 frames (500 ms)3.650.150.35
Multiple Offsets (1, 15, 60)3.840.160.33
1 frame (33 ms, default)4.130.170.38
Appendix
Table 8: Video-event curves across different temporal scales. We study video-event curves computed from different frame offsets in the similarity calculation using Stable-Audio-ControlNet [ 14 ] on OES-Pub.
Figure 4: Example event curves with different temporal dynamics. The blue solid curve corresponds to a video with frequent scene cuts, while the orange curve corresponds to a video with slower visual motion, showing distinct temporal structures. This supports our design choice of using event curves to represent relative timing, while text provides complementary semantic guidance.
More in this topic

Video-to-Music Generation

See all 10 papers

Explore similar work

CardsList
Apr 19, 2026

Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation

Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic and stylistic controllability to the end user. In this paper, we present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content. To balance musical fidelity and semantic understanding, Video-Robin integrates autoregressive planning with diffusion-based synthesis. Specifically, an autoregressive module models global structure by semantically aligning visual and textual inputs to produce high-level music latents. These latents are subsequently refined into coherent, high-fidelity music using local Diffusion Transformers. By factoring semantically driven planning into diffusion-based synthesis, Video-Robin enables fine-grained creator control without sacrificing audio realism. Our proposed model outperforms baselines that solely accept video input and additional feature conditioned baselines on both in-distribution and out-of-distribution benchmarks with a 2.21x speed in inference compared to SOTA. We will open-source everything upon paper acceptance.

Vaibhavi Lokegaonkar, Aryan Vijay Bhosale, Vishnu Raj +5

University of Maryland College Park, USA · Dolby Laboratories, USA · NVIDIA, USA

Controllable Music GenerationAudio Diffusion ModelsVideo-to-Music Generation
AskPDF
Aug 31, 2026

VIBE: Video Instruction-aligned Background music gEneration

Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment.

Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj +5

University of Maryland, College Park, USA · Dolby Laboratories, USA

Controllable Music GenerationMusic GenerationVideo-to-Music Generation
AskPDF
Aug 12, 2026

Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models' video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.

Haven Kim, Zachary Novack, Julian McAuley +1

University of California San Diego · University of Michigan

Audio-Video GenerationMusic GenerationVideo-to-Music Generation
AskPDF