Generating music that temporally aligns with video events is challenging for existing text-to-music models, which lack fine-grained temporal control. We introduce V2M-ZERO, a video-to-music generation approach that generates time-aligned music with disentangled time synchronization and semantic control (e.g., genre, mood) from video while requiring zero video-music pairs at training time. Our method is motivated by a key observation: temporal synchronization requires matching when and how much change occurs, not what changes. While musical and visual events differ semantically, they exhibit shared temporal structure that can be captured independently within each modality. We capture this structure through event curves computed from intra-modal similarity using pretrained music and video encoders. By measuring temporal change within each modality independently, these curves provide comparable representations across modalities. This enables a simple training strategy: fine-tune a text-to-music model on music-event curves, then substitute video-event curves at inference without cross-modal training or paired data. Across OES-Pub, MovieGenBench-Music, and AIST++, V2M-ZERO achieves state-of-the-art performance without any paired music-video data, surpassing the strongest prior baselines per metric with 5-9% higher audio quality, 13-15% better semantic alignment, 21-52% improved temporal synchronization, and 28% higher beat alignment on dance videos. We find similar results via a large crowd-source subjective listening test. Our results validate that temporal alignment through within-modality features is not only effective for video-to-music generation but also leads to better performance than paired cross-modal supervision. Furthermore, our approach enables independent controls for timing and music style (e.g., genre, mood) for more controllable generation.
Figures & tables
Figure 1: Why zero-pair video-to-music generation is possible. V2M-Zero is trained with music-event curves and, at inference, replaces them with video-event curves. This transfer is enabled by the shared temporal structure of event curves across music and video.
Figure 2: Method overview. Top : During training, V2M-Zero learns a rectified-flow diffusion process conditioned on text prompts and a music-event curve derived from intra-music similarity. Bottom : At inference, music conditioning is swapped with a video-event curve based on framewise similarity, enabling zero-pair, time-synchronized video-to-music generation. For semantic alignment, a text prompt is predicted from video and speech via Vibe [ 27 ] , which first summarizes video content and recommends a matching music text prompt.
OES-Pub.
MovieGenBench-Music
Audio Quality
High-Level Alignment
Low-Level Matching
Audio Quality
High-Level Alignment
Low-Level Matching
Method
V–M Pairs
FAD* ↓
CLAP ↑
SCH ↑
FAD ↓
CLAP ↑
SCH ↑
M 2 UGen [ 56 ]
✔
36.7h
6.67
0.07
0.35
5.84
0.02
0.24
GVMGen [ 118 ]
✔
147h
6.25
0.18
0.35
3.96
0.06
0.48
MTCV2M [ 101 ]
✔
147h
5.44
0.20
0.37
4.02
0.16
0.29
VidMuse [ 92 ]
✔
18000h
10.4
0.16
0.40
2.98
0.04
0.47
Table 1: Comparison with state of the art. We evaluate all methods [ 56 , 118 , 92 , 91 , 101 , 110 ] on OES-Pub [ 37 ] and MovieGenBench-Music [ 72 ] in audio quality (FAD), high-level semantic alignment (CLAP), and our proposed temporal metric (SCH). V2M-Zero outperforms all competing methods across all datasets and metrics. Note, OES-Pub reference data includes non-musical sounds, so we use SongDescriber [ 61 ] for the reference distribution for FAD results.
Table 2: Human evaluation. Pairwise win rates of our model against each baseline (1403 ratings, Bonferroni-corrected t -tests, p<0.0083 ). † above chance but not significant after correction. Scene cut subset comprises 67% of all votes.
Figure 3: Smoothing kernel size sweep on OES-Pub with MusicFM and DINOv2: audio quality (FAD*) improves with larger kernels; synchronization (SCH) peaks at intermediate sizes. We hyperparameter tune the kernel size and apply a standard (31) across all other datasets and encoders.
Table 3: Generalization across video types and model implementations. ( a ) Eval. on [ 95 , 45 ] using beat consistency (BCS, BHS, F1) and temporal deviation (TD). ( b ) V2M-Zero on a public text-to-music model evaluated on OES-Pub.
Table 4: (a) Encoder selection for event curves. Impact of music (training) and visual (inference) encoders on OES-Pub. MusicFM + DINOv2 achieves the best balance. (b) LLM selection for music prompts. Impact of LLM choice for music prompt generation on OES-Pub. V2M-Zero uses Vibe [ 27 ] based on Gemma-4B [ 89 ] .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
FAD ↓
CLAP ↑
SCH ↑
Text-only
3.63
0.23
0.35
V2M-Zero
4.95
0.23
0.61
Appendix
Table 5: Comparison with a text-only baseline. We compare V2M-Zero against the text-to-music model used for our finetuning, without event-curve conditioning, on OES-Pub.
Method
FAD* ↓
CLAP ↑
SCH ↑
HunyuanVideo-Foley [ 78 ]
15.02
0.12
0.36
V2M-Zero
4.95
0.23
0.61
Appendix
Table 6: Comparison with a large-scale open-source model. We compare V2M-Zero with HunyuanVideo-Foley [ 78 ] on OES-Pub.
OES-Pub
MovieGenBench-Music
Human Eval
Method
M
M+V
M-V
M∣V
M
M+V
M-V
M∣V
Temporal Align.
M 2 UGen [ 56 ]
6.29
6.96
18.26
1.33
1.96
1.99
24.76
1.52
Ours wins ∗
GVMGen [ 118 ]
3.45
4.04
10.10
0.48
1.31
1.34
20.09
1.02
Ours wins ∗
MTCV2M [ 101 ]
4.33
4.53
10.90
0.92
1.53
1.75
21.03
0.98
Ours wins ∗
VidMuse [ 92 ]
3.25
3.86
12.40
0.79
4.27
4.29
40.89
4.02
Ours wins
AudioX [ 91 ]
4.13
4.63
1.75
1.93
1.52
1.54
19.70
1.18
Ours wins ∗
Appendix
Table 7: Event-curve Fréchet distance comparison. M evaluates generated vs. ground-truth music event curves. M+V compares concatenated music–video event curves. M-V compares the distribution of generated music and ground-truth video. M ∣ V evaluates conditional music event distributions given video. We additionally report human preference for temporal alignment. ∗ denotes statistical significance.
Video-event curve design
FAD* ↓
CLAP ↑
SCH ↑
No video-event conditioning
4.86
0.18
0.28
60 frames (2 s)
3.76
0.15
0.29
15 frames (500 ms)
3.65
0.15
0.35
Multiple Offsets (1, 15, 60)
3.84
0.16
0.33
1 frame (33 ms, default)
4.13
0.17
0.38
Appendix
Table 8: Video-event curves across different temporal scales. We study video-event curves computed from different frame offsets in the similarity calculation using Stable-Audio-ControlNet [ 14 ] on OES-Pub.
Figure 4: Example event curves with different temporal dynamics. The blue solid curve corresponds to a video with frequent scene cuts, while the orange curve corresponds to a video with slower visual motion, showing distinct temporal structures. This supports our design choice of using event curves to represent relative timing, while text provides complementary semantic guidance.