Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typically assess diverse audio types under a unified protocol, overlooking the fine-grained requirements of distinct audio categories. To address this gap, we propose VidAudio-Bench, a multi-task benchmark for V2A evaluation with four key features: (1) Broad Coverage: It encompasses four representative audio categories - sound effects, music, speech, and singing - under both V2A and Video-Text-to-Audio (VT2A) settings. (2) Extensive Evaluation: It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models. (3) Comprehensive Metrics: It introduces 13 task-specific, reference-free metrics to systematically assess audio quality, video-audio consistency, and text-audio consistency. (4) Human Alignment: It validates all metrics through subjective studies, demonstrating strong consistency with human preferences. Experimental results reveal that current V2A models perform poorly in speech and singing compared to sound effects. Our VT2A results further highlight a fundamental tension between instruction following and visually grounded generation: stronger visual conditioning improves video-audio alignment, but often at the cost of generating the intended audio category. These findings establish VidAudio-Bench as a comprehensive and scalable framework for diagnosing V2A systems and provide new insights into multimodal audio generation.
Figures & tables
Figure 1. Overview of VidAudio-Bench. We categorize audio generation into four task types: sound effects , music , speech , and singing . We further introduce two input paradigms, V2A and VT2A, to analyze how adding textual descriptions changes generation behavior. Our evaluation suite spans Audio Quality, Video-Audio Consistency, and Text-Audio Consistency, covering 13 fine-grained dimensions. Human preference studies show strong correlation between our metrics and human perception.
Figure 2. Data distribution of the SFX subset. The inner ring denotes high-level categories, and the outer bars show the number of sound events in each subcategory.
Figure 3. (a) Distribution of categories in the Instrument Performance subset. (b) Gender and age group distributions in the Speech and Singing subsets.
Figure 4. Overview of the evaluation framework in VidAudio-Bench. For each audio generation task, we define a set of evaluation dimensions across Audio quality, Video-Audio Consistency, and Text-Audio Consistency. An MLLM-as-a-Judge framework is employed to assess the generated audio through multi-step reasoning based on the input video and audio.
Dimensions and Tasks
Fidelity ↓
Aesthetic ↑
Intelligibility ↑
Models
SFX
Music-I
Music-B
Speech
Singing
SFX
Music-I
Music-B
Speech
Singing
AudioX ( Tian et al., 2025b )
5.480 /9.675
10.533 /7.670
25.297 /19.726
7.300 / 5.607
13.787 / 6.217
4.587 /4.785
6.081 /6.133
4.485 /5.503
0.684 /0.626
0.613 /0.593
FoleyCrafter ( Zhang et al., 2026 )
11.847 /16.417
17.537 /24.443
20.100 /37.038
19.362 /28.316
22.894 /27.529
4.560 /4.951
5.821 /5.892
5.207 /4.369
0.552 /0.556
0.542 /0.528
HunyuanVideo-Foley ( Shan et al., 2025 )
8.205 /12.240
14.369 /7.721
10.187 /22.904
6.796 /21.030
58.529 /17.475
5.080 / 5.060
6.189 /6.000
6.659 /5.385
0.583 /0.572
0.557 /0.564
Kling-Foley ( Wang et al., 2025a )
9.090 /9.529
18.585 /17.114
19.767 / 19.150
35.304 /35.005
12.981 /13.118
4.855 /4.915
5.957 /6.176
6.035 / 6.006
0.554 /0.553
0.552 /0.553
MMAudio ( Cheng et al., 2025 )
13.499 /12.347
17.952 /16.763
24.060 /31.994
14.402 /13.279
20.109 /16.004
4.685 /4.740
6.161 /6.002
6.094 /4.937
0.548 /0.534
0.483 /0.468
Table 1. Evaluation results on VidAudio-Bench across multiple tasks and dimensions. Results are presented in the format of V2A / VT2A , where ↑ ( ↓ ) indicates higher (lower) values represent better performance. Best results in each category are highlighted in bold . Music-I : Instrument Performance; Music-B : Background Music.
Figure 5. Task-wise radar plots of eight representative models across four audio generation tasks using V2A results: (a) sound effects , (b) music , (c) speech , and (d) singing . Each subplot summarizes model performance over the task-specific evaluation dimensions.
Models
Fidelity ↓
Aesthetic ↑
Musicality ↑
Semantic-Corr ↑
Rhy-Sync ↑
GVMGen ( Zuo et al., 2025 )
12.171
6.598
0.746
0.100
0.151
SONIQUE ( Zhang and Fuentes, 2025 )
34.807
5.767
0.638
0.113
0.093
VidMuse ( Tian et al., 2025c )
10.960
7.080
0.717
0.141
0.128
Table 2. Evaluation results of three specialized BGM generation models on the BGM task across five dimensions.
Figure 6. Human preference correlation with VidAudio-Bench. This figure shows the Pearson correlation coefficients ( ρ ) between VidAudio-Bench scores (x-axis) and human win rates (y-axis) across various evaluation dimensions. High correlations demonstrate strong alignment with human perceptual judgments.
Category
Accuracy ↑
Precision ↑
Recall ↑
F1-score ↑
SFX
0.8625
0.8591
0.9922
0.9209
Music
0.8000
0.7253
0.9041
0.8042
Speech
0.8875
0.8855
0.9748
0.9280
Singing
0.8438
0.8000
0.9268
0.8588
Table 3. Binary classification performance of the automated Instruction Following metric against human-labeled audio categories across four task types.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Content category distribution of the BGM subset.
Domain
Task
Positive Instruction (Input Text)
Negative Instruction
SFX
V2A
Realistic foley sound synchronized with the video.
music, background music, speech, singing
VT2A
Realistic foley sound of {caption} .
music, background music, speech, singing
Music- Instrument Performance
V2A
Musical instrument performance synchronized with the video.
speech, singing, human voice
VT2A
Instrument performance of {caption} .
speech, singing, human voice
Music- BGM
V2A
Background music matching the video scene.
speech, sound effects, foley
VT2A
Background music fitting the scene: {caption} .
speech, sound effects, foley
Appendix
Table 4. Predefined positive and negative instruction templates used for the V2A and VT2A settings across four audio categories in VidAudio-Bench , where the Music category is further divided into Instrument Performance and BGM.
Figure 8. Prompt design for SFX visual captioning
Figure 9. Prompt design for Instrument Performance visual captioning
Figure 10. Prompt design for BGM visual captioning
Figure 11. Prompt design for Speech visual captioning
Figure 12. Prompt design for Singing visual captioning
Figure 13. Prompt design for Identity Consistency .
Figure 14. Prompt design for Affective Alignment on Speech .
Figure 15. Prompt design for Affective Alignment on Singing .
Figure 16. Prompt design for Instruction Following .
Dimension
SFX
Music
Speech
Singing
Realism
Fidelity: Focuses on the clarity, fidelity, and absence of artifacts (e.g., noise, distortion) of the generated audio.
Semantics
V-A Semantic-Corr : Match between audio and visual sound sources or key events.
V-A Semantic-Corr : Match with specific instruments or overall visual atmosphere.
Identity-Cons : Match between the voice and the person’s age and gender.
Identity-Cons : Match between the voice and the person’s age and gender.
Synchronization
Temp-Sync : Temporal alignment between visual actions and sound onsets.
Temp-Sync : Alignment with performance movements or rhythmic tempo.
Lip-Sync : Synchronization between mouth motion and speech articulation.
Lip-Sync : Synchronization between lip movements and melodic vocalization.
Instruction Following
Environmental/Event sounds; minimal music or speech.
Melodic/Harmonic content; absence of distinct speech or dialogue.
Intelligible human speech ; no dominant music or background SFX.
Melodic vocal performance ; distinct from plain speech or pure music.
Appendix
Table 5. Evaluation dimensions and task-specific criteria used in the human subjective study across four task categories. Each dimension was rated on a 5-point Likert scale (1–5), and the table summarizes the aspect emphasized for each task.
Figure 17. Annotation interface used in the human subjective study.
Figure 18. Examples of V2A and VT2A instruction prompts across four audio categories.