cs.CVApr 9, 2026
SaveAVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation
Organizations: Fudan University · University of Science and Technology of China · Microsoft Research Asia
Abstract
Text-to-Audio-Video (T2AV) generation is rapidly becoming a core interface for media creation, yet its evaluation remains fragmented. Existing benchmarks largely assess audio and video in isolation or rely on coarse embedding similarity, failing to capture the fine-grained joint correctness required by realistic prompts. We introduce AVGen-Bench, a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories. To support comprehensive assessment, we propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs), enabling evaluation from perceptual quality to fine-grained semantic controllability. Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control. Code and benchmark resources are available at http://aka.ms/avgenbench.
Figures & tables
Figure 1: Comparison between AVGen-Bench and existing benchmarks. Unlike prior works that rely on separate audio/visual evaluations and simple prompts, AVGen-Bench introduces (1) joint audio-visual evaluation, (2) fine-grained metrics across 10 dimensions, and (3) rich, complex prompts with high token counts to ensure a rigorous assessment.
Figure 2: Qualitative examples of failure modes across different fine-grained dimensions. (a1) Explicitly prompted text rendering. (a2) Incidental text rendering in background elements. (b) Fine-grained musical control (Pitch Accuracy). (c) Speech generation regarding incidental coherence and explicit instruction following. (d) Holistic semantic alignment in complex multi-shot narratives. (e) High-level physical plausibility and dynamic constraints. (f) Facial consistency failures, illustrating identity drift across shot transitions and degradation in multi-face crowd scenes. Red markings and crosses (✗) indicate generated errors.
Figure 3: Overview of the AVGen-Bench framework. The benchmark features a Task-Driven Prompt Set (left) categorized into three real-world application domains: Professional Media, Creator Economy, and World Simulation. The generated content is evaluated via our Multi-Granular Evaluation Suite (right), which employs a hybrid strategy combining lightweight specialist models (orange) for signal-level precision and MLLMs (purple) for high-level semantic reasoning and physical plausibility analysis.
| Benchmark | Task | #Metrics | Avg. Tokens | Audio Types |
|---|---|---|---|---|
| VBench 2.0 | T2V | 18 | 26.56 | - |
| TTA-Bench | T2A | 10 | 20.00 | SFX, Music, Speech |
| JavisBench | T2AV | 4 | 65.00 | SFX, |
| Harmony-Bench | TI2AV | 6 | - | SFX, Music, Speech |
| VerseBench | TI2AV | 4 | 68.00 | SFX, Speech |
| UniAVGen | TI2AV | 3 | - | Speech |
Table 1: Comparison with existing benchmarks. AVGen-Bench features the highest average prompt complexity (Avg. Tokens) and a comprehensive set of evaluation metrics covering all audio modalities.
Figure 4: Dataset-level statistics of the prompts used in AVGen-Bench.
Figure 5: Detailed workflows of the six Fine-grained Evaluation Modules in AVGen-Bench. The suite employs hybrid strategies combining specialist models (blue nodes) and MLLMs (purple nodes) to evaluate: (a) Scene Text Rendering (OCR + Verification); (b) Facial Consistency (InsightFace + DBSCAN); (c) Pitch Accuracy (Audio-to-MIDI + Theory Check); (d) Speech Intelligibility (ASR + Contextual Logic); (e) Physical Plausibility (Kinematics + Causal Reasoning); and (f) Holistic Semantic Alignment (Constraint Decomposition).
| Model | Basic Uni-modal | Basic Cross-modal | Fine-grained Visual | Fine-grained Audio | Fine-grained Macro | Overall | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Vis | Aud (PQ) | AV | Lip | Text | Face | Music | Speech | Lo-Phy | Hi-Phy | Holistic | Total | |
| Veo 3.1-fast | 0.960 | 6.64 | 0.21 | 2.39 | 75.10 | 52.77 | 3.13 | 94.53 | 3.68 | 67.43 | 86.27 | 67.87 |
| Veo 3.1-quality | 0.954 | 6.77 | 0.24 | 3.59 | 76.53 | 52.90 | 5.00 | 96.09 | 3.74 | 68.53 | 84.10 | 66.28 |
| Sora-2 | 0.848 | 5.91 | 0.25 | 4.50 | 74.84 | 51.17 | 7.81 | 88.63 | 4.05 | 78.95 | 88.89 | 64.16 |
| Wan2.6 | 0.959 | 7.15 | 0.30 | 4.32 | 76.95 | 49.27 | 1.75 | 89.33 | 3.69 | 66.92 | 80.98 | 62.97 |
| Seedance-1.5 Pro | 0.970 | 7.48 | 0.26 | 3.43 | 38.28 | 54.42 | 1.88 | 93.45 | 3.72 | 66.88 | 77.38 | 62.55 |
Table 2: Quantitative comparison on AVGen-Bench. We evaluate models across three granularities: Basic Uni-modal (Visual/Audio Aesthetic), Basic Cross-modal (Sync), and our proposed Fine-grained Modules. Best scores are highlighted in bold , and second-best are underlined . Note that for AV-Sync and Lip-Sync, lower ( ) is better; for others, higher ( ) is better. We also report an aggregate Total score (Scheme-2). Wan2.2+HunyuanVideo-Foley denotes a cascaded pipeline of T2V followed by V2A. Emu3.5+MOVA and NanoBanana2+MOVA are both T2Image+TI2AV cascaded pipelines. Proprietary components are marked with orange background, while open-source components are marked with blue background. Models are sorted by Overall score in descending order.
| Dimension | Protocol | Pearson |
|---|---|---|
| Text Rendering | Pointwise | 0.9657 |
| Pitch Accuracy | Pointwise | 0.5544 |
| Facial Consistency | Pairwise | 0.8270 |
| Speech Intelligibility & Coherence | Pairwise | 0.8300 |
| Physical Plausibility | Pairwise | 0.8290 |
| Holistic Semantic Alignment | Pairwise | 0.8402 |
Table 3: Human validation of fine-grained evaluation metrics. We report the Pearson correlation between our automated scores and expert human judgments across six fine-grained dimensions. All results are computed on a shared subset of 85 tasks annotated by 10 expert raters. Higher is better.
| Dimension | Agreement Metric | Score |
|---|---|---|
| Text Rendering | Weighted Cohen’s | 0.9116 |
| Pitch Accuracy | Weighted Cohen’s | 0.3156 |
| Facial Consistency | Cohen’s | 0.8511 |
| Speech Intelligibility & Coherence | Cohen’s | 0.9272 |
| Physical Plausibility | Cohen’s | 0.8455 |
| Holistic Semantic Alignment | Cohen’s | 0.8909 |
Table 4: Inter-rater agreement on the shared user-study subset. We report inter-rater reliability across 10 expert raters on the same 85 tasks. For pointwise dimensions, we use weighted Cohen’s ; for pairwise dimensions, we report Cohen’s . Higher is better.
| Model | Metric | Mean | Std. | Range |
|---|---|---|---|---|
| Veo 3.1 Fast | Text | 74.75 | 0.83 | 73.95–75.90 |
| Face | 52.97 | 0.00 | 52.97–52.97 | |
| Music | 2.76 | 0.08 | 2.65–2.81 | |
| Speech | 94.48 | 0.02 | 94.46–94.51 | |
| Lo-Phy | 74.79 | 0.00 | 74.79–74.79 | |
| Hi-Phy | 70.73 | 1.70 | 69.16–73.09 |
Table 5: Repeated-run stability of the MLLM-assisted evaluation. We repeat the full evaluation pipeline 3 times on the same generated outputs for two representative models, Veo 3.1 Fast and LTX-2 . We report the mean, standard deviation, and value range of each fine-grained metric across runs. Lower standard deviation indicates better stability.
Figure 6: Benchmark-scale robustness under prompt subset resampling. We repeatedly sample prompt subsets at different ratios and recompute the overall normalized score 200 times. The solid lines denote the mean score over random subsets, the error bars indicate one standard deviation, and the dashed lines mark the corresponding full-benchmark score. Results for both Veo 3.1 Fast and LTX-2 remain close to the full-score baseline, with smaller variance at larger subset ratios, indicating that AVGen-Bench yields stable model comparison under prompt subsampling.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Extended Examples of Text Rendering Failures. Top (Prompted Text): Models struggle with ”glyph collapse” and layout errors when prompted with specific strings like ”Your customers are talking” or ”EIGHTY-SEVEN SECONDS”. Even high-performing models like Veo 3.1 and Wan 2.6 often fail to render the text perfectly legible or place it on the correct object. Bottom (Incidental Text): A pervasive failure mode where models hallucinate gibberish for background text that was not explicitly prompted, such as website content, car license plates, or studio backdrops. This highlights a lack of ”world knowledge” regarding how text naturally appears in real-world scenes.
Figure 8: Extended Examples of Face Inconsistency and Speech Generation Errors. Top (Face Inconsistency): We observe two distinct patterns of identity loss: (1) Identity Drift across shot transitions, where a character’s appearance changes significantly after a cut; and (2) Crowd Degradation , where faces in multi-person scenes (e.g., boxing audience) become distorted. Bottom (Speech Generation): Models frequently fail to adhere to linguistic or speaker constraints. Failures include generating the wrong language (e.g., Spanish instead of English), producing rhythmic noise instead of dialogue, or assigning dialogue to the wrong speaker count.
Figure 9: Extended Examples of Physical Violations and Semantic Misalignment. Top (Violation of Physical Laws): Models fail to simulate complex physical phenomena driven by sound. Left (Chladni Plate): Models fail to generate the correct geometric sand patterns corresponding to resonant frequencies. Right (Chemical Reaction): Models fail to depict the correct color oscillations or liquid dynamics in a Briggs-Rauscher reaction setup. Bottom (Semantic Misalignment): In complex multi-shot narratives (e.g., a vacation ad), models often miss key semantic constraints, such as specific actions (”hitting a beach ball”) or correct text sequencing (”Book your family home now”).
Figure 10: Deep Dive into Pitch Accuracy Failures via Symbolic-Neural Verification. We illustrate the disconnect between visual realism and acoustic logic in music generation. Top (Piano): The prompt strictly requests a ”C-G-Am-F” chord progression. While models generate convincing visuals of hands on keys, the extracted MIDI data reveals that the audio contains wrong chords, random melodic noise, or chaotic note clusters, failing to follow basic music theory constraints. Bottom (Guitar): The prompt requests a specific single note (A4) plucked four times. Models fail to isolate the pitch, instead generating complex, unprompted chords or multi-string noise. This confirms that current T2AV models function as ”texture generators” rather than grounded simulators of physical acoustic events.
Figure 11: Overview of the Custom Gradio Annotation Suite. We tailored the annotation interface to the specific nature of the task. (a) For subjective dimensions, we enforce strict side-by-side comparison to reduce inter-rater variance. (b) For objective dimensions like text, we use absolute scoring to capture specific failure modes.
More in this topic
Audio-Video Generation
Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.
See all 154 papers