cs.CVAug 20, 2026
SaveVGI-Bench: Probing Visual Intelligence in Video Generation Models
Organizations: University of Illinois Urbana Champaign · University of Waterloo · Vector Institute · Tsinghua University · Massachusetts Institute of Technology · University of British Columbia · Etude AI · Independent · Microsoft Research
Abstract
Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/
Figures & tables
Figure 1: Overview of representative tasks in our benchmark. VGI-Bench adopts a double-level taxonomy : the first level groups every task into one of four mutually exclusive task domains , while the second level annotates each task with one or more (non-exclusive) skill tags , as shown in the legend at the bottom. Each panel shows a representative real-scene input image; the icons under the task name encode that task’s skill tags.
Table 1: Comparison with related video benchmarks. Each pie encodes the fraction of tasks that satisfy a desideratum versus do not . For input appearance, Real. denotes photorealistic-style inputs, while Abs. denotes synthetic inputs, like line-art or schematic images. Appendix E.1 details the ratio assignment.
| Category | Level | Commercial | Open Source | |||||||
| Sdce2.0 | Sora2 | Veo3.1 | Kling3.0 | Wan2.7 | Gen4.5 | Mnx-H3 | HY1.5 | Wan2.2 | ||
| Visual Org- anization | Easy | 72.6 | 55.6 | 50.3 | 67.6 | 37.0 | 60.5 | 42.8 | 26.6 | 30.4 |
| Mid | 56.4 | 37.0 | 43.3 | 47.5 | 32.2 | 38.3 | 42.3 | 17.9 | 15.3 | |
| Hard | 53.5 | 39.1 | 44.0 | 43.7 | 32.5 | 41.9 | 39.8 | 24.7 | 18.5 | |
| Avg. | 60.8 | 43.9 | 45.9 | 52.9 | 33.9 | 46.9 | 41.6 | 23.1 | 21.4 | |
| Spatiotemporal Dynamics | Easy | 54.6 | 45.3 | 28.1 | 45.2 | 38.9 | 39.6 | 44.3 | 34.7 | 38.7 |
Table 2: Evaluation results of video generation models, the best and second-best scores are highlighted. HY1.5 for HunyuanVideo-1.5 , Sdce2.0 for Seedance2.0 , Mnx-H3 for MiniMax-H3 .
Figure 2: Per-model performance across the skill tags.
| Judge Design | AUC | Pairwise Acc. |
| Main (w/ Gemini-3-Flash ) | 0.803 | 73.2% |
| w/o adaptive fps | 0.772 | 69.5% |
| w/o focus window | 0.753 | 68.3% |
| w/ GPT-5-mini | 0.690 | 64.3% |
| w/ Claude-Haiku-4.5 | 0.478 | 47.9% |
| w/ Qwen3.6-Plus | 0.624 | 59.1% |
Table 3: Reliability of our VLM-based evaluator compared with human annotations, including key component and base model ablations.
| Commercial | Open Source | ||||||||||
| Nano- Banana-Pro | Qwen- Image-3-Pro | GPT- Image-2 | MAI-Image -2.5-Pro | Seedream 4.5 | Flux.2 Max Edit | Sense- Nova-U1 | JoyAI- Image | Qwen- Image-Edit | BAGEL | Step1X- Edit | |
| Easy | 62.5 | 51.9 | 48.8 | 44.9 | 27.5 | 23.8 | 22.2 | 11.2 | 11.2 | 1.2 | 1.2 |
| Mid | 50.0 | 43.8 | 38.8 | 35.1 | 21.2 | 20.3 | 14.4 | 10.0 | 5.0 | 1.2 | 0.0 |
| Hard | 52.5 | 32.5 | 36.2 | 30.3 | 22.5 | 17.7 | 12.2 | 10.0 | 6.2 | 0.0 | 1.2 |
| Avg. | 55.0 | 42.7 | 41.2 | 36.8 | 23.8 | 20.6 | 16.3 | 10.4 | 7.5 | 0.8 | 0.8 |
Table 4: Evaluation results of image models on the adapted subset. Success Rate (%) is measured and reported (different from the video branch), against the ground truth image, the best and second-best scores are highlighted.
Figure 3: Representative cases in the failure modes.
Figure 4: Style variants (Real / Line-art) from 4 tasks: maze, hanoi tower, polyform tiling, clock running.
Figure 5: Performance comparison in oracle prompting and in varying input visual styles.
| Model | Overlap | Semi-overlap | Non-overlap |
| Wan2.2-I2V (base) | 15.3 | 17.8 | 29.2 |
| VBVR-Wan2.2 | 55.8 40.5 | 34.9 17.1 | 35.4 0 6.2 |
| Wan2.1-I2V (base) | 11.3 | 17.0 | 25.6 |
| VBVR-Wan2.1 | 25.1 13.8 | 20.2 0 3.2 | 22.0 0 3.6 |
| LTX-2.3 (base) | 15.0 | 18.2 | 15.5 |
| VBVR-LTX2.3 | 24.1 0 9.1 | 23.6 0 5.4 | 19.2 0 3.7 |
Table 5: Performance of the three released VBVR models and their respective base models, grouped by structural overlap with training distribution. Per-cell values are mean final score across tasks in the group.
| Model | Structured Puzzles | Visual Organization | Spatiotemporal Dynamics | Physical Manipulation | Overall |
| Wan2.2-I2V | 11.6 | 21.4 | 30.2 | 24.4 | 21.5 |
| VBVR-Wan2.2 | 53.2 41.6 | 37.6 16.2 | 29.0 1.2 | 42.6 18.1 | 41.2 19.7 |
Table 6: Per-domain (top) and per-skill (bottom) performance (final score) breakdown for VBVR-Wan2.2 vs. base Wan2.2-I2V .
| Transition | 1 2 | 2 3 | 3 4 | 4 10 | 10 20 | 20 40 |
| ♠ Unrecognizable | 69.2 | 41.9 | 22.2 | 0 6.8 | 0 0.0 | 0 0.0 |
| ♣ Stable | 17.9 | 49.6 | 70.1 | 69.2 | 75.2 | 90.6 |
| ◼ (a) Correct Wrong | 0 2.6 | 0 0.9 | 0 0.0 | 0 0.9 | 0 0.0 | 0 0.0 |
| ◼ (b) Wrong Correct | 0 0.9 | 0 0.0 | 0 0.9 | 0 0.0 | 0 0.0 | 0 0.0 |
| ◼ (c) Wrong Wrong ′ | 0 9.4 | 0 7.7 | 0 6.8 | 23.1 | 24.8 | 0 9.4 |
Table 7: Distribution (%) of solution-state transitions between consecutive decoded denoising steps. Row ◼ (b) (shaded) is self-correction. Protocols are in Appendix D.4 .
Figure 6: Decoded frames at different denoising stages.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Structured Puzzles | Visual Organization | Spatiotemporal Dynamics | Physical Manipulation | Total | |
| # tasks | 8 | 5 | 7 | 7 | 27 |
Table 8: Number of tasks per domain. Domains are mutually exclusive; each task belongs to exactly one domain.
| Planning | Physics | Spatial | Attribute Grounding | Affordance | Topology | Temporal | |
| # tasks | 12 | 8 | 7 | 6 | 5 | 3 | 3 |
Table 9: Number of tasks per skill tag. Skill tags are non-exclusive: each task carries one to three tags, so column counts sum to more than the total number of tasks.
Figure 7: Wordcloud of Task Prompts
Figure 8: Image-output examples on the adapted single-image subset (top to bottom: maze , recover_2d_to_3d , section_3d_figure ). Each row shows the input image, the Nano-Banana-Pro output, and the GPT-Image-2 output.
| Model | Resolution | Duration (s) | FPS | Price (USD / s) |
| Open Source | ||||
| HY1.5 | 1280 720 | 5/10 | 24 | – |
| Wan2.2 | 1280 720 | 5/10 | 16 | – |
| Mnx-H3 | 960 544 | 5/10 | 24 | – |
| Commercial | ||||
| Sora2 | 1280 720 | 8 | 30 | 0.10 |
Table 10: Generation configuration of the video models we evaluate, resolution defaults to . Price is the per-second generation cost. HY1.5 for HunyuanVideo-1.5 , Sdce2.0 for Seedance2.0 , Mnx-H3 for MiniMax-H3 .
| Category | Level | Commercial | Open Source | ||||||||||||||||||||||||||||
| Sdce2.0 | Sora2 | Veo3.1 | Kling3.0 | Wan2.7 | Gen4.5 | Mnx-H3 | HY1.5 | Wan2.2 | VBVR-Wan2.2 | ||||||||||||||||||||||
| Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | Comp. | Rub. | Final | ||
| Visual Org- anization | Easy | 82.0 | 84.8 | 72.6 | 70.0 | 76.1 | 55.6 | 66.0 | 72.5 | 50.3 | 82.0 | 80.4 | 67.6 | 52.0 | 68.6 | 37.0 | 76.0 | 77.5 | 60.5 | 56.0 | 73.4 | 42.8 | 36.0 | 66.7 | 26.6 | 46.0 | 60.2 | 30.4 | 58.0 | 73.6 | 46.1 |
| Mid | 70.0 | 77.3 | 56.4 | 58.0 | 62.7 | 37.0 | 66.0 | 64.4 | 43.3 | 62.0 | 73.2 | 47.5 | 44.0 | 67.1 | 32.2 | 54.0 | 68.1 | 38.3 | 56.0 | 72.6 | 42.3 | 28.0 | 61.0 | 17.9 | 24.0 | 59.9 | 15.3 | 46.0 | 65.7 | 32.4 | |
| Hard | 70.0 | 74.1 | 53.5 | 56.0 | 66.7 | 39.1 | 62.0 | 69.6 | 44.0 | 58.0 | 72.8 | 43.7 | 46.0 | 66.7 | 32.5 | 58.0 | 70.6 | 41.9 | 52.0 | 74.8 | 39.8 | 38.0 | 62.6 | 24.7 | 26.0 | 64.1 | 18.5 | 48.0 | 68.6 | 34.3 | |
| Avg. | 74.0 | 78.8 | 60.8 | 61.3 | 68.5 | 43.9 | 64.7 | 68.8 | 45.9 | 67.3 | 75.4 | 52.9 | 47.3 | 67.5 | 33.9 | 62.7 | 72.0 | 46.9 | 54.7 | 73.6 | 41.6 | 34.0 | 63.4 | 23.1 | 32.0 | 61.4 | 21.4 | 50.7 | 69.3 | 37.6 | |
Table 11: Full per-metric evaluation results of video generation models across our reasoning categories at Easy/Mid/Hard difficulty plus a per-category average, scored by three metrics: Completeness (Comp.), Rubric Score (Rub.), and the final aggregated score (Final). The main-paper Table 2 reports the final aggregated score only. Higher is better; the best and second-best Final scores in each row are highlighted. HY1.5 for HunyuanVideo-1.5 , Sdce2.0 for Seedance2.0 , Mnx-H3 for MiniMax-H3 . Note that, within each cell, generally, : this is because Comp. and Rub. are the mean Completeness and Rubric scores aggregated independently across instances, whereas Final is the mean of the per-instance product (i.e., the mean of products, not the product of means).
| Category | Level | Human | Commercial | Open Source | ||||||||
| Sdce2.0 | Sora2 | Veo3.1 | Kling3.0 | Wan2.7 | Gen4.5 | Mnx-H3 | HY1.5 | Wan2.2 | VBVR-Wan2.2 | |||
| Visual Org- anization | Easy | 100.0 | 40.0 | 16.0 | 4.0 | 24.0 | 8.0 | 8.0 | 12.0 | 4.0 | 4.0 | 16.0 |
| Mid | 97.5 | 16.0 | 4.0 | 4.0 | 12.0 | 4.0 | 8.0 | 8.0 | 0.0 | 0.0 | 4.0 | |
| Hard | 92.5 | 12.0 | 0.0 | 4.0 | 12.0 | 4.0 | 8.0 | 8.0 | 0.0 | 4.0 | 4.0 | |
| Avg. | 96.7 | 22.7 | 6.7 | 4.0 | 16.0 | 5.3 | 8.0 | 9.3 | 1.3 | 2.7 | 8.0 | |
| Spatiotemporal Dynamics | Easy | 100.0 | 5.4 | 5.7 | 0.0 | 10.0 | 5.7 | 5.0 | 8.6 | 0.0 | 2.9 | 5.7 |
Table 12: Strict success rate (%) of video generation models on VGI-Bench . Layout mirrors Table 2 . Per row, the best and second-best scores among the video models are highlighted (Human reference excluded); tied bests share the darker shade. Cells marked “-” have no judged instances yet. HY1.5 for HunyuanVideo-1.5 , Sdce2.0 for Seedance2.0 , Mnx-H3 for MiniMax-H3 .
Figure 9: Representative human failure cases, most cases are sourced from the highest difficulty level of the task.
| Video Model | Full (N=10) | Half (N=5, main) | (Half Full) |
| Kling3.0 | 34.8 | 33.2 | |
| Sora2 | 30.9 | 27.2 | |
| Veo3.1 | 27.8 | 25.1 | |
| Wan2.2 | 14.3 | 14.0 | |
| HY1.5 | 14.7 | 11.7 | |
| Mean | 24.5 | 22.2 |
Table 13: Stability of half-subset evaluation against full-instance evaluation. Scores are Final Score (rubric completeness) averaged per task then over the 6 tasks.
Figure 10: Generated result videos for the maze task, from three video models ordered low to high by final score.
Figure 11: Generated result videos for the recover_2d_net task, from three video models ordered low to high by final score.
Figure 12: Generated result videos for the section_3d_figure task, from three video models ordered low to high by final score.
Figure 13: Generated result videos for the block_assembly task, from three video models ordered low to high by final score.
Figure 14: Generated result videos for the object_packing task, from three video models ordered low to high by final score.
Figure 15: Generated result videos for the sorting task, from three video models ordered low to high by final score.
Figure 16: Screenshot of our human-annotation interface for resulting video preference. For each instance, the annotator watches the task instructions alongside three model-generated videos and either ranks them by reasoning progress from best to worst, or flags the triple as a tie / skip when no clear ordering exists.
| Judge Design | AUC | Pairwise Acc. |
| Main (w/ Gemini-3-Flash ) | 0.803 | 73.2% |
| w/o adaptive fps | 0.772 | 69.5% |
| w/o focus window | 0.753 | 68.3% |
| w/ GPT-5-mini | 0.690 | 64.3% |
| w/ Claude-Haiku-4.5 | 0.478 | 47.9% |
| w/ Qwen3.6-Plus | 0.624 | 59.1% |
Table 14: Reliability of our VLM-based evaluator compared with human annotations.
| Metric | Structured Puzzles | Visual Organization | Spatiotemporal Dynamics | Physical Manipulation |
| AUC | 0.851 | 0.800 | 0.814 | 0.783 |
| Pairwise Acc. | 75.0% | 69.2% | 74.1% | 70.1% |
Table 15: Judge–human agreement broken down by domain (top) and skill tag (bottom), for the main judge configuration of Table 14 . Skill tags are non-exclusive, so a task with tags contributes its pairs to skill pools.
Figure 18: Effect of appearance and style difference. Example task: maze .
Figure 19: Effect of appearance and style difference. Example task: hanoi tower .
Table 20: Each pie encodes fraction of tasks that satisfy a desideratum versus do not . For input appearance, Real. denotes photorealistic-style inputs, while Abs. denotes non-photorealistic inputs such as line-art or schematic renderings.