cs.CVMar 26, 2026
SaveViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?
Organizations: Tsinghua University · Meituan M17 · The University of Hong Kong · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
Abstract
Beneath the stunning visual fidelity of modern AIGC models lies a "logical desert", where systems fail tasks that require physical, causal, or complex spatial reasoning. Current evaluations largely rely on superficial metrics or fragmented benchmarks, creating a performance mirage'' that overlooks the generative process. To address this, we introduce ViGoR Vision-G}nerative Reasoning-centric Benchmark), a unified framework designed to dismantle this mirage. ViGoR distinguishes itself through four key innovations: 1) holistic cross-modal coverage bridging Image-to-Image and Video tasks; 2) a dual-track mechanism evaluating both intermediate processes and final results; 3) an evidence-grounded automated judge ensuring high human alignment; and 4) granular diagnostic analysis that decomposes performance into fine-grained cognitive dimensions. Experiments on over 20 leading models reveal that even state-of-the-art systems harbor significant reasoning deficits, establishing ViGoR as a critical stress test'' for the next generation of intelligent vision models. The demo have been available at https://vincenthancoder.github.io/ViGoR-Bench/
Figures & tables
Figure 1: An overview of ViGoR-Bench. (a) The data distribution across various domains. (b) Examples of the reasoning process from generation models. (c) Performance comparison of leading models on ViGoR-Bench.
| Benchmark | Tasks | Reference | Type | Evaluation | ||||
| Img | Text | I2I | I2Is ⋆ | I2V | Process | Result | ||
| RISE | – | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| KRIS | 22 | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| GIR-Edit | 3 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| UniREdit | 18 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ |
| WiseEdit | – | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
Table 1: Overview of benchmark properties and evaluation settings. Veo † ( Wiedemer et al., 2025 ) conducted quantitative evaluations across seven distinct task categories.
Figure 2: An overview of the ViGoR-Bench construction and evaluation pipelines. (a) The benchmark dataset is constructed through a three-pronged approach: generative synthesis, real-world acquisition, and algorithmic generation. All data undergoes human review to establish definitive image-ground truth (GT) pairs. (b) For evaluation, a Multimodal Large Language Model (MLLM) is employed as an automated judge. Conditioned on the ground-truth image, the MLLM assesses both the Chain-of-Thought (CoT) reasoning process and the final output of generative models for images and videos.
Figure 3: Overview of the ViGoR-Bench task suite. We present representative demo cases and their corresponding editing instructions across 20 distinct sub-tasks. These tasks are hierarchically organized into three primary reasoning domains: Physical Reasoning, Knowledge Reasoning, and Symbolic Reasoning.
| Type | Evaluator | GT Ref. | MAE | Acc | Var |
|---|---|---|---|---|---|
| Process | Human | ✓ | 0.000 | 1.000 | 0.051 |
| Gemini-2.5-Pro | ✗ | 0.319 | 0.680 | 0.039 | |
| Gemini-2.5-Pro | ✓ | 0.267 | 0.733 | 0.034 | |
| Result | Human | ✓ | 0.000 | 1.000 | 0.011 |
| Gemini-2.5-Pro | ✗ | 0.294 | 0.705 | 0.034 | |
| Gemini-2.5-Pro | ✓ | 0.213 | 0.786 | 0.029 |
Table 2: Reliability Analysis of VLM-as-a-Judge. We evaluate the alignment between Gemini-2.5-Pro ( Comanici et al., 2025 ) and human experts on a random subset. The table compares performance with and without Ground Truth references across both Process and Result metrics.
| Type | Process | Model | OS † | Process Metric (%) | Result Metric (%) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BC | RO | VQ | RA | Avg | BC | RO | VQ | RS | Avg | ||||
| Edit | w/o CoT | FLUX.1-Kontext-dev ( Labs, 2025b ) | ✓ | – | – | – | – | – | 65.1 | 13.1 | 75.9 | 1.6 | 38.9 |
| FLUX.2-dev ( Labs, 2025a ) | ✓ | – | – | – | – | – | 40.0 | 11.5 | 63.8 | 4.2 | 29.9 | ||
| Qwen-Image-Edit-2509 ( Wu et al., 2025a ) | ✓ | – | – | – | – | – | 52.4 | 14.3 | 60.7 | 1.4 | 32.2 | ||
| Qwen-Image-Edit-2511 ( Wu et al., 2025a ) | ✓ | – | – | – | – | – | 42.4 | 11.1 | 44.6 | 4.9 | 25.8 | ||
| LongCat-Image-Edit ( Ma et al., 2025 ) | ✓ | – | – | – | – | – | 53.6 | 11.4 | 60.5 | 3.3 | 32.2 | ||
Table 3: Main experimental results across different metrics. Process metrics evaluate generative process reasoning quality, while final result metrics assess the final output; all values are reported in percentage scale. OS † indicates open-sourced status. Within each model category, the best result for each metric is marked in bold . Bold values denote the overall best performance across all results.
Figure 4: Qualitative comparison of leading models. We present case studies across three representative reasoning domains.
Figure 5: Impact of problem complexity on Reasoning Success. We report the performance of evaluated models on Sudoku, Jigsaw Puzzle, and Maze Navigation tasks across varying grid dimensions.
Figure 6: Overview of results of SFT and RL
| Model | Maze Grid | BC | RO | VQ | RS | Avg |
| Qwen-Image-Edit-2509 | N/A | 0.0 | 0.0 | 32.0 | 0.0 | 8.0 |
| Qwen-Image-Edit-2511 | 66.0 | 3.0 | 74.0 | 2.0 | 36.3 | |
| Nano Banana | 38.0 | 37.0 | 96.0 | 3.0 | 43.5 | |
| GPT-Image-1 | 96.0 | 23.0 | 98.0 | 5.0 | 55.5 | |
| Nano Banana Pro | 93.0 | 50.0 | 98.0 | 11.0 | 63.0 | |
| Qwen-Image-Edit-2509-SFT | 65.0 | 65.0 | 68.0 | 27.0 | 56.3 |
Table 4: Comparison of SFT and RL fine-tuning stages. Models are fine-tuned on constructed maze datasets with grid dimensions of , , and . Evaluation is conducted on our benchmark test set, which covers maze grid dimensions ranging from to .
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Samples of ViGoR-Bench.
Figure 8: Statistic of ViGoR-Bench.
Figure 9: Comparison of qualitative results for Qwen-Image-Edit-2511 on ViGoR-Bench after SFT and RL.
Figure 10: Performance profiling on symbolic reasoning tasks. Comparison of five models across seven sub-tasks. The top and bottom rows report performance on four Process Metrics and four Result Metrics.
Figure 11: Performance profiling on physical reasoning tasks. Comparison of five models across seven sub-tasks. The top and bottom rows report performance on four Process Metrics and four Result Metrics.
Figure 12: Performance profiling on Knowledge reasoning tasks. Comparison of five models across seven sub-tasks. The top and bottom rows report performance on four Process Metrics and four Result Metrics.
Table 5: Knowledge Reasoning Binary Template.
Table 6: Knowledge Reasoning CoT Template.
Table 7: Physical Reasoning CoT Template.
Table 8: Physical Reasoning Binary Template.
Table 9: Symbolic Reasoning - Sudoku CoT Template.
Table 10: Symbolic Reasoning - Sudoku Binary Template.
Table 11: Symbolic Reasoning - Jigsaw Puzzle CoT Template.
Table 12: Symbolic Reasoning - Jigsaw Puzzle Binary Template.
Table 13: Symbolic Reasoning - Function Plotting CoT Template.
Table 14: Symbolic Reasoning - Function Plotting Binary Template.
Table 15: Symbolic Reasoning - Algebraic Calculation CoT Template.
Table 16: Symbolic Reasoning - Algebraic Calculation Binary Template.
Table 17: Symbolic Reasoning - Block Building CoT Template.
Table 18: Symbolic Reasoning - Block Building Binary Template.
Table 19: Symbolic Reasoning - Klotski Puzzle CoT Template.
Table 20: Symbolic Reasoning - Klotski Puzzle Binary Template.
Table 21: Symbolic Reasoning - Maze Navigation Binary Template.
Table 22: Maze Navigation CoT Template.