When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?
Organizations: Nanjing University · TongYi Lab · Tsinghua University · Fudan University
Abstract
Unified multimodal models (UMMs) can perform both understanding and generation, raising a central question: can visual generation improve understanding? Existing evaluations provide mixed evidence, but confound task difficulty, reasoning paradigms, and the closed-loop interaction between generation and understanding. We introduce VGAU-Diag, a fine-grained evaluation framework for vision generation-assisted understanding. It stratifies samples by difficulty, enables unified evaluation of multiple reasoning paradigms, and uses Oracle-Assisted Reference Protocols. Our analysis shows that generated visual aids help on easier instances but become unreliable as reasoning complexity increases. Oracle-assisted diagnosis further reveals that the main bottleneck often lies on the visual-understanding side rather than the visual-generation side, as current UMMs struggle to leverage even faithful visual aids. We also show that effective visual generation should target visual-understanding bottlenecks rather than add more reasoning steps, and identify a three-stage transition from task-irrelevant noise, to misleading plausible guidance, and finally to useful assistance. These findings would be useful to guide the development of better UMMs.
Figures & tables
| Model | Method | Maze | Sokoban | Path | Onet | Parking | Klotski | ||||||||||||
| Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | ||
| Open-source VLMs | |||||||||||||||||||
| Qwen2.5-VL-7B-Instruct | Direct | 10.0 | 0.0 | 0.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Text-CoT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 15.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Oracle-VCoT | 10.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Oracle-GtA | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Model | Setting | Maze | Sokoban | Path | Onet | Parking | Klotski |
| MammothModa2 | Self-GtA | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) |
| Self-VCoT | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | |
| Self-VR | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | |
| STAR-7B | Self-GtA | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (-5.0) | 0.0 (+0.0) |
| Self-VCoT | 1.6 (+1.6) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (-5.0) | 0.0 (+0.0) | |
| Self-VR | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (-5.0) | 0.0 (+0.0) |
| Method | Qwen3.5-0.8B | Qwen3.5-2B | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.6-plus |
| Text-CoT | 5.0 | 5.0 | 20.0 | 25.0 | 60.0 | 70.0 |
| Vision: Gaussian noise | ||||||
| Self-GtA | 0.0 (-5.0) | 1.6 (-3.4) | 8.3 (-11.7) | 8.3 (-16.7) | 20.0 (-40.0) | 60.0 (-10.0) |
| Self-VCoT | 6.6 (+1.6) | 6.6 (+1.6) | 11.6 (-8.4) | 13.3 (-11.7) | 35.0 (-25.0) | 61.6 (-8.4) |
| Self-VR | 0.0 (-5.0) | 0.0 (-5.0) | 6.6 (-13.4) | 6.6 (-18.4) | 18.3 (-41.7) | 53.3 (-16.7) |
| Vision: Mismatch Image | ||||||
| Model | Setting | Maze | Sokoban | Path | Onet | Parking | Klotski |
| MammothModa2 | Self-VCoT | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) |
| Self-VCoT-fixed | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | |
| STAR-7B | Self-VCoT | 1.6 (+1.6) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (-5.0) | 0.0 (+0.0) |
| Self-VCoT-fixed | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (-5.0) | 0.0 (+0.0) | |
| OmniGen2 | Self-VCoT | 0.0 (+0.0) | 5.0 (+5.0) | 5.0 (+5.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) |
| Self-VCoT-fixed | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) | 0.0 (+0.0) |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Average metric | Easy | Medium | Hard |
| Walkable cells | 20.30 | 30.10 | 39.12 |
| Branch nodes | 4.25 | 5.83 | 6.98 |
| Corridor nodes | 9.35 | 16.07 | 22.76 |
| Average path metric | Easy | Medium | Hard |
| Path nodes | 4.10 | 13.27 | 24.64 |
| Branch nodes on path | 1.55 | 3.70 | 6.00 |
| Corridor nodes on path | 2.00 | 8.97 | 18.00 |
| Task | Oracle-VCoT | Oracle-VCoT-last-image |
| Maze | 25.0 | 20.0 |
| Sokoban | 20.0 | 20.0 |
| Path | 25.0 | 20.0 |
| Model | Method | Maze | Sokoban | Parking | Klotski | ||||||||
| Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | ||
| Open-source VLMs | |||||||||||||
| Qwen2.5-VL-7B-Instruct | Direct | 10.0 | 0.0 | 0.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Text-CoT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 15.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Oracle-GtA | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Nemotron-nano-12b-v2-vl | Direct | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Model | Setting | Maze | Sokoban | Parking | Klotski |
| MammothModa2 | Self-GtA | 0.0 | 0.0 | 0.0 | 0.0 |
| Self-VR | 0.0 | 0.0 | 0.0 | 0.0 | |
| STAR-7B | Self-GtA | 0.0 | 0.0 | 0.0 | 0.0 |
| Self-VR | 0.0 | 0.0 | 0.0 | 0.0 | |
| OmniGen2 | Self-GtA | 0.0 | 0.0 | 0.0 | 0.0 |
| Self-VR | 0.0 | 0.0 | 10.0 | 0.0 |
| Method | Qwen3.5-0.8B | Qwen3.5-2B | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.6-plus |
| Text-CoT | 0.0 | 10.0 | 25.0 | 30.0 | 15.0 | 30.0 |
| Vision: Gaussian noise | ||||||
| Self-GtA | 0.0 (+0.0) | 8.3 (-1.7) | 11.6 (-13.4) | 21.6 (-8.4) | 20.0 (+5.0) | 23.3 (-6.7) |
| Self-VCoT | 6.6 (+6.6) | 8.3 (-1.7) | 15.0 (-10.0) | 15.0 (-15.0) | 21.6 (+6.6) | 41.6 (+11.6) |
| Self-VR | 0.0 (+0.0) | 3.3 (-6.7) | 11.6 (-13.4) | 16.6 (-13.4) | 18.3 (+3.3) | 21.6 (-8.4) |
| Vision: Mismatch Image | ||||||
| Method | Qwen3.5-0.8B | Qwen3.5-2B | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.6-plus |
| Text-CoT | 5.0 | 5.0 | 25.0 | 25.0 | 40.0 | 70.0 |
| Vision: Gaussian noise | ||||||
| Self-GtA | 3.3 (-1.7) | 3.3 (-1.7) | 20.0 (-5.0) | 33.3 (+8.3) | 6.6 (-33.4) | 88.3 (+18.3) |
| Self-VCoT | 3.3 (-1.7) | 26.6 (+21.6) | 8.3 (-16.7) | 23.3 (-1.7) | 10.0 (-30.0) | 75.0 (+5.0) |
| Self-VR | 0.0 (-5.0) | 3.3 (-1.7) | 10.0 (-15.0) | 16.6 (-8.4) | 30.0 (-10.0) | 56.6 (-13.4) |
| Vision: Mismatch Image | ||||||
Unified Multimodal Models
Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.
See all 290 papers