From Perception to Action: An Interactive Benchmark for Vision Reasoning
Authors: Yuhao Wu, Maojia Song, Yihuai Lan, Lei Wang, Zhiqiang Hu, Yao Xiao, Heng Zhou, Weihua Zheng, Dylan Raharja, Soujanya Poria, Roy Ka-Wei Lee
Organizations: Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · University of Science and Technology of China (USTC), China · Nanyang Technological University (NTU), Singapore
Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.
Figures & tables
Figure 1: Static vs. Interactive Evaluation of Physical Struction Reasoning. (a) Traditional VQA relies on passive observation of an image. (b) Our paradigm requires multi-step interaction, enabling procedural evaluation of planning and structural understanding.
Figure 2: Benchmark construction pipeline. We illustrate the end-to-end process for building our benchmark, including problem sourcing, document collection and filtering, concept annotation, regime construction, and final evaluation setup. The pipeline is designed to ensure controlled difficulty, minimize parametric leakage, and enable fine-grained analysis of reasoning and retrieval behaviors.
Models
Task Success
Plan Efficiency (Solved)
Token & Cost
Pass@1 ↑
Succ.Task ↑
Puzzle ↑
Stacking ↑
AvgSteps ↓
Dist2Opt ↓
N.Dist ↓
Solved/Tokens ↑
Solved/USD ↑
Closed-source Models
GPT-5.2
22.9
25
3.1
31.2
9.6
1.43
0.50
10.1
0.10
GPT-5-mini
11.0
12
3.1
14.3
6.8
1.08
0.57
19.3
0.19
OpenAI-o3
10.1
11
3.1
13.0
6.1
0.81
0.37
26.1
0.26
Claude-Opus-4.5
15.6
17
3.1
20.8
8.0
1.42
0.63
17.8
0.17
Table 1: Main model evaluation results. We report overall task success on the full benchmark (Pass@1 (All)) and subset success on Puzzle & Stacking. Plan-efficiency metrics (AvgSteps, Dist2Opt, NormDist; lower = better) are computed only for solved tasks. Token and monetary costs (Solved/Tokens, Solved/USD) are normalized by successful tasks.
Figure 3: Cost and token efficiency with solved tasks comparison between models
Figure 4: Qualitative results on the Luban puzzle subtask across world models. Top: Level 1 (two beams). Bottom: Level 2 (six beams). All models fail to produce a physically valid disassembly, either violating interlocking constraints or hallucinating (e.g., structural corruption and object insertion/removal), with failures worsening at higher complexity.
Models
Puzzle Acc ↑
Stacking Acc ↑
Easy
Mid
Hard
Easy
Mid
Hard
GPT5.2
10.0
0.0
0.0
100.0
55.0
6.3
Gemini-3-Pro
10.0
0.0
0.0
90.0
40.0
6.3
Sonnet-4.5
10.0
0.0
0.0
100.0
20.0
0.0
Table 2: Difficulty-stratified accuracy (Acc) on C H A IN . We report Easy/Mid/Hard tiers for both Puzzle and Stacking.
Models
Interactive (%) ↑
One-shot (%) ↑
Δ
Puzzle
Stacking
All
Puzzle
Stacking
All
GPT5.2
3.1
31.2
22.9
0.0
9.1
7.1
-15.8
Sonnet-4.5
3.1
18.2
13.8
0.0
10.3
8.1
-5.7
Gemini-3-Pro
3.1
26.0
19.3
0.0
9.1
7.1
-12.2
Table 3: Interactive vs. one-shot solving on C H A IN . In one-shot, the model receives only a single fixed-view image and must output a complete (or near-complete) solution with minimal or no intermediate feedback. Δ All Avg denotes the performance gap between interactive and one-shot settings (Interactive − One-shot).
Strategy
Puzzle(%) ↑
Stacking(%) ↑
All(%) ↑
Δ vs. Avg@4 ↑
Avg@4
3.1
15.5
9.3
–
Pass@1
3.1
15.6
9.4
+0.1
Pass@2
3.1
19.4
11.2
+1.9
Pass@4
3.1
19.4
11.2
+1.9
VLM judge
3.1
18.1
10.3
+1.3
Reward model
3.1
16.8
9.9
+0.6
Table 4: Comparison of multi-sample selection and reward-model (RM) reranking on C H A IN by kimi-k2.5. All strategies use the same base generator and differ only in candidate selection. Δ is computed w.r.t. Avg@4 on All Avg.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Models (abbr.)
Model Specs
Release Date
Context (tokens)
Model Size
Closed-source Models
GPT5.2
2025-12-11
400,000
–
o3 (optional)
2025-04-16
200,000
–
Opus4.5 (optional)
2025-11-24
200,000
–
Sonnet4.5
2025-09-29
1,000,000
–
Appendix
Table 5: Model release time, context window, and parameter scale for the evaluated model pool. For MoE models, AxxB denotes active parameters per forward pass when available. Unknown values are marked as -- .
Figure 5: Examples of different levels of puzzle task.
Figure 6: Examples of different levels of stacking task.
Figure 7: A case of successfully solving the stacking task.
Figure 8: A case of failed to solve the stacking task.
Figure 9: World-model prompt for generating physically valid disassembly videos of Kongming/Luban locks (interlocking burr puzzles). Constraints include rigidity, collision avoidance, continuous motion, and axis-aligned sliding.