Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on real-world data or procedurally generated tasks. Real-world data preserve natural complexity, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether abilities measured in toy games correspond to real-world visual reasoning and whether game examples can support transfer. We introduce Game2Real-Bench, a benchmark that studies these connections across four capability groups, including fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. We construct 1,000 procedurally generated game samples with executable verification and curate 400 real-world samples aligned by shared reasoning requirements rather than visual similarity. We evaluate 25 representative VLMs and observe strong game-to-real consistency, with Pearson and Spearman correlations of 0.935. We further study one-shot game-to-real transfer on four models. Constraint reasoning improves for several models, while fine-grained perception declines with game demonstrations. We also construct Game2Real-Dataset with nearly 24K verified game samples across 18 tasks and 65 QA families. Our benchmark, dataset, and code are available at https://github.com/QingAnLiu/Game2Real-Bench.
Figures & tables
Figure 1: Multi-stage construction pipeline of Game2Real-Bench. It comprises capability-guided task design, procedural generation and verification, and mechanism-aligned real-world curation.
Figure 2: Capability-aligned tasks in Game2Real-Bench. Toy-game and real-world tasks span fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. We present representative QA samples illustrating shared reasoning requirements across the two settings.
Model
Fine-grained Perception
Spatial Reasoning
Constraint Reasoning
Sequential Planning
Overall
▼ Proprietary Models
GPT-5.6-Sol
53.9 / 57.0
69.8 / 49.0
81.3 / 63.9
81.4 / 68.0
69.8
Gemini-3.1-Pro-Preview
36.0 / 66.0
59.0 / 55.0
56.7 / 51.1
75.2 / 73.0
59.6
Claude-Opus-4.8-thinking
22.5 / 50.0
50.5 / 44.0
62.0 / 46.8
70.8 / 61.0
53.7
▼ Large Open-Weight Models
Qwen3.8-27B
37.6 / 67.0
61.3 / 55.0
39.3 / 38.8
61.0 / 72.0
52.6
Table 1: Scores (%) on Game2Real-Bench, with a 1,000-sample Toy Game set and a 400-sample Real-World set. For each capability, results are reported as Toy Game / Real-World. Toy-game scores are accuracies; real-world scores incorporate SpatiaLQA F1 where applicable. Overall denotes the mean per-sample score over all 1,400 samples. Open-weight models are grouped by total parameter count as large (27–35B), medium (7–12B), and small (2–4B). For MoE models, active parameter counts are retained in the model names. The best Toy Game and Real-World results within each group are highlighted in bold ; models are ordered by Overall score.
Figure 3: Capability-wise game-to-real transfer with one-shot in-context learning. Compared with zero-shot evaluation, random game demonstrations improve constraint reasoning on average but reduce fine-grained perception accuracy for all four models on the 400 real-world targets.
Figure 5
Figure 6: Failure cases on Game2Real-Bench. Model reasoning traces are compared with verified solutions for Block Counting, Hexakai, and Mancala. These cases illustrate three recurring failures: misinterpreting spatial structure, inconsistent relational reasoning, and unreliable state updating.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Evaluation
Task
Families
Samples
Open
Choice
Images
C1: Fine-grained Perception
Missing Tiles
3
518
45
18
1, 7
Hidden Object
4
720
47
29
1, 2
Dot Shape
3
480
0
24
7
Color Swap
4
600
0
15
7
Appendix
Table 2: Task inventory and sample counts. Dataset columns report QA families and samples in Game2Real-Dataset, while evaluation columns report open-ended and single-choice samples in the toy-game set. Images reports the number of visual inputs per evaluation sample, including separately provided answer candidates.
Task
QA-family coverage
C1: Fine-grained Perception
Missing Tiles
Complete connector patterns; side states; port lists and degrees.
Hidden Object
Whole-scene and region-specific counts across target classes.
Dot Shape
Contour matching; vertex counts; counts by contour layer.
Color Swap
Transformed models; full or partial color histograms; changed and unchanged piece counts.
C2: Spatial Reasoning
Appendix
Table 3: QA-family coverage by task. Each row summarizes the query targets represented in Game2Real-Dataset. The number of families per task is reported in Table 2 .
Option
Candidate grid
Relation to target
Valid
A
6 6 4 / 1 3 2 / 4 2 5
Exact arrangement
Yes
B
5 2 4 / 2 3 1 / 4 6 6
180∘ rotation
No
C
4 5 4 / 6 1 1 / 2 6 3
Final Bottom grid
No
D
4 2 5 / 6 3 2 / 6 1 4
270∘ rotation
No
E
4 1 6 / 2 3 6 / 5 2 4
90∘ rotation
No
F
4 6 6 / 2 3 1 / 5 2 4
Left–right reflection
No
Appendix
Table 4: Candidate verification for Rolling Cube. The options are taken from Game2Real-Dataset sample GBT-COT-004927 ; slashes separate grid rows. Rotations are clockwise relative to the recomputed Top grid. Only A agrees at every position under the specified reading convention.
Source
N
Construction
GQA ( Hudson and Manning, 2019 )
12
10 original QA; 2 lightly reformulated.
OmniSpatial ( Jia et al., 2026 )
61
Original visual and spatial reasoning QA.
SPair-71k ( Min et al., 2019 )
27
QA derived from keypoint correspondences.
Visual-Jigsaw ( Wu et al., 2026 )
40
Lightly reformulated completion questions.
CAPTURe-real ( Pothiraj et al., 2025 )
30
Lightly reformulated counting questions.
All-Angles-Bench ( Yeh et al., 2026 )
20
Original synchronized-view questions.
Appendix
Table 5: Real-world sources and construction. Counts refer to retained Game2Real-Bench samples. Original QA is kept where suitable; light reformulation preserves source answers, while annotation-derived QA introduces questions grounded in source annotations.
Pre-step frames
Post-step frames
Time (s)
Label
Time (s)
Label
191.60
F2
256.48
F8
192.60
F7
257.48
F1
193.60
F5
258.48
F3
194.60
F6
259.48
F4
Appendix
Table 6: Frame selection for the IKEA applaro example. Timestamps are measured in seconds from the beginning of the source video. Each group contains four frames; labels identify their shuffled positions after the instruction diagram in the final nine-image input.
Setting
Value
Setting
Value
Backend
vLLM
Temperature
0
Sampling
Disabled
Context limit
65,536
Top- p / top- k
0.95 / 20
Presence penalty
1.5
Thinking budget
32k tokens
Answer budget
512 tokens
Custom prompts
Disabled
Answer stage
Conditional
Appendix
Table 7: Configuration of the controlled Qwen in-context learning runner. Paired zero-shot and one-shot runs share these inference settings. In thinking mode, the additional answer budget is used only when the initial budget is exhausted without a final answer.
Figure 13
Subset or answer type
Samples
Scoring criterion
Single-choice subsets
236
Exact option-label match.
GQA short answers
12
Normalized short-answer match.
CAPTURe-real counts
30
Exact count correctness.
VisualTrans and IKEA frame selection
8
Exact selected labels, ignoring order.
SSI-Bench rankings
50
Exact match of the full ranking.
ReasonMap routes
14
Match to an accepted structured route.
Appendix
Table 8: Scoring rules for the 400 real-world samples. Counts refer to the final selected set. All subsets use binary correctness except SpatiaLQA, which contributes a normalized graph score.
Toy task
Toy
Real
Real-world sources
Pool
C1: Fine-grained Perception
Missing Tiles
63
40
Visual-Jigsaw (40)
12
Hidden Object
76
21
OmniSpatial (21)
12
Dot Shape
24
27
SPair-71k (27)
12
Color Swap
15
12
GQA (12)
12
C2: Spatial Reasoning
Appendix
Table 9: Task correspondence and demonstration-pool composition. Toy and Real report evaluation counts; source counts in parentheses sum to Real in each row. Pool reports demonstration counts. Target-side primary task labels define these associations, independently of ICL assignments; zero denotes an absent primary alignment.
Model
Toy
Real
Model
Toy
Real
GPT-5.6-Sol
73.90
59.4750
Gemma4-12B-it
20.60
30.3300
Gemini-3.1-Pro-Preview
59.00
61.2750
GLM-4.6V-Flash
17.20
34.7400
Claude-Opus-4.8-thinking
55.00
50.4500
Ovis2.5-9B
17.00
28.2300
Qwen3.8-27B
50.30
58.2000
InternVL3.5-8B
15.10
28.0250
Qwen3.5-27B
42.40
51.1500
Qwen3-VL-8B-Instruct
13.90
28.1250
Gemma4-31B-it
34.70
47.8700
MiMo-VL-7B-RL
14.40
24.2750
Appendix
Table 10: The 25 paired scores used for cross-setting correlation. Values are the stored percentage scores used by the plotting script. Each model appears once; the two blocks are a compact presentation of the same 25-observation analysis.
Real-world aggregate or subset
Samples
Pearson r
95% CI
All samples, sample mean
400
0.960
[0.913,0.981]
All sources, source mean
400
0.964
[0.902,0.987]
Without SpatiaLQA, sample mean
350
0.946
[0.869,0.979]
Without SpatiaLQA, capability mean
350
0.933
[0.837,0.972]
Single-choice only, sample mean
236
0.939
[0.860,0.975]
Non-single-choice, sample mean
164
0.857
[0.719,0.929]
Appendix
Table 11: Sensitivity to real-world aggregation and answer format. All rows use the same 22 open-weight models and fixed toy-game capability-balanced scores. Sample means weight retained samples equally; capability means weight the four capability means equally. Source means weight the 11 source means equally. Intervals use 10,000 paired model-bootstrap replicates.
Removed source
Samples left
Sample mean r
Capability mean r
All-Angles-Bench
380
0.961
0.963
CAPTURe-real
370
0.958
0.960
GQA
388
0.959
0.959
IKEA Manuals at Work
395
0.956
0.956
OmniSpatial
339
0.965
0.964
ReasonMap
386
0.959
0.958
Appendix
Table 12: Leave-one-source-out correlations. Each row removes one real-world source while retaining all 22 models and the same toy-game capability-balanced scores. The two right columns differ only in how the remaining real-world samples are aggregated. The retained predictions and scoring rules are unchanged, allowing comparison under the two real-world aggregation rules.
Figure 7: Conversation structure for one-shot evaluation. Italic text denotes sample-specific content, not additional literal instructions. The demonstration’s final-answer instruction is reproduced verbatim; the target uses the unchanged benchmark prompt.
Model
Setting
C1
C2
C3
C4
Overall-400
Qwen3.5-2B
Zero-shot
41.00
23.00
16.51
47.00
31.88
1-shot seed 11
36.00
20.00
22.22
34.00
28.05
1-shot seed 23
38.00
23.00
22.08
41.00
31.02
1-shot seed 37
39.00
20.00
22.22
35.00
29.06
1-shot mean
37.67
21.00
22.17
36.67
29.38
Qwen3.5-4B
Zero-shot
59.00
38.00
28.87
54.00
44.97
Appendix
Table 13: Zero-shot and per-assignment one-shot scores. C1–C4 denote fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. Overall-400 averages the four equally sized target groups. Scores are percentages. Bold rows report the three-assignment means; the remaining rows show the zero-shot baseline or one-shot scores for seeds 11, 23, and 37.
Model
Mean Δ
SD
Observed Δ range
Gains / seeds
Qwen3.5-2B
−2.50
1.51
[−3.83,−0.86]
0/3
Qwen3.5-4B
−4.01
2.10
[−5.92,−1.76]
0/3
Qwen3.5-9B
+1.49
2.26
[−1.09,+3.14]
2/3
Qwen3.8-27B
−0.33
0.92
[−1.39,+0.31]
2/3
Appendix
Table 14: Overall changes and assignment variability. Δ is relative to the same model’s zero-shot score. SD is the sample standard deviation across the three one-shot scores. Changes and SD are in percentage points. Ranges are observed minima and maxima, not confidence intervals; the last column counts how many of the three assignments improve over the zero-shot baseline.
Model
Non-thinking
Main evaluation
Gain
Qwen3.5-2B
13.07
19.30
+6.23
Step3-VL-10B
18.25
24.20
+5.95
Gemma4-31B-it
27.41
38.50
+11.09
GPT-5.6-Sol
28.27
69.80
+41.53
Appendix
Table 15: Overall scores under different inference settings. Both columns use the full 1,400-sample benchmark. Main evaluation reports the scores from the main experiments; Gain is its difference from Non-thinking in percentage points.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
While Large Vision-Language Models (LVLMs) exhibit strong perceptual capabilities, they remain vulnerable in visual reasoning tasks. Existing benchmarks largely focus on symbolic mathematical or scientific problems and simple vision-centric tasks, offering limited assessment of complex visual reasoning and logical consistency, a critical requirement for reliable reasoning systems. We introduce ConVBench, a complex vision-centric reasoning benchmark in which each image is paired with two logically equivalent questions across six categories: action and state, complex counting, spatial reasoning, causal and intent understanding, commonsense reasoning, and temporal perception. To complement this benchmark, we define two evaluation metrics, logical consistency and robust accuracy, that jointly assess both the correctness and consistency of model responses. We further present ConVLM, which improves LVLM reasoning through Group Relative Policy Optimization (GRPO)-based reinforcement learning with a novel consistency reward. This method leverages automatically generated logically equivalent question-answer pairs and a dual-reward design combining accuracy- and consistency-based signals, encouraging agreement between paired responses. The framework functions effectively with or without strict answer supervision.
Liqiang Jing, Xiong Zhou, Siddharth Varia +3
University of Texas at Dallas · Amazon Web Services
What does it take to build a visual reasoner that works across charts, science, spatial understanding, and open-ended tasks? The strongest vision-language models (VLMs) suggest that broad visual reasoning is within reach, yet their closed data and reinforcement learning (RL) pipelines make their gains difficult to study, reproduce, or extend. We introduce Vero, a family of fully open VLMs that match or exceed existing open-weight models across diverse visual reasoning tasks. We scale RL data and rewards across six broad task categories, constructing Vero-600K, a 600K-sample dataset from 59 datasets, and designing task-routed rewards that handle heterogeneous answers. Across VeroEval, our 30-benchmark suite, Vero-600K outperforms existing RL datasets under controlled comparisons. Applied to five starting models, Vero variants gain 2.9-5.4 points on average over their initial models. Notably, Vero-Qwen3I-8B, trained on the Instruct model, surpasses Qwen3-VL-8B-Thinking by 3.8 points on average without additional distillation. Systematic ablations reveal that different task categories elicit distinct reasoning patterns and that broad gains depend on learning them jointly rather than in isolation. All data, code, and models are publicly available.
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences