Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · CCDS, Nanyang Technological University · Institute of Artificial Intelligence, Xiamen University
Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.
Figures & tables
Figure 1: RoTA-Bench overview. The benchmark contains 674 canonical tasks, with 100 test tasks selected by stratified sampling over reference path lengths. In the T-shirt example, the solver chooses discrete rotations from visual observations and receives updated views until projection IoU reaches 0.90 or another stopping rule applies. The right panel shows task success rates and final IoU on the test set; full results are reported in Table 1 .
Figure 2: RoTA-Bench construction and evaluation pipeline. (a) (1) Source 2D icons are (2) filtered, sampled, and normalized, then (3) converted into 3D objects through sphere-bounded extrusion. (4) Reversible initialization generates an initial pose with a feasible solution. (5) Pose-guided beam search and 30∘ -restricted breadth-first search provide verified reference paths, which are used to select 100 test tasks from 674 canonical tasks by stratified sampling over reference path lengths. (6) The action space contains 24 discrete rotations across three axes, two directions, and four angles. (b) In closed-loop evaluation, the solver selects rotations from visual observations and interaction history, then receives updated observations. Success requires projection IoU ≥0.90 .
Task Alignment Performance
Pose Action Decomposition
Method
Succ. (%)
Final IoU
Steps
Cov. (%)
Eff. dir. (%)
Best ang. (%)
Gap ( ∘ )
Loss ( ∘ )
Pose greedy
100.0
0.924
8.32
–
100.0
100.0
0.00
0.00
IoU greedy
39.0
0.725
16.49
–
63.86
40.46
15.45
8.00
Human
100.0
0.923
16.22
83.07
83.91
71.82
2.99
0.96
ReAct
16.0
0.628
24.62
79.08
59.59
58.01
7.44
2.81
Gemini 3.6 Flash
10.0
0.607
24.68
74.72
61.87
61.69
5.47
1.99
Table 1: Task alignment performance and action decomposition on 100 test tasks, sorted by Final IoU. Metrics include Task Success Rate (Succ.), final projection IoU, executed steps, visual surface coverage (Cov.), Effective Direction Rate (Eff. dir.), and for effective directions: Best Legal Angle Rate (Best ang.), Angle gap, and Angle loss (geometric penalty from sub-optimal step size). SpatioLM abbreviates SpatioLM-Understanding-InternVL3.5.
Figure 3: Mean IoU versus executed rotations for all methods with exported step trajectories on the same 100 tasks, including the 400 Human task runs and both extension runs. Each marker is the mean at one integer step (0–48), joined without smoothing; terminal IoU is retained after termination through step 48. The dashed line marks the success threshold.
Success
Final
Effective
Rotation angle (%)
IoU–angle correlation
Condition
(%)
IoU
direction (%)
1∘
5∘
10∘
30∘
ρ
95% CI
A: Full input
16.67
0.5656
56.52
12.18
7.51
17.56
62.75
-0.472
[-0.626, -0.272]
B: No aux. view
3.33
0.5490
55.90
3.54
3.24
7.08
86.14
-0.463
[-0.605, -0.238]
D: Neither input
6.67
0.5087
55.02
1.29
1.29
0.81
96.60
-0.295
[-0.468, -0.066]
C: No history
0.00
0.4822
54.85
0.16
0.49
2.59
96.76
-0.242
[-0.331, -0.131]
Uniform random
0.00
0.4808
53.12
24.24
25.27
27.10
23.39
-0.027
[-0.073, 0.019]
Table 2: Input ablation for GPT-5.6-Sol on 30 test tasks, with ablation conditions sorted by Final IoU. Columns report Task Success Rate (Succ.), final projection IoU, Effective Direction Rate (Eff. dir.), percentage distribution across the four legal rotation angles, and Spearman correlation ( ρ ) between pre-action IoU and rotation angle magnitude with 95% confidence intervals.
Figure 4: Absolute Spearman correlations between selected rotation angles and pre-action projection IoU or 3D pose error across the 100 test tasks. Bars show ∣ρ∣ ; whiskers show the original 95% bootstrap intervals mapped to absolute values. An interval crossing zero has a lower bound of zero after this transformation. SpatioLM abbreviates SpatioLM-Understanding-InternVL3.5.
Figure 5: Context and action outcomes across 211 matched states. Prior rotations and coverage are measured before each matched action. UC, DC, UF, and DF denote the four combinations of IoU increasing or decreasing and pose error decreasing or increasing, respectively.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Search steps
Tasks
GPT-5.6-Sol successes
GPT-5.6-Sol success rate
4–6 steps
4
8
3
12/55 (21.8%)
5
19
3
6
28
6
7–16 steps
7
19
2
5/45 (11.1%)
8
11
1
9
6
2
Appendix
Table 3: Test-task counts and GPT-5.6-Sol successes by Search-reference length, with success rates for the 4–6-step and 7–16-step groups.
Setting
Shared protocol
Tasks
Same 100-task test set; each participant completes one consecutive 20-task block
Current observations
Target projection, auxiliary view, current projection, and chronological history
Rotation actions
Six signed directions with 1∘ , 5∘ , 10∘ , or 30∘
Action representation
Human button selection and model JSON both map to the same action token and angle
Success criterion
Projection IoU ≥0.90
Stopping rules
Success, STOP, 48 decisions, or 12 rotations without a new best IoU
Appendix
Table 4: Shared human and model evaluation protocol. The interaction environment, visible observations, action space, and task rules are identical; humans use buttons to submit the same actions that models return in JSON.
Participant
Pass
Task block
Successes/tasks
Success (%)
Final IoU
Rotations
1
1
1
20/20
100.00
0.9198
13.05
2
1
2
20/20
100.00
0.9190
11.90
3
1
3
20/20
100.00
0.9162
13.15
4
1
4
20/20
100.00
0.9172
12.05
5
1
5
20/20
100.00
0.9246
13.20
6
2
1
20/20
100.00
0.9283
18.85
Appendix
Table 5: Human results for four passes over the 100-task test set. Each participant completed one consecutive 20-task block; participants with the same task-block index evaluated the same 20 tasks in different passes.
Figure 6: Human evaluation interface with the same prompt content, current observation, historical observations, and discrete action space used by the models. Humans submit actions through buttons that map to the model action tokens and angles.
Comparison
Tasks
Percentage
Search reference shorter
585
86.8%
Equal length
89
13.2%
Search reference longer
0
0.0%
Inverse mean / median / range: 11.04 / 11 / 6–16
Search mean / median / range: 6.54 / 6 / 2–16
Appendix
Table 6: Search-reference and inverse-initialization path lengths across 674 tasks. Counts and percentages compare the paired lengths; summary statistics report the mean, median, and range in rotation steps.
Reference rotations
All tasks
Test tasks
<4
32
0
4
53
8
5
122
19
6
181
28
7
124
19
8
69
11
Appendix
Table 7: Task counts in the 674-task dataset and 100-task test set by Search-reference length. Tasks with at least four reference rotation steps are eligible for test selection.
Method
Prior rotations
Prior coverage
Δ IoU
UC
DC
UF
DF
−Δ pose ( ∘ )
Human
2.40
42.01%
0.0207
50.24%
33.18%
10.90%
5.69%
10.0420
GPT-5.6-Sol
4.18
46.21%
0.0071
34.60%
23.70%
19.43%
22.27%
2.8740
Appendix
Table 8: Context before matched actions and their outcomes across 211 participant–task–state pairs. Prior rotations count completed rotations before each action. Prior coverage is cumulative direct surface coverage up to and including the current observation. All values are averaged over pairs. UC improves IoU and reduces pose error; DC reduces pose error only; UF improves IoU only; DF improves neither.
Method
Final pose error ( ∘ )
Pose-impr. steps (%)
Mean Δ IoU / step
Mean pose red. / step ( ∘ )
Human
4.44
77.88
0.0273
5.4238
IoU oracle
79.06
54.82
0.0280
2.8687
Pose oracle
1.67
100.00
0.0535
10.9053
Uniform random
102.11
48.94
-0.0003
-0.4365
Random + IoU rollback
88.83
52.14
0.0018
0.1002
ReAct
82.69
52.84
0.0060
0.3944
Appendix
Table 9: Additional pose and trajectory statistics on the same 100 tasks. Final pose error is the symmetry-adjusted angular distance from the target at the end of a trajectory. Pose-improving steps are the percentage of all executed steps that reduce pose error. Mean Δ IoU per step and mean pose reduction per step are computed over complete trajectories; positive values indicate progress toward the target. Human combines 400 task runs from 20 undergraduate participants, and random policies average three seeds within each task. SpatioLM-InternVL3.5 abbreviates SpatioLM-Understanding-InternVL3.5.
Method
IoU–angle ρ (95% CI)
Pose–angle ρ (95% CI)
Human
−0.696 [ −0.763,−0.633 ]
0.751 [ 0.675,0.828 ]
ReAct
−0.570 [ −0.674,−0.453 ]
0.320 [ 0.108,0.512 ]
Gemini 3.6 Flash
−0.562 [ −0.633,−0.480 ]
0.618 [ 0.496,0.709 ]
GPT-5.6-Sol
−0.536 [ −0.617,−0.443 ]
0.368 [ 0.166,0.532 ]
GPT-6-Astra
−0.533 [ −0.621,−0.427 ]
0.452 [ 0.266,0.607 ]
GPT-5.6-Terra
−0.438 [ −0.526,−0.335 ]
0.254 [ 0.056,0.440 ]
Appendix
Table 10: Spearman rank correlations ( ρ ) of selected rotation angles with pre-action projection IoU (IoU–angle) and 3D pose error (Pose–angle) across the 100 test tasks, sorted by Final IoU. Brackets report 95% bootstrap confidence intervals.
Method
Successes/tasks
Success (%)
95% CI (%)
ReAct
16/100
16.00
[10.10, 24.42]
Gemini 3.6 Flash
10/100
10.00
[5.52, 17.44]
GPT-5.6-Sol
17/100
17.00
[10.89, 25.55]
GPT-6-Astra
10/100
10.00
[5.52, 17.44]
GPT-5.6-Terra
8/100
8.00
[4.11, 15.00]
GPT-5.6-Luna
4/100
4.00
[1.57, 9.84]
Appendix
Table 11: Success counts and 95% Wilson confidence intervals for models and agents on the 100 test tasks. Rows follow the order in Table 1 .
Method
Success
Stagnation
STOP
Limit
Repeat (%)
Human-independent model methods
GPT-5.6-Sol
17
78
3
2
53.35
ReAct
16
75
5
4
49.41
Gemini 3.6 Flash
10
83
2
5
23.90
GPT-6-Astra
10
85
1
4
53.65
GPT-5.6-Terra
8
84
2
6
53.90
Appendix
Table 12: Episode outcomes and repeated actions on 100 tasks per method. The four outcome columns sum to 100. Repeat is measured over consecutive rotation pairs.
Method
Legal JSON
retries = 0 (%)
≤ 2 (%)
≤ 6 (%)
GPT-5.6-Sol
2369
99.87
100.00
100.00
GPT-5.6-Luna
2432
92.27
100.00
100.00
GPT-5.6-Terra
2293
100.00
100.00
100.00
Gemini 3.6 Flash
2470
99.92
100.00
100.00
Claude Opus 5
2212
99.55
100.00
100.00
Claude Sonnet 5
1944
100.00
100.00
100.00
Appendix
Table 13: Distribution of successfully returned actions by the number of retries needed to obtain valid JSON, over saved trajectories. Only responses that contain output tokens count as valid returns; transmission failures (no returned tokens) are excluded. “retries ≤k ” is the cumulative share of actions produced within k retries (i.e. on the k+1 -th content-bearing response).
Method/comparison
Effective Direction Rate (%)
Angle Gap ( ∘ )
Human
83.91 [82.63, 85.22]
2.99 [2.70, 3.30]
GPT-5.6-Sol
58.75 [56.62, 60.87]
6.49 [5.38, 7.64]
Human minus Sol
25.16 [22.79, 27.56]
−3.50 [ −4.60 , −2.42 ]
Appendix
Table 14: Action metrics on 100 tasks, with paired 95% bootstrap intervals. Angle Gap uses effective directions only. Differences are Human minus Sol; direction differences are in percentage points.
Input
Successes
Success (%; 95% CI)
1∘
5∘
10∘
30∘
A: Full
5/30
16.67 [7.34, 33.56]
86
53
124
443
B: No auxiliary
1/30
3.33 [0.59, 16.67]
24
22
48
584
C: No history
0/30
0.00 [0.00, 11.35]
1
3
16
598
D: Neither
2/30
6.67 [1.85, 21.32]
8
8
5
597
Appendix
Table 15: Successes on 30 tasks, with Wilson 95% intervals. The last four columns count rotations at each angle.
Comparison
Success difference
Mean Final IoU difference
A minus B
13.33 [0.00, 30.00]
0.0166 [ −0.0598 , 0.0961]
A minus C
16.67 [3.33, 30.00]
0.0834 [0.0255, 0.1477]
A minus D
10.00 [0.00, 20.00]
0.0568 [0.0138, 0.1107]
Appendix
Table 16: Full input minus each ablation on the same 30 tasks, with paired 95% bootstrap intervals. Success differences are in percentage points.
Policy
Successor error ( ∘ )
Improvement ( ∘ )
Sol actual action
91.99
0.00
Uniform random
93.60
-1.61
Oracle direction
76.38
15.61
Oracle angle
86.29
5.70
Joint oracle
72.00
19.99
Appendix
Table 17: Oracle decomposition on 2,366 GPT-5.6-Sol states. Successor pose errors and improvements are averaged over states. For Uniform random, the successor error is the average over all 24 legal actions in each state. Oracle actions are selected by minimizing the symmetry-adjusted successor pose error.
Task set
Success
Mean final IoU
Mean steps
Icon-derived tasks
5/30 (16.7%)
0.5656
23.53
Poly Haven meshes
6/30 (20.0%)
0.7179
22.17
Appendix
Table 18: GPT-5.6-Sol on two 30-task sets with exactly matched reference-path length distributions. The icon-derived set is the original ablation subset; the comparison set uses 30 distinct Poly Haven CC0 meshes based on real-world object categories.
Component
Setting
Rendering
Python 3.11.14; NumPy 2.3.3; OpenCV 4.13.0; PyTorch 2.10.0+cu130; CUDA 13.0
SpatioLM-Understanding-InternVL3.5: bfloat16 , greedy decoding ( do_sample=False ), official architecture; local HTTP server
Response format
JSON; no streaming
Appendix
Table 19: Software versions, hardware specifications, model hyperparameters, agent configurations, and interaction settings used for evaluation.
Condition
Input/output rate
Input tokens
Output tokens
Total cost (USD)
GPT-5.6-Sol
5.00 / 30.00
28,883,151
853,753
170.0283
GPT-5.6-Terra
2.50 / 15.00
27,905,743
1,062,892
85.7077
GPT-5.6-Luna
1.00 / 6.00
22,630,532
1,087,401
29.1549
GPT-6-Astra
10.00 / 50.00
29,975,049
1,187,433
359.1221
Gemini 3.6 Flash
1.50 / 7.50
45,392,248
4,555,620
102.2555
Claude Opus 5
5.00 / 25.00
17,014,514
700,389
102.5823
Appendix
Table 20: Published API prices, recorded token counts, and estimated costs. Main evaluations use 100 tasks; the three GPT-5.6-Sol input ablations use the same 30-task subset. Rates are USD per million tokens using uncached input pricing; costs are in USD.
Figure 7: First-turn prompt for single_00627, showing the initial observation, user input, and an example JSON response.
Figure 8: Second-turn prompt for single_00627, showing the first observation and action–COT record, current observation, user input, and an example JSON response.
Figure 9: Third-turn prompt for single_00627, showing both earlier observations and action–COT records, current observation, user input, and an example JSON response.
Figure 10: GPT-5.6-Sol’s successful trajectory on single_00270: rotation steps 1–4 of 8, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.
Figure 11: GPT-5.6-Sol’s successful trajectory on single_00270: rotation steps 5–8 of 8 and the final state, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.
Figure 12: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 1–4 of 12, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.
Figure 13: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 5–8 of 12, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.
Figure 14: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 9–12 of 12 and the final state, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
Current Large Language Models have achieved Olympiad-level logic, yet Vision-Language Models paradoxically falter on elementary spatial tasks like block counting. This capability mismatch reveals a critical spatial intelligence gap,'' where models fail to construct coherent 3D mental representations from 2D observations. We uncover this gap via diagnostic analyses showing the bottleneck is a missing view-consistent spatial interface rather than insufficient visual features or weak reasoning. To bridge this, we introduce **3ViewSense**, a framework that grounds spatial reasoning in Orthographic Views. Drawing on engineering cognition, we propose a Simulate-and-Reason'' mechanism that decomposes complex scenes into canonical orthographic projections to resolve geometric ambiguities. By aligning egocentric perceptions with these allocentric references, our method facilitates explicit mental rotation and reconstruction. Empirical results on spatial reasoning benchmarks demonstrate that our method significantly outperforms existing baselines, with consistent gains on occlusion-heavy counting and view-consistent spatial reasoning. The framework also improves the stability and consistency of spatial descriptions, offering a scalable path toward stronger spatial intelligence in multimodal systems.~\footnote{https://github.com/Jasaxion/3ViewSense}
Shaoxiong Zhan, Yanlin Lai, Zheng Liu +6
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Software Engineering, Chongqing University, Chongqing, China
Are current Vision Language Models (VLMs) ready to comprehend and reason about complex embodied interactions in 3D environments? We introduce Embodied3DBench, a robot-centric benchmark targeting low-level spatial intelligence in embodied 3D environments. To systematically evaluate these foundational perceptual capabilities, the benchmark includes 6 task categories divided into two core groups: Spatial Structural Understanding (Grounding, Spatial Relation Prediction, and Multi-view Correspondence) and Interaction-Oriented Perception (Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction). The benchmark spans 12 subcategories and contains over 21k high-quality question-answer pairs. We evaluate 13 state-of-the-art models, and the results show that while current models exhibit relatively strong high-level spatial reasoning, such as understanding object-to-object positional relations, they remain fragile in interaction-oriented perception, highlighting a significant lack of robust 3D-aware interaction priors. To actively bridge this capability gap revealed by our benchmark, we further synthesize a large-scale training dataset comprising 1.3M QA pairs. Notably, fine-tuning on this dataset yields significant improvements in low-level spatial intelligence. Ultimately, Embodied3DBench fills a critical gap by providing both a systematic evaluation framework and a scalable data solution, setting a clear target for the development of interaction-aware multimodal systems.
Jiyao Zhang, Mingxu Zhang, Yitong Peng +8
CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2
Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.
Xianzheng Ma, Tao Sun, Shuai Chen +7
VGG, University of Oxford · AVL, University of Oxford · Stanford University +2