RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

Qingchuan Ma; Yuexiao Ma; Xiang Meng; Yixuan Zou; Tianyu Xie; Wang Chen; Yuhui Zeng; Yuhang Wu

cs.AIOct 13, 2026

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

Authors: Qingchuan Ma, Yuexiao Ma, Xiang Meng, Yixuan Zou, Tianyu Xie, Wang Chen, Yuhui Zeng, Yuhang Wu,

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · CCDS, Nanyang Technological University · Institute of Artificial Intelligence, Xiamen University

Abstract

Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.

Figures & tables

Figure 1: RoTA-Bench overview. The benchmark contains 674 canonical tasks, with 100 test tasks selected by stratified sampling over reference path lengths. In the T-shirt example, the solver chooses discrete rotations from visual observations and receives updated views until projection IoU reaches $0.90$ or another stopping rule applies. The right panel shows task success rates and final IoU on the test set; full results are reported in Table 1 .

Figure 2: RoTA-Bench construction and evaluation pipeline. (a) (1) Source 2D icons are (2) filtered, sampled, and normalized, then (3) converted into 3D objects through sphere-bounded extrusion. (4) Reversible initialization generates an initial pose with a feasible solution. (5) Pose-guided beam search and $30^{\circ}$ -restricted breadth-first search provide verified reference paths, which are used to select 100 test tasks from 674 canonical tasks by stratified sampling over reference path lengths. (6) The action space contains 24 discrete rotations across three axes, two directions, and four angles. (b) In closed-loop evaluation, the solver selects rotations from visual observations and interaction history, then receives updated observations. Success requires projection IoU $\geq 0.90$ .

	Task Alignment Performance				Pose Action Decomposition
Method	Succ. (%)	Final IoU	Steps	Cov. (%)	Eff. dir. (%)	Best ang. (%)	Gap ( ∘ )	Loss ( ∘ )
Pose greedy	100.0	0.924	8.32	–	100.0	100.0	0.00	0.00
IoU greedy	39.0	0.725	16.49	–	63.86	40.46	15.45	8.00
Human	100.0	0.923	16.22	83.07	83.91	71.82	2.99	0.96
ReAct	16.0	0.628	24.62	79.08	59.59	58.01	7.44	2.81
Gemini 3.6 Flash	10.0	0.607	24.68	74.72	61.87	61.69	5.47	1.99

Table 1: Task alignment performance and action decomposition on 100 test tasks, sorted by Final IoU. Metrics include Task Success Rate (Succ.), final projection IoU, executed steps, visual surface coverage (Cov.), Effective Direction Rate (Eff. dir.), and for effective directions: Best Legal Angle Rate (Best ang.), Angle gap, and Angle loss (geometric penalty from sub-optimal step size). SpatioLM abbreviates SpatioLM-Understanding-InternVL3.5.

Panel of Figure 3 — Figure 3: Mean IoU versus executed rotations for all methods with exported step trajectories on the same 100 tasks, including the 400 Human task runs and both extension runs. Each marker is the mean at one integer step (0–48), joined without smoothing; terminal IoU is retained after termination through step 48. The dashed line marks the success threshold.

	Success	Final	Effective	Rotation angle (%)				IoU–angle correlation
Condition	(%)	IoU	direction (%)	$1^{\circ}$	$5^{\circ}$	$10^{\circ}$	$30^{\circ}$	$\rho$	95% CI
A: Full input	16.67	0.5656	56.52	12.18	7.51	17.56	62.75	-0.472	[-0.626, -0.272]
B: No aux. view	3.33	0.5490	55.90	3.54	3.24	7.08	86.14	-0.463	[-0.605, -0.238]
D: Neither input	6.67	0.5087	55.02	1.29	1.29	0.81	96.60	-0.295	[-0.468, -0.066]
C: No history	0.00	0.4822	54.85	0.16	0.49	2.59	96.76	-0.242	[-0.331, -0.131]
Uniform random	0.00	0.4808	53.12	24.24	25.27	27.10	23.39	-0.027	[-0.073, 0.019]

Table 2: Input ablation for GPT-5.6-Sol on 30 test tasks, with ablation conditions sorted by Final IoU. Columns report Task Success Rate (Succ.), final projection IoU, Effective Direction Rate (Eff. dir.), percentage distribution across the four legal rotation angles, and Spearman correlation (

\rho

) between pre-action IoU and rotation angle magnitude with 95% confidence intervals.

Panel of Figure 4 — Figure 4: Absolute Spearman correlations between selected rotation angles and pre-action projection IoU or 3D pose error across the 100 test tasks. Bars show $|\rho|$ ; whiskers show the original 95% bootstrap intervals mapped to absolute values. An interval crossing zero has a lower bound of zero after this transformation. SpatioLM abbreviates SpatioLM-Understanding-InternVL3.5.

Panel of Figure 5 — Figure 5: Context and action outcomes across 211 matched states. Prior rotations and coverage are measured before each matched action. UC, DC, UF, and DF denote the four combinations of IoU increasing or decreasing and pose error decreasing or increasing, respectively.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Search steps	Tasks	GPT-5.6-Sol successes	GPT-5.6-Sol success rate
4–6 steps	4	8	3	12/55 (21.8%)
	5	19	3
	6	28	6
7–16 steps	7	19	2	5/45 (11.1%)
	8	11	1
	9	6	2

Appendix

Table 3: Test-task counts and GPT-5.6-Sol successes by Search-reference length, with success rates for the 4–6-step and 7–16-step groups.

Setting	Shared protocol
Tasks	Same 100-task test set; each participant completes one consecutive 20-task block
Current observations	Target projection, auxiliary view, current projection, and chronological history
Rotation actions	Six signed directions with $1^{\circ}$ , $5^{\circ}$ , $10^{\circ}$ , or $30^{\circ}$
Action representation	Human button selection and model JSON both map to the same action token and angle
Success criterion	Projection IoU $\geq 0.90$
Stopping rules	Success, STOP, 48 decisions, or 12 rotations without a new best IoU

Appendix

Table 4: Shared human and model evaluation protocol. The interaction environment, visible observations, action space, and task rules are identical; humans use buttons to submit the same actions that models return in JSON.

Participant	Pass	Task block	Successes/tasks	Success (%)	Final IoU	Rotations
1	1	1	20/20	100.00	0.9198	13.05
2	1	2	20/20	100.00	0.9190	11.90
3	1	3	20/20	100.00	0.9162	13.15
4	1	4	20/20	100.00	0.9172	12.05
5	1	5	20/20	100.00	0.9246	13.20
6	2	1	20/20	100.00	0.9283	18.85

Appendix

Table 5: Human results for four passes over the 100-task test set. Each participant completed one consecutive 20-task block; participants with the same task-block index evaluated the same 20 tasks in different passes.

Figure 6: Human evaluation interface with the same prompt content, current observation, historical observations, and discrete action space used by the models. Humans submit actions through buttons that map to the model action tokens and angles.

Comparison	Tasks	Percentage
Search reference shorter	585	86.8%
Equal length	89	13.2%
Search reference longer	0	0.0%
Inverse mean / median / range: 11.04 / 11 / 6–16
Search mean / median / range: 6.54 / 6 / 2–16

Appendix

Table 6: Search-reference and inverse-initialization path lengths across 674 tasks. Counts and percentages compare the paired lengths; summary statistics report the mean, median, and range in rotation steps.

Reference rotations	All tasks	Test tasks
$<4$	32	0
4	53	8
5	122	19
6	181	28
7	124	19
8	69	11

Appendix

Table 7: Task counts in the 674-task dataset and 100-task test set by Search-reference length. Tasks with at least four reference rotation steps are eligible for test selection.

Method	Prior rotations	Prior coverage	$\Delta$ IoU	UC	DC	UF	DF	$-\Delta$ pose ( ∘ )
Human	2.40	42.01%	0.0207	50.24%	33.18%	10.90%	5.69%	10.0420
GPT-5.6-Sol	4.18	46.21%	0.0071	34.60%	23.70%	19.43%	22.27%	2.8740

Appendix

Table 8: Context before matched actions and their outcomes across 211 participant–task–state pairs. Prior rotations count completed rotations before each action. Prior coverage is cumulative direct surface coverage up to and including the current observation. All values are averaged over pairs. UC improves IoU and reduces pose error; DC reduces pose error only; UF improves IoU only; DF improves neither.

Method	Final pose error ( ∘ )	Pose-impr. steps (%)	Mean $\Delta$ IoU / step	Mean pose red. / step ( ∘ )
Human	4.44	77.88	0.0273	5.4238
IoU oracle	79.06	54.82	0.0280	2.8687
Pose oracle	1.67	100.00	0.0535	10.9053
Uniform random	102.11	48.94	-0.0003	-0.4365
Random + IoU rollback	88.83	52.14	0.0018	0.1002
ReAct	82.69	52.84	0.0060	0.3944

Appendix

Table 9: Additional pose and trajectory statistics on the same 100 tasks. Final pose error is the symmetry-adjusted angular distance from the target at the end of a trajectory. Pose-improving steps are the percentage of all executed steps that reduce pose error. Mean

\Delta

IoU per step and mean pose reduction per step are computed over complete trajectories; positive values indicate progress toward the target. Human combines 400 task runs from 20 undergraduate participants, and random policies average three seeds within each task. SpatioLM-InternVL3.5 abbreviates SpatioLM-Understanding-InternVL3.5.

Method	IoU–angle $\rho$ (95% CI)	Pose–angle $\rho$ (95% CI)
Human	$\mathbf{-0.696}$ [ $-0.763,-0.633$ ]	$\mathbf{0.751}$ [ $0.675,0.828$ ]
ReAct	$-0.570$ [ $-0.674,-0.453$ ]	$0.320$ [ $0.108,0.512$ ]
Gemini 3.6 Flash	$-0.562$ [ $-0.633,-0.480$ ]	$0.618$ [ $0.496,0.709$ ]
GPT-5.6-Sol	$-0.536$ [ $-0.617,-0.443$ ]	$0.368$ [ $0.166,0.532$ ]
GPT-6-Astra	$-0.533$ [ $-0.621,-0.427$ ]	$0.452$ [ $0.266,0.607$ ]
GPT-5.6-Terra	$-0.438$ [ $-0.526,-0.335$ ]	$0.254$ [ $0.056,0.440$ ]

Appendix

Table 10: Spearman rank correlations (

\rho

) of selected rotation angles with pre-action projection IoU (IoU–angle) and 3D pose error (Pose–angle) across the 100 test tasks, sorted by Final IoU. Brackets report 95% bootstrap confidence intervals.

Method	Successes/tasks	Success (%)	95% CI (%)
ReAct	16/100	16.00	[10.10, 24.42]
Gemini 3.6 Flash	10/100	10.00	[5.52, 17.44]
GPT-5.6-Sol	17/100	17.00	[10.89, 25.55]
GPT-6-Astra	10/100	10.00	[5.52, 17.44]
GPT-5.6-Terra	8/100	8.00	[4.11, 15.00]
GPT-5.6-Luna	4/100	4.00	[1.57, 9.84]

Appendix

Table 11: Success counts and 95% Wilson confidence intervals for models and agents on the 100 test tasks. Rows follow the order in Table 1 .

Method	Success	Stagnation	STOP	Limit	Repeat (%)
Human-independent model methods
GPT-5.6-Sol	17	78	3	2	53.35
ReAct	16	75	5	4	49.41
Gemini 3.6 Flash	10	83	2	5	23.90
GPT-6-Astra	10	85	1	4	53.65
GPT-5.6-Terra	8	84	2	6	53.90

Appendix

Table 12: Episode outcomes and repeated actions on 100 tasks per method. The four outcome columns sum to 100. Repeat is measured over consecutive rotation pairs.

Method	Legal JSON	retries $=$ 0 (%)	$\leq$ 2 (%)	$\leq$ 6 (%)
GPT-5.6-Sol	2369	99.87	100.00	100.00
GPT-5.6-Luna	2432	92.27	100.00	100.00
GPT-5.6-Terra	2293	100.00	100.00	100.00
Gemini 3.6 Flash	2470	99.92	100.00	100.00
Claude Opus 5	2212	99.55	100.00	100.00
Claude Sonnet 5	1944	100.00	100.00	100.00

Appendix

Table 13: Distribution of successfully returned actions by the number of retries needed to obtain valid JSON, over saved trajectories. Only responses that contain output tokens count as valid returns; transmission failures (no returned tokens) are excluded. “retries

\leq k

” is the cumulative share of actions produced within

k

retries (i.e. on the

k{+}1

-th content-bearing response).

Method/comparison	Effective Direction Rate (%)	Angle Gap ( ∘ )
Human	83.91 [82.63, 85.22]	2.99 [2.70, 3.30]
GPT-5.6-Sol	58.75 [56.62, 60.87]	6.49 [5.38, 7.64]
Human minus Sol	25.16 [22.79, 27.56]	$-3.50$ [ $-4.60$ , $-2.42$ ]

Appendix

Table 14: Action metrics on 100 tasks, with paired 95% bootstrap intervals. Angle Gap uses effective directions only. Differences are Human minus Sol; direction differences are in percentage points.

Input	Successes	Success (%; 95% CI)	$1^{\circ}$	$5^{\circ}$	$10^{\circ}$	$30^{\circ}$
A: Full	5/30	16.67 [7.34, 33.56]	86	53	124	443
B: No auxiliary	1/30	3.33 [0.59, 16.67]	24	22	48	584
C: No history	0/30	0.00 [0.00, 11.35]	1	3	16	598
D: Neither	2/30	6.67 [1.85, 21.32]	8	8	5	597

Appendix

Table 15: Successes on 30 tasks, with Wilson 95% intervals. The last four columns count rotations at each angle.

Comparison	Success difference	Mean Final IoU difference
A minus B	13.33 [0.00, 30.00]	0.0166 [ $-0.0598$ , 0.0961]
A minus C	16.67 [3.33, 30.00]	0.0834 [0.0255, 0.1477]
A minus D	10.00 [0.00, 20.00]	0.0568 [0.0138, 0.1107]

Appendix

Table 16: Full input minus each ablation on the same 30 tasks, with paired 95% bootstrap intervals. Success differences are in percentage points.

Policy	Successor error ( ∘ )	Improvement ( ∘ )
Sol actual action	91.99	0.00
Uniform random	93.60	-1.61
Oracle direction	76.38	15.61
Oracle angle	86.29	5.70
Joint oracle	72.00	19.99

Appendix

Table 17: Oracle decomposition on 2,366 GPT-5.6-Sol states. Successor pose errors and improvements are averaged over states. For Uniform random, the successor error is the average over all 24 legal actions in each state. Oracle actions are selected by minimizing the symmetry-adjusted successor pose error.

Task set	Success	Mean final IoU	Mean steps
Icon-derived tasks	5/30 (16.7%)	0.5656	23.53
Poly Haven meshes	6/30 (20.0%)	0.7179	22.17

Appendix

Table 18: GPT-5.6-Sol on two 30-task sets with exactly matched reference-path length distributions. The icon-derived set is the original ablation subset; the comparison set uses 30 distinct Poly Haven CC0 meshes based on real-world object categories.

Component	Setting
Rendering	Python 3.11.14; NumPy 2.3.3; OpenCV 4.13.0; PyTorch 2.10.0+cu130; CUDA 13.0
Supporting packages	CairoSVG 2.8.2; cairocffi 1.7.1; Pillow 12.1.1; Numba 0.64.0
API and agent packages	OpenAI 2.48.0; HTTPX 0.28.1; Microsoft Agent Framework core 1.12.1, OpenAI 1.11.0, orchestrations 1.0.1
Local GPU server	8 $\times$ NVIDIA A100-PCIE-40GB; 2 $\times$ Intel Xeon Silver 4314 (32 cores / 64 threads); 256 GB RAM; Driver 615.71.09
Open-source model serving	SpatioLM-Understanding-InternVL3.5: bfloat16 , greedy decoding ( do_sample=False ), official architecture; local HTTP server
Response format	JSON; no streaming

Appendix

Table 19: Software versions, hardware specifications, model hyperparameters, agent configurations, and interaction settings used for evaluation.

Condition	Input/output rate	Input tokens	Output tokens	Total cost (USD)
GPT-5.6-Sol	5.00 / 30.00	28,883,151	853,753	170.0283
GPT-5.6-Terra	2.50 / 15.00	27,905,743	1,062,892	85.7077
GPT-5.6-Luna	1.00 / 6.00	22,630,532	1,087,401	29.1549
GPT-6-Astra	10.00 / 50.00	29,975,049	1,187,433	359.1221
Gemini 3.6 Flash	1.50 / 7.50	45,392,248	4,555,620	102.2555
Claude Opus 5	5.00 / 25.00	17,014,514	700,389	102.5823

Appendix

Table 20: Published API prices, recorded token counts, and estimated costs. Main evaluations use 100 tasks; the three GPT-5.6-Sol input ablations use the same 30-task subset. Rates are USD per million tokens using uncached input pricing; costs are in USD.

Figure 7: First-turn prompt for single_00627, showing the initial observation, user input, and an example JSON response.

Panel of Figure 8 — Figure 8: Second-turn prompt for single_00627, showing the first observation and action–COT record, current observation, user input, and an example JSON response.

Panel of Figure 9 — Figure 9: Third-turn prompt for single_00627, showing both earlier observations and action–COT records, current observation, user input, and an example JSON response.

Panel of Figure 10 — Figure 10: GPT-5.6-Sol’s successful trajectory on single_00270: rotation steps 1–4 of 8, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.

Panel of Figure 11 — Figure 11: GPT-5.6-Sol’s successful trajectory on single_00270: rotation steps 5–8 of 8 and the final state, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.

Panel of Figure 12 — Figure 12: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 1–4 of 12, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.

Panel of Figure 13 — Figure 13: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 5–8 of 12, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.

Panel of Figure 14 — Figure 14: GPT-5.6-Sol’s unsuccessful trajectory on single_00039: rotation steps 9–12 of 12 and the final state, with pre-action observations and original JSON responses. Panels show the target projection, auxiliary view, and current projection from left to right; IoU annotations were not provided to the model.

3D Spatial Reasoning

3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.

See all 202 papers

3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
Mar 8, 2026Shaoxiong Zhan, Yanlin Lai, Zheng Liu +6Visual Spatial Reasoning Spatial Reasoning in Language Models
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Software Engineering, Chongqing University, Chongqing, China
Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
May 27, 2026Jiyao Zhang, Mingxu Zhang, Yitong Peng +8VLM Evaluation Spatial Reasoning Benchmarks
CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2
Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Mar 6, 2026Xianzheng Ma, Tao Sun, Shuai Chen +73D Spatial Reasoning 3D VQA
VGG, University of Oxford · AVL, University of Oxford · Stanford University +2

Mar 8, 2026J/K move · Enter open · S save

3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models

Shaoxiong Zhan, Yanlin Lai, Zheng Liu, Hai Lin+5

Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Software Engineering, Chongqing University, Chongqing, China

Visual Spatial Reasoning Spatial Reasoning in Language Models 3D Spatial Reasoning Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Current Large Language Models have achieved Olympiad-level logic, yet Vision-Language Models paradoxically falter on elementary spatial tasks like block counting. This capability mismatch reveals a critical

spatial intelligence gap,'' where models fail to construct coherent 3D mental representations from 2D observations. We uncover this gap via diagnostic analyses showing the bottleneck is a missing view-consistent spatial interface rather than insufficient visual features or weak reasoning. To bridge this, we introduce **3ViewSense**, a framework that grounds spatial reasoning in Orthographic Views. Drawing on engineering cognition, we propose a

Simulate-and-Reason'' mechanism that decomposes complex scenes into canonical orthographic projections to resolve geometric ambiguities. By aligning egocentric perceptions with these allocentric references, our method facilitates explicit mental rotation and reconstruction. Empirical results on spatial reasoning benchmarks demonstrate that our method significantly outperforms existing baselines, with consistent gains on occlusion-heavy counting and view-consistent spatial reasoning. The framework also improves the stability and consistency of spatial descriptions, offering a scalable path toward stronger spatial intelligence in multimodal systems.~\footnote{https://github.com/Jasaxion/3ViewSense}

Abstract

3D Spatial Reasoning

Explore similar work

3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models

Do 3D Large Language Models Really Understand 3D Spatial Relationships?