Fine-Tuning VLM for Enhancing AI's Spatial Intelligence: Understanding 3D and 2D Rotations

Uttamasha Monjoree; Wei Yan

cs.AIOct 3, 2026

Fine-Tuning VLM for Enhancing AI's Spatial Intelligence: Understanding 3D and 2D Rotations

Authors: Uttamasha Monjoree, Wei Yan

Organizations: Department of Architecture Texas A&M University College Station, USA

Abstract

Spatial intelligence is a fundamental skill in multiple domains, such as Science, Technology, Engineering, and Mathematics (STEM), Medicine, Architecture, and Construction. Recent studies indicate that Vision-Language Models (VLMs) still face limitations in spatial reasoning, which inhibits artificial intelligence (AI) from performing practical spatial tasks. Using multiple object-rotation datasets developed for training and evaluation, our experiments demonstrated promising improvements in both 2D and 3D rotation detection. Fine-tuned Google DeepMind-built Gemma-4 mixture-of-experts (MoE) models significantly outperformed fine-tuned Gemma-4 generalist models in predicting rotations defined by both their axes and angles. Fine-tuning also substantially improved angle estimation for 2D representation without requiring an explicit coordinate system. Furthermore, identifiable objects did not improve angle-detection accuracy; instead, objects with prominent linear features showed improved performance.

Figures & tables

Fig. 1 : Workflow of rotation image generation in Grasshopper

Fig. 2 : Example from 3D voxel dataset (used in Experiment 1.1 & 1.2) where the object is rotating along the X-axis

Panel of Figure 3 — Fig. 3 : Image Examples from Dataset of Identifiable Objects Used in Experiment 2.1.

Panel of Figure 4 — Fig. 4 : Image Examples from Dataset of Arbitrary Objects Used in Experiment 2.2.

Fig. 5 : Multimodal image-text LoRA fine-tuning architecture for angle and axis of rotation prediction.

Model	Baseline					Fine-tuned
Model	Axis Acc.	$\pm 0^{\circ}$	$\pm 3^{\circ}$	$\pm 5^{\circ}$	$\pm 10^{\circ}$	Axis Acc.	$\pm 0^{\circ}$	$\pm 3^{\circ}$	$\pm 5^{\circ}$	$\pm 10^{\circ}$
Generalist	34.0%	0.0%	2.1%	2.1%	4.3%	31.9%	0.0%	8.5%	17.0%	31.9%
MoE	53.0%	6.3%	11.8%	12.5%	18.8%	98.0%	18.8%	58.8%	87.5%	93.8%

TABLE I : Performance comparison of Experiment 1.1 (Generalist) & Experiment 1.2 (MoE) before and after fine-tuning. The

\pm k^{\circ}

columns report accuracy within an angular tolerance of

k

degrees.

Object	Baseline			Fine-tuned Model
Object	$\pm 0^{\circ}$	$\pm 5^{\circ}$	$\pm 10^{\circ}$	$\pm 0^{\circ}$	$\pm 5^{\circ}$	$\pm 10^{\circ}$
Identifiable (Exp 2.1)	2.58%	4.50%	7.30%	62.36%	84.40%	87.86%
Arbitrary (Exp 2.2)	2.49%	3.98%	7.32%	82.15%	85.73%	85.92%

TABLE II : Mean performance comparison of Experiment 2.1 (Identifiable objects) & Experiment 2.2 (Arbitrary objects) before and after fine-tuning.

Panel of Figure 8 — Fig. 6 : Experiment 2.1 Results: Baseline and fine-tuned accuracy for five identifiable objects under angular tolerances of $\pm 0^{\circ}$ , $\pm 5^{\circ}$ , and $\pm 10^{\circ}$ . Dashed lines represent baseline (B) performance, while solid lines represent fine-tuned (FT) performance.

Panel of Figure 9 — Fig. 7 : Experiment 2.2 Results: Baseline and fine-tuned accuracy for five arbitrary objects under angular tolerances of $\pm 0^{\circ}$ , $\pm 5^{\circ}$ , and $\pm 10^{\circ}$ . Dashed lines represent baseline (B) performance, while solid lines represent fine-tuned (FT) performance.

Explore similar work

Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs
Oct 5, 2026Zhaochen Wang, Yujun Cai, Huangbo Zou +4VLM Evaluation Object Pose Estimation
The University of Queensland · Westlake University
How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study
Apr 16, 2026Zhen Yang, Ping Jian, Zhongbin Guo +5Visual Spatial Reasoning Large Vision-Language Models
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, China
Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation
Oct 3, 2026Uttamasha Monjoree, Wei YanSpatial Reasoning Benchmarks 3D Spatial Reasoning
Department of Architecture Texas A&M University College Station, USA

Oct 5, 2026, cs.CVJ/K move · Enter open · S save

Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs

Zhaochen Wang, Yujun Cai, Huangbo Zou, Hower Yang+3

The University of Queensland · Westlake University

VLM Evaluation Object Pose Estimation VLM Reasoning

Open paper AskPDF arXiv

Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much. We introduce OR-Bench, a fine-grained benchmark for object-rotation reasoning with eight tasks covering rotation detection, rotation magnitude estimation, and multi-view rotation reasoning. Across 12 VLMs, the gap is stark: the strongest models approach 100% accuracy on detection, yet even coarse magnitude estimation is near chance. When asked for exact angles, models place 91.8--100% of their predictions on just

0^\circ

90^\circ

, and

180^\circ

, a failure we term canonical-angle collapse. This collapse persists even without visual input. Representation probing shows that missing information is only part of the explanation. Although rotation information becomes less recoverable at finer granularity, substantial coarse-grained information remains, and a simple linear probe outperforms the models' generated answers. This suggests that VLMs underuse rotation information they already encode. We therefore propose RotationCue, a lightweight decoder that recovers coarse rotation information from the VLM's own frozen representations and feeds it back to the model as intermediate textual context. Across three VLMs, RotationCue improves every model--task combination on OR-Bench, raising macro-average accuracy by 7.9--12.6 points while preserving general capabilities.

Abstract

Explore similar work

Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs

How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study

Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation