cs.CVOct 15, 2026

Learning to Think and Explore in 3D World

Authors: Yijiang Li, Bingyang Wang, Nuno Vasconcelos

Organizations: University of California San Diego · Georgia Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.

Figures & tables

Figure 1: Learning to reason and explore in 3D. Unlike prior methods with passive perception which assumes receive complete 3D representations as input, Act3D treats 3D representations as interactive environments, learning to alternate between reasoning and actions to actively explore the 3D world before committing an answer.
Figure 3: Training data composition. Distribution of SFT trajectories and RL questions across datasets, with representative examples of curated spatial reasoning questions.
modelScanQASQA3DSPB MVSPB SIMP3DARKitAvg.
Passive perception
Video-R1-7B30.841.142.039.642.663.443.2
ViLaSR-7B33.347.652.643.434.666.046.2
SpatialLadder-3B29.340.461.245.245.258.746.7
InternVL3.5-38B36.247.036.938.751.175.447.6
SpaceR-7B31.644.752.641.052.070.148.7
Table 1: Performance comparison on six 3D reasoning benchmarks. We compare Act3D with existing active and passive methods. Green highlights indicate benchmark domains included in model training.
Figure 4: Learning dynamics of Act3D. (a) ScanQA-only validation. (b) Mixed-data validation gains relative to SFT initialization. (c) Mean number of interactions for the final reward. Panel (c) is in-training rollout statistics; panels (a) and (b) are validation accuracies, and no value should be read across the two.
Figure 5: Performance gains across different exploration demands. ScanQA validation questions are grouped by a geometry-based estimate of the interactions needed to reach the answer object from the initial viewpoint. Stacked bars show the accuracy improvements from SFT over the base model and subsequent RL over SFT.
Figure 6: Qualitative analysis. We highlight emergent exploration behaviors. Arrows denote the executed actions, and text excerpts beneath illustrate the reasoning of the policy.
Figure 7: Rendered versus retrieved observations. Comparison of pose-aligned renders and nearest available real images from ScanNet. Distance and angular offsets indicate discrepancies between the requested and retrieved camera poses.
Initial visual inputAccuracy (%)
BEV + 1 rendered frames36.736.7
BEV + 4 rendered frames41.4
1 rendered frame (Ours)41.0‾\underline{41.0}
Table 2: ScanQA Acc. given different initial observation.
Figure 8: ScanQA Acc. of different action parameterizations.
Figure 9: Effect of supervised initialization on RL. We report ScanQA Acc. vs. average number of interactions per episode with and without SFT initialization. RL-only results are shown after 80 and 470 training steps.
Figure 10: Effect of interaction reward. ScanQA Acc. with and without the per-interaction reward across held-out and training benchmarks. Numbers below the bars indicate the average number of interactions per episode.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

metricpublishedreproduced
EM30.830.7
CIDEr107.0107.1
BLEU-417.918.2
METEOR20.920.8
ROUGE-L50.750.7
Appendix
Table 3: Agreement with the published Ross3D results on 4,6754{,}675 ScanQA validation questions.
More in this topic

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Explore similar work

CardsList
Oct 17, 2026

From Views to Worlds: Active Exploration over 3D Worlds for Vision-Language Models

Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning. Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. Project page: https://fangzhou2000.github.io/projects/v2w.

Qijian Tian, Jiayu Ying, Ke Fan +2

Shanghai Jiao Tong University, Shanghai, China · East China Normal University, Shanghai, China

Visual Spatial Reasoning3D Scene Understanding
AskPDF
Jan 19, 2026

Think3D: Thinking with Space for Spatial Reasoning

While Vision-Language Models (VLMs) excel at 2D visual understanding, they remain constrained by 2D-centric paradigm that severely limits genuine 3D spatial reasoning. To bridge this gap, we introduce Think3D, a novel framework that equips VLM agents with interactive, 3D chain-of-thought reasoning capabilities. By integrating a suite of 3D manipulation tools, Think3D transforms perception into active spatial exploration, mirroring human geometric reasoning. Think3D consistently improves proprietary models, including GPT-4.1 and Gemini 2.5 Pro, across BLINK Multi-view, MindCube-1K, and VSI-Bench-Tiny. We further propose Think3D-RL to teach smaller open-weight models how to manipulate 3D space effectively. Using only final-answer rewards, without process supervision or handcrafted exploration trajectories, Think3D-RL enables Qwen3-VL-4B to autonomously learn effective 3D exploration strategies. After training, the model exhibits tool-use patterns similar to those of stronger proprietary models, while shifting the effect of 3D tool use on MindCube-1K from a performance drop to a substantial improvement. These results show that active exploration in 3D space provides an effective and general paradigm for improving spatial reasoning in multimodal agents. Code, models, and data are available at https://github.com/zhangzaibin/spagent.

Zaibin Zhang, Yuhan Wu, Lianjie Jia +10

Dalian University of Technology · University of California San Diego · Independent Researcher +1

3D Spatial ReasoningVLM Reasoning3D VLMsTool Use in VLMsRL for VLMs
AskPDF
Jun 3, 2026

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and injected at different feature hierarchies. During training, three tightly coupled components work in concert primarily within the latent space: (1) To gain geometric priors, a latent 3D geometry perception module aligns intermediate visual features with a 3D foundation model, acquiring low-level geometric cues without architectural modifications to the VLM backbone. (2) Complementing this, an online 3D reasoning distillation module mitigates the prompt-induced reasoning gap via a shared reasoning anchor token. During 3D VLM co-training, this anchor is emitted as the first output token to robustly encode spatial priors. During VLA training, it serves as an input token inserted between the task and action instructions, transferring high-level spatial thinking from explicit teacher reasoning prompts to student action prompts without chain-of-thought text generation. (3) These disentangled geometric and reasoning features are then united by a spatially augmented action integration, which jointly injects them into the action-query tokens as hierarchical spatial conditions to prevent action shortcuts. At deployment, our method retains only its lightweight adapters to perform implicit 3D reasoning, discarding the 3D foundation model and the teacher branch used for supervision. Consequently, it operates purely on 2D images without 3D sensors, external models, or explicit text generation while preventing catastrophic forgetting of the pretrained VLM, achieving state-of-the-art performance on LIBERO, LIBERO-PLUS, SimplerEnv, and real-world manipulation tasks.

Jiaxin Shi, Xidong Zhang, Fucai Zhu +3

Shanghai Jiao Tong University · Daimon Robotics · Harbin Institute of Technology +4

3D Spatial ReasoningRobotic ManipulationVision-Language-Action Models3D Representation LearningReasoning for VLA Models
AskPDF