Thinking with Spatial Code for Physical-World Video Reasoning

Jieneng Chen; Wenxin Ma; Ruisheng Yuan; Yunzhi Zhang; Jiajun Wu; Alan Yuille

cs.CVMar 5, 2026

Thinking with Spatial Code for Physical-World Video Reasoning

Authors: Jieneng Chen, Wenxin Ma, Ruisheng Yuan, Yunzhi Zhang, Jiajun Wu, Alan Yuille

Organizations: Johns Hopkins University · Stanford University

Abstract

We introduce Thinking with Spatial Code, a framework that transforms RGB video into explicit, temporally coherent 3D representations for physical-world visual question answering. We highlight the empirical finding that our proposed spatial encoder can parse videos into structured spatial code with explicit 3D oriented bounding boxes and semantic labels, enabling large language models (LLMs) to reason directly over explicit spatial variables. Specifically, we propose the spatial encoder that encodes image and geometric features by unifying 6D object parsing and tracking backbones with geometric prediction, and we further finetuning LLMs with reinforcement learning using a spatial rubric reward that encourages perspective-aware, geometrically grounded inference. As a result, our model outperforms proprietary vision-language models on VSI-Bench, setting a new state-of-the-art. Code is available at https://github.com/Beckschen/spatialcode.

Figures & tables

Figure 1: Thinking with Spatial Code enables superior spatial reasoning from video. Left: Unlike current state-of-the-art multimodal LLMs (MLLMs) that reason directly over the raw RGB image or video, our approach first parses video into explicit 3D spatial codes, then prompts a text-only LLM to reason over these symbolic descriptions. Right: On VSI-Bench ( Yang et al., 2025b ) , our method fine-tuned on Qwen3-4B ( Yang et al., 2025a ) significantly outperforms leading MLLMs including GPT-5o, Gemini-2.5, and Qwen3-VL in video-spatial reasoning accuracy. Reinforcement learning with spatial rubric rewards further improves performance. Dot size indicates model scale (4B–230B parameters; GPT and Gemini sizes are undisclosed). This demonstrates that the quality of 3D spatial representation, rather than model scale alone, is the key bottleneck for spatial reasoning.

Figure 2: Overview. (a) Encoding Video to Spatial Code: The Spatial Encoder processes video through a dual-encoder architecture. The SAM-2 ( Ravi et al., 2024 ) encoder extracts object-level features $F_{\text{sam}}$ with temporal attention, while the Depth Encoder (from Depth Anything 3 ( Lin et al., 2025 ) ) extracts spatial features $F_{\text{dep}}$ . Cross-attention fuses these representations into $F_{\text{ca}}$ , which feeds into a 3D Head for predicting 3D object bounding boxes with 3D orientation and a Depth Head for dense geometric supervision. Outputs are structured into symbolic spatial codes encoding object categories, positions, sizes, and orientations. (b) Prompting LLMs with Spatial Code: The spatial codes serve as explicit, interpretable inputs to LLMs for spatial reasoning. Given a query requiring perspective-aware understanding ( e.g., “Where is Table1 relative to the Sofa, from sofa’s perspective?”), the LLM reasons directly over the structured 3D representations to produce geometrically grounded answers.

Figure 3: Comparison of Spatial Rubric Reward (c) against conventional SFT (a) and RL (b). Unlike traditional methods, our framework utilizes the 3D spatial codes as primary input. Applying a structured spatial rubric reward to model rollouts significantly improves the quality of spatial reasoning.

Methods	Size	VSI-Bench ( Yang et al., 2025b )									Video-RoboSpatial
Methods	Size	Avg.	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appear. Order	Config.
Human Level	–	79.2	84.3	47.0	60.4	45.9	94.7	95.8	95.8	100	92.1
Proprietary MLLMs
Seed-1.6	230B	49.9	43.5	34.3	66.1	52.8	55.0	35.7	44.3	67.9	54.3
Gemini-1.5 Pro	–	48.8	49.6	30.9	64.1	49.4	51.3	48.1	42.0	68.0	–
Gemini-2.5-Pro	–	53.5	46.0	37.3	68.7	54.3	61.9	43.9	47.4	68.7	53.3

Table 1: Video spatial reasoning results on VSI-Bench and Video-RoboSpatial. Thinking with Spatial Code achieves state-of-the-art performance, outperforming proprietary MLLMs ( e.g., GPT-5o) and spatial-centric methods. Gray rows indicate fair comparisons.

Input	Methods	ARKitScenes		ScanNet
Input	Methods	[email protected]	[email protected]	[email protected]	[email protected]
Image detector	3D-MOOD ( Yang et al., 2025c )	0.066	0.009	0.083	0.025
Image detector	DetAny3D ( Zhang et al., 2025 )	0.094	0.012	0.119	0.024
Point-cloud detector	SpatialLM ( Mao et al., 2025 )	0.122	0.055	0.134	0.025
Point-cloud detector	SceneScript ( Avetisyan et al., 2024 )	0.020	0.014	0.101	0.043
MLLMs	Qwen3-VL-4B ( Yang et al., 2025a )	0.041	0.011	0.077	0.023

Table 2: Video 3D perception performance comparison. Our Spatial Encoder achieves state-of-the-art performance on scene-level F1, demonstrating strong spatial-temporal consistency. Image-based detectors operate on single frames and to support video input we aggregate predictions across frames. Point-cloud-based detectors such as SceneScript require ground-truth point clouds as input (video support is not yet open-sourced). MLLMs process videos as sequences of image frames. No existing open-source method supports video 3D perception; our approach fills this gap.

SAM2 Enc.	DA3 Enc.	Depth Head	[email protected]	3D IoU $\uparrow$	Dep. Err. $\downarrow$	Scale Err. $\downarrow$
✓	-	-	0.27	0.10	0.50	0.63
✓	✓	-	0.31	0.12	0.42	0.57
✓	✓	✓	0.32	0.14	0.31	0.55

Table 3: Ablation on encoder design. Dep. Err.: Depth Error. Scale Err.: Scale Error.

Training	Avg.	Rel. Dir.	Abs. Dist.	Config.
w/o spatial code	51.8	48.8	42.6	47.0
w/o RL	56.5	55.5	50.0	65.3
w/ Accuracy Reward	57.6	59.3	52.3	63.0
w/ Spatial Rubric	60.0	87.1	60.7	67.0

Table 4: Ablation on RL reward. Spatial rubric rewards yield large gains on direction and distance tasks.

Spatial Code Source	RL	Perception	Reasoning
Ground-truth codes		1.00	68.9
Spatial Encoder (ours)		0.52	56.5
Ground-truth codes	$\checkmark$	1.00	73.2
Spatial Encoder (ours)	$\checkmark$	0.52	60.0
Human performance		–	79.2

Table 5: GT vs. predicted spatial codes. The same LLM achieves 73.2% reasoning accuracy on VSI-Bench with perfect perception.

Figure 4: Qualitative comparison. We show three examples where Thinking with Spatial Code succeeds while Gemini 2.5 Pro fails. (a) Perspective-aware reasoning with detailed reasoning trace: The question requires model to reason from a specific observer viewpoint. Video-based models confuse absolute positions with observer-relative directions, while our spatial codes enable step-by-step coordinate transformation with precise calculation and significantly improves reasoning accuracy. (b) Orientation-aware reasoning: The question requires understanding object orientation (yaw angles). MLLMs rely on visual appearance, while our spatial codes provide explicit orientation parameters for accurate inference. (c) 3D distance estimation: The task requires metric depth measurements. MLLMs use ambiguous 2D visual cues, while our spatial codes provide precise 3D coordinates for reliable distance calculation.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Symbol	Value
GRPO Algorithm
Group size (samples per prompt)	$G$	16
KL penalty coefficient	$\beta$	0.01
Sampling temperature	$\tau$	0.7
Reward clipping range	–	$[-0.5,1.8]$
Optimization

Appendix

Table 6: GRPO training hyperparameters.

Panel of Figure 5 — Figure 5: High-level prompt structure for spatial reasoning tasks.

Task Type	Key Instruction	Answer
pairwise_configuration	Focus on relative positions, orientations, left/right, front/behind	Yes/No
object_rel_direction	Analyze relative positions from object-centric perspective	A/B/C/D
object_rel_distance	Compare closest-point distances to multiple candidates	A/B/C/D
object_abs_distance	Calculate min distance between closest bbox points	Numeric
object_counting	Count objects matching specified category	Numeric
object_size_estimation	Report max(x_size, y_size, z_size) in requested unit	Numeric

Appendix

Table 7: Task-specific prompt instructions and answer formats.

Panel of Figure 6 — Figure 6: Complete prompt example for the object_rel_direction task.

Component	Range	Description
$r_{\text{acc}}$	$\{0,1\}$	Exact match with ground truth $a^{*}$
$r_{\text{format}}$	$[-0.7,+0.1]$	Structure compliance and penalties
$r_{\text{rubric}}$	$[-0.65,+0.8]$	Task-specific reasoning quality
$r_{\text{total}}$	$[-0.5,1.8]$	Clipped sum of all components

Appendix

Table 8: Reward function components and their ranges.

Indicator	Condition	Weight
Valid format	Single “Final Answer:” with valid response	$+0.1$
Missing format	No “Final Answer:” detected	$-0.2$
Degenerate repetition	Pathological output patterns (e.g., repeated phrases)	$-0.5$
Excessive length	Response $>4000$ characters	$\leq-0.3$
Too short	Response $<200$ characters	$-0.2$

Appendix

Table 9: Format compliance indicators.

Indicator $\phi_{i}$	Condition	$w_{i}$
Structured reasoning	$\geq 2$ step indicators (“Step 1”, “First”, “Then”)	$+0.1$
Conclusion statement	Contains conclusion phrase (“therefore”, “thus”)	$+0.1$
Reasoning consistency	Analysis logically supports final answer	$+0.1$
Reasoning inconsistency	Analysis contradicts final answer	$-0.3$

Appendix

Table 10: Base reasoning quality indicators (all tasks).

Indicator $\phi_{i}$	Condition	$w_{i}$
Positive Indicators
Coordinate extraction	Extracts $\geq 3$ coordinate values	$+0.10$
Facing vector computation	Computes observer-to-target direction	$+0.15$
Local coordinate system	Constructs forward/right basis vectors	$+0.25$
Vector projection	Uses dot product for local components	$+0.20$
Quadrant determination	Maps component signs to quadrant	$+0.10$

Appendix

Table 11: Task-specific rubrics for object_rel_direction .

Indicator $\phi_{i}$	Condition	$w_{i}$
Positive Indicators
Perspective-based reasoning	Uses object’s perspective, not viewer’s	$+0.15$
Orientation awareness	Considers yaw angle in analysis	$+0.15$
Coordinate analysis	Compares object coordinates correctly	$+0.10$
Directional consistency	Handles opposite pairs (left/right)	$+0.10$
Negative Indicators

Appendix

Table 12: Task-specific rubrics for pairwise configuration .

Panel of Figure 7 — Figure 7: Rubric scoring comparison for object_rel_direction . Both responses achieve the correct answer (A: front-right), but the rubric reward assigns $3\times$ higher reward to proper spatial reasoning methodology.

Component	Parameters
SAM2 Encoder (Hiera-ViT-L)	214M
DA3 Encoder (ViT-G)	1.43B
Qwen3-VL-4B (LM backbone)	$\sim$ 4B
Total	$\sim$ 5.6B

Appendix

Table 13: Parameter breakdown of our full model.

Model	Total Param.	Spatial Enc.	LM	VSI-Bench (Avg 8)
Qwen3-VL-8B	8B	0	8B	55.1
Ours	5.6B	1.6B	4B	60.8 (+2.9%)

Appendix

Table 14: Parameter allocation comparison. Despite fewer total parameters (

\sim

5.6B vs.

\sim

8B), allocating capacity to the spatial encoder outperforms scaling the language model.

Figure 8: The details of 3D Head implementation.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

Geometric Encoding for Spatial Reasoning in Vision-Language Models
Sep 28, 2026Antonio Jun, Haoshui Yu, Zhengyi Lu +2Visual Spatial Reasoning Representation Geometry in VLMs
School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States
3D Primitives are a Spatial Language for VLMs
May 12, 2026Junze Liu, Kun Qian, Florian Dubost +8Visually Grounded Reasoning 3D Spatial Reasoning
Unity Technologies
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
May 28, 2026Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang +3Vision-Language Models 3D Spatial Reasoning
FAIR at Meta · UC Berkeley · HKU

Sep 28, 2026J/K move · Enter open · S save

Geometric Encoding for Spatial Reasoning in Vision-Language Models

Antonio Jun, Haoshui Yu, Zhengyi Lu, Huirong Fu+1

School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States

Visual Spatial Reasoning Representation Geometry in VLMs VLM Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are far more reliable at recognizing what appears in a video than at reasoning about its spatial and temporal properties, such as metric distances, object dimensions, and consistent object identities across frames. We present Geometric Code, a perception-to-geometry pipeline that computes explicit spatial structure from video and supplies it to VLMs as context to augment reasoning. A perception layer segments and classifies objects and recovers depth, camera pose, and intrinsics from monocular RGB video. A deterministic geometric engine then back-projects, merges, and cleans these outputs into a spatial code, including per-object positions, dimensions, counts, inter-object distances, appearance order, and room geometry. The code is serialized into VLMs' prompts, either alongside the video or replacing it entirely. Specifically, there is no component trained or fine-tuned in our approach. On VSI-Bench, augmenting 2B and 4B open models with the spatial code improves average accuracy by +4.1 points over the frames-only baseline, with the largest gains on numeric estimation tasks such as absolute distance (+24.1 points). The results suggest that explicitly computed geometry, delivered through the language channel, recovers spatial competence that small VLMs cannot extract from pixels alone.

Abstract

Visual Spatial Reasoning

Explore similar work

Geometric Encoding for Spatial Reasoning in Vision-Language Models

3D Primitives are a Spatial Language for VLMs

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning