Spatial reasoning asks a model to work out where objects are in 3D space, how they relate to one another, and how they move. Agentic VLMs have recently brought large improvements on such tasks by letting the model actively gather evidence with 3D perception tools instead of answering in a single pass. Even so, on many questions the tools return the right evidence and the model still does not arrive at the correct answer: the gap between model and tools, where raw readings become usable conclusions, is left unfilled. We present S-Agent 2.0, a spatial reasoning agent that equips an off-the-shelf general coding agent with a command-line perception toolkit, and we design the interface around one principle: help the VLM close the evidence loop: acquire scene evidence, remember it in a persistent scene memory, and interpret each reading in plain spatial language. This design keeps the full flexibility of a coding agent, and every reading it acts on stays grounded in the tools' geometry. On MMSI-Bench and ReVSI, the toolkit improves a strong open-source model, Qwen3.8-27B, by +13.9 and +20.5 points over the same agent without tools, and a strong closed-source model, Gemini-3.8-Flash, by +11.2 and +9.6 points. With the open backbone S-Agent 2.0 reaches 66.5 on MMSI-Bench and 82.3 on ReVSI, ahead of the specialized system SpatialClaw on the same backbone (62.6 and 74.5), and with Gemini-3.8-Flash it reaches 74.1 and 90.5 against its 67.0 and 79.7. Both tool gains shrink as the backbone grows more capable, but SpatialClaw's shrinks far more on ReVSI, from +13.2 to +2.7, while ours stays at +9.6. S-Agent 2.0 also works across different general coding agent harnesses, such as Codex and Claude Code.
Figures & tables
Figure 1: S-Agent 2.0 is a spatial reasoning agent built on a general coding agent. A command-line perception toolkit lets the agent close the evidence loop, acquiring scene evidence, remembering it in a persistent scene memory, and interpreting every reading in plain spatial language. S-Agent 2.0 delivers strong performance: it leads MMSI-Bench and ReVSI and stays ahead of the specialized agentic system SpatialClaw on every benchmark and backbone we test.
Figure 2: Overview of the framework. The agent keeps a coding agent’s full generic freedom over the workspace (viewing images, reading and writing files, writing and running code) and composes perception through the same shell; the toolkit registers everything it perceives into the scene memory and returns interpreted, answer-ready readings.
Figure 3: An example question used throughout this section. Left to right: the question, the agent’s command sequence colour-coded by who performs the perception, the outputs those calls produce and the scene memory they write, and the readouts that answer the question.
Model
Positional Relationship
Geometric Attr.
Motion
MSR
Avg.
C-C
O-O
R-R
C-O
O-R
C-R
Meas.
Appr.
Cam.
Obj.
Proprietary models
GPT-5.4
41.9
33.0
35.8
49.8
42.4
68.7
54.7
37.4
28.3
40.8
36.4
41.9
GPT-5.6
64.5
42.6
55.6
41.9
44.7
59.0
67.2
40.9
59.5
48.7
57.1
53.2
GPT-6
73.1
61.7
72.8
58.1
67.1
72.3
73.4
69.7
89.2
65.8
75.3
71.0
Grok 4
36.6
35.1
39.5
34.9
45.9
50.6
21.9
22.7
40.5
43.4
38.4
37.8
Table 1: MMSI-Bench results by question category (C/O/R denote camera/object/region).
Model
Frames
Numerical Question
Multiple-Choice Question
Avg.
Obj. Cnt.
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Proprietary models
GPT-5.2
64
56.2
41.5
73.9
63.0
48.4
34.9
38.2
50.9
GPT-5.4
64
70.3
40.2
73.3
59.8
57.8
43.5
33.3
54.0
GPT-5.6
64
80.6
64.4
80.3
71.7
71.8
71.0
63.9
72.0
Gemini 3 Flash
1 FPS
65.7
53.1
77.6
52.8
64.6
47.9
41.8
57.6
Table 2: ReVSI results by question type. Numerical types (object counting, absolute distance, object size, room size) use the official mean relative accuracy; the remaining types are multiple choice.
Table 6Table 7
Figure 4: Qualitative analysis. Left: two items where SpatialClaw calls the right tools and gets the right perception, and still answers wrong, with S-Agent 2.0 correct on both (gemini-3.8-flash on both sides). Top right: S-Agent 2.0 cross-checking its own perception against the scene memory, dropping the wrong masks before grounding the observer and reading the answer. Bottom right: the two remaining grounding failure classes, a wrongly imagined viewpoint and a quantity read in the wrong sense. Zoom in for better readability.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Command
Arguments
Effect on S and reply
Tool perception ( Ttool )
recon
none (all frames under /app )
Runs Depth Anything 3 [ 20 ] on all frames for metric depth and camera poses, GeoCalib [ 32 ] for the gravity direction and πmos3 [ 37 , 54 ] for per-pixel motion masks; fuses them into the metric, gravity-aligned point cloud P and registers camera:f<N> with its pose and scene . Returns the instance list, with an overlap_warning naming the seam when two views share almost no covisible surface.
segment
--prompt TEXT[@f<N>] + , --max-masks K
Runs SAM 3 [ 5 ] with each text prompt on the named frame or on all frames, lifts every mask onto P and registers each label as an instance with extent Ei (same-label regions within 0.75 m merge), tagged static or moving from the motion masks. Returns, per frame, the overlay PNG path and the region ids with label and centroid, then the updated registry.
Model perception ( Tmodel )
add_instance
--name , --source ; position --at-map f<N>:PX,PY ∣ --view N --pixel X,Y ∣ --at-camera N ; orientation --facing V ∣ --primary A=V --secondary A=V
Registers an instance at the chosen pi , with pose Ti when an orientation is given; called again with the same name and only orientation flags, re-orients it in place. Returns the updated registry.
drop
--instance f<N>:<id> +
Removes those instances permanently. Returns the registry after the drop.
Appendix
Table 7: The toolkit’s command-line interface. Flags marked + are repeatable. P , I and the instance registry are the components of the scene memory S (Eq. 2 ).
Qwen3.8-27B, MMSI
Gemini-3.8-Flash, MMSI
Gemini-3.8-Flash, ReVSI
recon
1.00 (100%)
1.00 (100%)
1.00 (100%)
segment
1.75 (93%)
1.47 (92%)
2.25 (100%)
add_instance
1.47 (36%)
0.72 (26%)
1.48 (48%)
drop
0.37 (26%)
0.27 (21%)
0.53 (40%)
list_instance
0.29 (24%)
0.52 (45%)
1.11 (91%)
bev
1.66 (94%)
1.67 (94%)
1.74 (98%)
Appendix
Table 8: Toolkit usage per episode. Calls per episode and, in parentheses, the share of episodes that issue the command at least once. Image views count the agent’s own view calls on benchmark frames, segmentation overlays and top-down maps. MMSI-Bench and ReVSI, 1000 items each.
System
ReVSI
MMSI
SpatialClaw
74.5
62.6
SpatialClaw + our readouts
75.5
62.9
S-Agent 2.0
82.3
66.5
Appendix
Table 9: SpatialClaw with our readouts transplanted into its kernel, against SpatialClaw and S-Agent 2.0. Qwen3.8-27B backbone.
Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce S-Agent, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, S-Agent reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, S-Agent casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (e.g., counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that S-Agent consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on S-Agent-generated spatial trajectories S-300K yields S-Agent-8B, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).
Spatial reasoning, the ability to determine where objects are, how they relate, and how they move in 3D, remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked. In this work, we study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning. Existing spatial agents either employ single-pass code execution, which commits to a full analysis strategy before any intermediate result is observed, or rely on a structured tool-call interface that often offers less flexibility for freely composing operations or tailoring the analysis to each task. Both designs offer limited flexibility for open-ended, complex 3D/4D spatial reasoning. We therefore propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface. SpatialClaw maintains a stateful Python kernel pre-loaded with input frames and a suite of perception and geometry primitives, letting a VLM-backed agent write one executable cell per step conditioned on all prior outputs, enabling the agent to flexibly compose and manipulate perception results and adapt its analysis to both intermediate text and visual observations and the demands of each problem. Evaluated across 20 spatial reasoning benchmarks spanning a broad range of static and dynamic 3D/4D spatial reasoning tasks, SpatialClaw achieves 59.9% average accuracy, outperforming the recent spatial agent by +11.2 points, with consistent gains across six VLM backbones from two model families without any benchmark- or model-specific adaptation.
Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce SpatialAct, a simulator-grounded benchmark for probing action-conditioned spatial reasoning in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear reasoning-to-action gap: current VLMs can perform well on isolated spatial reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.
Tianhui Liu, Jie Feng, Zhiheng Zheng +6
1The Hong Kong University of Science and Technology (Guangzhou) · 2Zhongguancun Academy · 3Tsinghua University +1