Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives

Xiwei Liu; Yulong Li; Xinling Zhuang; Xuhui Li; Yichen Li; Haolin Yang; Zhixiang Lu; Yutong Xie

cs.CVOct 30, 2026

Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives

Authors: Xiwei Liu, Yulong Li, Xinling Zhuang, Xuhui Li, Yichen Li, Haolin Yang, Zhixiang Lu, Yutong Xie

Organizations: Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · Huazhong University of Science and Technology · University of Liverpool

Abstract

Spatial reasoning in Vision-Language Models (VLMs) has increasingly benefited from tool-integrated pipelines that ground computation in explicit geometry. However, existing systems often compile queries into relatively open-ended artifacts, such as code, task constraints, tool plans, or gathered evidence, leaving what must be resolved before execution under-specified. This can lead to unnecessary tool acquisition and iterative refinement, while still producing plausible reasoning traces from incomplete spatial commitments. We identify this failure mode as under-committed query compilation. To address it, we propose PEPS, a training-free agentic paradigm Parsing Executable Primitives for Spatial reasoning under a typed FESM schema over Frame, Entity, State, and Metric. PEPS follows a parse-before-execute principle: the Parser commits each query to a compact set of executable requirements; the Executor selectively grounds the required primitives through fixed geometric APIs and deterministic computation; and the Verifier checks execution sufficiency, enabling early termination or targeted repair. Extensive experiments across challenging spatial reasoning benchmarks show that PEPS consistently improves reasoning accuracy while substantially reducing refinement rounds, API cost, and end-to-end latency over strong tool-integrated baselines. These results demonstrate that explicit requirement commitment can make tool-integrated spatial reasoning not only more reliable, but also more execution-efficient.

Figures & tables

Figure 1: Plausible but incomplete requirement commitments can mislead spatial reasoning.

Figure 2: Overall Paradigm. The Parser retrieves Frame, Entity, State, and Metric primitives from the query. The Executor grounds them with a fixed geometric toolbox and deterministic code, producing a candidate answer and trace. The Verifier outputs <Decision> , <Score> , and <Feedback> ; rejected candidates are refined, and high-scoring accepted ones are stored for reference.

Figure 3: FESM-dimensional decomposition of a query.

API Tools	Visual Models	Skills
reconstruct	VGGT Wang et al. (2025a)	3D reconstruction; camera parameters; world points; object alignment
detect	GroundingDINO Liu et al. (2024)	Object detection; bounding boxes; labels
project_box_to_3d_points	SAM-2 Ravi et al. (2024)	2D $\rightarrow$ 3D projection; pixel mask filtering; point extraction
predict_obj_pose	Orient Anything Wang et al. (2024)	6-DoF pose estimation; object-to-world transformation; frame setup
estimate_scale	MoGe-2 Wang et al. (2025c) + VGGT	Metric scale computation; align depth maps; real-world units
ocr	EasyOCR Jaided (2020)	Optical character recognition; bounding boxes

Table 1: Overview of Tool APIs, underlying visual foundation models, and their core skills.

Method	MMSI-Bench					ViewSpatial-Bench
Method	Positional Relationship	Multi-Step Reasoning	Attribute	Motion	Overall	Camera Based	Person Based	Overall
Random Guessing	25.0	25.0	25.0	25.0	25.0	25.5	27.1	26.3
Proprietary LLMs
o3 OpenAI (2025)	45.8	34.9	41.6	32.0	41.0	51.3	51.0	51.1
Gemini 2.5 Pro Google DeepMind (2025)	39.0	34.3	36.1	33.3	36.9	44.3	41.6	43.0
Gemini 2.5 Flash Google DeepMind (2025)	37.4	30.3	34.5	33.4	35.0	41.6	36.9	38.4

Table 2: Comparison with baselines on MMSI-Bench and ViewSpatial-Bench. The best results are shown in bold , and the second-best are underlined . Results for PEPS and MSSR are reported as

\text{Mean}_{\pm\text{Std Dev}}

over 3 independent runs; other entries are single-run or reported baseline results.

Method	Counting	Float-MRA	Yes/No	Multiple Choice	Overall
Generic VLMs
GPT-4o	28.1	35.5	66.7	57.2	42.9
Claude3.5-Sonnet	22.4	20.6	62.2	50.6	32.2
Llama-3.2	24.3	19.3	47.5	27.4	25.6
Gemini1.5-Pro	25.2	28.1	46.2	37.6	32.0
Gemini1.5-Flash	24.3	27.6	51.1	52.9	35.0

Table 3: Performance comparison on OMNI3D-Bench .

Method	API Cost (USD/query)	API Calls (calls/query)	Non-cached Input (tokens/query)	Cached Input (tokens/query)	Total Input (tokens/query)	Output (tokens/query)
MSSR (GPT-4o)	0.1800	4.30	7,200.00	1,600.00	8,800.00	16,000.00
PEPS (GPT-4o)	0.1530 $\downarrow$ 15.0%	3.60 $\downarrow$ 16.3%	6,000.00 $\downarrow$ 16.7%	4,800.00 $\uparrow$ 200.0%	10,800.00 $\uparrow$ 22.7%	13,200.00 $\downarrow$ 17.5%
GPT-5	0.0862	1.00	4,900.00	900.00	5,800.00	8,000.00
MSSR (GPT-5)	0.1562	3.90	6,400.00	1,800.00	8,200.00	14,800.00
PEPS (GPT-5)	0.1231 $\downarrow$ 21.2%	3.30 $\downarrow$ 15.4%	5,200.00 $\downarrow$ 18.8%	4,800.00 $\uparrow$ 166.7%	10,000.00 $\uparrow$ 22.0%	11,600.00 $\downarrow$ 21.6%
Gemini 3.6 Flash	0.0557	1.00	4,500.00	1,000.00	5,500.00	6,500.00

Table 4: Average Per-query API Usage and Cost on MMSI-Bench.

Method	GPU Memory (GB/process)	Mean E2E Latency (s/query)	Median E2E Latency (s/query)	P95 E2E Latency textbf(s/query)	Average Iterations (rounds/query)	1-Iteration (%)	2-Iteration (%)	3-Iteration (%)
MSSR (GPT-4o)	7.20	86.75	79.60	137.40	2.15	15.0%	55.0%	30.0%
PEPS (GPT-4o)	9.80 $\uparrow$ 36.1%	53.60 $\downarrow$ 38.2%	46.20 $\downarrow$ 42.0%	92.80 $\downarrow$ 32.5%	1.20 $\downarrow$ 44.2%	83.0% $\uparrow$ 68.0%	14.0% $\downarrow$ 41.0%	3.0% $\downarrow$ 27.0%
GPT-5	0.00 $\dagger$	34.80	31.20	60.50	1.00	100.0%	0.0%	0.0%
MSSR (GPT-5)	7.20	98.75	96.40	165.20	1.95	25.0%	55.0%	20.0%
PEPS (GPT-5)	9.80 $\uparrow$ 36.1%	66.83 $\downarrow$ 32.3%	61.50 $\downarrow$ 36.2%	117.80 $\downarrow$ 28.7%	1.10 $\downarrow$ 43.6%	91.5% $\uparrow$ 66.5%	7.0% $\downarrow$ 48.0%	1.5% $\downarrow$ 18.5%
Gemini 3.6 Flash	0.00	23.40	20.80	42.60	1.00	100.0%	0.0%	0.0%

Table 5: Average Per-query Computational Overhead on MMSI-Bench.

Figure 4: Ablation studies. We evaluate three scenarios: (a) prepending primitive parsing to existing spatial reasoning agentic baselines; and (b) applying PEPS to different VLM backbones; and (c) varying the maximum loop budget, with or without corrective Verifier feedback.

Method	MindCube-Tiny	OmniSpatial
GPT-4o	35.8	51.5
Free-form	61.3	62.1
PEPS	73.7	69.8

Table 8: Cross-Dataset Generalizability of Primitive Schema.

Method	MMSI-Bench
GPT-4o	30.3
PEPS w/ VGGT	51.1
PEPS w/ $\pi^{3}$	51.8
PEPS w/ CUT3R	50.5

Table 9: Sensitivity analysis of different 3D reconstruction backbones.

Figure 5: Compact Qualitative Example. More detailed examples are provided in Appendix F .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 10 — Table 10: FESM Schema: Frame Primitives.

Panel of Table 11 — Table 11: FESM Schema: Entity Primitives.

Panel of Table 12 — Table 12: FESM Schema: State Primitives.

Panel of Table 13 — Table 13: FESM Schema: Metric Primitives.

Panel of Table 14 — Table 14: Prompt Used for Per-Question Primitive Decomposition. This prompt is applied to each sampled question independently to decompose it into candidate spatial primitives across the four FESM layers. Here, {question} is the placeholder that will be replaced.

Panel of Table 15 — Table 15: Prompt Used for Cross-Question Schema Aggregation. After per-question decomposition (Table 14 ), this prompt aggregates the decomposition results across all sampled questions to induce the initial FESM schema. Here, {per_question_decompositions} is the placeholder for the concatenated JSON outputs from Stage A.

Panel of Table 16 — Table 16: Prompt Used for Retrieving FESM Requirements Library. Here, {library} is the predefined FESM library, {examples} is the in-context examples, and {question} is the placeholder that will be replaced by the user’s query with input images.

Panel of Table 17 — Table 17: Prompt Used for Requirement-Guided Tool Orchestration. Here, {api_documents} is the documentation of available APIs, {requirements} is the retrieved requirement set, and {history} contains the user question, current workspace, previous plans, and execution results.

Panel of Table 18 — Table 18: Prompt Used for Coder. Here, {question} is the original user query, {requirements} is the retrieved Frame–Entity–State–Metric requirement set, {workspace_docs} documents the currently available variables in the workspace, and {knowledge} contains the fixed geometric formulas and rules relevant to the required metric.

Panel of Table 19 — Table 19: Prompt for Verifier Candidate Evaluation

Tool Use in VLMs

Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.

See all 119 papers

Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning
Jun 11, 2026Changye Li, Meng Lu, Yi Wu +1Tool Use in VLMs Visual Spatial Reasoning
Tsinghua University · Virginia Tech · NVIDIA
LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
Apr 8, 2026Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu +5Visual Spatial Reasoning Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China
EgoP $^{2}$ C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Oct 15, 2026Chaoyu Li, Arka Sadhu, Yuliang Li +3Visual Spatial Reasoning Multimodal Spatial Reasoning
Arizona State University · Meta

Jun 11, 2026J/K move · Enter open · S save

Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning

Changye Li, Meng Lu, Yi Wu, Ligeng Zhu

Tsinghua University · Virginia Tech · NVIDIA

Tool Use in VLMs Visual Spatial Reasoning Tool-Using Vision-Language Agents Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

While recent vision-language models (VLMs) demonstrate strong multimodal understanding, they remain limited in spatial reasoning tasks that require active evidence acquisition and multi-step visual interaction. This limitation suggests that relying solely on implicit visual representations from vision encoders is insufficient for recovering fine-grained spatial evidence. We introduce PERception-Interaction-reason Agent (PERIA), a tool-augmented visual agent for spatial reasoning tasks across map reasoning, visual probing, and vision reconstruction. PERIA uses two lightweight tool families: vision perception tools for exposing textual, symbolic, and spatial evidence, and vision interaction tools for manipulating visual context, tracing paths, and verifying spatial relations. To train PERIA, we develop a unified recipe that combines supervised tool-use trajectory synthesis, composite rewards, and Observation-Relaxed Group-in-Group Policy Optimization (OR-GIGPO) for effective multi-tool behavior. Experiments on 13 benchmarks from 8 datasets show that PERIA-8B improves over the Qwen3-8B backbone by 10.0 percentage points on in-distribution benchmarks and 4.4 percentage points on out-of-distribution benchmarks, while outperforming previous state-of-the-art baselines of similar size by 6.9--14.8 percentage points. It also achieves performance comparable to much larger models such as Qwen3-VL-235B-A22B-Thinking and GPT-5, demonstrating the effectiveness of PERIA in enhancing spatial reasoning capabilities.

Abstract

Tool Use in VLMs

Explore similar work

Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

EgoP2^{2}2C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video

EgoP $^{2}$ C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video