Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action

Yihang Liu; Yiming Chen; Mingzhe Zheng

cs.RONov 3, 2026

Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action

Authors: Yihang Liu, Yiming Chen, Mingzhe Zheng

Organizations: Independent Researcher · Georgia Institute of Technology · The Hong Kong University of Science and Technology

Abstract

Recent multimodal language models can reason directly about robot scenes, but it is unclear whether they need to repeatedly reinterpret raw visual observations at every step of an interaction. We propose shifting routine spatial reasoning from images to text and symbols through a 3D geometric abstraction that is refreshed locally from observations and read directly by the language models. Specifically, our interface represents scene geometry via typed, evidence-aware symbolic references to object parts, grounds them from calibrated RGB-D, and returns compact textual checkpoints with metric residuals after each action. Symbolic task structure can thus be reused across decisions while metric values are rebound to the current scene, with direct visual reasoning invoked only on demand. Across offline policy synthesis and continuous online execution, reasoning over grounded text and symbols improves multi-step manipulation completion while substantially reducing repeated multimodal queries, positioning explicit geometric state as an effective substrate for agentic robot policies.

Figures & tables

Figure 1: Geometric abstraction between observation, reasoning, and action. Visual observations are grounded locally into metric 3D geometry and exposed to the language model through symbolic references and compact textual checkpoints. Policies reason in text and symbols over stable geometric roles whose metric values are rebound to the current scene during execution.

Figure 2: Geometric abstraction between perception and reasoning. RGB-D observations are grounded into object geometry and exposed to the LLM through typed symbolic references and compact textual checkpoints. In policy synthesis, the LLM revises executable policies between rollouts, while perception and control run locally. In online execution, the same interface supplies updated geometry for successive decisions within a continuing episode.

Operation	Inputs	Output
reference	object and role	geometric entity with evidence
relation	predicate and objects	residuals and true / false / unknown
align	source and target geometry	desired rigid transform
checkpoint	active object and goal	compact textual state

Table 1: Operators on the geometric state.

Figure 3: Single-policy acquisition. Policies are revised between reset rollouts and evaluated after training freezes. Geometric Abstraction additionally uses supplied geometric references. Right: illustrative start and goal configurations, constructed in simulation rather than recorded policy trajectories.

Figure 4: Test success and acquisition cost. Lines show mean held-out test SR of training-retained policies; bars show cumulative input-plus-output tokens through each iteration, including rejected proposals. Both average over five chains. Ring-on-peg Pixel Reasoning excludes unknown usage from two interrupted requests.

Figure 5: Online execution and tasks. The agent acts in a single continuing scene until completion or the 200 s timeout; Geometric Abstraction receives grounded symbolic references and textual checkpoints. Right: illustrative start and goal states.

Figure 6: Online outcomes and cost. Completion rate within 200 s (left), mean capped completion time (middle; timeouts assigned 200 s), and mean recorded tokens (right) over 50 paired layouts per task ( $\dagger$ : excludes unknown interrupted tokens).

Figure 7: Model interaction cost. Mean (bars) and per-episode (markers) request wall-clock time (left) and recorded tokens (right) over 50 paired episodes per task ( $\triangle$ : excludes unknown interrupted tokens).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Condition	R1	R2	R3	R1	R1–R2	R1–R3
		Retained test SR (%)			Cumulative tokens (k)
Bridge + cylinder	Pixel Reasoning	$21.2\pm 22.0$	$53.0\pm 11.7$	$56.0\pm 9.1$	24.6	66.2	125.7
	Geometric Abstraction (ours)	$\boldsymbol{45.2\pm 25.4}$	$\boldsymbol{63.4\pm 5.9}$	$\boldsymbol{65.4\pm 3.5}$	25.8	67.3	127.0
Three-cube stacking	Pixel Reasoning	$17.2\pm 29.4$	$53.6\pm 26.2$	$70.8\pm 26.5$	28.0	74.1	138.1
	Geometric Abstraction (ours)	$\boldsymbol{57.2\pm 32.6}$	$\boldsymbol{72.2\pm 7.2}$	$\boldsymbol{81.4\pm 12.3}$	25.7	64.7	123.4
Ring-on-peg	Pixel Reasoning	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	37.8	93.5	166.8

Appendix

Table 2: Single-policy acquisition outcomes and token usage. Held-out test SR (mean

\pm

sample SD) across five training chains on 100 shared held-out layouts per task, using training-only policy retention. Tokens report cumulative completed-request input plus output per chain (in thousands), including rejected proposals and counting cached input once. Ring-on-peg Pixel Reasoning excludes unknown token usage from two interrupted requests. Bold highlights where Geometric Abstraction (ours) improves over Pixel Reasoning.

Task	Condition	Success	Time (s)	Model (s)	Tokens (k)	Requests	Images
Bridge-tower	Pixel Reasoning	12/50	198.0	55.5	112.5	7.88	7.50
	Geometric Abstraction (ours)	28/50	185.3	51.8	129.5	7.32	4.28
Ring-on-peg	Pixel Reasoning	49/50	115.8	90.0	214.5	11.88	10.94
	Geometric Abstraction (ours)	50/50	77.2	45.6	101.3	6.90	2.72
Tower relocation	Pixel Reasoning	15/50	186.4	80.3	185.9	11.58	11.34
	Geometric Abstraction (ours)	42/50	139.2	31.7	88.3	5.36	3.00

Appendix

Table 3: Online outcomes and interaction cost. Each task evaluates 50 paired layouts. Success reports completed episodes out of 50; all other columns report per-episode means across all 50 episodes, including timeouts. Time is capped completion time, assigning 200 seconds to timeouts. Model is cumulative request wall-clock time, including interrupted waits. Tokens are recorded input plus output tokens (in thousands), counting cached input once and excluding unknown interrupted usage. Requests counts completed model calls; Images counts newly attached camera images across all request attempts (excluding local sensing frames not sent to the LLM). Both conditions use audited repeats for nine runtime-flagged pairs. Bold highlights where Geometric Abstraction (ours) improves over Pixel Reasoning.

Figure 8: Cumulative success over elapsed wall-clock time. Each curve summarizes the 50 paired 200-second episodes for Pixel Reasoning and Geometric Abstraction (ours), stepping at the first verified completion time. Timeouts do not convert to successes at the 200 s deadline. Intermediate values are read directly from the 200-second episodes rather than separate evaluations under shorter budgets.

Agentic AI for Robotics

See all 60 papers

Agentic Scene Policies
Sep 23, 2025Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4Robot Policy Generalization Robot Learning
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome
Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization
Apr 8, 2026Wenhao Yang, Yu Xia, Jinlong Huang +10Multimodal Reasoning Multimodal Agents
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · AI Business, Alibaba Group +5
Agent Explorative Policy Optimization for Multimodal Agentic Reasoning
May 27, 2026Minki Kang, Shizhe Diao, Ryo Hachiuma +4Multimodal Reasoning Agentic Reasoning
1NVIDIA · 2KAIST

Sep 23, 2025J/K move · Enter open · S save

Agentic Scene Policies

Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu, Charlie Gauthier+3

Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome

Robot Policy Generalization Robot Learning Zero-Shot Robotic Manipulation Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

Designing or learning robot policies that generalize zero-shot across a range of language instructions and objects is a core problem in robotics. Vision-Language-Action models (VLAs) learn such policies end-to-end by repurposing existing Vision-Language Models (VLMs), but generalization to new instructions and objects remains challenging. An alternative is to implement a modular policy by leveraging an explicit VLM-based 3D scene representation and motion planning. While modular policies show strong zero-shot potential, they typically retrieve objects based on semantics without explicit spatial reasoning, severely restricting their overall grounding capabilities. They also interact with objects using basic grasping and navigation skills. In this work, we address these limitations by unifying grounding capabilities and robot skills in a single agentic action space through a scene-agent tool interface. By leveraging part-level affordances, our skills generalize across diverse objects and enable zero-shot interactions such as unplugging chargers and opening drawers. We name the resulting framework Agentic Scene Policies (ASP). Through extensive real-world experiments, we show how ASP consistently outperforms leading VLAs in the zero-shot setting. We also demonstrate the extensibility of our framework by introducing a mobile version of ASP to tackle room-level queries. See our project page (https://montrealrobotics.ca/agentic-scene-policies.github.io/) for more results.

Abstract

Agentic AI for Robotics

Explore similar work

Agentic Scene Policies

Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning