SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision Insertion

Jiaming Wang

cs.ROOct 30, 2026

SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision Insertion

Authors: Jiaming Wang

Abstract

Inserting an unseen part requires sub-millimetre precision and the correct orientation of keyed features. Current zero-shot agentic manipulation methods built on vision-language models (VLMs) produce spatial outputs that are far too coarse for this. We propose to let a VLM build a sketch model of the task, i.e., a few geometric primitives and two mate frames, which a VLM agent measures with geometric tools in multi-view depth and which a standard planner and compliant controller execute. Because a sketch describes the parts independently of their placement, the robot needs to build it only once per task. A sketch memory stores each agent run and transfers the stored sketch to later placements by point-cloud registration, so that the VLM is called only for unfamiliar parts. After grasping the part, the robot inspects both mating faces at close range and corrects the sketch by shape matching. On 69 simulated insertion episodes, our method inserts 88% of the parts at the assembled pose in a strict kinematic test, while ReKep and MOKA with the same VLM insert none. On new placements of eight tasks, reusing one stored run inserts as many parts as calling the agent in each episode, and it produces the sketch in 3.4 s instead of about 90 s.

Figures & tables

Figure 2: Overview. A VLM agent builds the sketch for a new task, and the sketch memory supplies it for a known task. The agent (purple) calls geometric tools on three RGB-D views, whereas the memory registers a stored run to the new scene by geometry alone (green). After the grasp, we match the mating end (dark orange) to the height map of the fixed part and insert the part with compliance. The panels show one episode with GPT-6.1-Sol and the NEMA 5-15 task of Section 4.5 , with symbols as in Section 3.2 .

Figure 3: A sketch model is a few primitives and two mate frames. Agent sketches for four tasks beside crops of the camera images that the agent received (orange: moving part $A$ , blue: fixed part $B$ ). Primitives of $B$ are optional, and the agent added one only for the post. Each $B$ is turned so that $\mathbf{s}\parallel\mathbf{r}$ , as in the assembled state, and each sketch alone inserts the part.

Tool	Input	Returns	Runs
view	camera, optional pixel box	the image or a crop of it, with a grid labelled in camera pixels	97%, 4
xyz	camera, pixels	the 3D point, normal and object at each pixel, and the near and far points at depth edges	77%, 2
region	pixel or object	the object fused over all cameras: height range, upright box, vertical cylinder, heights of flat upward faces and radius profile	100%, 2
hole	pixel inside an opening	the rim of the opening: centre, circle radius, rectangle, rim height and the depth visible below the rim	100%, 2
top	square area, height range	an orthographic height map with a millimetre grid, in which keys, flats and slots appear without perspective	98%, 3
fit	3D points, model type	a plane, circle, line, cylinder, upright box or rigid pose fitted to the points	24%, 2

Table 1: Tools of the agent. Each tool answers one geometric question on the fused depth data of the episode and returns JSON in millimetres in the world frame. view , top and check also return an image. Runs: share of the 141 agent runs that call the tool, with the median number of calls in these runs.

Figure 4: Geometry recovers the rotations that the VLM misreads. Each column shows the mating end seen by the upward camera (dark orange, fingers in grey), the height map of the fixed part with this end at the sketch pose (purple, dashed) and at the best fit (green), and the best score per rotation. A symmetric mate gives equal peaks (triangle), and a flip that differs only in a small key scores lower (SFP).

	Kinematic, strict (development)						Held-out	Physics
Method	AutoMate	Conn.	Board	AIC	All	Search	strict	Devel.	Held-out
	(42)	(13)	(10)	(4)	(69)	range	(18)	(66)	(18)
ReKep ( Huang et al., 2024b )	0	0	0	0	0	0	0	0/10 †	–
MOKA ( Fang et al., 2024 )	0	0	0	0	0	2	0	0/12 †	–
Ours, sketch only	32	11	6	1	50	60	6	54	10
Ours	35	13	9	4	61	64	18	60	18

Table 2: Insertion success (number of episodes) in the kinematic test, strict or within reach of the compliant search, and in physics from valid starts ( Section 4.1 ). All methods share GPT-6.1-Sol, the planner and the controller, and Oracle runs our method on ground-truth sketches. † Search-range episodes and ten random others. ‡ Valid in this run and ours.

	Median	Episodes (of 69)
Method	error	success	place	rotat.	impr.	grasp
ReKep	25	0	54	1	0	14
MOKA	13	0	67	1	1	0
Sketch only	0.15	50	3	6	10	0
Ours	0.10	61	4	1	3	0
Oracle	0.01	64	0	1	4	0

Table 3: Failure causes (development episodes). Median lateral error of the final pose (mm) and the cause of each strict failure: more than

3\text{\,}\mathrm{m}\mathrm{m}

from the opening (place), more than

5\text{\,}\mathrm{\SIUnitSymbolDegree}

from any fitting rotation (rotation), a smaller miss (imprecise) or the wrong object grasped (grasp).

Variant	Auto.	Conn.	Board	AIC	Dev.	Held.	Err.
Sketch only	32	11	6	1	50	6	0.15
Scene map, keyed	31	13	7	4	55	11	0.18
Scene map, all	31	13	7	4	55	11	0.17
Close view, keyed	33	13	8	4	58	18	0.11
Close view, all	35	13	9	4	61	18	0.10
Oracle, no look	39	13	10	4	66	18	0.00

Table 4: Ablations (kinematic, strict) on development (Dev.) and held-out (Held.) episodes, with median final error (mm, development). The look uses the height map of the scene cameras or of the close view, for keyed sketches or all parts (ours: close view, all). Bottom: ground-truth sketches.

Sketch source	Sketch	Strict	Search	Physics	Time
Agent	13	30	31	28	$90\text{\,}\mathrm{s}$
Memory, $k=1$	9	30	31	32	$3.4\text{\,}\mathrm{s}$
Memory, $k=2$	9	29	30	–	$6.6\text{\,}\mathrm{s}$
Memory, $k=4$	9	28	29	–	$15.5\text{\,}\mathrm{s}$
Averaged, $k=2$	4	30	30	–	$3.1\text{\,}\mathrm{s}$
Averaged, $k=4$	4	29	29	–	$3.4\text{\,}\mathrm{s}$

Table 5: Reusing sketches on four new placements of eight tasks (32 episodes). The memory stores

k

agent runs per task and uses the best-registering one or, in the averaged variant, their average. Sketch: strict success of the sketch alone. Time: median time to produce the sketch.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The VLM decides what to measure, and most numbers of its sketch come from the tools. One typical run of GPT-6.1-Sol on a held-out task, a DC barrel plug and its jack. The strip lists the 15 calls of the run, grouped by the code block that sent them, and the cards show the calls with filled chips, with what the VLM sent (purple) and what the tool returned (green), shortened with “…”. Marks show where each number of the sketch comes from, a tool (green) or the VLM (purple). check scores each primitive by the share of its surface seen by the cameras that lies within $2\text{\,}\mathrm{m}\mathrm{m}$ of the observed depth ( match ). The body cylinder of the draft, with a radius set by the VLM itself, scores 0.61, and 0.88 after xyz and fit measure the radius. The insets are drawn from the logs: the pixel sent to hole and the rim it returned on a zoom of the crop of call 4, and the points, draft and fit seen from above.

Search	Development (34)	Held-out (14)	Reuse tasks (50)
Gripper offset, $8\text{\,}\mathrm{m}\mathrm{m}$	22	6	35
Gripper offset, $3\text{\,}\mathrm{m}\mathrm{m}$	18	–	–
Tip position, $8\text{\,}\mathrm{m}\mathrm{m}$	24	7	–
Tip position, $3\text{\,}\mathrm{m}\mathrm{m}$	25	8	30
Outline centre, $3\text{\,}\mathrm{m}\mathrm{m}$ (ours)	25	8	34

Appendix

Table 6: Search frame and window of the look (kinematic, strict, height map of the scene cameras) on the episodes with a keyed sketch: development, held-out, and the placements of the reuse tasks that are not in the development set.

Figure 6: Tasks. Examples of the five task sets, rendered from the assets of the simulator (moving part in orange, lifted above its assembled pose, and fixed part in blue). The held-out connectors were built after the method was fixed and were not used during development.

Set	Moving part	Sym.	Clear. (mm)	Lat. (mm)	Rot. (deg)	Start
Connectors	NEMA 5-15 power plug	1	0.3	0.34	1.8	holder
	NEMA 1-15 power plug	1	0.3	0.34	2.8	holder
	USB Type-A	1	0.15	0.20	2.2	holder
	RJ45 (8P8C)	1	0.15	0.20	2.1	holder
	D-sub 9	1	0.15	0.20	1.5	stands
	rectangular peg (Factory)	2	0.124	0.17	1.3	stands

Appendix

Table 7: Procedural and AIC tasks: rotational symmetry of the mate, clearance per side, lateral and rotational tolerance, and whether the moving part stands on the table or in a holder.

Robotic Insertion

Robotic insertion studies how robots place objects through openings or into tissue despite tight tolerances, uncertain alignment, and contact forces.

See all 52 papers

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation
Oct 31, 2026Jiaming WangAgentic AI for Robotics World Models for Robotics
Generalizable Robotic Insertion with World Models
Sep 23, 2026Nicklas Hansen, Iretiayo Akinola, Yijie Guo +7Zero-Shot Robotic Manipulation World Models for Robotics
NVIDIA · University of California San Diego · Work completed during an internship at NVIDIA +1
GRAFT: Graph-Based Affordance Transfer via Part Correspondence
Jun 23, 2026Mengying Lin, Utkarsh Mishra, Ajay Mandlekar +1Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
Georgia Institute of Technology · NVIDIA

Oct 31, 2026J/K move · Enter open · S save

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Jiaming Wang

Agentic AI for Robotics World Models for Robotics Zero-Shot Robotic Manipulation World Model-Based Planning Embodied World Models

Open paper AskPDF arXiv

Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.

Abstract

Robotic Insertion

Explore similar work

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Generalizable Robotic Insertion with World Models

GRAFT: Graph-Based Affordance Transfer via Part Correspondence