Spatial Grafting: Grounding 3D Features for Flow-Matching Robot Policies

Dingsheng Liu; Yangzheng Wu; Mahboubeh Asadi; Zhiyuan Li; Jinbang Huang; Yixin Xiao; Tongtong Cao; Yingxue Zhang

cs.ROSep 28, 2026

Spatial Grafting: Grounding 3D Features for Flow-Matching Robot Policies

Authors: Dingsheng Liu, Yangzheng Wu, Mahboubeh Asadi, Zhiyuan Li, Jinbang Huang, Yixin Xiao, Tongtong Cao, Yingxue Zhang

Organizations: University of Toronto · Huawei Noah’s Ark Lab · Department of Foundation Model, 2012 Labs

Abstract

Pretrained robot manipulation policies such as vision-language-action models (VLAs) or world-action models (WAMs) leave interaction-relevant metric geometry implicit. Recent breakthroughs in spatial reconstruction can supply the necessary geometry reliably, but their features describe local shape without stating where it lies with respect to the robot. How best to deliver these features to a pretrained policy remains unresolved. We propose Spatial Grafting, a versatile, lightweight spatial module that binds frozen reconstruction features to metric, robot-relative geometry. Spatial Grafting constructs metric-grounded spatial tokens and injects them into the flow-matching action expert through cross-attention, without modifying the host's perceptual pathway, so the host retains the full benefit of its pretraining. We evaluate it more broadly than any geometry-aware policy we compare against: one graft architecture, with no per-host redesign, on two VLAs and two WAMs, across four simulation benchmarks that span short-horizon manipulation, visual robustness, clutter and long-horizon mobile manipulation, and on three real-robot platforms with single- and dual-arm configurations. On RoboTwin 2.0, a dual-arm manipulation benchmark, the graft improves every host across VLAs and WAMs. Grafted $π_{0.5}$ gains 11.3% and 15.6% on clean and randomized scenes, reaching 94.0% and 92.4%, above the strongest published 3D-conditioned policy, WAM4D (93.8% and 89.9%). The margin widens as the horizon lengthens: on tasks from BEHAVIOR-1K, a dual-arm mobile manipulation challenge scored by average task progress, it surpasses the 2025 challenge winner on five of six tasks,by up to 0.47 Q-score, and exceeds a map-conditioned spatial policy on average across the three tasks both report.

Figures & tables

Fig. 1: Overview of Spatial Grafting . A frozen geometry foundation model extracts multi-level features from each RGB view. Depth, calibrated rays, and end-effector anchors bind those features to metric, robot-relative 3D locations, forming a spatial bank per view. The magnified graft layer acts only on the flow-matching action expert: action tokens first cross-attend to the primary-view bank; the updated states then query the auxiliary-view banks. A single bias-free linear projection fuses the auxiliary attention outputs into the action update. This layer repeats in selected late action blocks, leaving the host’s vision-language or video pathway ungrafted.

Host	Grafted blocks	Bank builder	Injection	Bridges	Total (share)
$\pi_{0.5}$	6	46.0M	100.7M	–	146.8M ( $4.4\%$ of 3.3B)
X-VLA	6	46.0M	100.7M	–	146.8M
$\pi_{0.5}$ , VGGT- $\Omega$	6	48.1M	100.7M	–	148.9M
Fast-WAM	10	46.0M	168.0M	–	214.0M
LingBot-VA	10	46.0M	168.0M	62.9M	276.9M ( $5.4\%$ of 5.09B)

TABLE I: Trainable parameters added by the graft on RoboTwin. Counts are measured by building the modules. The share is relative to the host’s own parameter count.

Model	LIBERO Avg.		RoboTwin Clean		RoboTwin Rand.
GLaD	94.1		NR		NR
GeoVLA	97.7		NR		NR
SpatialVLA	78.1		NR		NR
Spatial Forcing	98.5		66.2		58.9
WAM4D	NR		93.8		89.9
$\pi_{0.5}\rightarrow\pi_{0.5}^{\mathrm{graft}}$	96.9 $\rightarrow$	99.6 $+2.7$	82.7 $\rightarrow$	94.0 $+11.3$	76.8 $\rightarrow$	92.4 $+15.6$

TABLE II: LIBERO and RoboTwin SR (%). Above the line: published 3D-conditioned policies, quoted from their own papers, except Spatial Forcing on RoboTwin, which we trained ourselves (Section VII ). Below the line: our four hosts. LIBERO bases are quoted from each host’s published results; the

\pi_{0.5}

entry is from the openpi release [ 41 ] , as its paper reports no LIBERO result. On RoboTwin, the

\pi_{0.5}

base is cited from [ 14 ] and the X-VLA base from [ 42 ] , both trained on the same RoboTwin demonstrations as our grafts, so the pairs are directly comparable; the Fast-WAM and LingBot-VA bases are our evaluations of their public checkpoints with our seeds and pipeline. Every grafted entry is our run. Green/red: absolute change from the host’s own base. NR: not reported in the cited paper.

Task	SERF	$\pi_{0.5\text{-}\mathrm{Comet}}$	$\pi_{0.5\text{-}\mathrm{RLC}}$ $\rightarrow$	$\pi_{0.5\text{-}\mathrm{RLC}}^{\mathrm{graft}}$
Assembling gift baskets	0.525	0.312	0.281 $\rightarrow$	0.632 $+0.351$
Collecting children’s toys	0.635	0.583	0.500 $\rightarrow$	0.673 $+0.173$
Putting shoes on rack	0.601	0.470	0.720 $\rightarrow$	0.590 $-0.130$
Putting up Christmas decorations	NR	0.356	0.478 $\rightarrow$	0.510 $+0.032$
Clean boxing gloves	NR	0.000	0.200 $\rightarrow$	0.600 $+0.400$
Cleaning up plates and food	NR	0.043	0.243 $\rightarrow$	0.714 $+0.471$

TABLE III: B1K Q-score on six tasks. Q-score is average task progress.

\pi_{0.5\text{-}\mathrm{RLC}}

and

\pi_{0.5\text{-}\mathrm{Comet}}

are the #1 and #2 entries of the 2025 challenge [ 8 ] and SERF [ 25 ] is quoted on the three tasks it reports; all three are taken from their published scores. The grafted policy is evaluated on the 20 public evaluation instances of each task; official challenge scoring uses only 10 of those 20. Green/red: change from the host’s own base.

Panel of Figure 5 — Fig. 2: Grafted success (top) against base failure (bottom), one column per benchmark. Each column is the task discussed in the corresponding paragraph. R1Pro: the grafted policy estimates plate height and releases the orange with clearance, where the base drops it short. Piper: plate–rack alignment avoids the collision the base drives into. B1K: the base shuts its jaws beside the candy cane’s shaft; the grafted policy closes on it. RoboTwin: rack height is estimated well enough to clear the hook. RoboPRO: a lower, better-placed grasp seats the plate in the sink instead of catching its rim.

Spatial Grafting : base $\rightarrow$ graft
Model	Clean (Easy/Hard)		Clutter (Easy/Hard)
$\pi_{0.5}\rightarrow\pi_{0.5}^{\mathrm{graft}}$	70.3 / 65.7 $\rightarrow$	78.8 / 73.7 $+8.5/{+}8.0$	60.9 / 47.8 $\rightarrow$	69.8 / 49.4 $+8.9/{+}1.6$
X-VLA $\rightarrow$ X-VLA graft	49.5 / 46.7 $\rightarrow$	60.9 / 49.9 $+11.4/{+}3.2$	39.7 / 27.6 $\rightarrow$	45.3 / 35.3 $+5.6/{+}7.7$

TABLE IV: RoboPRO success (%). Each cell reads Easy/Hard. Easy is RoboPRO’s success rate (SR); Hard is its stricter harsh success rate (HSR), scored on the same episodes. Bases are reported by [ 7 ] and grafted scores are ours. Green: change from the host’s own base.

Platform	Task	SR		Trials
UR5e	Pick up marker and place in pen cup (thin objects, constrained placement)	30 $\rightarrow$	80 $+50$	10
Piper	Clean up table (transparent objects, long horizon)	20 $\rightarrow$	70 $+50$	10
	Open tea can (bimanual dexterous manipulation)	10 $\rightarrow$	40 $+30$	10
	Place plate on rack (thin objects, constrained placement)	70 $\rightarrow$	90 $+20$	10
R1Pro	Pick up tangerine and place on plate (thin objects, constrained placement)	50 $\rightarrow$	90 $+40$	10
R1Pro	Clean up table (deformable object, long horizon)	50 $\rightarrow$	80 $+30$	10

TABLE V: Real-robot evaluation (SR, %). Matched base/graft comparisons on three platforms, with 10 trials per task. A trial is considered successful only if all sub-tasks are completed. Green: change from the base.

Fig. 3: Ablations on RoboTwin with

\pi_{0.5}

(SR %). (a) Reference, ungrafted base, and bank interventions on a fixed checkpoint. (b) One design choice retrained at a time. Equal-width Clean (green) and Randomized (terracotta) bars overlap at each configuration and share a 25% baseline.

Panel of Figure 9 — Fig. 4: Success and failure modes across platforms. Left and right show successful and failed rollouts. For R1Pro fruit placement, improved plate-height estimation leaves clearance above the rim, while the failure approaches too low. For Piper plate placement, better plate–rack alignment avoids collision; the two panels use left- and right-wrist fisheye cameras, respectively. In B1K candy pickup and RoboPRO plate pickup, target depth supports a lower grasp pose. In RoboTwin mug hanging, estimating the rack height positions the mug at the support. The added Piper tea-can opening and UR5e marker-grasp panels are cropped and AI-enhanced for visual clarity; generative deblurring may reconstruct fine details, so these panels are illustrative rather than unaltered experimental frames.

Method	Status	Geometry signal	Route into the model	Test-time inputs	Evaluation
Hosts: VLAs and WAMs
$\pi_{0.5}$	CoRL’25	None	VLM trunk co-trained with a flow-matching action expert	RGB	Real
X-VLA	ICLR’26	None	Embodiment-specific soft prompts into a flow-matching transformer	RGB	LIBERO; SimplerEnv; CALVIN; VLABench; RoboTwin; Real
Fast-WAM	arXiv’26	None	Video and action experts in a shared mixture of transformers; no video generation at inference	RGB	LIBERO; RoboTwin; Real
LingBot-VA	RSS’26	None	Autoregressive video/action tokens in a mixture of transformers	RGB	LIBERO; RoboTwin; Real
Spatial action models compared in Section V

TABLE VI: Host policies and geometry-aware policies. Rows follow Section II . Status is the venue as of submission; “Real” denotes real-robot experiments; test-time inputs are what the deployed policy consumes beyond proprioception. The four hosts carry no geometry module. Every geometry-aware method reports at most three evaluation settings; Spatial Grafting reports five, on four hosts.

Method	Spat.	Obj.	Goal	Long	Avg.
$\pi_{0}$	96.8	98.8	95.8	85.2	94.1
OpenVLA-OFT	97.6	98.4	97.9	94.5	97.1
GLaD	95.0	97.4	94.4	89.4	94.1
GeoVLA	98.4	99.0	96.6	96.6	97.7
Spatial Forcing	99.4	99.6	98.8	96.0	98.5
SpatialVLA	88.2	89.9	78.6	55.5	78.1

TABLE VII: Detailed LIBERO SR (%). Results across the four standard suites. Values for external methods are reported by their respective works; the

\pi_{0.5}

base is from the openpi release [ 41 ] . Host averages match Table II ; dashes indicate unavailable per-suite results, not zero success. Blocks group other baselines, existing 3D methods, and our hosts. Green/red: change from the host’s own base.

Task	$\pi_{0.5}^{\mathrm{graft}}$	X-VLA graft	Fast-WAM	Fast- WAM graft	LingBot-VA	LingBot- VA graft	$\pi_{0.5}^{\mathrm{graft}}$ (VGGT- $\Omega$ )
Adjust bottle	100/100	100/95	100/95	100/100	100/100	95/95	100/100
Beat block hammer	100/100	90/85	100/100	100/95	100/100	100/100	100/95
Blocks ranking RGB	100/95	100/85	95/85	100/90	75/90	100/100	100/100
Blocks ranking size	85/70	75/85	60/65	65/85	85/50	95/90	85/75
Click alarmclock	95/100	10/30	100/100	100/100	100/100	100/100	95/100
Click bell	100/100	70/90	100/100	100/100	100/100	100/100	95/100

TABLE VIII: Per-task RoboTwin SR (%), Clean/Randomized. Each cell reads Clean/Randomized over exactly 20 seeds per task; every model and evaluation uses the same seeds. DA3 is the spatial backbone unless marked VGGT-

\Omega

; Fast-WAM and LingBot-VA without a superscript are the ungrafted public checkpoints. The average is the mean of the 50 per-task rates.

Task	Clean	Clutter	Clean	Clutter
	$\pi_{0.5}^{\mathrm{graft}}$		X-VLA graft
Chain apple bin bowl rack spoon sink	15/15	10/10	0/0	0/0
Chain apple sink plate bread board	95/85	100/90	0/0	0/0
Chain bowl rack apple sink	0/0	0/0	0/0	0/0
Chain heat hamburger	45/45	85/75	20/20	0/0
Chain serve hamburger	45/45	0/0	0/0	0/0

TABLE IX: Per-task RoboPRO success (%), Easy/Hard. Each cell reads Easy (SR)/Hard (HSR) over exactly 20 seeds per task for all 80 tasks in each configuration, the same seeds for every model; averages are the mean of the 80 per-task rates and match Table IV .

Explore similar work

STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation
Apr 29, 2026Yuxuan Tian, Yurun Jin, Bin Yu +5Robotic Manipulation World Models
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +4
Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluation
Sep 27, 2026Sichao Liu, Zekun Wang, Lixuan Tang +6Robotic Manipulation Policies Robot Systems
KTH · EPFL · Idiap Research Institute +3
Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization
Jun 26, 2026Shiang-Feng Tsai, Jin-Cheng Jhang, Yen-Ling Tai +5Robotic Manipulation Recent Vision-Language Models
National Tsing Hua University · National Yang Ming Chiao Tung University

Apr 29, 2026, cs.ROJ/K move · Enter open · S save

STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

Yuxuan Tian, Yurun Jin, Bin Yu, Yukun Shi+4

Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence+4

Robotic Manipulation World Models World Geometry-Aware

Open paper AskPDF arXiv

Robotic manipulation requires reasoning about future spatial-temporal interactions and geometric constraints, yet existing Vision-Language-Action (VLA) policies often leave predictive representation weakly coupled with action execution, causing failures in tasks requiring precise spatial-temporal coordination. We propose STARRY, a world-model-enhanced action-generation policy that aligns spatial-temporal prediction and action generation by jointly denoising future spatial-temporal latents and actions through a unified diffusion process. To bridge 2D visual tokens and 3D metric control, STARRY introduces Geometry-Aware Selective Attention Modulation (GASAM), which converts predicted depth and end-effector geometry into token-aligned weights for selective action-attention modulation. On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings across 50 bimanual tasks. Real-world experiments show that STARRY improves average success from 42.5% to 70.8% compared with

π_{0.5}

. These results demonstrate the effectiveness of action-centric spatial-temporal world modeling for spatially and temporally demanding robotic manipulation.

Abstract

Explore similar work

STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluation

Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization