Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Pham Tri Quang; Bao-Ngoc Dao; Minh Duc Nguyen; Duc-Duy Nguyen; Hung T. Ho; Linh Dang Le; Trong-Bao Ho; Thien-Loc Ha

cs.ROOct 27, 2026

Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Authors: Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen, Duc-Duy Nguyen, Hung T. Ho, Linh Dang Le, Trong-Bao Ho, Thien-Loc Ha,

Organizations: Center for AI Research, VinUniversity · VinRobotics · Hanoi University of Science and Technology · German Research Center for Artificial Intelligence (DFKI) ·

Abstract

Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, $π_{0.5}$ , and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/

Figures & tables

Figure 1: R2A organizes offline training demonstrations into a pure kinematic motion primitive graph. At test time, it projects OOD initial states back to the training manifold via diffusion, continuously retrieves context-aligned geometric priors $\mathbf{A}_{t}^{\text{ref}}$ via a kinematic query $Q_{t}$ , and harmonizes them with the frozen VLA’s predictions $\mathbf{A}_{t}^{\pi}$ via a dynamic gating matrix $\mathbf{S}_{t}$ to ensure robust execution under severe visual shifts.

Model	Camera	Robot	Language	Lighting	Background	Noise	Layout	Avg.
OpenVLA	0.8	3.5	23.0	8.1	34.8	15.2	28.5	16.3
OpenVLA-OFT	56.4	31.9	79.5	88.7	93.3	75.8	74.2	71.4
NORA	2.2	37.0	65.1	45.7	58.6	12.8	62.1	40.5
WorldVLA	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.6
UniVLA	1.8	46.2	69.6	69.0	81.0	21.2	31.9	45.8
$\pi_{0}$	13.8	6.0	58.8	85.5	81.4	79.0	68.9	56.2

Table 1: Success rates (%) under seven LIBERO-Plus perturbation categories.

Figure 3: R2A under Sensor Noise (GR00T N1.7). The query retrieves a demonstration going down to the book, and the executed action blends it with the policy. Background shown from a clean frame for readability.

Table 5

Figure 4: Comparison of Inference Latency and Task Success Rate across methods

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Paired success outcomes. (a) Paired outcomes for each perturbation axis. Each bar shows the fraction of episodes where both policies succeed, R2A fixes a GR00T N1.7 failure, R2A breaks a GR00T N1.7 success, or both policies fail. The value on the right is the net success-rate gain of R2A. (b) Per-task success-rate gain of R2A over GR00T N1.7 for each perturbation axis. Blue indicates improvement, orange indicates regression, and gray indicates no change.

Figure 6: Solving earlier. Cumulative fraction of episodes solved by each environment step, pooled across all perturbation axes and shown separately for each axis. The shaded region shows the gap between R2A and GR00T N1.7.

\Delta

median denotes the change in median solve step (R2A minus GR00T N1.7), where negative values indicate earlier success.

Figure 7: Robustness under increasing perturbation. Success rate of GR00T N1.7 and R2A as a function of robot start offset, camera perturbation type, camera orbit angle, and sensor-noise type. Numbers above the bars indicate the success-rate gain of R2A in percentage points.

Figure 8: Level-5 LIBERO-Long rollouts for Camera Viewpoints (top) and Robot Initial States (bottom). In each block, the first two rows compare GR00T N1.7 with and without R2A, and the third row details one R2A step. Camera Viewpoints: alone, the policy keeps pushing after placing the soup and never returns for the cream cheese; at step 104, the retrieved segment releases and rises (1.2 cm executed vs. 0.2 cm from the policy), and R2A finishes at step 242. Robot Initial States: alone, the policy heads for the microwave with an empty gripper; R2A first restores the start pose, then at step 48 retrieval moves the open gripper down toward the mug, and it finishes at step 225.

Figure 9: Real-world demonstration samples for Banana2Pot (top) and Cube2Drawer (bottom). Each row shows one recorded episode (000, 075, and 149), and the columns go from the initial scene to the final scene. Frames come from the third-person camera and are cropped to reduce background.

Policy	Benchmark	Checkpoint
GR00T N1.7	LIBERO-Plus	nvidia/GR00T-N1.7-LIBERO (per suite)
$\pi_{0.5}$	LIBERO-Plus	pi05_libero (openpi)
StableVLA	LIBERO-Plus	DAGroup-PKU/StableVLA (per suite)
$\pi_{0.5}$	RoboTwin2.0-Plus	RoboTwin2.0 co-train, joint †

Appendix

Table 6: Base policy checkpoints, all used as released. Checkpoint names link to their sources.

Module	Architecture	Params	Training
Locator (GNN)	3 message-passing layers, width 64, 16 relation types	0.11M	30 epochs
Retriever	2-layer Transformer encoder (width 64, 8 heads) over the last 8 poses; segment key and action MLPs	0.22M	jointly, 30 epochs, lr $10^{-3}$ , weight decay $10^{-5}$ , 4,096 queries per epoch in batches of 256, 8 hard negatives per query; harmonization loss from epoch 5
Harmonization	shape head (MLP, width 128) and trust head (MLP, width 64)	5k
Start-state alignment	score network (width 128, denoising score matching) and inverse-dynamics network (width 64)	42k	6,000 steps each
Total		0.37M

Appendix

Table 7: R2A modules, sizes and training settings, identical for LIBERO-Plus and RoboTwin2.0-Plus. Adam throughout.

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 221 papers

ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency
Jul 31, 2026Wenda Yu, Tianshi Wang, Fengling Li +3Robot Safety Robotic Manipulation
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · Shanghai Artificial Intelligence Laboratory +1
PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation
May 27, 2026Yutai Li, Shaohui Peng, Jiaming Guo +8Long-Horizon Robotic Manipulation Robot Skill Learning
State Key Lab of Processors, Institute of Computing Technology, CAS · Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS · University of Chinese Academy of Sciences +3
Reflective VLA: In-Context Action Consequences Make VLAs Generalize
Jun 23, 2026Qing Lian, Kent Yu, Lei ZhangICL for Robotics Generalization in Robotic Manipulation
Futian Laboratory · International Digital Economy Academy (IDEA) · Visincept

Jul 31, 2026J/K move · Enter open · S save

ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency

Wenda Yu, Tianshi Wang, Fengling Li, Xin Li+2

Tongji University · Mohamed bin Zayed University of Artificial Intelligence · Shanghai Artificial Intelligence Laboratory+1

Robot Safety Robotic Manipulation Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) policies achieve strong performance in robotic manipulation but remain vulnerable to runtime disturbances that break the temporal alignment among visual observations, robot states, and executed actions. We introduce ActFovea, a plug-and-play safeguarding framework that detects and mitigates such failures without retraining or modifying the underlying VLA policy. ActFovea uses robot kinematics, proprioceptive states, and recent actions to construct action-conditioned foveated regions that retain contact-relevant areas and predicted motion corridors while suppressing task-irrelevant visual content. It detects runtime risks by evaluating whether visual motion and observation freshness remain consistent with geometric, proprioceptive, and action transitions. For recoverable disturbances, ActFovea constructs disturbance-specific candidate observations and accepts a recovery only after verifying the resulting action chunk. When stale or replayed observations make reliable recovery impossible, it invokes a bounded safe-failure procedure. In closed-loop evaluations of

π_0

across multiple LIBERO suites, ActFovea increases success under localized visual overlays from 49.3% to 90.3%, closing 93.7% of the gap to clean performance. It further improves success under action drift and visual delay by 7.0 and 9.8 percentage points, respectively, while preserving clean-task performance. Under frozen-observation replay, ActFovea triggers timely safe failure in all trials, with no unprotected failures. These results demonstrate that spatiotemporal visual-action consistency provides an effective basis for runtime safeguarding of VLA policies.

Abstract

Robustness of VLA Models

Explore similar work

ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency

PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation

Reflective VLA: In-Context Action Consequences Make VLAs Generalize