Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution
Authors: Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen, Duc-Duy Nguyen, Hung T. Ho, Linh Dang Le, Trong-Bao Ho, Thien-Loc Ha, Tan Q. Nguyen, Tuan Dam, Vu N. Duong, Daniel Sonntag, Ngan Le, Khoa D. Doan, Jan Peters, An Thai Le, Duy M. H. Nguyen, Vien Anh Ngo
Organizations: Center for AI Research, VinUniversity · VinRobotics · Hanoi University of Science and Technology · German Research Center for Artificial Intelligence (DFKI) · University of Oldenburg · University of Arkansas · Technische Universität Darmstadt · Universität Stuttgart · Max Planck Research School for Intelligent Systems (IMPRS-IS)
Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, π0.5, and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/
Figures & tables
Figure 1: R2A organizes offline training demonstrations into a pure kinematic motion primitive graph. At test time, it projects OOD initial states back to the training manifold via diffusion, continuously retrieves context-aligned geometric priors Atref via a kinematic query Qt , and harmonizes them with the frozen VLA’s predictions Atπ via a dynamic gating matrix St to ensure robust execution under severe visual shifts.
Model
Camera
Robot
Language
Lighting
Background
Noise
Layout
Avg.
OpenVLA
0.8
3.5
23.0
8.1
34.8
15.2
28.5
16.3
OpenVLA-OFT
56.4
31.9
79.5
88.7
93.3
75.8
74.2
71.4
NORA
2.2
37.0
65.1
45.7
58.6
12.8
62.1
40.5
WorldVLA
0.1
27.9
41.6
43.7
17.1
10.9
38.0
25.6
UniVLA
1.8
46.2
69.6
69.0
81.0
21.2
31.9
45.8
π0
13.8
6.0
58.8
85.5
81.4
79.0
68.9
56.2
Table 1: Success rates (%) under seven LIBERO-Plus perturbation categories.
Figure 3
Figure 3: R2A under Sensor Noise (GR00T N1.7). The query retrieves a demonstration going down to the book, and the executed action blends it with the policy. Background shown from a clean frame for readability.
Table 5
Figure 4: Comparison of Inference Latency and Task Success Rate across methods
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Paired success outcomes. (a) Paired outcomes for each perturbation axis. Each bar shows the fraction of episodes where both policies succeed, R2A fixes a GR00T N1.7 failure, R2A breaks a GR00T N1.7 success, or both policies fail. The value on the right is the net success-rate gain of R2A. (b) Per-task success-rate gain of R2A over GR00T N1.7 for each perturbation axis. Blue indicates improvement, orange indicates regression, and gray indicates no change.
Figure 6: Solving earlier. Cumulative fraction of episodes solved by each environment step, pooled across all perturbation axes and shown separately for each axis. The shaded region shows the gap between R2A and GR00T N1.7. Δ median denotes the change in median solve step (R2A minus GR00T N1.7), where negative values indicate earlier success.
Figure 7: Robustness under increasing perturbation. Success rate of GR00T N1.7 and R2A as a function of robot start offset, camera perturbation type, camera orbit angle, and sensor-noise type. Numbers above the bars indicate the success-rate gain of R2A in percentage points.
Figure 8: Level-5 LIBERO-Long rollouts for Camera Viewpoints (top) and Robot Initial States (bottom). In each block, the first two rows compare GR00T N1.7 with and without R2A, and the third row details one R2A step. Camera Viewpoints: alone, the policy keeps pushing after placing the soup and never returns for the cream cheese; at step 104, the retrieved segment releases and rises (1.2 cm executed vs. 0.2 cm from the policy), and R2A finishes at step 242. Robot Initial States: alone, the policy heads for the microwave with an empty gripper; R2A first restores the start pose, then at step 48 retrieval moves the open gripper down toward the mug, and it finishes at step 225.
Figure 9: Real-world demonstration samples for Banana2Pot (top) and Cube2Drawer (bottom). Each row shows one recorded episode (000, 075, and 149), and the columns go from the initial scene to the final scene. Frames come from the third-person camera and are cropped to reduce background.
Policy
Benchmark
Checkpoint
GR00T N1.7
LIBERO-Plus
nvidia/GR00T-N1.7-LIBERO (per suite)
π0.5
LIBERO-Plus
pi05_libero (openpi)
StableVLA
LIBERO-Plus
DAGroup-PKU/StableVLA (per suite)
π0.5
RoboTwin2.0-Plus
RoboTwin2.0 co-train, joint †
Appendix
Table 6: Base policy checkpoints, all used as released. Checkpoint names link to their sources.
2-layer Transformer encoder (width 64, 8 heads) over the last 8 poses; segment key and action MLPs
0.22M
jointly, 30 epochs, lr 10−3 , weight decay 10−5 , 4,096 queries per epoch in batches of 256, 8 hard negatives per query; harmonization loss from epoch 5
Harmonization
shape head (MLP, width 128) and trust head (MLP, width 64)
Vision-language-action (VLA) policies achieve strong performance in robotic manipulation but remain vulnerable to runtime disturbances that break the temporal alignment among visual observations, robot states, and executed actions. We introduce ActFovea, a plug-and-play safeguarding framework that detects and mitigates such failures without retraining or modifying the underlying VLA policy. ActFovea uses robot kinematics, proprioceptive states, and recent actions to construct action-conditioned foveated regions that retain contact-relevant areas and predicted motion corridors while suppressing task-irrelevant visual content. It detects runtime risks by evaluating whether visual motion and observation freshness remain consistent with geometric, proprioceptive, and action transitions. For recoverable disturbances, ActFovea constructs disturbance-specific candidate observations and accepts a recovery only after verifying the resulting action chunk. When stale or replayed observations make reliable recovery impossible, it invokes a bounded safe-failure procedure. In closed-loop evaluations of π0 across multiple LIBERO suites, ActFovea increases success under localized visual overlays from 49.3% to 90.3%, closing 93.7% of the gap to clean performance. It further improves success under action drift and visual delay by 7.0 and 9.8 percentage points, respectively, while preserving clean-task performance. Under frozen-observation replay, ActFovea triggers timely safe failure in all trials, with no unprotected failures. These results demonstrate that spatiotemporal visual-action consistency provides an effective basis for runtime safeguarding of VLA policies.
Wenda Yu, Tianshi Wang, Fengling Li +3
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · Shanghai Artificial Intelligence Laboratory +1
Vision-Language-Action (VLA) models offer a promising paradigm for generalist robotic policies, yet their adaptation is hindered by data inefficiency and poor generalization. We argue that these bottlenecks stem from the prevailing Direct Instruction-to-Control Mapping, which forces models to memorize monolithic trajectories rather than reusable motion patterns, i.e., primitives. We propose PrimitiveVLA, a framework that shifts this paradigm toward a Primitive-Centric Disassemble & Assemble paradigm. Supported by a shared Multimodal Canonical Representation (MCR), PrimitiveVLA unifies two phases: (1) Fine-tuning-phase Disassembly, which uses an automated pipeline to disassemble demonstrations into reusable primitives; and (2) Inference-phase Assembly, which employs a VLM-based planner and an LLM-generated switch module for robust closed-loop execution. By disassembling tasks into reusable primitives, PrimitiveVLA enables VLA models to learn invariant motion patterns instead of task-specific trajectories. Extensive experiments show that our framework improves data efficiency and achieves superior zero-shot generalization across unseen and long-horizon tasks.
Yutai Li, Shaohui Peng, Jiaming Guo +8
State Key Lab of Processors, Institute of Computing Technology, CAS · Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS · University of Chinese Academy of Sciences +3
Most vision-language-action (VLA) models are reactive: they predict the next action from the current instruction and observation, implicitly assuming that the current observation fully specifies the action-relevant state. In embodied control, however, embodiment-specific factors such as camera-to-robot geometry, robot calibration, or systematic actuation bias are often hard to identify from a single observation. As a result, reactive policies cannot reliably disambiguate these factors in general, overfitting to training environments and generalizing poorly at deployment. We propose Reflective VLA, which conditions each decision on a context of observation-action-consequence triplets. Each triplet records not only what the robot observed and executed, but also how the scene changed afterward, exposing the deployment-specific mapping from actions to observed effects. Architecturally, Reflective VLA routes all observation modalities through the VLM under shared attention, so the action expert reasons directly over past triplets and the current observation. A block-causal mask enables parallel multi-frame training without leakage and supports KV-cached real-time inference. On standard LIBERO and SimplerEnv-Bridge, Reflective VLA preserves strong in-distribution performance. Under distribution shift on LIBERO-Plus and the harder LIBERO-Plus-Hard, it improves average success rate by 5.4 and 4.2 percentage points over a matched reactive baseline. Ablations with a matched history-only baseline further show that action consequences -- rather than additional context length alone -- are the key to cross-environment generalization. Project page: https://lianqing11.github.io/reflective-vla-page/
Qing Lian, Kent Yu, Lei Zhang
Futian Laboratory · International Digital Economy Academy (IDEA) · Visincept