cs.RONov 7, 2026

Zero-Shot VLA Adaptation with Retrieved Guidance

Authors: Subin Kim, Anh-Quan Pham, Dinesh Jayaraman

Organizations: University of Pennsylvania

Abstract

Pre-training large parametric "Vision-Language-Action" (VLA) models on large demonstration datasets has recently yielded promising results for general-purpose manipulation. However, these models still struggle under even small domain shifts relative to the training data, such as changes in camera viewpoint, background, object identity, or robot morphology. Motivated by the success of Retrieval-Augmented Generation (RAG) in language models, we pursue retrieval to boost VLA performance. Specifically, we seek to leverage retrieval from domain-mismatched, "in-the-wild" demonstration datasets like DROID to guide a VLA's behavior in a new target domain. To achieve this, we introduce RAGu, which overcomes robotics-specific challenges for RAG: (1) It incorporates simple geometric calibration to transform source demonstrations into the target setup and reduce cross-domain differences. (2) It uses a coarse-to-fine retrieval strategy that selects, from a large pool of pre-warped candidates, the trajectory most compatible with the base VLA. (3) It guides the VLA's flow-matching inference procedure rather than seeking to retrain the base model or expand its limited context window. We evaluate RAGu across real-world, simulation, sim-to-real transfer, and cross-embodiment humanoid manipulation. We find it improves real-world VLA success rates from 20% to 80% and outperforms the closest-related baselines in sim. Project website: raguvla.github.io

Figures & tables

Explore similar work

CardsList
  1. InteractVLA: Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

    Nov 17, 2026Chen Zhao, Zipeng Ye, Shifeng Bao +5Agentic AI for RoboticsTool-Using Vision-Language Agents

  2. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Long-Horizon Robotic ManipulationLanguage-Conditioned Robot Manipulation

  3. SimVLA: A Simple VLA Baseline for Robotic Manipulation

    Feb 20, 2026Yuankai Luo, Woping Chen, Tong Liang +2Vision-Language-Action ModelsVLMs for Robotics