Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs

Jensen Gao; Dorsa Sadigh; Sandy Huang; Dhruv Shah

cs.ROMar 12, 2026

Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs

Authors: Jensen Gao, Dorsa Sadigh, Sandy Huang, Dhruv Shah

Organizations: Stanford University · Google DeepMind · Princeton University

Abstract

Recent work on robot manipulation has advanced policy generalization to novel scenarios. However, it is often difficult to characterize how different evaluation settings actually represent generalization from the training distribution of a given policy. To work towards more precise evaluation of generalization in robotics, we propose RADAR, a scalable framework for directly comparing test-time evaluation tasks to policy training data, to determine what form of policy generalization is required. RADAR consists of a two-stage pipeline: first, retrieval using generalist policy embeddings identifies which training examples are relevant for a given evaluation task. Next, vision-language models (VLMs) analyze the evaluation task against the retrieved data, outputting interpretable analysis on how they compare along a variety of axes, and an overall classification of what type of policy generalization is required. Through controlled experiments, we demonstrate that VLMs are effective at analyzing data for generalization, and that our retrieval step effectively identifies examples needed to make accurate classifications with respect to the training data. Furthermore, we scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions from prior work. We provide demonstrations at radar-analysis.github.io.

Figures & tables

Fig. 1 : We outline the two-stage pipeline of RADAR. Given an evaluation task, we first identify relevant examples from training data using nearest neighbors with embeddings from a generalist robot policy. Next, we use vision-language models to analyze the evaluation task against the retrieved examples to categorize what generalization it represents w.r.t. the training data: in-distribution (ID) , visual , or behavioral .

Fig. 2 : (Left) Test example $\tau^{\text{test}}$ for the task “put the lemon on the plate” . (Right) If $d^{*}$ (e.g., minor lighting change) is found in $\mathcal{D}$ , then $\tau^{\text{test}}$ is in-distribution (ID) . Otherwise, if $d^{v}$ (e.g., distractor objects) is found in $\mathcal{D}$ , then this is visual generalization. If neither case applies, then all $d\in\mathcal{D}$ involve different optimal behavior than $\tau^{\text{test}}$ (e.g., changed object poses), and this is behavioral generalization.

Fig. 3 : We visualize the three task families in our controlled experiments ( Pick-and-Place , Unzip Lunchbag , and Fold Dress ). We provide an example instance for each task (left), and variations of that task across different axes (right), with the specific change noted in parentheses.

Fig. 4 : Recall rate scaling (

y

-axis) of

d^{*}

for

\mathcal{D}_{\text{ID}}

as the size of the retrieval set (

x

-axis) increases. We find that VLA-based embeddings and DINOv3 significantly outperform the other retrieval methods.

Fig. 5 : Recall rate scaling (

y

-axis) of

d^{v}

for

\mathcal{D}_{\text{visual}}

as the size of the retrieval set (

x

-axis) increases.

\pi_{0.5}

and GROD outperform their counterparts trained on less robot data (

\pi_{0}

, GROD (25% Data)), and are much better than the baselines that do not use robot data.

Panel of Figure 6 — Fig. 6 : Distribution of embedding distances (L2 with unit-normalized embeddings, equivalent to cosine distance) induced by task perturbations across different axes for Pick-and-Place . VLA embeddings, especially for GROD, generally have larger distances for behavioral axes (blue) than visual axes (orange), unlike the other embeddings.

Task	VLM	Image Aug	Visual Task Obj	Visual Scene	Obj Poses	Morphed Obj	New Obj	Interacting Scene	Perturb Class	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Pick-and-Place	Gemini 3.1 Flash-Lite	39.6	81.9	39.7	29.3	–	87.5	–	78.9	95.0	7.0	78.0	60.0
	Gemini 3 Flash	94.7	67.3	92.7	38.2	–	97.5	–	96.7	95.6	64.4	97.9	86.2
	Gemini 3.1 Pro	79.1	69.4	98.9	73.9	–	99.4	–	98.3	94.9	82.7	100.0	92.5
Unzip Lunchbag	Gemini 3.1 Flash-Lite	52.8	84.9	75.6	75.0	55.3	–	–	83.4	100.0	4.0	82.0	62.0
	Gemini 3 Flash	95.4	96.6	89.5	81.2	65.8	–	–	88.3	95.9	58.3	51.0	68.5
	Gemini 3.1 Pro	74.8	97.3	100.0	88.8	69.5	–	–	89.7	93.8	71.4	54.0	72.8

TABLE I: (Left) VLM performance (F1 ×100) on pairwise prediction of per-axis differences and perturbation classification between

\tau^{\text{test}}

and

d\in\mathcal{D}_{\text{retrieval}}

. We do not report axes for which the task has no variation. (Right) Overall RADAR classification accuracy after aggregating VLM predictions across all

d\in\mathcal{D}_{\text{retrieval}}

, for each version of

\mathcal{D}

Fig. 7 : Examples of successful (green) and failed (red) predictions of Gemini 3 Flash for “Object Poses” and “Morphed Objects”. The VLM struggles more when comparing examples with more subtle visual changes (lunchbag rotation, slightly different dress size).

Task	Method	Image Aug	Visual Task Obj	Visual Scene	Obj Poses	Morphed Obj	Perturb Class	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Unzip Lunchbag	Gemini 3.1 Pro	74.8	97.3	100.0	88.8	69.5	89.7	93.8	71.4	54.0	72.8
	– Obj Detect	75.5	97.7	89.9	86.7	66.0	87.2	93.9	55.1	33.3	61.0
	– Obj Detect, Axes	–	–	–	–	–	79.9	90.0	40.0	6.0	45.3
	+ Longer Context ( $k=40$ )	73.1	95.4	99.4	87.8	68.1	89.7	96.0	62.5	44.0	67.6

TABLE II: Ablations of VLM analysis in RADAR. (Left) VLM performance (F1 ×100) on pairwise prediction of per-axis differences and perturbation classification between

\tau^{\text{test}}

and

d\in\mathcal{D}_{\text{retrieval}}

. (Right) Overall RADAR classification accuracy after aggregating VLM predictions.

Fig. 8 : Examples of RADAR applied to benchmarking generalization for a large-scale ALOHA 2 dataset. For a given benchmark evaluation task (left), we provide three instances of retrieved training data examples (middle). Then, we provide the VLM analysis comparing the evaluation task to the leftmost retrieved example, as well as the overall generalization classification (right).

VLM	Gemini 3 Flash	Gemini 3.0 Pro	Gemini 3.1 Pro
Agreement	60.0	71.4	77.1

TABLE III: Classification agreement of different VLMs in RADAR on BridgeV2-

\bigstar

[ 5 ] , compared against their original categorizations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Retrieval
Embedding feature	KV cache of image tokens from the 2nd-to-last transformer backbone layer (PaliGemma VLAs); cognition feature (CogACT)
Raw dimensionality	131072 ( $\pi_{0}$ , $\pi_{0.5}$ ); 4096 (CogACT)
Dim. reduction	Gaussian random projection to 2048 (KV-cache embeddings)
Distance metric	Cosine
Embedding inputs	VLA embeddings: camera views, language instruction $l$ , and 14-d proprioception (per arm: 6-d joint position + 1-d gripper position, as in ALOHA); CogACT omits proprioception. SigLIP 2/DINOv3 embed the overhead view only (DINOv3 omits language)
Retrieval size $k$	20 (main); 40 (long-context ablation); 50 (large-scale case studies)

Appendix

TABLE IV : Summary of retrieval and VLM analysis hyperparameters and inputs for RADAR.

Panel of Figure 13 — Fig. 9 : VLM prompt with $\bigstar$ -Gen axes guidance (Part 1).

Panel of Figure 14 — Fig. 10 : VLM prompt with $\bigstar$ -Gen axes guidance and object detection (Part 2).

Panel of Figure 15 — Fig. 11 : VLM prompt with $\bigstar$ -Gen axes guidance, but without object detection (Part 2).

Panel of Figure 16 — Fig. 12 : VLM prompt without $\bigstar$ -Gen axes guidance or object detection.

Fig. 13 : Set-level retrieval success for

\mathcal{D}_{\text{ID}}

: the fraction of evaluation tasks whose top-

k

retrieval set contains at least one

d^{*}

, as a function of the retrieval set size

k

, for GROD and

\pi_{0.5}

Fig. 14 : Set-level retrieval success for

\mathcal{D}_{\text{visual}}

: the fraction of evaluation tasks whose top-

k

retrieval set contains at least one

d^{v}

, as a function of the retrieval set size

k

, for GROD and

\pi_{0.5}

Panel of Figure 19 — Fig. 15 : Distinct object classes with functionally identical behavior, for the instruction “put utensil on plate” . Starting from a base scene with a knife (a), we swap in a fork (b) and spoon (c) while keeping the handle geometry and pose fixed, yielding visual perturbations that require the same behavior. Moving the original knife to a new pose (d) is a behavioral perturbation. We report the cosine distance ( $d_{\cos}$ ) of each image to the base scene in the CogACT policy embedding space used for retrieval. The different-class fork and spoon are closer to the base than the same-class repositioned knife, so embedding proximity tracks the required behavior rather than object-class identity.

Panel of Figure 20 — Fig. 16 : Stress-testing retrieval under increasing visual shifts, for the instruction “put knife on plate” . Starting from the base scene (a), we apply progressively larger, compounding visual changes (b–f), and report their cosine distances to the base in CogACT embedding space. Only the most extreme visual shift (+ granite counter, f) exceeds the embedding distance for the behavioral example (knife moved, g).

Fig. 17 : Per-task family RADAR classification confusion matrices using Gemini 3.1 Pro. Rows are ground truth, columns are the predicted overall generalization type; diagonal entries (correct) are bold .

Task	Aggregation Method	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Pick-and-Place	Original	94.9	82.7	100.0	92.5
	Multi-Support	92.9	61.2	100.0	84.7
	Rank-Weighted	93.9	53.1	100.0	82.4
	Uncertainty (87.8% cov.)	94.8	84.4	100.0	94.2
Unzip Lunchbag	Original	93.8	71.4	54.0	72.8
	Multi-Support	93.8	65.3	58.0	72.1

Appendix

TABLE V: Comparing methods for aggregating per-neighbor predictions in RADAR. We compare the Original method described in Algorithm 1 with three alternate methods. Uncertainty accuracies (italic) are computed over a subset of examples (coverage shown in parentheses), and are therefore not directly comparable.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

Jun 19, 2026

Inductive Generalization for Robotic Manipulation

Understanding the generalization capabilities of visuomotor policies is essential in the development of capable robotic agents. Generalizable models learn structures that transfer across domains. However, in practice, visuomotor policies test performance by interpolation on known distributions using unstructured domain shifts (e.g. lighting, clutter, diverse objects). We argue that to measure generalization capabilities we must instead test the inductive capacity of policies on progressively harder, out-of-distribution task variants. We call this inductive generalization, drawing directly on how axis-based evaluation has revealed inherent generalization limitations in language models (e.g. sequence length, counting) arXiv:2502.00197 . We provide a reusable and formal evaluation protocol for measuring inductive generalization in any manipulation policy, and establish baselines showing that existing paradigms fail this test; e.g. SoTA Vision-Language-Action models and find that policies that appear to generalize to prior domain shifts (distractors, etc) fail inductive generalization tests. These results expose a class of learning challenges orthogonal to those addressed by data and model scaling in robot learning, yet are imperative to solve in order to realize general purpose robots.

Annabella Macaluso, Haochen Zhang, Ishaan Masilamony +2

Carnegie Mellon University

OOD Generalization Generalization in Robotic Manipulation Robotic Policy Evaluation Visuomotor Policy Learning Robotic Manipulation Robot Manipulation Benchmarks+1

Ask PDF

Jun 28, 2026

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks. These gains are commonly interpreted as evidence that semantic representations learned from internet-scale data transfer to physical execution generalization. This position paper argues that the assumption underlying this interpretation -- that semantic generalization is sufficient to support physical action decisions -- has not been independently verified and cannot be tested under current evaluation protocols. We support this claim by decomposing VLA policies into semantic mapping and physical action decision, and showing that task success rate -- the dominant evaluation metric -- cannot distinguish between these two sources of capability. As a result, improvements in benchmark performance are consistent with multiple competing explanations, including semantic matching, distributional overlap, and genuine physical generalization. We further argue that this identifiability gap has been reinforced through narrative drift, whereby successive systems inherit and strengthen prior interpretations of performance gains without isolating the underlying causal mechanism. To address this limitation, we propose a research direction based on evaluation designs that introduce controlled variation to separately measure semantic and physical generalization. Such designs make it possible to causally attribute performance without requiring access to model internals, and to empirically assess the role of VLM backbones as semantic interfaces rather than implicit sources of physical competence. Our goal is not to refute the role of VLMs in robotics, but to clarify the conditions under which claims of physical generalization can be meaningfully evaluated.

Taozhao Chen, Ian Manchester, Huaming Chen

School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia

VLM Evaluation Physical Reasoning Generalization in Robotic Manipulation Robotics Vision-Language-Action Models Robot Manipulation Benchmarks+1

Ask PDF

Mar 30, 2026

ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

Vision-Language-Action (VLA) models and world-action models have emerged as central paradigms for general-purpose robotic intelligence, yet their empirical progress remains constrained by the absence of evaluation protocols that are both physically realistic and diagnostically controlled. Simulator-centric benchmarks provide scale and reproducibility, but cannot fully capture the reality gap induced by perception noise, contact dynamics, latency, calibration error, and hardware constraints. Conversely, real-robot evaluations are often fragmented across platforms, scenes, objects, and scoring rules, making fair comparison and failure attribution difficult. We introduce ManipArena, a standardized real-robot evaluation framework for studying manipulation generalization under matched physical conditions. ManipArena comprises 20 tasks, 10,812 expert trajectories, 13.5M frames, and approximately 188 robot hours across tabletop and mobile manipulation. The framework combines schema-defined task variation, stratified in-domain, visualshift, and semantic-OOD trials, subtask-level partial-credit scoring, three-level language annotations, low-level motor signals, and paired real-to-sim environments reconstructed from physical scenes. Using ManipArena, we evaluate seven tabletop configurations spanning VLA and world-action-model policies. The results show that real-robot conclusions depend not only on architecture, but also on model provenance, fine-tuning regime, data sampling, and annotation granularity. ManipArena thus provides a reproducible and interpretable foundation for diagnosing capability boundaries and failure modes in embodied generalization.

Yu Sun, Meng Cao, Yang Ping +24

Sun Yat-sen University · X Square Robot · MBZUAI +2

Generalization in Robotic Manipulation Robotic Manipulation Robot Manipulation Benchmarks VLMs for Robotics

Ask PDF

Abstract

Robotic Policy Evaluation

Explore similar work

Inductive Generalization for Robotic Manipulation

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation