Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs

Jensen Gao; Dorsa Sadigh; Sandy Huang; Dhruv Shah

cs.ROMar 12, 2026

Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs

Authors: Jensen Gao, Dorsa Sadigh, Sandy Huang, Dhruv Shah

Organizations: Stanford University · Google DeepMind · Princeton University

Abstract

Recent work on robot manipulation has advanced policy generalization to novel scenarios. However, it is often difficult to characterize how different evaluation settings actually represent generalization from the training distribution of a given policy. To work towards more precise evaluation of generalization in robotics, we propose RADAR, a scalable framework for directly comparing test-time evaluation tasks to policy training data, to determine what form of policy generalization is required. RADAR consists of a two-stage pipeline: first, retrieval using generalist policy embeddings identifies which training examples are relevant for a given evaluation task. Next, vision-language models (VLMs) analyze the evaluation task against the retrieved data, outputting interpretable analysis on how they compare along a variety of axes, and an overall classification of what type of policy generalization is required. Through controlled experiments, we demonstrate that VLMs are effective at analyzing data for generalization, and that our retrieval step effectively identifies examples needed to make accurate classifications with respect to the training data. Furthermore, we scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions from prior work. We provide demonstrations at radar-analysis.github.io.

Figures & tables

Fig. 1 : We outline the two-stage pipeline of RADAR. Given an evaluation task, we first identify relevant examples from training data using nearest neighbors with embeddings from a generalist robot policy. Next, we use vision-language models to analyze the evaluation task against the retrieved examples to categorize what generalization it represents w.r.t. the training data: in-distribution (ID) , visual , or behavioral .

Fig. 2 : (Left) Test example $\tau^{\text{test}}$ for the task “put the lemon on the plate” . (Right) If $d^{*}$ (e.g., minor lighting change) is found in $\mathcal{D}$ , then $\tau^{\text{test}}$ is in-distribution (ID) . Otherwise, if $d^{v}$ (e.g., distractor objects) is found in $\mathcal{D}$ , then this is visual generalization. If neither case applies, then all $d\in\mathcal{D}$ involve different optimal behavior than $\tau^{\text{test}}$ (e.g., changed object poses), and this is behavioral generalization.

Fig. 3 : We visualize the three task families in our controlled experiments ( Pick-and-Place , Unzip Lunchbag , and Fold Dress ). We provide an example instance for each task (left), and variations of that task across different axes (right), with the specific change noted in parentheses.

Fig. 4 : Recall rate scaling (

y

-axis) of

d^{*}

for

\mathcal{D}_{\text{ID}}

as the size of the retrieval set (

x

-axis) increases. We find that VLA-based embeddings and DINOv3 significantly outperform the other retrieval methods.

Fig. 5 : Recall rate scaling (

y

-axis) of

d^{v}

for

\mathcal{D}_{\text{visual}}

as the size of the retrieval set (

x

-axis) increases.

\pi_{0.5}

and GROD outperform their counterparts trained on less robot data (

\pi_{0}

, GROD (25% Data)), and are much better than the baselines that do not use robot data.

Panel of Figure 6 — Fig. 6 : Distribution of embedding distances (L2 with unit-normalized embeddings, equivalent to cosine distance) induced by task perturbations across different axes for Pick-and-Place . VLA embeddings, especially for GROD, generally have larger distances for behavioral axes (blue) than visual axes (orange), unlike the other embeddings.

Task	VLM	Image Aug	Visual Task Obj	Visual Scene	Obj Poses	Morphed Obj	New Obj	Interacting Scene	Perturb Class	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Pick-and-Place	Gemini 3.1 Flash-Lite	39.6	81.9	39.7	29.3	–	87.5	–	78.9	95.0	7.0	78.0	60.0
	Gemini 3 Flash	94.7	67.3	92.7	38.2	–	97.5	–	96.7	95.6	64.4	97.9	86.2
	Gemini 3.1 Pro	79.1	69.4	98.9	73.9	–	99.4	–	98.3	94.9	82.7	100.0	92.5
Unzip Lunchbag	Gemini 3.1 Flash-Lite	52.8	84.9	75.6	75.0	55.3	–	–	83.4	100.0	4.0	82.0	62.0
	Gemini 3 Flash	95.4	96.6	89.5	81.2	65.8	–	–	88.3	95.9	58.3	51.0	68.5
	Gemini 3.1 Pro	74.8	97.3	100.0	88.8	69.5	–	–	89.7	93.8	71.4	54.0	72.8

TABLE I: (Left) VLM performance (F1 ×100) on pairwise prediction of per-axis differences and perturbation classification between

\tau^{\text{test}}

and

d\in\mathcal{D}_{\text{retrieval}}

. We do not report axes for which the task has no variation. (Right) Overall RADAR classification accuracy after aggregating VLM predictions across all

d\in\mathcal{D}_{\text{retrieval}}

, for each version of

\mathcal{D}

Fig. 7 : Examples of successful (green) and failed (red) predictions of Gemini 3 Flash for “Object Poses” and “Morphed Objects”. The VLM struggles more when comparing examples with more subtle visual changes (lunchbag rotation, slightly different dress size).

Task	Method	Image Aug	Visual Task Obj	Visual Scene	Obj Poses	Morphed Obj	Perturb Class	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Unzip Lunchbag	Gemini 3.1 Pro	74.8	97.3	100.0	88.8	69.5	89.7	93.8	71.4	54.0	72.8
	– Obj Detect	75.5	97.7	89.9	86.7	66.0	87.2	93.9	55.1	33.3	61.0
	– Obj Detect, Axes	–	–	–	–	–	79.9	90.0	40.0	6.0	45.3
	+ Longer Context ( $k=40$ )	73.1	95.4	99.4	87.8	68.1	89.7	96.0	62.5	44.0	67.6

TABLE II: Ablations of VLM analysis in RADAR. (Left) VLM performance (F1 ×100) on pairwise prediction of per-axis differences and perturbation classification between

\tau^{\text{test}}

and

d\in\mathcal{D}_{\text{retrieval}}

. (Right) Overall RADAR classification accuracy after aggregating VLM predictions.

Fig. 8 : Examples of RADAR applied to benchmarking generalization for a large-scale ALOHA 2 dataset. For a given benchmark evaluation task (left), we provide three instances of retrieved training data examples (middle). Then, we provide the VLM analysis comparing the evaluation task to the leftmost retrieved example, as well as the overall generalization classification (right).

VLM	Gemini 3 Flash	Gemini 3.0 Pro	Gemini 3.1 Pro
Agreement	60.0	71.4	77.1

TABLE III: Classification agreement of different VLMs in RADAR on BridgeV2-

\bigstar

[ 5 ] , compared against their original categorizations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Retrieval
Embedding feature	KV cache of image tokens from the 2nd-to-last transformer backbone layer (PaliGemma VLAs); cognition feature (CogACT)
Raw dimensionality	131072 ( $\pi_{0}$ , $\pi_{0.5}$ ); 4096 (CogACT)
Dim. reduction	Gaussian random projection to 2048 (KV-cache embeddings)
Distance metric	Cosine
Embedding inputs	VLA embeddings: camera views, language instruction $l$ , and 14-d proprioception (per arm: 6-d joint position + 1-d gripper position, as in ALOHA); CogACT omits proprioception. SigLIP 2/DINOv3 embed the overhead view only (DINOv3 omits language)
Retrieval size $k$	20 (main); 40 (long-context ablation); 50 (large-scale case studies)

Appendix

TABLE IV : Summary of retrieval and VLM analysis hyperparameters and inputs for RADAR.

Panel of Figure 13 — Fig. 9 : VLM prompt with $\bigstar$ -Gen axes guidance (Part 1).

Panel of Figure 14 — Fig. 10 : VLM prompt with $\bigstar$ -Gen axes guidance and object detection (Part 2).

Panel of Figure 15 — Fig. 11 : VLM prompt with $\bigstar$ -Gen axes guidance, but without object detection (Part 2).

Panel of Figure 16 — Fig. 12 : VLM prompt without $\bigstar$ -Gen axes guidance or object detection.

Fig. 13 : Set-level retrieval success for

\mathcal{D}_{\text{ID}}

: the fraction of evaluation tasks whose top-

k

retrieval set contains at least one

d^{*}

, as a function of the retrieval set size

k

, for GROD and

\pi_{0.5}

Fig. 14 : Set-level retrieval success for

\mathcal{D}_{\text{visual}}

: the fraction of evaluation tasks whose top-

k

retrieval set contains at least one

d^{v}

, as a function of the retrieval set size

k

, for GROD and

\pi_{0.5}

Panel of Figure 19 — Fig. 15 : Distinct object classes with functionally identical behavior, for the instruction “put utensil on plate” . Starting from a base scene with a knife (a), we swap in a fork (b) and spoon (c) while keeping the handle geometry and pose fixed, yielding visual perturbations that require the same behavior. Moving the original knife to a new pose (d) is a behavioral perturbation. We report the cosine distance ( $d_{\cos}$ ) of each image to the base scene in the CogACT policy embedding space used for retrieval. The different-class fork and spoon are closer to the base than the same-class repositioned knife, so embedding proximity tracks the required behavior rather than object-class identity.

Panel of Figure 20 — Fig. 16 : Stress-testing retrieval under increasing visual shifts, for the instruction “put knife on plate” . Starting from the base scene (a), we apply progressively larger, compounding visual changes (b–f), and report their cosine distances to the base in CogACT embedding space. Only the most extreme visual shift (+ granite counter, f) exceeds the embedding distance for the behavioral example (knife moved, g).

Fig. 17 : Per-task family RADAR classification confusion matrices using Gemini 3.1 Pro. Rows are ground truth, columns are the predicted overall generalization type; diagonal entries (correct) are bold .

Task	Aggregation Method	$\mathcal{D}_{\text{ID}}$ Acc	$\mathcal{D}_{\text{visual}}$ Acc	$\mathcal{D}_{\text{behavioral}}$ Acc	Overall Acc
Pick-and-Place	Original	94.9	82.7	100.0	92.5
	Multi-Support	92.9	61.2	100.0	84.7
	Rank-Weighted	93.9	53.1	100.0	82.4
	Uncertainty (87.8% cov.)	94.8	84.4	100.0	94.2
Unzip Lunchbag	Original	93.8	71.4	54.0	72.8
	Multi-Support	93.8	65.3	58.0	72.1

Appendix

TABLE V: Comparing methods for aggregating per-neighbor predictions in RADAR. We compare the Original method described in Algorithm 1 with three alternate methods. Uncertainty accuracies (italic) are computed over a subset of examples (coverage shown in parentheses), and are therefore not directly comparable.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

Inductive Generalization for Robotic Manipulation
Jun 19, 2026Annabella Macaluso, Haochen Zhang, Ishaan Masilamony +2OOD Generalization Generalization in Robotic Manipulation
Carnegie Mellon University
Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning
Jun 28, 2026Taozhao Chen, Ian Manchester, Huaming ChenVLM Evaluation Physical Reasoning
School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia
ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
Mar 30, 2026Yu Sun, Meng Cao, Yang Ping +24Generalization in Robotic Manipulation Robotic Manipulation
Sun Yat-sen University · X Square Robot · MBZUAI +2

Jun 19, 2026J/K move · Enter open · S save

Inductive Generalization for Robotic Manipulation

Annabella Macaluso, Haochen Zhang, Ishaan Masilamony, Yingshan Chang+1

Carnegie Mellon University

OOD Generalization Generalization in Robotic Manipulation Robotic Policy Evaluation Visuomotor Policy Learning Robotic Manipulation Robot Manipulation Benchmarks+1

Open paper AskPDF arXiv

Understanding the generalization capabilities of visuomotor policies is essential in the development of capable robotic agents. Generalizable models learn structures that transfer across domains. However, in practice, visuomotor policies test performance by interpolation on known distributions using unstructured domain shifts (e.g. lighting, clutter, diverse objects). We argue that to measure generalization capabilities we must instead test the inductive capacity of policies on progressively harder, out-of-distribution task variants. We call this inductive generalization, drawing directly on how axis-based evaluation has revealed inherent generalization limitations in language models (e.g. sequence length, counting) arXiv:2502.00197 . We provide a reusable and formal evaluation protocol for measuring inductive generalization in any manipulation policy, and establish baselines showing that existing paradigms fail this test; e.g. SoTA Vision-Language-Action models and find that policies that appear to generalize to prior domain shifts (distractors, etc) fail inductive generalization tests. These results expose a class of learning challenges orthogonal to those addressed by data and model scaling in robot learning, yet are imperative to solve in order to realize general purpose robots.

Abstract

Robotic Policy Evaluation

Explore similar work

Inductive Generalization for Robotic Manipulation

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation