Retrieval-augmented vision-language-action (VLA) policies fill their context with the demonstrations whose frames look most like the current observation under frozen DINOv2 embeddings. We ask whether visual similarity selects the demonstrations that make the policy succeed. For a frozen retrieval-augmented VLA (RICL on π0-FAST, LIBERO), we make the demonstration pool swappable at run time and score each demonstration's utility by regressing the closed-loop success of 60 fixed-size pools on their membership. On initial states disjoint from those used for scoring, pools of the five highest-utility demonstrations succeed on 66.4% of 760 paired episodes across 19 tasks, pools of the five most similar on 55.7%, and random pools on 55.5%. Utility beats similarity by 10.8 points (p=3.2×10−6; positive on 15 of 18 untied tasks), including on 12 tasks the policy was never primed on, and similarity-selected pools do no better than random (+0.1 points). Within a task, a pool's visual similarity to the test scenes tracks its success at Spearman 0.10; across tasks, similarity separates tasks whose scenes differ and falls to chance when two tasks share a scene. The gain comes from on-task closed-loop evidence: best-of-ten direct search over pools recovers two-thirds of it with 100 rollouts. Against the shipped 20-demonstration pool, utility-curated pools of five lead by 7.0 points over 560 paired episodes from two evaluation runs (p=0.006), and utility-ordered retrieval inside the full pool leads by 4.5 points (p=0.055).
Figures & tables
Figure 2: Protocol. Top: the frozen policy’s inference loop. The demonstration pool feeding the FAISS index is set per episode, while per-step frame retrieval inside the pool is unchanged. Bottom: utility is estimated on initial states 0–9 by regressing the success rate of 60 random size-5 pools (10 rollouts each) on membership; pool-selection rules are then evaluated, all at size 5, on initial states 10–49.
Figure 3: Context necessity per task: matched versus mismatched pools, 20 tasks, 20 episodes per cell. No discordant episode favoured mismatched context; held-out tasks fail at the floor and primed tasks saturate.
7 study tasks
12 unseen tasks
all 19 pooled
top
77.5
60.0
66.4
dino
65.7
49.8
55.7
random
67.5
48.5
55.5
bottom
54.6
37.5
43.8
top − dino
+11.8 (3e − 3)
+10.2 (4e − 4)
+10.8 (3e − 6)
dino − random
−1.8 (0.70)
+1.3 (0.70)
+0.1 (1.0)
Table 1: Held-out success (%) at pool size 5 on initial states 10–49, and paired contrasts in percentage points with the exact McNemar p in parentheses. top : five highest measured utility; dino : five most similar to the initial observation; bottom : five lowest utility (anti-selected). 280 episodes per cell for the 7 attribution-study tasks, 480 for the 12 unseen tasks, 760 pooled.
Figure 4: Per-task top − dino , 19 tasks , with ±1.96 standard errors computed from each task’s discordant counts as b+c/n (a null-based paired standard error; Tab. S2 ). The rose line is the inverse-variance common effect ( +11.0pp ); Cochran’s Q does not reject a common effect ( p=0.31 , I2=12% ).
Figure 5: Same initial state, same frozen policy, different five demonstrations in context ( libero_object:4 ): success by step 174 with the five highest-utility demonstrations, a knocked-over bottle and timeout with the five lowest. Table 1 gives the statistics over 760 paired episodes per cell.
Figure 6: Context size. The four cells at pool sizes 3, 5 and 10 on the same 7 tasks, 280 episodes per cell, with Wilson 95% intervals; Δ is top − dino with its exact McNemar p .
Figure 7: All strategies on the 7 attribution-study tasks, 280 episodes each; the pool size is in parentheses. Strategies at pool size 20 use the full pool; rerank orders retrieval slots by φ^ fit on the same tasks; uricl is the rollout-free utility predictor of Sec. 4.7 . One evaluation run per strategy; Sec. 4.4 adds a second run of full , rerank and top .
Figure 8: One initial state, four pools. Recorded rollouts of the frozen policy on libero_object:4 from the same initial state with the top , dino , uricl and full pools: the similarity-selected pool times out while the other three succeed. This initial state is one of four declared before recording and is shown for illustration; frame times are approximate control steps read from the recorded videos. Success rates over 280 episodes per rule are in Fig. 7 .
Figure 9: The observed variance of V^(S) over pools against the binomial reference variance from 10 rollouts per pool; the estimated share above the reference ranges from 0% to 27% .
Figure 10: Pool similarity weakly correlates with measured pool success. Spearman correlation between a pool’s mean DINOv2 similarity to the test scenes and its estimated success V^(S) , 60 pools per task at fixed size 5 (standard error about 0.13 per task); the six tasks whose per-pool records survive are shown, and the mean over all seven (including goal:9 , ρ=+0.134 ) is +0.10 .
Figure 11: Similarity identifies the task when scenes differ and not when they coincide. (a) Each task’s pool is doubled with the 20 demonstrations of its same-suite partner among the attributed tasks: three task pairs, each tested in both directions (chance share of own-task demonstrations among the five most similar: 0.5). In the object and libero_10 pairs, the five nearest demonstrations are own-task in 40 of 40 scenes; in the goal pair, whose two tasks share one scene, the share is at chance. No rollouts are needed for this panel. (b) On the two goal tasks, held-out success of similarity and random selection from the mixed pool (hatched) beside the own-pool cells; 40 episodes each.
Claim
Evidence motivating withdrawal
V^(S) spans 0.30–1.00, so demonstration choice is decisive
the binomial reference accounts for 73–100% of the observed variance at 10 rollouts per pool
Suite structure (goal < spatial < long < object) explains when similarity works
The pre-declared confirmation task goal:9 returned ρ=+0.13 against goal:0 ’s −0.14 ; within-suite spread exceeds the between-suite gap
The effect is monotone in selection pressure (size 3>5>10 )
Held at 3 tasks per condition, not at 7
corr(ρ,dino−random) predicts where similarity helps
Decayed 0.999→0.925→0.837 as tasks were added; sign flips on the size-10 subset
φ^ from 15 pools matches φ^ from 60
Single draw; at five draws it is 3.4 pp lower
Direct search fails, so the attribution machinery buys the gain
Single draw; repeated-draw mean 73.8%, versus 65.7% for similarity and 74.1% for 15-pool φ^
Table 2: Hypotheses that looked supported mid-study and were withdrawn.
Figure 12: The suite-structure hypothesis and the pre-declared confirmation task that overturned it ( Tab. 2 ).
Figure 13: Held-out success against attribution rollouts spent per task , descriptive. Grey: best-of-10 direct search. Green: φ^ fit from 15, 30 or 60 pools. Points at 100–300 rollouts are means over 5 sub-samples per task (35 task-draw cells per ridge budget; 29 for direct search, whose draws are de-duplicated to 3–5 per task) with the mean within-task SD across draws; endpoints are single cells with Wilson intervals. Sub-samples are compared with the same 280-episode reference cells.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Policy
RICL- π0 -FAST-LIBERO, checkpoint step 10 000 of the third-party LIBERO port published on Hugging Face by user ryanhoangt ( pi0_fast_libero_icl_priming_ckpt_10000 , image-key-fixed normalisation statistics); config pi0_fast_libero_ricl , four retrieved frames
Pools
20 processed demonstrations per task from the port’s preprocessed data (DINOv2 per-patch frame embeddings, 49 152-d)
Attribution
60 pools of size 5 sampled uniformly (seed 0); 10 rollouts each on initial states 0–9; ridge λ=1 on centred indicators
Selection cells
size 5; initial states 10–49; 40 paired episodes per cell; random seeded per (evaluation seed, task, scene); the second evaluation seed reuses the fitted scores and the utility-selected pools and resamples the random pools
Similarity
per scene, minimum L2 between any frame of a demonstration and the scene’s first-frame embedding; the five nearest demonstrations form dino
Step caps
spatial 220, object 280, goal 300, long 520, plus 10 settling steps
Screening
full-pool success in [0.25,0.85] on 10 (LIBERO-90) or 20 (standard suites) screening trials; exclusion if pool-estimate SD <0.10 or >25% of pools at 1.0
Appendix
Table S1 : Evaluation contract.
task
top
dino
random
bottom
b
c
task
top
dino
random
bottom
b
c
spatial:0
90.0
65.0
80.0
62.5
12
2
l90:0
77.5
67.5
72.5
55.0
7
3
object:2
57.5
65.0
50.0
42.5
10
13
l90:10
55.0
25.0
17.5
17.5
17
5
object:4
87.5
60.0
55.0
35.0
14
3
l90:12
42.5
35.0
35.0
10.0
11
8
goal:0
80.0
70.0
75.0
67.5
8
4
l90:15
42.5
35.0
42.5
25.0
11
8
goal:9
67.5
47.5
70.0
35.0
15
7
l90:20
32.5
20.0
22.5
15.0
9
4
long:0
85.0
75.0
75.0
62.5
9
5
l90:47
67.5
80.0
67.5
70.0
6
11
Appendix
Table S2 : Per-task held-out success (%) at pool size 5, 40 episodes per cell, initial states 10–49, with the discordant counts b ( top wins) and c ( dino wins) for top − dino . Left block: the seven attribution-study tasks (second evaluation seed). Right block: the twelve unseen LIBERO-90 tasks.
task
ρ
task
ρ
task
ρ
task
ρ
spatial:0
0.663
goal:9
0.268
l90:0
0.316
l90:47
0.367
object:4
0.068
spatial:7
−0.474
l90:55
0.002
l90:59
0.083
object:2
0.021
long:0
0.383
l90:72
0.329
l90:12
0.535
goal:0
−0.042
l90:20
0.107
l90:10
−0.200
long:4
0.039
l90:60
0.027
l90:15
0.244
l90:56
0.065
l90:61
−0.020
Appendix
Table S3 : Split-half Spearman agreement of φ^ (initial states 0–4 against 5–9, same pools) for the 20 attributed tasks, including the excluded spatial:7 . Mean 0.139 ; mean top-five Jaccard 0.20 .
Figure S1 : Seven-task family. Left: second evaluation seed (bars) with the four first-seed tasks whose episode files are recoverable (diamonds). Right: per-task cells; long:4 is the documented outlier.
Figure S2 : Left: the prune control against random and bottom from the same run, with the discordant-pair panel. Right: pairwise contrasts; the four size-5 cells come from the second-seed run and the prune contrast from its own run (hence random 67.9% there against 67.5%) ( Sec. 4.4 ).
Figure S3 : Left: sorted relative ridge scores φ^ per task; shaded ends form top and bottom . Right: split-half agreement of V^ per task and the reliability-adjusted similarity correlation, which is unstable.
Figure S4 : Left: cumulative success by step under top and bottom ; the curves are similar below about 130 steps. Right: the 73 unseen LIBERO-90 tasks by full-pool success under the screen; 12 met the criterion.
Vision-language-action (VLA) models have shown strong potential as generalist robot policies, but adapting them to unseen tasks often requires costly parameter updates. Recent work such as RICL introduces in-context adaptability by retrieving expert demonstrations based on the current VLA observation and providing them as additional context at test time. The effectiveness of this adaptation therefore depends critically on the retrieval mechanism. In this work, we systematically study how different retrieval methods affect both retrieval quality and task performance within the RICL framework. Specifically, we compare four different methods: image-based retrieval, retrieval augmented with VLA's state, retrieval using features from the VLA backbone, and random retrieval. Our experiments yield three main findings. First, no retrieval method consistently dominates the others in task success, while surprisingly, random retrieval achieves a non-trivial success rate. Second, standard retrieval-quality diagnostics do not reliably reflect downstream VLA performance. Third, demonstrations from different but related tasks can provide useful transferable information. Together, these results provide an initial step toward understanding how retrieval mechanisms shape the in-context learning capability of VLA models and their downstream task performance, while highlighting the need for more careful design and evaluation of retrieval mechanisms for reliable test-time adaptation.
Zixuan Liu, Joris Köster, Zizhan Zheng +1
Department of Computer Science, Tulane University · Department of Computer Science, Aalto University · Department of Industrial Engineering and Management, Aalto University
Vision-language-action (VLA) models now exceed 95% success on the LIBERO simulation benchmark, yet it remains unclear how much of this performance requires generalization beyond the training demonstrations. In this work, we show that retrieving nearby demonstrations and reusing their action chunks achieves a success rate of 91.7 % on LIBERO without policy training. Under LIBERO-Plus perturbations, retrieval shows the smallest drop of all evaluated policies. Across nine publicly available VLAs, fine-tuning mostly makes the vision-language backbone a better retrieval key, and the actions of the VLAs remain close to the demonstrations. In addition, we introduce Local Action Consistency (LAC), an offline metric that measures whether nearby observations in representation space are also associated with similar future actions. We show that LAC tracks the success rate of retrieval and can guide the training of a retrieval key. Our evaluation and source code are available at https://github.com/sereact/lerobot_knn.
Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.