Similar Is Not Useful: Selecting In-Context Demonstrations by Closed-Loop Utility
Organizations: University of Texas at Austin · Georgia Institute of Technology · General Intelligent Machines, Inc
Abstract
Retrieval-augmented vision-language-action (VLA) policies fill their context with the demonstrations whose frames look most like the current observation under frozen DINOv2 embeddings. We ask whether visual similarity selects the demonstrations that make the policy succeed. For a frozen retrieval-augmented VLA (RICL on -FAST, LIBERO), we make the demonstration pool swappable at run time and score each demonstration's utility by regressing the closed-loop success of 60 fixed-size pools on their membership. On initial states disjoint from those used for scoring, pools of the five highest-utility demonstrations succeed on 66.4% of 760 paired episodes across 19 tasks, pools of the five most similar on 55.7%, and random pools on 55.5%. Utility beats similarity by 10.8 points (; positive on 15 of 18 untied tasks), including on 12 tasks the policy was never primed on, and similarity-selected pools do no better than random ( points). Within a task, a pool's visual similarity to the test scenes tracks its success at Spearman 0.10; across tasks, similarity separates tasks whose scenes differ and falls to chance when two tasks share a scene. The gain comes from on-task closed-loop evidence: best-of-ten direct search over pools recovers two-thirds of it with 100 rollouts. Against the shipped 20-demonstration pool, utility-curated pools of five lead by 7.0 points over 560 paired episodes from two evaluation runs (), and utility-ordered retrieval inside the full pool leads by 4.5 points ().
Figures & tables
| 7 study tasks | 12 unseen tasks | all 19 pooled | |
| top | 77.5 | 60.0 | 66.4 |
| dino | 65.7 | 49.8 | 55.7 |
| random | 67.5 | 48.5 | 55.5 |
| bottom | 54.6 | 37.5 | 43.8 |
| top dino | (3e 3) | (4e 4) | (3e 6) |
| dino random | (0.70) | (0.70) | (1.0) |
| Claim | Evidence motivating withdrawal |
|---|---|
| spans 0.30–1.00, so demonstration choice is decisive | the binomial reference accounts for 73–100% of the observed variance at 10 rollouts per pool |
| Suite structure (goal spatial long object) explains when similarity works | The pre-declared confirmation task goal:9 returned against goal:0 ’s ; within-suite spread exceeds the between-suite gap |
| The effect is monotone in selection pressure (size ) | Held at 3 tasks per condition, not at 7 |
| predicts where similarity helps | Decayed as tasks were added; sign flips on the size-10 subset |
| from 15 pools matches from 60 | Single draw; at five draws it is 3.4 pp lower |
| Direct search fails, so the attribution machinery buys the gain | Single draw; repeated-draw mean 73.8%, versus 65.7% for similarity and 74.1% for 15-pool |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Policy | RICL- -FAST-LIBERO, checkpoint step 10 000 of the third-party LIBERO port published on Hugging Face by user ryanhoangt ( pi0_fast_libero_icl_priming_ckpt_10000 , image-key-fixed normalisation statistics); config pi0_fast_libero_ricl , four retrieved frames |
| Pools | 20 processed demonstrations per task from the port’s preprocessed data (DINOv2 per-patch frame embeddings, 49 152-d) |
| Attribution | 60 pools of size 5 sampled uniformly (seed 0); 10 rollouts each on initial states 0–9; ridge on centred indicators |
| Selection cells | size 5; initial states 10–49; 40 paired episodes per cell; random seeded per (evaluation seed, task, scene); the second evaluation seed reuses the fitted scores and the utility-selected pools and resamples the random pools |
| Similarity | per scene, minimum between any frame of a demonstration and the scene’s first-frame embedding; the five nearest demonstrations form dino |
| Step caps | spatial 220, object 280, goal 300, long 520, plus 10 settling steps |
| Screening | full-pool success in on 10 (LIBERO-90) or 20 (standard suites) screening trials; exclusion if pool-estimate SD or of pools at 1.0 |
| task | top | dino | random | bottom | task | top | dino | random | bottom | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| spatial:0 | 90.0 | 65.0 | 80.0 | 62.5 | 12 | 2 | l90:0 | 77.5 | 67.5 | 72.5 | 55.0 | 7 | 3 |
| object:2 | 57.5 | 65.0 | 50.0 | 42.5 | 10 | 13 | l90:10 | 55.0 | 25.0 | 17.5 | 17.5 | 17 | 5 |
| object:4 | 87.5 | 60.0 | 55.0 | 35.0 | 14 | 3 | l90:12 | 42.5 | 35.0 | 35.0 | 10.0 | 11 | 8 |
| goal:0 | 80.0 | 70.0 | 75.0 | 67.5 | 8 | 4 | l90:15 | 42.5 | 35.0 | 42.5 | 25.0 | 11 | 8 |
| goal:9 | 67.5 | 47.5 | 70.0 | 35.0 | 15 | 7 | l90:20 | 32.5 | 20.0 | 22.5 | 15.0 | 9 | 4 |
| long:0 | 85.0 | 75.0 | 75.0 | 62.5 | 9 | 5 | l90:47 | 67.5 | 80.0 | 67.5 | 70.0 | 6 | 11 |
| task | task | task | task | ||||
|---|---|---|---|---|---|---|---|
| spatial:0 | 0.663 | goal:9 | 0.268 | l90:0 | 0.316 | l90:47 | 0.367 |
| object:4 | 0.068 | spatial:7 | l90:55 | 0.002 | l90:59 | 0.083 | |
| object:2 | 0.021 | long:0 | 0.383 | l90:72 | 0.329 | l90:12 | 0.535 |
| goal:0 | l90:20 | 0.107 | l90:10 | ||||
| long:4 | 0.039 | l90:60 | 0.027 | l90:15 | 0.244 | ||
| l90:56 | 0.065 | l90:61 |
Retrieval-Augmented ICL
Retrieval-augmented in-context learning (ICL) selects relevant examples or knowledge to help models adapt without updating their weights.
See all 49 papers