Reuse Before You Retrieve: Diagnosing Headroom and Complementarity for Test-Time Augmentation of Embodied Multimodal Policies

Yuhwan Jeong; Kuk-Jin Yoon

cs.ROAug 18, 2026

Reuse Before You Retrieve: Diagnosing Headroom and Complementarity for Test-Time Augmentation of Embodied Multimodal Policies

Authors: Yuhwan Jeong, Kuk-Jin Yoon

Organizations: KAIST, Visual Intelligence Lab.

Abstract

Frozen vision-language-action (VLA) policies are increasingly improved at test time by sampling additional policy behaviors or introducing external demonstrations. Yet there is little guidance for deciding which intervention a deployed policy actually needs. Additional sampling is useful only when better behavior already exists within the policy's stochastic rollouts and can be identified, whereas retrieval is most useful when the relevant action prior is not reliably represented by the policy. We study this decision through two measurable factors, recoverable headroom and retrieval complementarity, which characterize how much useful behavior is already available to recover and whether an external action prior fills a measurable gap. We evaluate an episode-level retry selector under retryable or parallel execution, together with retrieval across multiple frozen VLA policies and environments. The selector consistently recovers substantial latent capability across all tested VLA backbones on LIBERO, with gains of up to 21.0 success-rate points that closely track recoverable headroom. It also transfers to a different robot and simulator and remains effective under degraded observations, while experiments with autoregressive OpenVLA illustrate the distinction between available headroom and the ability to rank candidate rollouts. Retrieval behaves differently, improving the policy with the largest measured action-prior gap and providing further gains when combined with selection. Together, these results provide an empirical basis for characterizing test-time augmentation opportunities by separating capability that can be recovered from the frozen policy from behavioral priors that may need to be introduced externally.

Figures & tables

Figure 1: The headroom relation. Selector gain against recoverable headroom (pass@

N-

pass@1). Small dots are the 12 backbone

\times

suite cells, and large markers are the three backbone means. The dashed fit gives

r{=}0.954

, bootstrap 95% CI

[0.89,0.99]

, and

0.91{\pm}0.04

on disjoint episode splits. The training-free score recovers about 90% of the available headroom across backbones.

Figure 2: Episode-level retry selection. From the same initial state

s_{0}

, the frozen policy

\pi

is run

N

times with different seeds, producing candidate rollouts

\tau_{1},\dots,\tau_{N}

. Each rollout is scored by its average proximity to successful demonstrations in a frozen embedding space

\phi

(Eq. 4 ), and the highest-scoring rollout is selected. The selector is training-free, uses no success labels, and assumes repeated or parallel execution.

Backbone	pass@1 (frozen)	$+$ Retrieval	$+$ Selector (ours)
$\pi_{0}$	64.2	70.6 ( $+6.4$ )	85.2 ( $\mathbf{+21.0}$ )
SmolVLA	65.6	65.9 ( $+0.3$ )	85.5 ( $\mathbf{+19.9}$ )
$\pi_{0.5}$	88.7	89.2 ( $+0.5$ )	96.5 ( $\mathbf{+7.8}$ )

Table 1: The episode-level selector improves every tested backbone, while retrieval provides consistent gains only for

\pi_{0}

. LIBERO 4-suite mean success rate (%) with gains over the frozen pass@1 baseline in parentheses. Selector gains are positive in all 12 backbone

\times

suite cells, while retrieval is positive across all four suites only for

\pi_{0}

Backbone	Mean pass@1	Spatial	Object	Goal	10
$\pi_{0}$	64.2	$\mathbf{+4.5{\pm}1.5}$	$\mathbf{+9.8{\pm}2.5}$	$\mathbf{+5.5{\pm}0.8}$	$\mathbf{+6.0{\pm}2.3}$
SmolVLA	65.6	$+0.5{\pm}0.9$	$-0.2{\pm}2.0$	$+1.6{\pm}0.7$	$-0.9{\pm}1.6$
$\pi_{0.5}$	88.7	$-1.1{\pm}0.9$	$+0.6{\pm}0.6$	$+2.6{\pm}2.0$	$-0.1{\pm}2.8$

Table 2: Retrieval is backbone-specific rather than determined by baseline capability alone. Retrieval gain over pass@1 across the four LIBERO suites, reported as mean

\pm

std over three seeds. The retrieval mechanism, demonstration library, and hyperparameters are fixed across backbones. Bold marks the only backbone with positive gains across all four suites.

Suite	pass@1	$+$ retrieval	$+$ selector	retrieval $+$ selector	combine $-$ selector
spatial	63.6	68.1 ( $+4.5$ )	89.6 ( $+26.0$ )	92.4 ( $+28.8$ )	$+2.8$
object	77.1	86.9 ( $+9.8$ )	96.0 ( $+18.9$ )	97.2 ( $+20.1$ )	$+1.2$
goal	80.4	85.9 ( $+5.5$ )	97.6 ( $+17.2$ )	96.8 ( $+16.4$ )	$-0.8$
10	35.7	41.7 ( $+6.0$ )	57.6 ( $+21.9$ )	63.8 ( $+28.1$ )	$+6.2$
mean	64.2	70.6 ( $+6.4$ )	85.2 ( $+21.0$ )	87.5 ( $+23.3$ )	$+2.4$

Table 3: Retrieval and selection provide additional gains when combined on average.

\pi_{0}

on LIBERO with

n{=}500{\times}3

rollouts. Values report success rate (%) with gains over pass@1 in parentheses. The final column reports the additional gain over selection alone.

Backbone	pass@1	rand-of-3	selector ( $N{=}2$ )	selector ( $N{=}3$ )	oracle pass@3
$\pi_{0}$	64.2	64.5	79.2	85.2	87.8
SmolVLA	65.6	65.3	81.0	85.5	87.0
$\pi_{0.5}$	88.7	89.0	94.8	96.5	98.3

Table 4: The gain comes from selecting additional samples rather than sampling alone. LIBERO 4-suite mean success rate (%). Randomly retaining one of three rollouts remains near pass@1, while manifold scoring recovers a large fraction of the available pass@3 headroom.

Task	pass@1	$+$ retrieval	$+$ selector (ours)	oracle pass@ $N$
carrot	34.7	30.6 ( $-4.2$ )	41.7 ( $+6.9$ )	50.0
spoon	45.8	47.2 ( $+1.4$ )	54.2 ( $+8.3$ )	62.5
stack	31.9	31.9 ( $-0.0$ )	54.2 ( $+22.2$ )	54.2
overall	37.5	36.6 ( $-0.9$ )	50.0 ( $\mathbf{+12.5}$ )	55.6

Table 5: The selector transfers to a different robot and simulator, while retrieval does not improve overall performance. SimplerEnv-Bridge with

\pi_{0}

-Bridge and WidowX using

n{=}24{\times}3

rollouts. Values report success rate (%) with gains over pass@1 in parentheses, and the oracle reports pass@

N

	Configuration	Success
baseline	greedy decoding	0.40
	sample@1 ( $T{=}1$ , 3-seed mean)	0.40 ( $+0.00$ )
per-step	re-rank ( $N{=}4$ )	0.36 ( $-0.04$ )
episode-level (ours)	selector ( $N{=}3$ )	0.44 ( $+0.04$ )
	oracle pass@3	0.72

Table 6: On an autoregressive VLA, available headroom and realized selection can differ substantially. OpenVLA-7B on the SimplerEnv coke-can task using 25 initial states. Gains are relative to greedy decoding, and oracle pass@3 indicates whether any of the three sampled episodes succeeds.

	$\pi_{0}$			$\pi_{0.5}$
Suite	pass@1	$+$ retr	$+$ sel	pass@1	$+$ retr	$+$ sel
spatial	61.6	$+6.2$	$\mathbf{+18.4}$	80.8	$+3.4$	$\mathbf{+14.0}$
object	81.0	$+6.6$	$\mathbf{+9.4}$	74.0	$+0.6$	$\mathbf{+4.8}$
goal	74.4	$+2.8$	$\mathbf{+9.6}$	74.8	$+2.0$	$\mathbf{+12.0}$
10	20.8	$+1.8$	$\mathbf{+10.4}$	42.4	$-2.6$	$\mathbf{+14.0}$
mean	59.5	$+4.3$	$\mathbf{+11.9}$	68.0	$+0.8$	$\mathbf{+11.2}$

Table 7: Under scene-induced occlusion, selector gains change together with the available opportunity. LIBERO-Occ using

n{=}250{\times}2

rollouts with

N{=}2

. Pass@1 reports the occluded baseline, while retrieval and selection columns report gains over it.

Setting	Value
Visual encoder $\phi$	frozen DINOv2 $+$ SigLIP, $1{,}536$ -d, $\ell_{2}$ -normalized
Camera view for scoring	wrist camera when available, else third-person
Selector candidates $N$	$3$ , from $3$ seeds (LIBERO)
Demonstration library	$50$ demos/task (LIBERO); $510$ / $1{,}360$ frames (Bridge/coke-can)
Retrieval strength $\alpha$	$0.3$
Demonstration-fit samples $K$	$3$ , over $400$ demonstration states

Table S1 : Fixed hyperparameters shared across backbones and suites.

Test-Time Adaptation of VLA Models

Test-time adaptation of vision-language-action models helps robot policies adjust to new views, tasks, objects, and execution errors during deployment.

See all 52 papers

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
Jun 14, 2026Jeongeun Park, Juhan Park, Taekyung Kim +3Test-Time Adaptation Visuomotor Policy Learning
NAVER AI Lab · Korea University
Retrieve in Time, Correct in Frequency
Aug 5, 2026Yuze Fan, Yue Cao, Pengjie Gao +7Robotic Manipulation Vision-Language-Action Models
Research Institute of Tsinghua University in Shenzhen · Tongji University · Everwise-Tech Co., Ltd. +2
What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies
May 27, 2026Jiachen Zhang, Junnan Nie, Junyi Lao +4Value Function Estimation Robotic Manipulation
Peking University · China Agricultural University · 1Peking University. +1

Jun 14, 2026J/K move · Enter open · S save

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi+2

NAVER AI Lab · Korea University

Test-Time Adaptation Visuomotor Policy Learning Vision-Language-Action Models Test-Time Adaptation of VLA Models Cross-Embodiment Transfer

Open paper AskPDF arXiv

Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.

Abstract

Test-Time Adaptation of VLA Models

Explore similar work

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

Retrieve in Time, Correct in Frequency

What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies