Reuse Before You Retrieve: Diagnosing Headroom and Complementarity for Test-Time Augmentation of Embodied Multimodal Policies

Yuhwan Jeong; Kuk-Jin Yoon

cs.ROAug 18, 2026

Reuse Before You Retrieve: Diagnosing Headroom and Complementarity for Test-Time Augmentation of Embodied Multimodal Policies

Authors: Yuhwan Jeong, Kuk-Jin Yoon

Organizations: KAIST, Visual Intelligence Lab.

Abstract

Frozen vision-language-action (VLA) policies are increasingly improved at test time by sampling additional policy behaviors or introducing external demonstrations. Yet there is little guidance for deciding which intervention a deployed policy actually needs. Additional sampling is useful only when better behavior already exists within the policy's stochastic rollouts and can be identified, whereas retrieval is most useful when the relevant action prior is not reliably represented by the policy. We study this decision through two measurable factors, recoverable headroom and retrieval complementarity, which characterize how much useful behavior is already available to recover and whether an external action prior fills a measurable gap. We evaluate an episode-level retry selector under retryable or parallel execution, together with retrieval across multiple frozen VLA policies and environments. The selector consistently recovers substantial latent capability across all tested VLA backbones on LIBERO, with gains of up to 21.0 success-rate points that closely track recoverable headroom. It also transfers to a different robot and simulator and remains effective under degraded observations, while experiments with autoregressive OpenVLA illustrate the distinction between available headroom and the ability to rank candidate rollouts. Retrieval behaves differently, improving the policy with the largest measured action-prior gap and providing further gains when combined with selection. Together, these results provide an empirical basis for characterizing test-time augmentation opportunities by separating capability that can be recovered from the frozen policy from behavioral priors that may need to be introduced externally.

Figures & tables

Figure 1: The headroom relation. Selector gain against recoverable headroom (pass@

N-

pass@1). Small dots are the 12 backbone

\times

suite cells, and large markers are the three backbone means. The dashed fit gives

r{=}0.954

, bootstrap 95% CI

[0.89,0.99]

, and

0.91{\pm}0.04

on disjoint episode splits. The training-free score recovers about 90% of the available headroom across backbones.

Figure 2: Episode-level retry selection. From the same initial state

s_{0}

, the frozen policy

\pi

is run

N

times with different seeds, producing candidate rollouts

\tau_{1},\dots,\tau_{N}

. Each rollout is scored by its average proximity to successful demonstrations in a frozen embedding space

\phi

(Eq. 4 ), and the highest-scoring rollout is selected. The selector is training-free, uses no success labels, and assumes repeated or parallel execution.

Backbone	pass@1 (frozen)	$+$ Retrieval	$+$ Selector (ours)
$\pi_{0}$	64.2	70.6 ( $+6.4$ )	85.2 ( $\mathbf{+21.0}$ )
SmolVLA	65.6	65.9 ( $+0.3$ )	85.5 ( $\mathbf{+19.9}$ )
$\pi_{0.5}$	88.7	89.2 ( $+0.5$ )	96.5 ( $\mathbf{+7.8}$ )

Table 1: The episode-level selector improves every tested backbone, while retrieval provides consistent gains only for

\pi_{0}

. LIBERO 4-suite mean success rate (%) with gains over the frozen pass@1 baseline in parentheses. Selector gains are positive in all 12 backbone

\times

suite cells, while retrieval is positive across all four suites only for

\pi_{0}

Backbone	Mean pass@1	Spatial	Object	Goal	10
$\pi_{0}$	64.2	$\mathbf{+4.5{\pm}1.5}$	$\mathbf{+9.8{\pm}2.5}$	$\mathbf{+5.5{\pm}0.8}$	$\mathbf{+6.0{\pm}2.3}$
SmolVLA	65.6	$+0.5{\pm}0.9$	$-0.2{\pm}2.0$	$+1.6{\pm}0.7$	$-0.9{\pm}1.6$
$\pi_{0.5}$	88.7	$-1.1{\pm}0.9$	$+0.6{\pm}0.6$	$+2.6{\pm}2.0$	$-0.1{\pm}2.8$

Table 2: Retrieval is backbone-specific rather than determined by baseline capability alone. Retrieval gain over pass@1 across the four LIBERO suites, reported as mean

\pm

std over three seeds. The retrieval mechanism, demonstration library, and hyperparameters are fixed across backbones. Bold marks the only backbone with positive gains across all four suites.

Suite	pass@1	$+$ retrieval	$+$ selector	retrieval $+$ selector	combine $-$ selector
spatial	63.6	68.1 ( $+4.5$ )	89.6 ( $+26.0$ )	92.4 ( $+28.8$ )	$+2.8$
object	77.1	86.9 ( $+9.8$ )	96.0 ( $+18.9$ )	97.2 ( $+20.1$ )	$+1.2$
goal	80.4	85.9 ( $+5.5$ )	97.6 ( $+17.2$ )	96.8 ( $+16.4$ )	$-0.8$
10	35.7	41.7 ( $+6.0$ )	57.6 ( $+21.9$ )	63.8 ( $+28.1$ )	$+6.2$
mean	64.2	70.6 ( $+6.4$ )	85.2 ( $+21.0$ )	87.5 ( $+23.3$ )	$+2.4$

Table 3: Retrieval and selection provide additional gains when combined on average.

\pi_{0}

on LIBERO with

n{=}500{\times}3

rollouts. Values report success rate (%) with gains over pass@1 in parentheses. The final column reports the additional gain over selection alone.

Backbone	pass@1	rand-of-3	selector ( $N{=}2$ )	selector ( $N{=}3$ )	oracle pass@3
$\pi_{0}$	64.2	64.5	79.2	85.2	87.8
SmolVLA	65.6	65.3	81.0	85.5	87.0
$\pi_{0.5}$	88.7	89.0	94.8	96.5	98.3

Table 4: The gain comes from selecting additional samples rather than sampling alone. LIBERO 4-suite mean success rate (%). Randomly retaining one of three rollouts remains near pass@1, while manifold scoring recovers a large fraction of the available pass@3 headroom.

Task	pass@1	$+$ retrieval	$+$ selector (ours)	oracle pass@ $N$
carrot	34.7	30.6 ( $-4.2$ )	41.7 ( $+6.9$ )	50.0
spoon	45.8	47.2 ( $+1.4$ )	54.2 ( $+8.3$ )	62.5
stack	31.9	31.9 ( $-0.0$ )	54.2 ( $+22.2$ )	54.2
overall	37.5	36.6 ( $-0.9$ )	50.0 ( $\mathbf{+12.5}$ )	55.6

Table 5: The selector transfers to a different robot and simulator, while retrieval does not improve overall performance. SimplerEnv-Bridge with

\pi_{0}

-Bridge and WidowX using

n{=}24{\times}3

rollouts. Values report success rate (%) with gains over pass@1 in parentheses, and the oracle reports pass@

N

	Configuration	Success
baseline	greedy decoding	0.40
	sample@1 ( $T{=}1$ , 3-seed mean)	0.40 ( $+0.00$ )
per-step	re-rank ( $N{=}4$ )	0.36 ( $-0.04$ )
episode-level (ours)	selector ( $N{=}3$ )	0.44 ( $+0.04$ )
	oracle pass@3	0.72

Table 6: On an autoregressive VLA, available headroom and realized selection can differ substantially. OpenVLA-7B on the SimplerEnv coke-can task using 25 initial states. Gains are relative to greedy decoding, and oracle pass@3 indicates whether any of the three sampled episodes succeeds.

	$\pi_{0}$			$\pi_{0.5}$
Suite	pass@1	$+$ retr	$+$ sel	pass@1	$+$ retr	$+$ sel
spatial	61.6	$+6.2$	$\mathbf{+18.4}$	80.8	$+3.4$	$\mathbf{+14.0}$
object	81.0	$+6.6$	$\mathbf{+9.4}$	74.0	$+0.6$	$\mathbf{+4.8}$
goal	74.4	$+2.8$	$\mathbf{+9.6}$	74.8	$+2.0$	$\mathbf{+12.0}$
10	20.8	$+1.8$	$\mathbf{+10.4}$	42.4	$-2.6$	$\mathbf{+14.0}$
mean	59.5	$+4.3$	$\mathbf{+11.9}$	68.0	$+0.8$	$\mathbf{+11.2}$

Table 7: Under scene-induced occlusion, selector gains change together with the available opportunity. LIBERO-Occ using

n{=}250{\times}2

rollouts with

N{=}2

. Pass@1 reports the occluded baseline, while retrieval and selection columns report gains over it.

Setting	Value
Visual encoder $\phi$	frozen DINOv2 $+$ SigLIP, $1{,}536$ -d, $\ell_{2}$ -normalized
Camera view for scoring	wrist camera when available, else third-person
Selector candidates $N$	$3$ , from $3$ seeds (LIBERO)
Demonstration library	$50$ demos/task (LIBERO); $510$ / $1{,}360$ frames (Bridge/coke-can)
Retrieval strength $\alpha$	$0.3$
Demonstration-fit samples $K$	$3$ , over $400$ demonstration states

Table S1 : Fixed hyperparameters shared across backbones and suites.

Test-Time Adaptation of VLA Models

Test-time adaptation of vision-language-action models helps robot policies adjust to new views, tasks, objects, and execution errors during deployment.

See all 52 papers

Jun 14, 2026

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.

Jeongeun Park, Juhan Park, Taekyung Kim +3

NAVER AI Lab · Korea University

Test-Time Adaptation Visuomotor Policy Learning Vision-Language-Action Models Test-Time Adaptation of VLA Models Cross-Embodiment Transfer

Ask PDF

Aug 5, 2026

Retrieve in Time, Correct in Frequency

Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. Successful rollouts provide useful corrective evidence, yet current frame retrieval can return progress-misaligned actions,while direct replay or time-domain fusion can overwrite the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.RTCF separates which experience to retrieve from which part of its action to transfer. Progressive Memory Alignment (PMA) causally aligns the growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and the current aligned memory position without stage labels. From the aligned action chunk,RTCF transfers a coefficient-wise-clipped low-frequency residual on motion channels. Higher-frequency components and gripper decisions remain inherited from the frozen policy. Across four LIBERO suites and 2,000 episodes per condition, RTCF raises aggregate success from 86.4% to 88.4% and improves LIBERO-Long from 61.6% to 68.6%.These gains require no parameter updates, repeated VLA inference, or additional GPU resources: correction can be performed on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk

Yuze Fan, Yue Cao, Pengjie Gao +7

Research Institute of Tsinghua University in Shenzhen · Tongji University · Everwise-Tech Co., Ltd. +2

Robotic Manipulation Vision-Language-Action Models

Ask PDF

May 27, 2026

What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies

Vision--language--action (VLA) policies are trained to imitate actions; their loss never asks them to estimate reward, progress, or future success. Their frozen representations nevertheless carry such information, and it can be read out and used to guide action choice without retraining the policy. From mixed successful and failed manipulation trajectories on LIBERO-Goal, we recover Monte-Carlo outcome targets using lightweight linear probes on frozen features. The targets are consistently predictable from OpenVLA, Pi0.5, DINOv2, and CLIP features, and substantially less so from baselines built on progress, time-to-go, task identity, or proprioception. To rule out task and temporal shortcuts, we evaluate the probes under same-task, same-timestep matched comparisons: Pi0.5 probes still reach roughly 92% pairwise ordering accuracy, while label-shuffled controls stay at chance. Used as a test-time selector over sampled Pi0.5 action prefixes, the same probe turns this offline finding into behavior: on push-plate, success rises from 26.7% under greedy decoding to 44.3%, with a second positive case on wine-rack. The gains are not universal and require additional inference compute, but the underlying finding is clean: frozen VLAs already encode information about success that their imitation objective never explicitly demands.

Jiachen Zhang, Junnan Nie, Junyi Lao +4

Peking University · China Agricultural University · 1Peking University. +1

Value Function Estimation Robotic Manipulation Vision-Language-Action Models Representation Probing

Ask PDF

Abstract

Test-Time Adaptation of VLA Models

Explore similar work

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

Retrieve in Time, Correct in Frequency

What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies