RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

Xuhui Kang; Sung-Wook Lee; Yan Zhuang; Tianle Zhong; Yen-Ling Kuo

cs.ROOct 9, 2026

RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

Authors: Xuhui Kang, Sung-Wook Lee, Yan Zhuang, Tianle Zhong, Yen-Ling Kuo

Organizations: University of Virginia

Abstract

Pre-trained robot policies, such as Vision-Language-Action (VLA) models, provide a strong foundation for generalist robotic manipulation, but adapting them to new tasks with only a few demonstrations remains challenging. Directly fine-tuning a pre-trained policy often achieves strong in-distribution performance, yet it can distort the representations learned during pre-training and weaken the model's generalization ability under out-of-distribution shifts. We propose Retrieval-Anchored Representation Alignment (RARA), a two-stage adaptation method, to address this issue. First, RARA retrieves representation features from pretraining with similar task intent and motion, and aligns the policy's representations of the target demonstrations with the pre-trained representations of the retrieved demonstrations. Second, RARA fine-tunes the policy on the target demonstrations while limiting how much its representations of the retrieved demonstrations change during fine-tuning. By reducing the representation mismatch before action learning and preserving pre-trained knowledge during adaptation, RARA limits unnecessary representation drift during fine-tuning. Experiments with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa, and on a real robot show that RARA improves the average success of fine-tuning by 9.5 points in-distribution and 13.6 points out-of-distribution, and outperforms all compared baselines. Project page: https://shua-kang.github.io/RARA_robot/

Figures & tables

Figure 1: Overview of RARA. (Left) A policy is first pre-trained on diverse robot data. (Middle) To adapt it to a target task, RARA first retrieves pre-training demonstrations of a similar skill, such as removing a lid from a flask for the target task of opening a blender lid. It then aligns the target representation with their pre-trained representation, which it keeps as an anchor during fine-tuning. (Right) Across 13 simulated and real-world tasks, RARA improves the average ID and OOD success over fine-tuning by 11.6 points.

Figure 2: Stage 1 of RARA. (a) For a target frame, frames of the retrieved demonstrations with similar action chunks (small DTW distance) are pulled closer, and frames with different action chunks are pushed away. (b) UMAP of the representations before (left) and after (right) Stage 1: the target representations (orange) move toward those of the retrieved pre-training data (purple).

Figure 3: Pre-training demonstration retrieval for the target task: Open Blender Lid. Selecting by action similarity alone can return similar action chunks from a different task (grey); retrieving by the task instruction returns the same kind of task (purple).

Figure 4: ID (top) and OOD (bottom) scenes for one RoboTwin, one RoboCasa and one real-robot task.

	Pre-training feature retention						Target-data feature distortion
	Move pill bottle			Place container			Move pill bottle		Place container
Recipe	Rank $\uparrow$	CKA $\uparrow$	$R^{2}$ $\uparrow$	Rank $\uparrow$	CKA $\uparrow$	$R^{2}$ $\uparrow$	ID $\downarrow$	OOD $\downarrow$	ID $\downarrow$	OOD $\downarrow$
Pre-trained $P$	58.8	1.00	0.71	58.8	1.00	0.71	0.00	0.00	0.00	0.00
FT	50.2	0.68	0.67	53.1	0.87	0.64	0.50	0.49	0.47	0.48
Co-FT	51.4	0.93	0.72	49.1	0.94	0.72	0.57	0.57	0.62	0.63
RARA, after Stage 1	73.2	0.70	0.74	68.6	0.76	0.75	0.48	0.46	0.56	0.53

Table 1: Representation metrics at the anchored layer on RoboTwin (mean over five pre-training seeds). Left: computed on held-out pre-training frames; higher values mean that more of the pre-trained structure remains. Right: feature distortion on held-out frames of the target task in the ID and OOD conditions; lower values mean smaller changes from the pre-trained policy

P

. Bold marks the best adapted value in each column. “After Stage 1” is the policy before Stage-2 fine-tuning.

RoboTwin	Place Container		Move Pill Bottle		Click Alarm Clock		Shake Bottle		Task average
Method	ID	OOD	ID	OOD	ID	OOD	ID	OOD	ID	OOD
FT	82.0 $\,\pm$ 0.7	60.7 $\,\pm$ 2.4	32.9 $\,\pm$ 3.7	9.6 $\,\pm$ 2.3	80.4 $\,\pm$ 6.3	52.4 $\,\pm$ 4.1	74.2 $\,\pm$ 2.1	66.4 $\,\pm$ 5.7	67.4 $\,\pm$ 1.4	47.3 $\,\pm$ 2.4
Co-FT	62.7 $\,\pm$ 5.8	43.3 $\,\pm$ 3.3	32.4 $\,\pm$ 8.1	10.0 $\,\pm$ 2.4	81.8 $\,\pm$ 3.4	61.3 $\,\pm$ 1.8	80.7 $\,\pm$ 6.8	75.6 $\,\pm$ 5.7	64.4 $\,\pm$ 2.8	47.6 $\,\pm$ 1.3
STRAP	81.1 $\,\pm$ 3.8	59.3 $\,\pm$ 2.4	44.4 $\,\pm$ 5.1	13.8 $\,\pm$ 1.4	81.3 $\,\pm$ 1.3	61.6 $\,\pm$ 1.7	83.6 $\,\pm$ 7.3	74.9 $\,\pm$ 7.7	72.6 $\,\pm$ 2.0	52.4 $\,\pm$ 2.3
RETAIN	55.3 $\,\pm$ 10.1	40.7 $\,\pm$ 4.2	29.3 $\,\pm$ 6.1	13.3 $\,\pm$ 3.1	53.3 $\,\pm$ 6.1	36.0 $\,\pm$ 4.0	50.0 $\,\pm$ 6.0	46.7 $\,\pm$ 6.4	47.0 $\,\pm$ 6.7	34.2 $\,\pm$ 2.3
BlindVLA	69.3 $\,\pm$ 3.1	52.9 $\,\pm$ 2.0	47.6 $\,\pm$ 1.4	15.8 $\,\pm$ 0.8	74.0 $\,\pm$ 6.7	46.9 $\,\pm$ 1.0	80.7 $\,\pm$ 4.8	73.3 $\,\pm$ 2.9	67.9 $\,\pm$ 1.0	47.2 $\,\pm$ 0.8

Table 2: Simulation results (success rate, %, mean

\pm

std). RoboTwin uses the Diffusion Policy; RoboCasa uses GR00T N1.7 pre-trained on DROID. Bold: best; underline: second best.

Figure 5: Real-robot results (success rate, %, mean

\pm

std over three runs) with GR00T N1.7 pre-trained on DROID. RARA outperforms baselines in all target tasks, ID and OOD settings.

Figure 6: Choice of anchored layer on Move Pill Bottle: mean of ID and OOD success. Left: Diffusion Policy. Right: GR00T N1.7, where every arm trains the vision encoder and the action head with the LLM frozen, as FT does (Appx. E ). Bars and band: std over the last three checkpoints.

	RoboTwin (avg.)		Open Blender Lid
Method	ID	OOD	ID	OOD
FT	67.4 $\,\pm$ 1.4	47.3 $\,\pm$ 2.4	76.0 $\,\pm$ 3.5	44.7 $\,\pm$ 9.0
RARA	76.0 $\,\pm$ 1.0	58.3 $\,\pm$ 1.2	80.7 $\,\pm$ 3.1	56.7 $\,\pm$ 9.0
Random pre-training data	50.3 $\,\pm$ 1.7	34.9 $\,\pm$ 1.7	67.7 $\,\pm$ 1.3	44.2 $\,\pm$ 1.2
Random Stage-1 pairs	65.3 $\,\pm$ 3.1	40.6 $\,\pm$ 1.2	76.7 $\,\pm$ 2.3	45.5 $\,\pm$ 4.2
Co-FT in Stage 2	65.8 $\,\pm$ 3.3	48.1 $\,\pm$ 2.9	72.9 $\,\pm$ 5.3	53.6 $\,\pm$ 3.7

Table 3: Ablations of RARA (success rate, %, mean

\pm

std). RoboTwin: average over its four tasks. Bold: best; underline: second best.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: PCA visualization of visual features of ID and OOD tasks.

Figure 8: Additional examples of retrieval for RARA. Without the instruction filter, selection by action similarity can return pre-training demonstrations with similar action chunks but a different task. Filtering by the task instruction returns demonstrations that better match the target task.

Figure 9: RoboCasa training and OOD evaluation environments for all simulated robot tasks. OOD settings include variations in camera view, background, distractors, objects, clutter, and layouts.

Figure 10: Real-world Franka training and OOD evaluation environments for all robot tasks. OOD settings include variations in camera view, background, distractors, and objects.

Figure 11: RoboTwin training and OOD evaluation environments for the four target tasks. OOD settings widen the object layout and vary the background and distractors.

Stage 1 (alignment)
Anchored representation $h^{\ell}$ (default)	pooled encoder output of both cameras, concatenated (1024-d)
Other locations (RQ3)	ResNet stages 1–4, FiLM conditioning, UNet middle block, action output
Trainable	encoder (both cameras) and projector
Projector $g_{\psi}$	MLP $1024\!\to\!512\!\to\!256$ , GELU, $\ell_{2}$ -normalized output
Positives / negatives	$m_{+}=4$ nearest by action-chunk DTW; 64 negatives per step, sampled beyond the median distance
Temperature, $\lambda_{1}$	$\tau=0.07$ , $\lambda_{1}=1$

Appendix

Table 4: Diffusion Policy settings on RoboTwin.

	Target task
Pre-training data	Move Pill Bottle	Place Container	Click Alarm Clock
Pick-and-place	$\mathbf{+18.8}$	$\mathbf{+4.9}$	$+0.2$
Click Bell	$-6.7$	$-20.0$	$\mathbf{+10.3}$

Appendix

Table 5: Gain of RARA over FT (points, RoboTwin) when only the pre-training data is changed. Bold: larger gain for each target.

		FT		RETAIN
Setting	Pre-training $\rightarrow$ target	ID	OOD	ID	OOD
RoboTwin	RoboTwin $\rightarrow$ RoboTwin	67.4 $\pm$ 1.4	47.3 $\pm$ 2.4	47.0 $\pm$ 6.7	34.2 $\pm$ 2.3
RoboCasa	DROID (real) $\rightarrow$ RoboCasa (simulated)	69.2 $\pm$ 2.5	44.3 $\pm$ 2.0	31.1 $\pm$ 1.8	14.8 $\pm$ 2.9

Appendix

Table 6: RETAIN and FT in the two simulated settings (task-average success rate, %, mean

\pm

std).

RoboCasa	Close Fridge Drawer		Open Blender Lid		Open Cabinet		Turn On Toaster		Pick-Place to Drawer		Task average
Method	ID	OOD	ID	OOD	ID	OOD	ID	OOD	ID	OOD	ID	OOD
FT	86.0 $\,\pm$ 8.7	62.7 $\,\pm$ 3.1	76.0 $\,\pm$ 3.5	44.7 $\,\pm$ 9.0	72.0 $\,\pm$ 2.0	44.0 $\,\pm$ 10.6	72.0 $\,\pm$ 5.3	46.0 $\,\pm$ 6.0	40.0 $\,\pm$ 2.0	24.0 $\,\pm$ 8.0	69.2 $\,\pm$ 2.5	44.3 $\,\pm$ 2.0
RETAIN	33.3 $\,\pm$ 5.8	22.0 $\,\pm$ 6.0	13.3 $\,\pm$ 7.6	9.3 $\,\pm$ 9.5	46.7 $\,\pm$ 3.1	19.3 $\,\pm$ 4.2	46.0 $\,\pm$ 2.0	20.0 $\,\pm$ 2.0	16.0 $\,\pm$ 3.5	3.3 $\,\pm$ 3.1	31.1 $\,\pm$ 1.8	14.8 $\,\pm$ 2.9

Appendix

Table 7: RETAIN and FT on the five RoboCasa tasks (success rate, %, mean

\pm

std).

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 173 papers

Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment
Jul 15, 2026Dwip Dalal, Shivansh Patel, Chahit Jain +8VLA Model Training Vision-Language-Action Models
University of Illinois Urbana-Champaign · Texas A&M University · University of California, Irvine
Semantic Anchoring for Robotic Action Representations
Jul 15, 2026Yuan Xu, Youheng Shi, Chengyang Li +2Action Representation Learning Generalization in Robotic Manipulation
Peking University · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University
RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation
Aug 26, 2026Sanghwan Jang, Minjin Jeon, Minsoo Kim +3Test-Time Adaptation of VLA Models Vision-Language-Action Models
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea

Jul 15, 2026J/K move · Enter open · S save

Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

Dwip Dalal, Shivansh Patel, Chahit Jain, Jeonghwan Kim+7

University of Illinois Urbana-Champaign · Texas A&M University · University of California, Irvine

VLA Model Training Vision-Language-Action Models Vision-Language Alignment Robustness of VLA Models VLA Model Adaptation

Open paper AskPDF arXiv

Finetuning a pretrained vision-language model (VLM) on robot demonstrations via behavior cloning (BC) has become the standard recipe for vision-language-action (VLA) policies. However, BC finetuning progressively overwrites the pretrained representations that support visual and semantic generalization. Co-training on web image-text data, a common remedy, applies language and action losses to separate observations, leaving VLAs with language-action misalignment that standard manipulation benchmarks do not expose. We propose Anchor-Align, which augments BC with two objectives: Vision-Language Anchoring distills layer-wise representations from a frozen VLM copy to prevent this drift, while Language-Action Alignment converts each action target into a discrete motion-direction label and jointly trains language and action prediction on the same robot observation. We conduct real-world evaluations across eight manipulation settings on single-arm xArm7 and bimanual YAM robots, using two VLA architectures with regression and flow-matching action heads. Across these settings, Anchor-Align consistently improves over BC on novel targets, layouts, and motion-sensitive bimanual tasks requiring coordinated control. At scale in simulation, we demonstrate consistent improvements on OOD perturbations, perceptual robustness, and long-horizon control across LIBERO-PRO, LIBERO-Plus, and CALVIN, respectively, suggesting that preserving pretrained representations and effective action learning are not fundamentally at odds. Project page: anchoralignvla.github.io

Abstract

Robot Policy Adaptation

Explore similar work

Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

Semantic Anchoring for Robotic Action Representations

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation