While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at https://hoar012.github.io/HSR-Project.
Figures & tables
Fig. 1: H ierarchical S kill R etrieval (HSR): LLM-based task decomposition enables structured demonstration retrieval for data-efficient VLA adaptation.
Fig. 2: Overview of the Proposed HSR Framework . HSR identifies existing skills in the prior dataset based on task-instruction clustering. Given a long-horizon target task, 1. HSR first decomposes the task into several subtasks, evaluates and selects the plan with the highest score; 2. for each subtask, HSR retrieves data based on the subtask instruction, and further reranks using representative demonstrations from the target task to filter out low-consistency data; 3. HSR first pretrains the policy on retrieved data to learn general skills, and then finetunes it on task-related data to adapt to the target task.
Fig. 3: Illustration of Plan Evaluation . We first cluster Dprior into reusable skills. Each clustered skill is then evaluated by finetuning a pretrained VLA model on the training split and measuring the BC loss on its validation set. The resulting loss serves as a lightweight proxy for skill reliability, enabling efficient plan evaluation without costly environment rollouts.
Category
Pick-Place
Spatial Understanding
Composite Manipulation
Task
Soup-S
Cheese-B
Book
Soup-C
Moka-M
Mug-Mug
Mug-P
Stove-M
Bowl-C
Mug-M
Average
BC
2.0 ± 2.0
18.0 ± 3.5
20.7 ± 4.2
3.3 ± 1.2
0.0 ± 0.0
0.0 ± 0.0
12.7 ± 3.1
68.0 ± 5.3
77.3 ± 4.2
24.7 ± 5.8
22.7 ± 1.1
Random
10.0 ± 5.3
18.0 ± 3.5
72.0 ± 9.2
6.7 ± 2.3
0.7 ± 1.2
11.3 ± 2.3
8.0 ± 5.3
58.7 ± 4.2
44.0 ± 2.0
30.7 ± 9.2
26.0 ± 3.7
All Data
16.7 ± 1.2
19.3 ± 3.1
81.3 ± 6.1
11.3 ± 2.3
3.3 ± 4.2
12.7 ± 3.1
6.7 ± 2.3
62.7 ± 6.4
44.7 ± 3.1
31.3 ± 11.7
29.0 ± 2.4
Language
16.7 ± 8.1
20.0 ± 5.3
88.7 ± 5.0
14.7 ± 8.1
2.7 ± 3.1
16.0 ± 3.5
10.7 ± 2.3
68.0 ± 5.3
45.3 ± 9.0
32.7 ± 6.1
31.5 ± 1.1
FR
16.0 ± 5.3
20.0 ± 0.0
76.0 ± 7.2
9.3 ± 5.0
0.7 ± 1.2
14.0 ± 0.0
11.3 ± 5.8
63.3 ± 6.1
46.7 ± 7.6
25.3 ± 10.3
28.3 ± 1.6
TABLE I: Comparison on LIBERO Benchmark . We compare our method with behavior cloning (BC), training with all data, language-based retrieval, and four retrieval baselines: FR [ 27 ] , STRAP [ 14 ] , BR [ 10 ] , and IWR [ 11 ] . We report the success rate (in %) and standard deviation across 3 seeds. The best results are in bold and the second-best results are underlined .
Fig. 4: Experimental Setup of Real-robot Tasks . We evaluate using a 7-DoF xArm on several manipulation tasks: (a) Drawer : Close the drawer, (b) Trashcan : Pick up the crumpled paper and throw it in the trash can, (c) Cup-Drawer : Put the cup into the drawer and close it, (d) Cup-Tea : Put the teabag next to the cup and pour water into the cup.
Task
Drawer
Trashcan
Cup-Drawer
Cup-Tea
Overall
#Subtask
1
2
3
4
BC
12/20
7/20
5/20
3/20
27/80
BR
15/20
3/20
4/20
5/20
27/80
IWR
16/20
4/20
7/20
4/20
31/80
HSR
18/20
8/20
13/20
9/20
48/80
TABLE II: Experimental Results on Real-robot Manipulation . For each task, we report the success rate over 20 trials. We also report the overall success rate across all tasks. The best results are in bold .
Fig. 5: Results of Ablation Studies. We conduct ablation studies on four LIBERO tasks to evaluate the effectiveness of key components of HSR, including task decomposition, feature reranking, and the pretraining-finetuning strategy. The results show that each component improves performance, with the full HSR framework achieving the highest success rates across all tasks.
Method
Avg. Succ. (%)
LLM Only
37.1 ± 2.5
LLM+Skill Set
37.7 ± 1.2
HSR
43.5 ± 3.3
Human
44.9 ± 1.0
TABLE III: Ablation on Task Decomposition .
Fig. 6: Comparison of Retrieved Data. Distribution of retrieved data for the task “put the yellow and white mug in the microwave and close it” . HSR effectively retrieves demonstrations across subtasks covering both object-placing and microwave-closing skills.
Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, yet adapting them to new tasks and domains remains inefficient: existing methods often rely on parameter tuning, incurring substantial costs and risking catastrophic forgetting of previously learned tasks. To address this, we propose Optimus-R, a memory-centric VLA framework that formulates robotic adaptation as explicit query-skill memory tuning. Optimus-R introduces: (i) An Inline Memory Interface for skill extraction. It inserts learnable memory tokens into the VLA prefix stream, allowing the backbone to derive control-aware query and skill representations within the native action-conditioning pathway. (ii) A Query-Skill Memory Bank for skill learning. It externalizes skills into query prototypes for deciding what to retrieve and skill values for specifying how to act, supporting skill reuse and expansion with limited parameter updates. (iii) A lightweight Bridge-and-Adapt mechanism for skill updating. It aligns target-domain queries and skills with the existing memory space through a lightweight adapter and residual memory updates. Experiments on in-domain adaptation, cross-domain adaptation, and lifelong learning show that Optimus-R enables data-efficient skill learning while mitigating catastrophic forgetting.
Zaijing Li, Rui Shao, Bing Hu +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Vision-language-action (VLA) models have shown strong potential as generalist robot policies, but adapting them to unseen tasks often requires costly parameter updates. Recent work such as RICL introduces in-context adaptability by retrieving expert demonstrations based on the current VLA observation and providing them as additional context at test time. The effectiveness of this adaptation therefore depends critically on the retrieval mechanism. In this work, we systematically study how different retrieval methods affect both retrieval quality and task performance within the RICL framework. Specifically, we compare four different methods: image-based retrieval, retrieval augmented with VLA's state, retrieval using features from the VLA backbone, and random retrieval. Our experiments yield three main findings. First, no retrieval method consistently dominates the others in task success, while surprisingly, random retrieval achieves a non-trivial success rate. Second, standard retrieval-quality diagnostics do not reliably reflect downstream VLA performance. Third, demonstrations from different but related tasks can provide useful transferable information. Together, these results provide an initial step toward understanding how retrieval mechanisms shape the in-context learning capability of VLA models and their downstream task performance, while highlighting the need for more careful design and evaluation of retrieval mechanisms for reliable test-time adaptation.
Zixuan Liu, Joris Köster, Zizhan Zheng +1
Department of Computer Science, Tulane University · Department of Computer Science, Aalto University · Department of Industrial Engineering and Management, Aalto University
Vision-Language-Action (VLA) models provide a versatile foundation for general robotic manipulation, yet they exhibit significant brittleness when confronted with novel task distributions. While In-Context Imitation Learning (ICIL) offers a training-free alternative, existing frameworks suffer from an adaptation bottleneck that hinders the effective translation of expert context to executable actions. This failure originates from superficial retrieval mechanisms and an inherent behavioral inertia that anchors the policy to its pre-trained priors. To address these limitations, we present RA-VLA, a retrieval-augmented VLA framework that integrates behavior-aligned context retrieval with a grounded execution pipeline. By enforcing faithful adherence to functional cues within a scalable architecture, RA-VLA facilitates seamless task adaptation while preserving inference efficiency. Our empirical evaluations across the LIBERO benchmark and a real-world UR5e environment demonstrate that RA-VLA achieves superior success rates and computational efficiency, establishing a robust framework for training-free robotic adaptation.
Sanghwan Jang, Minjin Jeon, Minsoo Kim +3
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea