ROSER: Few-Shot Robotic Sequence Retrieval for Scalable Robot Learning

Zillur Rahman; Eddison Pham; Alejandro Daniel Noel; Cristian Meo

cs.ROMar 2, 2026

ROSER: Few-Shot Robotic Sequence Retrieval for Scalable Robot Learning

Authors: Zillur Rahman, Eddison Pham, Alejandro Daniel Noel, Cristian Meo

Organizations: University of Nevada, Las Vegas · University of Toronto · LatentWorlds AI, NL

Abstract

A critical bottleneck in robot learning is the scarcity of task-labeled, segmented training data, despite the abundance of large-scale robotic datasets recorded as long, continuous interaction logs. Existing datasets contain vast amounts of diverse behaviors, yet remain structurally incompatible with modern learning frameworks that require cleanly segmented, task-specific trajectories. We address this data utilization crisis by formalizing robotic sequence retrieval: the task of extracting reusable, task-centric segments from unlabeled logs using only a few reference examples. We introduce ROSER, a lightweight few-shot retrieval framework that learns task-agnostic metric spaces over temporal windows, enabling accurate retrieval with as few as 3-5 demonstrations, without any task-specific training required. To validate our approach, we establish comprehensive evaluation protocols and benchmark ROSER against classical alignment methods, learned embeddings, and language model baselines across three large-scale datasets (e.g., LIBERO, DROID, and nuScenes). Our experiments demonstrate that ROSER consistently outperforms all prior methods in both accuracy and efficiency, achieving sub-millisecond per-match inference while maintaining superior distributional alignment. By reframing data curation as few-shot retrieval, ROSER provides a practical pathway to unlock underutilized robotic datasets, fundamentally improving data availability for robot learning.

Figures & tables

Figure 1: Retrieval Framework. The time-series encoder $f_{\theta}$ uses labeled dataset $\mathcal{S}$ and create a metric space where similar tasks are grouped together and create prototypes while different tasks are separated from each others. The embeddings of unlabeled trajectories $\mathcal{U}$ are compared with prototypes using learned metric $Dist(.)$ to find the closest match and create retrieved set $\mathcal{R}$ .

Model	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
gemma	0.17	12.98	0.0028	0.51	0.32	16.29
llama	0.16	12.08	0.0024	0.57	0.37	14.61
momentfm	0.14	11.04	0.0023	0.62	0.41	15.10
qwen	0.17	13.04	0.0029	0.51	0.29	15.07
shapelet	0.19	11.81	0.0034	0.56	0.40	11.74
dtaidistance	0.092	6.42	0.0022	0.70	0.75	7.27

Table 1: Performance of baselines and ROSER on LIBERO dataset. Dark color represent best result and light color represent second best result. All models have same top-k results. Time-series models like Stumpy, Dtaidistance, and ROSER perform better compared to Large Models. ROSER performs best or second best in all of the metrics.

Model	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
gemma	1.02	36.56	0.00071	0.35	0.59	51.69
llama	0.73	30.74	0.00073	0.37	0.79	50.05
momentfm	0.67	24.09	0.00086	0.42	0.79	48.86
qwen	0.96	43.95	0.00083	0.28	0.49	74.38
shapelet	0.93	32.25	0.0015	0.14	0.46	48.82
dtaidistance	0.54	13.19	0.00069	0.48	1.07	19.98

Table 2: Performance of baselines and ROSER on nuScenes dataset. Time-series models like Stumpy, Dtaidistance, and ROSER perform better compared to Large Models. ROSER performs best or second best in most of the metrics.

Model	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
Gemma	0.23	30.33	0.0021	0.35	0.69	49.54
Llama	0.23	29.56	0.0018	0.37	0.72	48.66
Momentfm	0.23	29.88	0.0019	0.36	0.64	48.87
Qwen	0.24	31.03	0.0023	0.33	0.65	50.05
Shapelet	0.26	29.64	0.0028	0.37	0.72	46.50
Dtaidistance	0.21	25.23	0.0017	0.37	0.80	34.97

Table 3: Performance of baselines and ROSER on DROID dataset. Time-series models like Stumpy, Dtaidistance, and ROSER perform better compared to Large Models. ROSER performs best or second best in most of the metrics.

Figure 2: Feature-level distribution visualization for task ”regular stop” in the nuScenes benchmark.

Dataset	K-Shot	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
Libero	10	0.086	6.57	0.0019	0.72	0.69	8.8
	7	0.10	6.84	0.0022	0.71	0.59	8.38
	5	0.12	8.41	0.0026	0.68	0.49	9.61
	3	0.11	7.06	0.0025	0.69	0.55	8.72
nuScenes	10	0.27	12.84	0.00056	0.53	1.30	26.69
	7	0.35	12.62	0.00050	0.54	1.24	23.65

Table 4: ROSER performance when number of target samples is varied. We train ROSER in K-3 to K-10 shot manner. Performance degradation is insignificant when we lower K, indicating the robustness of the encoder’s ability to learn discriminative features from a few samples.

Model	LIBERO $\left(\downarrow\right)$	nuScenes $\left(\downarrow\right)$	DROID $\left(\downarrow\right)$
Gemma	0.1558	0.1348	0.1347
LLaMA	0.8566	0.7929	0.7919
MomentFM	0.1088	0.2092	0.1040
Qwen	0.4449	0.4225	0.4176
Shapelet	0.002526	0.001717	0.001617
Dtaidistance	0.09030	0.1805	0.1770

Table 5: Average retrieval time per match (seconds) across datasets. Each entry reports the mean wall-clock time required to perform a single query–candidate matching operation, measured per offline demonstration and averaged over all retrievals.

Figure 3: Relationship between Wasserstein Distance and Intra-class distance (diversity). Each point corresponds to a single retrieval model evaluated on the dataset. For each dataset, we report Spearman’s rank correlation (

\rho

), Pearson correlation (

r

), and Kendall Tau (

\tau

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{R},\mathcal{S}$	Offline, Retrieved and reference trajectory datasets (collections of tasks)
$\mathcal{R}^{(t)},\mathcal{S}^{(t)}$	Trajectories for task $t$
$\mathbf{r}_{r}^{(t)},\mathbf{s}_{i}^{(t)}$	Individual trajectories from task $t$ of $\mathcal{R}^{(t)}$ or $\mathcal{S}^{(t)}$
$L_{r}^{(t)},{L_{i}}^{(t)}$	Lengths of the r-th retrieved and i-th reference trajectories in task $t$
$F$	Number of features per time step
$r_{r,l,f}^{(t)},s_{i,l,f}^{(t)}$	Feature $f$ at time $l$ of trajectory $\mathbf{r}_{r}^{(t)}$ or $\mathbf{s}_{i}^{(t)}$

Appendix

Table 6: Notation used throughout the evaluation metrics

Dataset	Top- $k$	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
Libero	50	0.080	4.96	0.0017	0.78	0.82	6.15
	70	0.081	5.27	0.0018	0.77	0.77	6.75
	90	0.083	5.70	0.0019	0.75	0.71	7.46
	110	0.085	6.09	0.0019	0.74	0.67	8.11
	130	0.086	6.36	0.0020	0.73	0.64	8.62
	150	0.086	6.57	0.0019	0.72	0.69	8.8

Appendix

Table 7: ROSER Metrics for Varying Top-

k

. We retrieve a range of samples for each task. Performance is better when the retrieval number is small. Our assumption is tasks are imbalanced and retrieving more samples may include false positives.

Dataset	Removed Feature	WD $\left(\downarrow\right)$	DTW NN $\left(\downarrow\right)$	Spectral WD $\left(\downarrow\right)$	Temp Corr. $\left(\uparrow\right)$	Density $\left(\uparrow\right)$	Diversity $\left(\downarrow\right)$
Libero	N/A	0.086	6.57	0.0019	0.72	0.69	8.8
	EE Pos	0.12	7.9	0.0023	0.67	0.51	9.2
	Gripper States	0.10	7.52	0.0023	0.67	0.56	9.12
	Joint States	0.12	9.28	0.0022	0.69	0.53	12.13
nuScenes	N/A	0.27	12.84	0.00056	0.53	1.30	26.69
	Velocity	0.41	17.22	0.0005	0.54	0.95	40.7

Appendix

Table 8: ROSER performance when one of the features is removed. For manipulation tasks, end-effector pose and joint states are the most important. For navigation, velocity and acceleration play major roles.

Figure 4: Feature-level distribution visualization for microwave open task in the LIBERO benchmark.

Figure 5: LIBERO qualitative results for microwave open task. ROSER retrieves correct sample while Dtaidistance and Stumpy retrieves drawer close task

Figure 6: Feature-level distribution visualization for microwave close task in the LIBERO benchmark.

Figure 7: LIBERO qualitative results for microwave close task. All the models fail to retrieve correct tasks on this.

Figure 8: Feature-level distribution visualization for stove on task in the LIBERO benchmark.

Figure 9: LIBERO qualitative results for stove on task. All models successfully retrieve samples for this task.

Figure 10: Feature-level distribution visualization for stove off task in the LIBERO benchmark.

Figure 11: LIBERO qualitative results for stove off. ROSER successfully retrieves stove off tasks while Dtaidistance and Stumpy retrieves pick and place tasks

Figure 12: Feature-level distribution visualization for top drawer open task in the LIBERO benchmark.

Figure 13: LIBERO qualitative results for top drawer open. ROSER and Stumpy successfully retrieve the task while Dtaidistance retrieves pick and place task.

Figure 14: Feature-level distribution visualization for top drawer close task in the LIBERO benchmark.

Figure 15: Libero qualitative results for top drawer close. ROSER and Stumpy successfully retrieve the task while Dtaidistance retrieves pick and place task.

Figure 16: Feature-level distribution visualization for bottom drawer open task in the LIBERO benchmark.

Figure 17: LIBERO qualitative results for bottom drawer open. This task is challenging since robot arms need to avoid collding with top drawer handle to reach bottom drawer handle. ROSER successfully retrieves similar task while Stumpy and Dtaidistance retrieve other tasks.

Figure 18: Feature-level distribution visualization for bottom drawer close task in the LIBERO benchmark.

Figure 19: LIBERO qualitative results for bottom drawer close. ROSER and Dtaidistance successfully retrieves similar task while Stumpy retrieves pick and place task.

Figure 20: Feature-level distribution visualization for pick and place task in the LIBERO benchmark.

Figure 21: LIBERO qualitative results pick and place. All models can successfully retrieve pick an place tasks

Figure 22: Feature-level distribution visualization for drawer close task in the DROID benchmark.

Figure 23: DROID qualitative results for drawer close. ROSER can successfully retrieve similar task even when target environment is different.

Figure 24: Feature-level distribution visualization for pick and place task in the DROID benchmark.

Figure 25: DROID qualitative results for pick and place task. All models successfully retrieve similar tasks.

Figure 26: Feature-level distribution visualization for turn faucet task in the DROID benchmark.

Figure 27: DROID qualitative results on turn faucet. All models successfully retrieve similar tasks.

Figure 28: Feature-level distribution visualization for close cabinet task in the DROID benchmark.

Figure 29: DROID qualitative results for close cabinet task. ROSER successfully retrieves similar task from a different environment while Stumpy retrieves a pick and place task and Dtaidistance retrieves a close cabinet task.

Figure 30: Feature-level distribution visualization for open cabinet task in the DROID benchmark.

Figure 31: DROID qualitative results for open cabinet. ROSER and Dtaidistance successfully retrieve similar task while Stumpy retrieves a pick and place task.

Figure 32: Feature-level distribution visualization for left turn maneuver in the nuScenes benchmark.

Figure 33: nuScenes qualitative results for left turn maneuver. ROSER successfully retrieves similar task while Dtaidistance retrieves a regular stop and Stumpy retrieves a straight driving scene.

Figure 34: Feature-level distribution visualization for right turn maneuver in the nuScenes benchmark.

Figure 35: nuScenes qualitative results for right turn maneuver. ROSER and Dtaidistance successfully retrieves similar task while Stumpy retrieves a straight driving scene.

Figure 36: Feature-level distribution visualization for straight driving maneuver in the nuScenes benchmark.

Figure 37: nuScenes qualitative results for straight driving maneuver. All models successfully retrieve similar scene for this task.

Figure 38: Feature-level distribution visualization for regular stop maneuver in the nuScenes benchmark.

Figure 39: nuScenes qualitative results for regular stop maneuver. ROSER successfully retrieves similar task as the vehicle stops at the crosswalk while Dtaidistance and Stumpy retrieve straight driving scenes.

Explore similar work

RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning
Sep 2, 2026Howard Qian, Yiting Chen, Yunfei Xie +6Robot Skill Learning Robotic Data Collection
Rice University · NVIDIA
EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World
Apr 8, 2026Ryan Punamiya, Simar Kareer, Zeyi Liu +37Robot Skill Learning Robotic Data Collection
Georgia Institute of Technology · Stanford University · University of California San Diego +5
Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons
Mar 2, 2026Anthony Liang, Yigit Korkmaz, Jiahui Zhang +14Reward Modeling
Univ. of Southern California · UT Dallas · MIT +4

Sep 2, 2026J/K move · Enter open · S save

RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning

Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren+5

Rice University · NVIDIA

Robot Skill Learning Robotic Data Collection Robotic Manipulation Motion Representation Learning Robot Learning

Open paper AskPDF arXiv

Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and difficult to scale across the wide range of real-world tasks. To address this bottleneck, we introduce RoboTok, a scalable data engine that uses a query human manipulation video to retrieve manipulation-relevant internet demonstrations for training dexterous robot policies. Specifically, we learn a latent motion space from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient indexing and retrieval over internet video collections. We evaluate RoboTok against existing robot-data retrieval approaches using retrieval metrics and downstream robot policy performance, showing that RoboTok retrieves more manipulation-relevant demonstrations and improves downstream task success. In real-world robot experiments, RoboTok-guided policies achieve a mean success improvement of 42.2 percentage points over the strongest baseline for each task, establishing hand-pose trajectory-aware retrieval as a scalable way to leverage continuously growing web video for robot learning.

Abstract

Explore similar work

RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning

EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons