From Teleoperation to Long-Horizon Dexterity: Evaluating Visuomotor Skills and Offline Proxies for Dexterous Manipulation

Ammar N. Abbas; Aayush Jain; Prashanth Viswanath; Nivesh Gadipudi; Giulia Vilone; Jim Gibbons

cs.ROOct 29, 2026

From Teleoperation to Long-Horizon Dexterity: Evaluating Visuomotor Skills and Offline Proxies for Dexterous Manipulation

Authors: Ammar N. Abbas, Aayush Jain, Prashanth Viswanath, Nivesh Gadipudi, Giulia Vilone, Jim Gibbons

Organizations: Analog Devices, Inc., EdgeAI Robotics, Limerick, Ireland

Abstract

Imitation learning policies are intended to make robotic manipulation robust to variations in object pose, appearance, and environment that challenge pre-programmed motion. However, real-world performance remains difficult to predict, as conventional optimization signals such as validation loss show no consistent relationship with robot success. To identify the design choices that govern real-world performance, 281 imitation-learning policies spanning seven policy families were evaluated in approximately 1,700 real-robot trials of RJ45 connector grasping and insertion using a five-fingered robotic hand. Controlled ablations showed that observation and action representations dominated conventional optimizer and sampling choices. For point-cloud policies, region-of-interest cropping, representation in the robot-base frame, and normalization within the point encoder each improved success by 70-100 percentage points. For the vision-language-action policy, absolute rather than relative action prediction improved insertion success by up to 70 percentage points. Skill decomposition, multi-view perception, and recovery demonstrations were associated with the strongest-performing systems, although their effects were not independently isolated. Of 67 offline metrics evaluated, only five measures of predicted-motion smoothness and directness reliably correlated with insertion success (r = -0.70 to -0.80), and only within a single policy family. These results identify representation choices as key determinants of real-world imitation-learning performance and show that offline metrics can shortlist policies, but cannot replace real-robot evaluation.

Figures & tables

Figure 1: Skill-orchestration workflow: (a) teleoperated data collection, (b) segmentation into motion primitives and learned skills, (c) execution, where the learned skill acts after classical point-to-point motion.

Panel of Figure 2 — Figure 2: (a) Before/after success of paired runs, both jigs, with Fisher exact $p$ ; labels name every setting that differed. † Pick rate of the grasp skill; other rows: insert rate of the grasp-then-insert pairing. (b) Pearson $r$ between offline metrics and the insert-given-pick rate, one metric per cell within one policy group (centre-only or both-jig checkpoints); coloured cells are reliable.

Family (best config.; obs.)	Pick [CI]	Ins. $\|$ pick [CI]	Insert	Typical	Params	s/ep	GB tr./inf.
DP, ImageNet, 6D; RGB	100 [95,100]	82 [48,98]	70	60 (5)	72 M	71	18/1.1
DP two-view insert † ; 2 RGB	98 [94,100]	100 [69,100]	100	–	281 M	208	34/2.1
DP3 PointNet, robot frame; PCD	90 [55,100]	78 [40,97]	70	0 (26)	67 M	18	5.2/1.0
iDP3, ROI crop; PCD	100 [95,100]	100 [69,100]	100	57 (22)	69 M	18	6.0/1.0
$\pi_{0.5}$ LoRA, 2-view insert; 2 RGB	98 [94,100]	100 [69,100]	100	10 (20)	3.3 B	1.6 ‡	24/8
FM ResNet; RGB	100 [69,100]	40 [12,74]	40	40 (1)	260 M	66	21/1.9

Table 1: Best configuration per family, both jigs: pick and insert-given-pick rates [95% CI], best pairing’s insert rate, median over the family’s pairings (Typical, count), parameters of the larger skill (

\pi_{0.5}

: total, LoRA-finetuned), summed grasp+insert training time per epoch on four H100s, and larger per-GPU memory at batch 32 (training) / batch 1 (inference), which at equal batch scales mainly with model size. ‡ Per step (30k per skill). † Insert only, after the

\pi_{0.5}

grasp. p three picks.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Episodes	Used for
Image-based skills
Full grasp set	1472	RGB grasp skills
Balanced-186	186	RGB DP, $\pi_{0.5}$ grasp
Small insert set $+$ recovery	294	RGB DP, $\pi_{0.5}$ insert
Two-view balanced-200 $+$ recovery	180 $+$ 108 (insert)	two-view inserts
Point-cloud skills

Appendix

Table 2: Datasets, with grasp / insert episode counts.

Figure 3: Start positions (

x

y

of the end effector in the robot base frame) of the grasp episodes in the all-episode and balanced-200 sets, with their convex hulls. Stars mark the two test start positions. The balanced set keeps the spread of start positions while evening out the clusters formed by the jig placements.

Condition	Centre	Left
Nominal	5/5	5/5
Light variation	5/5	5/5
Background variation	5/5	4/5
New environment	5/5	4/5
New environment $+$ light variation	3/5 †	4/5
Sunlight ‡	4/5	4/5

Appendix

Table 3: Robustness. Top: the iDP3 point-cloud pipeline, insert successes out of five trials at the centre and left jig; rows marked ‡ use the large-UNet grasp and batch-320 insert checkpoints of the same recipe. Bottom: RGB DP pipelines, picks, inserts / trials pooled over both jigs.

Family	Encoder and head	Training
DP (RGB)	ResNet with one block per stage, spatial softmax; 1D UNet, DDIM; $H{=}4$ , $n_{\text{a}}{=}4$	1000 epochs; insert skill also at batch 32 (small insert set)
DP3	PointNet (LayerNorm in its layers), output dim 64; UNet, DDIM; $H{=}4$ , $n_{\text{a}}{=}3$	1000 epochs; both skills at batch 160 on the balanced set
iDP3	multi-stage PointNet, output dim 128; UNet, DDIM; $H{=}16$ , $n_{\text{a}}{=}8$	1000–3000 epochs
Flow matching	ResNet, PointNet or multi-stage encoder; UNet; 5–10 integration steps	100–1000 epochs; both skills at batch 160
ACT	ResNet backbone, CVAE with a transformer decoder	150 epochs; both skills at batch 128
$\pi_{0.5}$	PaliGemma 2B with a 300M action expert, LoRA; action horizon 50, chunks of 10 executed	30k steps per skill, batch 32

Appendix

Table 4: Default settings of each family’s best configuration (

H

: prediction horizon;

n_{\text{a}}

: executed steps). Unless noted, grasp skills are trained on the balanced set at batch 32 per GPU and insert skills on all episodes at batch 160, on four H100s.

Setting	Values tried
Data
Episode selection	all demonstrations; subset balanced across start positions
Extra episodes	recovery episodes; light-variation episodes
Cameras	wrist only; wrist and static third-person camera
Observation processing
Image augmentation	none; spotlight; spotlight and noise injection; plus colour jitter; plus cutout

Appendix

Table 5: Settings varied in the ablation study, with the values tried.

Figure 4: Training loss (solid) and validation loss (dotted) of the deployed skill of each policy family, from the runs’ training logs; each panel states how often the validation loss was computed and the skill’s robot rate. In every family the validation loss reaches its minimum early and is then flat or rising while the training loss keeps falling, regardless of the robot rate (36–100% insert-given-pick);

\pi_{0.5}

finetuning uses no validation split.

Group	Metrics
Training (1)	best validation loss
Open-loop error (10)	first-step position error, horizon error growth, average displacement error, cumulative position error, Fréchet distance (position; all dimensions), MMD and KL between predicted and demonstrated actions, consistency of overlapping action chunks, sensitivity to small state perturbations
Uncertainty (4)	spread of actions and of positions over 8 denoising samples, denoising confidence, denoising loss
Encoder (4)	keypoint jitter, number of peaked keypoints, cosine similarity of consecutive features, input saliency
Goal region (10)	mean and median position error, median goal error, mean and median orientation error, reach within 1, 2 and 5 mm, translational and rotational drift
Motion quality (9)	jerk, path-length ratio, dimensionless-jerk smoothness, SPARC, mean speed, Cartesian and orientation path length, spectral power of the predicted path and its ratio to the demonstration

Appendix

Table 6: The 67 offline metrics, by group (count in brackets).

Visuomotor Policy Learning

Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.

See all 293 papers

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds
Sep 1, 2026Clinton Enwerem, John S. Baras, Calin BeltaContact-Rich Robotic Manipulation Robotic Policy Evaluation
Institute for Systems Research University of Maryland College Park, MD, U.S.A.
Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Sep 23, 2026Tara Sadjadpour, Siming He, C. K. Wolfe +5Robot Imitation Learning Motion Retargeting
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Jun 17, 2026Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik +2RL for Robotics Reinforcement Learning
Department of Computing Science, University of Alberta · National Research Council Canada · School of Electrical Engineering and Computer Science, University of Ottawa +3

Sep 1, 2026J/K move · Enter open · S save

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

Clinton Enwerem, John S. Baras, Calin Belta

Institute for Systems Research University of Maryland College Park, MD, U.S.A.

Contact-Rich Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.

Abstract

Visuomotor Policy Learning

Explore similar work

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation