Imitation learning policies are intended to make robotic manipulation robust to variations in object pose, appearance, and environment that challenge pre-programmed motion. However, real-world performance remains difficult to predict, as conventional optimization signals such as validation loss show no consistent relationship with robot success. To identify the design choices that govern real-world performance, 281 imitation-learning policies spanning seven policy families were evaluated in approximately 1,700 real-robot trials of RJ45 connector grasping and insertion using a five-fingered robotic hand. Controlled ablations showed that observation and action representations dominated conventional optimizer and sampling choices. For point-cloud policies, region-of-interest cropping, representation in the robot-base frame, and normalization within the point encoder each improved success by 70-100 percentage points. For the vision-language-action policy, absolute rather than relative action prediction improved insertion success by up to 70 percentage points. Skill decomposition, multi-view perception, and recovery demonstrations were associated with the strongest-performing systems, although their effects were not independently isolated. Of 67 offline metrics evaluated, only five measures of predicted-motion smoothness and directness reliably correlated with insertion success (r = -0.70 to -0.80), and only within a single policy family. These results identify representation choices as key determinants of real-world imitation-learning performance and show that offline metrics can shortlist policies, but cannot replace real-robot evaluation.
Figures & tables
Figure 1: Skill-orchestration workflow: (a) teleoperated data collection, (b) segmentation into motion primitives and learned skills, (c) execution, where the learned skill acts after classical point-to-point motion.
Figure 2: (a) Before/after success of paired runs, both jigs, with Fisher exact p ; labels name every setting that differed. † Pick rate of the grasp skill; other rows: insert rate of the grasp-then-insert pairing. (b) Pearson r between offline metrics and the insert-given-pick rate, one metric per cell within one policy group (centre-only or both-jig checkpoints); coloured cells are reliable.
Family (best config.; obs.)
Pick [CI]
Ins. ∣ pick [CI]
Insert
Typical
Params
s/ep
GB tr./inf.
DP, ImageNet, 6D; RGB
100 [95,100]
82 [48,98]
70
60 (5)
72 M
71
18/1.1
DP two-view insert † ; 2 RGB
98 [94,100]
100 [69,100]
100
–
281 M
208
34/2.1
DP3 PointNet, robot frame; PCD
90 [55,100]
78 [40,97]
70
0 (26)
67 M
18
5.2/1.0
iDP3, ROI crop; PCD
100 [95,100]
100 [69,100]
100
57 (22)
69 M
18
6.0/1.0
π0.5 LoRA, 2-view insert; 2 RGB
98 [94,100]
100 [69,100]
100
10 (20)
3.3 B
1.6 ‡
24/8
FM ResNet; RGB
100 [69,100]
40 [12,74]
40
40 (1)
260 M
66
21/1.9
Table 1: Best configuration per family, both jigs: pick and insert-given-pick rates [95% CI], best pairing’s insert rate, median over the family’s pairings (Typical, count), parameters of the larger skill ( π0.5 : total, LoRA-finetuned), summed grasp+insert training time per epoch on four H100s, and larger per-GPU memory at batch 32 (training) / batch 1 (inference), which at equal batch scales mainly with model size. ‡ Per step (30k per skill). † Insert only, after the π0.5 grasp. p three picks.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Episodes
Used for
Image-based skills
Full grasp set
1472
RGB grasp skills
Balanced-186
186
RGB DP, π0.5 grasp
Small insert set + recovery
294
RGB DP, π0.5 insert
Two-view balanced-200 + recovery
180 + 108 (insert)
two-view inserts
Point-cloud skills
Appendix
Table 2: Datasets, with grasp / insert episode counts.
Figure 3: Start positions ( x , y of the end effector in the robot base frame) of the grasp episodes in the all-episode and balanced-200 sets, with their convex hulls. Stars mark the two test start positions. The balanced set keeps the spread of start positions while evening out the clusters formed by the jig placements.
Condition
Centre
Left
Nominal
5/5
5/5
Light variation
5/5
5/5
Background variation
5/5
4/5
New environment
5/5
4/5
New environment + light variation
3/5 †
4/5
Sunlight ‡
4/5
4/5
Appendix
Table 3: Robustness. Top: the iDP3 point-cloud pipeline, insert successes out of five trials at the centre and left jig; rows marked ‡ use the large-UNet grasp and batch-320 insert checkpoints of the same recipe. Bottom: RGB DP pipelines, picks, inserts / trials pooled over both jigs.
Family
Encoder and head
Training
DP (RGB)
ResNet with one block per stage, spatial softmax; 1D UNet, DDIM; H=4 , na=4
1000 epochs; insert skill also at batch 32 (small insert set)
DP3
PointNet (LayerNorm in its layers), output dim 64; UNet, DDIM; H=4 , na=3
1000 epochs; both skills at batch 160 on the balanced set
iDP3
multi-stage PointNet, output dim 128; UNet, DDIM; H=16 , na=8
1000–3000 epochs
Flow matching
ResNet, PointNet or multi-stage encoder; UNet; 5–10 integration steps
100–1000 epochs; both skills at batch 160
ACT
ResNet backbone, CVAE with a transformer decoder
150 epochs; both skills at batch 128
π0.5
PaliGemma 2B with a 300M action expert, LoRA; action horizon 50, chunks of 10 executed
30k steps per skill, batch 32
Appendix
Table 4: Default settings of each family’s best configuration ( H : prediction horizon; na : executed steps). Unless noted, grasp skills are trained on the balanced set at batch 32 per GPU and insert skills on all episodes at batch 160, on four H100s.
Setting
Values tried
Data
Episode selection
all demonstrations; subset balanced across start positions
Extra episodes
recovery episodes; light-variation episodes
Cameras
wrist only; wrist and static third-person camera
Observation processing
Image augmentation
none; spotlight; spotlight and noise injection; plus colour jitter; plus cutout
Appendix
Table 5: Settings varied in the ablation study, with the values tried.
Figure 4: Training loss (solid) and validation loss (dotted) of the deployed skill of each policy family, from the runs’ training logs; each panel states how often the validation loss was computed and the skill’s robot rate. In every family the validation loss reaches its minimum early and is then flat or rising while the training loss keeps falling, regardless of the robot rate (36–100% insert-given-pick); π0.5 finetuning uses no validation split.
Group
Metrics
Training (1)
best validation loss
Open-loop error (10)
first-step position error, horizon error growth, average displacement error, cumulative position error, Fréchet distance (position; all dimensions), MMD and KL between predicted and demonstrated actions, consistency of overlapping action chunks, sensitivity to small state perturbations
Uncertainty (4)
spread of actions and of positions over 8 denoising samples, denoising confidence, denoising loss
Encoder (4)
keypoint jitter, number of peaked keypoints, cosine similarity of consecutive features, input saliency
Goal region (10)
mean and median position error, median goal error, mean and median orientation error, reach within 1, 2 and 5 mm, translational and rotational drift
Motion quality (9)
jerk, path-length ratio, dimensionless-jerk smoothness, SPARC, mean speed, Cartesian and orientation path length, spectral power of the predicted path and its ratio to the demonstration
Appendix
Table 6: The 67 offline metrics, by group (count in brackets).
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.
Clinton Enwerem, John S. Baras, Calin Belta
Institute for Systems Research University of Maryland College Park, MD, U.S.A.
Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO outperforms five baselines in contact F1, improving on the strongest ones by 8 to 28 points, while improving the success rate of downstream dynamic retargeting by as much as 35 points. On a Sharpa hand, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects spanning 10 categories. Project page: https://morphometricimitation.github.io
Tara Sadjadpour, Siming He, C. K. Wolfe +5
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.
Department of Computing Science, University of Alberta · National Research Council Canada · School of Electrical Engineering and Computer Science, University of Ottawa +3