Human data can support robot policy pretraining, but gains attributed to a data recipe may also reflect subset size, training variability, or downstream adaptation. In an HPT implementation on a 20-task MetaWorld suite with three training blocks, a constrained random 5,000-recording subset matches most of the task-informed subset's gain over the 40,000-recording pool at 20,000 updates (3.69 of 4.29 percentage points), leaving equal-size selection differences of 0.60 points at 5,000 recordings and 0.45 points at 1,250 recordings, with wide paired intervals. Across 12 comparisons, task orderings repeat across seeds for several contrasts, but a fixed predictor built from the other two seeds trails the mean or difficulty reference in at least one held-out fold for every comparison. Neither mechanism probe shows its predicted positive interaction: task-slot mismatch yields an interaction of −0.0347 points after adaptation, and in a six-task observation experiment the observation-by-pretraining interaction changes sign between seeds (−18.29 and +1.04 points) while full-state pretraining gains are positive in both (+6.37 and +2.31 points). This retrospective study shows how equal-size and capacity controls, seed-level reporting, and targeted interventions change the interpretation of apparent transfer gains.
Figures & tables
Figure 1: One transfer pipeline, three questions. Human recordings pretrain the HPT trunk under data recipes from three separate protocol families; only the trunk transfers to robot adaptation on 20 Meta-World tasks, and every model is evaluated on 144 initial states per task across three training seeds. The analyses then ask whether a recipe beats a matched control, whether its task-level effects repeat across seeds, and whether targeted interventions support the proposed mechanism.
Condition
Seed 1
Seed 2
Seed 3
Mean
SD
Selection / training horizon
Full40000
53.51
46.91
49.48
49.97
3.33
U5000
60.03
50.14
52.60
54.26
5.15
Random5000
58.75
55.38
46.84
53.66
6.14
Loss5000
47.01
45.10
47.01
46.38
1.10
Full160k
57.08
47.12
46.56
50.25
5.92
Table 1: Task-macro success rates (%) by training block. Standard deviation is across the three block values. The three experiment families use separate protocols. Random5000 is reused in the compression comparisons and is listed once.
Figure 2: Within-family paired effects in percentage points. Small markers show the three training-block differences; the aggregate marker and interval show their mean and descriptive 95% paired t interval ( n=3 ). Every interval crosses zero.
Figure 3: Task-rank repeatability and gain-magnitude prediction across all 12 comparisons for the two-seed same-task predictor; positive MSE improvement favors same-task prediction. No comparison beats both references in all three folds. Protocol families are separated.
Comparison
ρ1
ρ2
ρ3
Δ MSE: mean
Δ MSE: difficulty
Folds
U5000 - Random5000
-0.211
-0.478
-0.427
-359.4
-295.8
0/3
U5000 - Loss5000
+0.151
+0.462
-0.478
-111.0
-95.8
1/3
U5000 - Full40000
+0.232
+0.503
+0.390
-57.2
-67.9
1/3
Random5000 - Full40000
+0.537
+0.394
+0.661
+107.0
+113.2
2/3
Loss5000 - Random5000
+0.628
+0.422
+0.269
+89.6
+90.8
2/3
Full160k - Full40000
-0.053
-0.279
+0.006
-307.5
-254.8
0/3
Table 2: All cross-seed comparisons. Positive MSE improvement favors the fixed same-task predictor over the specified reference. The final column counts held-out folds in which its MSE is lower than both the mean and difficulty references.
Figure 4: The six observation-probe tasks. Illustrative keyframes of scripted expert demonstrations, rendered with one fixed camera in the current Meta-World release (v3 tasks); the study used the v2 tasks, and these frames do not show its trained policies. Under each task, the study’s 20,000-update observation-by-pretraining interaction for seeds 2 and 3, in percentage points ( Tab. 3 ).
Figure 5: Separate mechanism probes. The slot intervention uses one parent/downstream seed and one fixed derangement after 20,000 adaptation updates. The observation intervention uses six tasks and two training seeds; each seed’s full-input gain, reduced-input gain, and interaction are shown.
Task
Seed
B - A
D - C
Interaction
button-press-topdown
2
+0.00
+0.00
+0.00
button-press-topdown
3
+0.00
-8.33
-8.33
button-press
2
+0.00
+0.69
+0.69
button-press
3
+0.00
+1.39
+1.39
coffee-push
2
+0.00
-4.17
-4.17
coffee-push
3
+0.00
-0.69
-0.69
Table 3: Observation probe at 20,000 updates: all six tasks and both training seeds. Values are percentage-point effects, with interaction (D−C)−(B−A) .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Human pretraining
Robot adaptation
Trunk
16 blocks; width 256; 8 heads; drop path 0.1
Batch size
64 windows
256 windows
Optimizer
AdamW
AdamW
Peak learning rate
10−4
10−5
Adam β1,β2
(0.9,0.95)
(0.9,0.999)
Adam ϵ
10−8
10−7
Appendix
Table S1 : Training settings used by the ordinary human and robot policies. D4 changes only the specified human trunk layer.
Comparison
Seed 1
Seed 2
Seed 3
Mean
95% interval
U5000 - Random5000
+1.28
-5.24
+5.76
+0.60
[-13.15, +14.35]
U5000 - Loss5000
+13.02
+5.03
+5.59
+7.88
[-3.19, +18.96]
U5000 - Full40000
+6.53
+3.23
+3.12
+4.29
[-0.51, +9.10]
Random5000 - Full40000
+5.24
+8.47
-2.64
+3.69
[-10.51, +17.89]
Loss5000 - Random5000
-11.74
-10.28
+0.17
-7.28
[-23.42, +8.86]
Full160k - Full40000
+3.58
+0.21
-2.92
+0.29
[-7.78, +8.36]
Appendix
Table S2 : All 12 within-family paired contrasts. Differences and interval endpoints are in percentage points.
Step
Arm
State
Human
Seed 2
Seed 3
Mean
5000
A
Full39
No
71.99
77.31
74.65
5000
B
Full39
Yes
70.37
77.08
73.73
5000
C
Proprio8
No
67.59
50.23
58.91
5000
D
Proprio8
Yes
50.69
64.12
57.41
20000
A
Full39
No
75.00
75.81
75.41
20000
B
Full39
Yes
81.37
78.12
79.75
Appendix
Table S3 : Observation-by-pretraining cells at the diagnostic and primary readouts. All cells retain the same frozen ImageNet visual encoder and RGB input; the pretraining factor is human policy pretraining.
Step
Comparison
Seed 2
Seed 3
Mean
5000
B - A
-1.62
-0.23
-0.93
5000
D - C
-16.90
+13.89
-1.50
5000
Interaction
-15.28
+14.12
-0.58
20000
B - A
+6.37
+2.31
+4.34
20000
D - C
-11.92
+3.36
-4.28
20000
Interaction
-18.29
+1.04
-8.62
Appendix
Table S4 : Observation pretraining effects and interactions, in percentage points. The 5,000-update readout is diagnostic; 20,000 updates is the primary endpoint.
Arm
Seed
Successes / episodes
Success (%)
D4-Query
3
1350 / 2880
46.875
D4-Query-Mismatched
3
1350 / 2880
46.875
D4-Uniform
3
1386 / 2880
48.125
D4-Uniform-Mismatched
3
1385 / 2880
48.090
Appendix
Table S5 : Task-slot probe after 20,000 downstream updates. Correct-assignment endpoints are reused. This probe contains one parent/downstream training seed and one fixed derangement.
Task
Query loss
Uniform loss
Interaction
assembly
+0.000
+0.000
+0.000
basketball
+0.000
+0.000
+0.000
bin-picking
+0.000
+0.000
+0.000
box-close
+1.389
-0.694
+2.083
button-press-topdown
+0.000
+0.000
+0.000
button-press
+0.000
+0.000
+0.000
Appendix
Table S6 : Slot probe at 20,000 updates: all 20 tasks, training seed 3. Loss is correct minus mismatched success; interaction is Query loss minus Uniform loss. Values are percentage points.
Figure S1 : Task-by-seed effects for the first six comparisons in the fixed analysis order. Color and numeric annotations show percentage-point differences. Zero denotes an exact tied success count.
Figure S2 : Task-by-seed effects for the remaining six comparisons. The data-selection, compression, and routing families retain their own controls.
Variant
Designated control
Seeds
Blocks
Δ (pp)
Q
H_TBE-v4
1
1
-1.77
H_TBE
H_TBE-v4
1
1
-1.42
Q-v4-query
H_TBE-v4
1
1
-3.26
Q-query-ensemble
H_TBE-v4
1
1
-9.76
H-v4-shift50
H_TBE-v4
1
1
+0.83
H_TBE-v1
H_TBE-v4
1
1
-1.42
Appendix
Table S7 : Historical development contrasts with their designated controls. Training-seed identifiers distinguish individual late4 runs from the seed-2/3 summary; these rows are not independent experiments. D4 contrasts superseded by the three-seed confirmation are reported in the main comparison tables.
Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing different human-to-robot (H2R) transfer methods remains challenging, as existing approaches are evaluated under different settings, including differing task suites, scene layouts, object instances, and amounts of robot supervision. To address this challenge, we present H2RBench, a Real2Sim benchmark for evaluating H2R transfer methods. H2RBench provides a standardized protocol built on real human video demonstrations and simulated robot demonstrations, and includes four manipulation tasks spanning diverse interaction requirements. We evaluate multiple representative H2R transfer methods, each adopting a different strategy for bridging the embodiment gap. Using H2RBench, we systematically characterize how each method scales with the amount of human demonstrations, revealing that methods differ substantially in their ability to leverage additional human data. We further show that simulation performance is broadly predictive of real-world robot performance, with an overall Pearson correlation of r = 0.89, Spearman correlation of \r{ho} = 0.85 and Mean Maximum Rank Violation (MMRV) of 0.06 across method-task configurations. These results establish H2RBench as a practical and scalable benchmark for comparative H2R evaluation prior to real-world deployment.
Chuyang Xiao, Haotian Zhan, Sriram Krishna +4
Robotics Institute, Carnegie Mellon University · University of Michigan · Toyota Research Institute
Human video datasets used for cotraining robot manipulation policies largely consist of curated demonstrations where motions are orchestrated to resemble robot behavior and 3D hand poses are captured with specialized hardware. A more plentiful source of data is everyday Internet video, but it is an open question what factors enable transfer from such videos to robots. We investigate this using a new dataset of 532 human videos with 28 hours of high-quality triangulated hand labels and natural motions. We find that hand pose quality affects transfer, but even with accurate hands, the inherent motion gap hinders transfer unless the vision and policy networks specialize to each embodiment. Our cotraining recipe yields consistent improvements, with an absolute success rate gain of 29.7% in the low-robot-data regime across six manipulation tasks.
Richard Li, Aditya Prakash, Andrew Wen +3
1Massachusetts Institute of Technology · University of Illinois Urbana-Champaign · 3Harvard University
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.