Human data can support robot policy pretraining, but gains attributed to a data recipe may also reflect subset size, training variability, or downstream adaptation. In an HPT implementation on a 20-task MetaWorld suite with three training blocks, a constrained random 5,000-recording subset matches most of the task-informed subset's gain over the 40,000-recording pool at 20,000 updates (3.69 of 4.29 percentage points), leaving equal-size selection differences of 0.60 points at 5,000 recordings and 0.45 points at 1,250 recordings, with wide paired intervals. Across 12 comparisons, task orderings repeat across seeds for several contrasts, but a fixed predictor built from the other two seeds trails the mean or difficulty reference in at least one held-out fold for every comparison. Neither mechanism probe shows its predicted positive interaction: task-slot mismatch yields an interaction of −0.0347 points after adaptation, and in a six-task observation experiment the observation-by-pretraining interaction changes sign between seeds (−18.29 and +1.04 points) while full-state pretraining gains are positive in both (+6.37 and +2.31 points). This retrospective study shows how equal-size and capacity controls, seed-level reporting, and targeted interventions change the interpretation of apparent transfer gains.
Figures & tables
Figure 1: One transfer pipeline, three questions. Human recordings pretrain the HPT trunk under data recipes from three separate protocol families; only the trunk transfers to robot adaptation on 20 Meta-World tasks, and every model is evaluated on 144 initial states per task across three training seeds. The analyses then ask whether a recipe beats a matched control, whether its task-level effects repeat across seeds, and whether targeted interventions support the proposed mechanism.
Condition
Seed 1
Seed 2
Seed 3
Mean
SD
Selection / training horizon
Full40000
53.51
46.91
49.48
49.97
3.33
U5000
60.03
50.14
52.60
54.26
5.15
Random5000
58.75
55.38
46.84
53.66
6.14
Loss5000
47.01
45.10
47.01
46.38
1.10
Full160k
57.08
47.12
46.56
50.25
5.92
Table 1: Task-macro success rates (%) by training block. Standard deviation is across the three block values. The three experiment families use separate protocols. Random5000 is reused in the compression comparisons and is listed once.
Figure 2: Within-family paired effects in percentage points. Small markers show the three training-block differences; the aggregate marker and interval show their mean and descriptive 95% paired t interval ( n=3 ). Every interval crosses zero.
Figure 3: Task-rank repeatability and gain-magnitude prediction across all 12 comparisons for the two-seed same-task predictor; positive MSE improvement favors same-task prediction. No comparison beats both references in all three folds. Protocol families are separated.
Comparison
ρ1
ρ2
ρ3
Δ MSE: mean
Δ MSE: difficulty
Folds
U5000 - Random5000
-0.211
-0.478
-0.427
-359.4
-295.8
0/3
U5000 - Loss5000
+0.151
+0.462
-0.478
-111.0
-95.8
1/3
U5000 - Full40000
+0.232
+0.503
+0.390
-57.2
-67.9
1/3
Random5000 - Full40000
+0.537
+0.394
+0.661
+107.0
+113.2
2/3
Loss5000 - Random5000
+0.628
+0.422
+0.269
+89.6
+90.8
2/3
Full160k - Full40000
-0.053
-0.279
+0.006
-307.5
-254.8
0/3
Table 2: All cross-seed comparisons. Positive MSE improvement favors the fixed same-task predictor over the specified reference. The final column counts held-out folds in which its MSE is lower than both the mean and difficulty references.
Figure 4: The six observation-probe tasks. Illustrative keyframes of scripted expert demonstrations, rendered with one fixed camera in the current Meta-World release (v3 tasks); the study used the v2 tasks, and these frames do not show its trained policies. Under each task, the study’s 20,000-update observation-by-pretraining interaction for seeds 2 and 3, in percentage points ( Tab. 3 ).
Figure 5: Separate mechanism probes. The slot intervention uses one parent/downstream seed and one fixed derangement after 20,000 adaptation updates. The observation intervention uses six tasks and two training seeds; each seed’s full-input gain, reduced-input gain, and interaction are shown.
Task
Seed
B - A
D - C
Interaction
button-press-topdown
2
+0.00
+0.00
+0.00
button-press-topdown
3
+0.00
-8.33
-8.33
button-press
2
+0.00
+0.69
+0.69
button-press
3
+0.00
+1.39
+1.39
coffee-push
2
+0.00
-4.17
-4.17
coffee-push
3
+0.00
-0.69
-0.69
Table 3: Observation probe at 20,000 updates: all six tasks and both training seeds. Values are percentage-point effects, with interaction (D−C)−(B−A) .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Human pretraining
Robot adaptation
Trunk
16 blocks; width 256; 8 heads; drop path 0.1
Batch size
64 windows
256 windows
Optimizer
AdamW
AdamW
Peak learning rate
10−4
10−5
Adam β1,β2
(0.9,0.95)
(0.9,0.999)
Adam ϵ
10−8
10−7
Appendix
Table S1 : Training settings used by the ordinary human and robot policies. D4 changes only the specified human trunk layer.
Comparison
Seed 1
Seed 2
Seed 3
Mean
95% interval
U5000 - Random5000
+1.28
-5.24
+5.76
+0.60
[-13.15, +14.35]
U5000 - Loss5000
+13.02
+5.03
+5.59
+7.88
[-3.19, +18.96]
U5000 - Full40000
+6.53
+3.23
+3.12
+4.29
[-0.51, +9.10]
Random5000 - Full40000
+5.24
+8.47
-2.64
+3.69
[-10.51, +17.89]
Loss5000 - Random5000
-11.74
-10.28
+0.17
-7.28
[-23.42, +8.86]
Full160k - Full40000
+3.58
+0.21
-2.92
+0.29
[-7.78, +8.36]
Appendix
Table S2 : All 12 within-family paired contrasts. Differences and interval endpoints are in percentage points.
Step
Arm
State
Human
Seed 2
Seed 3
Mean
5000
A
Full39
No
71.99
77.31
74.65
5000
B
Full39
Yes
70.37
77.08
73.73
5000
C
Proprio8
No
67.59
50.23
58.91
5000
D
Proprio8
Yes
50.69
64.12
57.41
20000
A
Full39
No
75.00
75.81
75.41
20000
B
Full39
Yes
81.37
78.12
79.75
Appendix
Table S3 : Observation-by-pretraining cells at the diagnostic and primary readouts. All cells retain the same frozen ImageNet visual encoder and RGB input; the pretraining factor is human policy pretraining.
Step
Comparison
Seed 2
Seed 3
Mean
5000
B - A
-1.62
-0.23
-0.93
5000
D - C
-16.90
+13.89
-1.50
5000
Interaction
-15.28
+14.12
-0.58
20000
B - A
+6.37
+2.31
+4.34
20000
D - C
-11.92
+3.36
-4.28
20000
Interaction
-18.29
+1.04
-8.62
Appendix
Table S4 : Observation pretraining effects and interactions, in percentage points. The 5,000-update readout is diagnostic; 20,000 updates is the primary endpoint.
Arm
Seed
Successes / episodes
Success (%)
D4-Query
3
1350 / 2880
46.875
D4-Query-Mismatched
3
1350 / 2880
46.875
D4-Uniform
3
1386 / 2880
48.125
D4-Uniform-Mismatched
3
1385 / 2880
48.090
Appendix
Table S5 : Task-slot probe after 20,000 downstream updates. Correct-assignment endpoints are reused. This probe contains one parent/downstream training seed and one fixed derangement.
Task
Query loss
Uniform loss
Interaction
assembly
+0.000
+0.000
+0.000
basketball
+0.000
+0.000
+0.000
bin-picking
+0.000
+0.000
+0.000
box-close
+1.389
-0.694
+2.083
button-press-topdown
+0.000
+0.000
+0.000
button-press
+0.000
+0.000
+0.000
Appendix
Table S6 : Slot probe at 20,000 updates: all 20 tasks, training seed 3. Loss is correct minus mismatched success; interaction is Query loss minus Uniform loss. Values are percentage points.
Figure S1 : Task-by-seed effects for the first six comparisons in the fixed analysis order. Color and numeric annotations show percentage-point differences. Zero denotes an exact tied success count.
Figure S2 : Task-by-seed effects for the remaining six comparisons. The data-selection, compression, and routing families retain their own controls.
Variant
Designated control
Seeds
Blocks
Δ (pp)
Q
H_TBE-v4
1
1
-1.77
H_TBE
H_TBE-v4
1
1
-1.42
Q-v4-query
H_TBE-v4
1
1
-3.26
Q-query-ensemble
H_TBE-v4
1
1
-9.76
H-v4-shift50
H_TBE-v4
1
1
+0.83
H_TBE-v1
H_TBE-v4
1
1
-1.42
Appendix
Table S7 : Historical development contrasts with their designated controls. Training-seed identifiers distinguish individual late4 runs from the seed-2/3 summary; these rows are not independent experiments. D4 contrasts superseded by the three-seed confirmation are reported in the main comparison tables.