Robot policies increasingly act through a small vocabulary of discrete latent actions learned from unlabeled video, yet training leaves open which motion receives which code. We find this choice highly unstable: two independently trained runs of Moto's latent motion tokenizer assign the same robot clips to nearly unrelated codes (adjusted Rand index 0.001), and evaluating one tokenizer in bf16 instead of fp32 reassigns 74% of its tokens. We use this freedom to impose the representational geometry of macaque ventral premotor cortex (PMv), recorded while a monkey drew shapes, on the tokenizer's code assignment, with a straight-through similarity loss trained alongside its reconstruction objective on robot video. In one training run per target, on five shape categories the loss never saw, the PMv-trained tokenizer reaches a representational similarity of 0.52 with PMv, against 0.20-0.26 for released and no-prior tokenizers and 0.42 for the 95th percentile of random kinematic encoders. Kinematic, language, and random targets do not transfer (-0.17 to -0.03). Two results limit the claim: a hand-picked kinematic descriptor, the stroke's initial direction, matches PMv better (0.65), and a label-shuffled PMv control scores 0.41. We specify the downstream manipulation test, on a CALVIN pipeline that reproduces Moto's published results, before running it.
Figures & tables
Comparison
Same model?
Agreement
Seed A vs. seed B, Moto’s objective only
no
ARI 0.001 ; 4.1% after best relabeling (random pairing 3.9%)
Released tokenizer, bf16 vs. fp32 evaluation
yes
26% same code (74% reassigned)
Released tokenizer, TF32 vs. fp32 evaluation
yes
79% same code (21% reassigned)
Random-init tokenizer, bf16 vs. fp32 evaluation
yes
98% same code
200 training steps, bf16 vs. fp32 (same init, same batches)
—
46% same code; loss 0.145 vs. 0.147
Table 1: The partition of motions into codes is not determined by the objective. Agreement between two token assignments of the same inputs. Chance agreement for a 128-code vocabulary is about 0.8%; an adjusted Rand index (ARI) of 0 is the value for unrelated partitions. Seed comparisons use 10,240 tokens from 1,280 CALVIN validation clips (seed A at 150k steps, seed B at 99k); precision comparisons use 2,048 tokens.
Tokenizer
Target RDM
Tests whether transfer comes from…
PMv (ours)
PMv, per session
—
Shuffled PMv
PMv with category labels permuted (over all categories; Appendix B )
PMv’s distance statistics, not which shape has which (imperfect, Section 4 )
Kinematics
random nonlinear encoders of pen displacement
the motion of the pen
Language
text embeddings of shape descriptions
the shapes’ names or semantics
Random
random matrix with PMv’s spectrum
imposing any structured target
No prior (2 seeds)
none ( λ=0 )
training on robot video alone
Table 2: Tokenizers and the explanation each one tests. All are trained for 150k steps with the same recipe; the four control targets use the same loss weight, seed, and data order as PMv.
Figure 2: Premotor geometry transfers to held-out shapes. (A) RSA between each tokenizer’s code geometry and PMv on the five held-out shape categories, at the final checkpoint. Bars: mean over the four recording sessions of one training run; black lines: range over sessions; numbers: means (s.d. in Table 3 ). Vertical lines are untrained references. Dashed ( 0.42 ): the kinematic yardstick, each session’s 95th percentile of 100 random nonlinear encoders of pen motion (mean 0.26 ), averaged over sessions ( Appendix A ); only the PMv tokenizer exceeds it. The 95th percentile of the encoders’ four-session means is lower ( 0.39 ), and the shuffled tokenizer also exceeds that. Dotted ( 0.65 ): a hand-picked kinematic descriptor, the stroke’s initial direction, which exceeds every tokenizer. (B) The same score at every scored checkpoint (mean over sessions); PMv and shuffled PMv are emphasized, and the dashed line is the same yardstick. Single checkpoints are noisy because each session’s score is a rank correlation over ten distances, which is why the final checkpoint was fixed as the read-out in advance.
Tokenizer or reference
Held-out RSA
Sessions > encoders
Reconstruction error
PMv (ours)
0.52±0.08
3/4
1.76×
Shuffled PMv
0.41±0.12
2/4
1.46×
Random
−0.03±0.18
0/4
1.44×\lx@sectionsign
Language
−0.06±0.11
0/4
1.35×
Kinematics
−0.17±0.22
0/4
2.05×
No prior, seed A
0.24±0.55
2/4
0.88×
Table 3: Held-out alignment and cost at the final checkpoint. RSA: mean ± s.d. over the four sessions of one training run, on the five held-out categories. “Sessions > encoders” counts sessions above that session’s 95th percentile of 100 random kinematic encoders ( Appendix A ). Reconstruction error is relative to the mean of the two no-prior runs ( 0.0088 MSE). § Computed from the training log with the same definition.
Figure 3: Alignment costs reconstruction. (A) Training reconstruction error (running mean over 5k steps); dashed lines are the two no-prior runs, and labels give each line’s tokenizer. (B) Held-out RSA with PMv (dashed: kinematic yardstick) against final reconstruction error (mean of the last five logged values, relative to the no-prior mean of 0.0088 ). Every targeted tokenizer raises reconstruction error, to 1.35 – 2.05× the no-prior level; the PMv tokenizer reaches the highest alignment, at the second-highest cost.
Labeled data
Policy
1
2
3
4
5
Avg. Len.
100%
Published [ 3 ]
0.897
0.729
0.601
0.484
0.386
3.10
Released checkpoint, our evaluation
0.899
0.748
0.610
0.491
0.380
3.13±0.06
1%
Published [ 3 ]
0.525 †
–
–
–
–
–
Released Stage 2 + our 1% Stage 3
0.507
0.206
0.071
0.027
0.005
0.82±0.03
Table 4: Our CALVIN pipeline reproduces Moto. Success rate at each step of 1,000 five-task chains (ABC → D) and the average number of consecutive tasks completed (Avg. Len., ± s.e.). † Moto reports a 52.5% “success rate” with 1% of labeled data without specifying the step [ 3 ] ; we list it under the first task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Session 1
Session 2
Session 3
Session 4
Yardstick
0.479
0.576
0.358
0.248
0.415
Appendix
Table 5: The kinematic yardstick. 95th percentile of 100 random kinematic encoders on the five held-out categories, per recording session, and their average.
Tokenizer
10k
40k
50k
60k
70k
80k
90k
100k
110k
130k
140k
150k
PMv
0.47
0.35
0.53
0.43
0.16
0.56
0.57
0.27
0.57
0.60
0.62
0.52
Shuffled PMv
0.56
0.31
0.37
0.25
0.29
0.33
0.21
0.36
0.29
0.40
0.43
0.41
Random
−0.13
−0.25
–
–
−0.28
–
–
−0.06
−0.13
−0.08
–
−0.03
Language
0.13
−0.19
−0.43
–
−0.50
−0.45
−0.39
−0.31
−0.11
−0.11
–
−0.06
Kinematics
0.07
0.28
0.05
–
0.27
0.23
−0.12
0.26
−0.13
−0.18
0.08
−0.17
Appendix
Table 6: Held-out RSA with PMv at every scored checkpoint (mean over four sessions; “–” = not scored).