Compressing continuous human motion into a finite vocabulary of discrete tokens is now the first stage of many motion understanding and generation pipelines. Vector quantization partitions a latent space into Voronoi cells around a finite codebook, but the geometry of that latent space is inherited from the input pose encoding, and it is unclear which encoding admits an efficient discretization. We study four encodings that place skeletal motion in different geometric spaces: Gaussian joint heatmaps on the pixel lattice, which we show to be a morphological dilation of the joint point set by a quadratic structuring function; root-relative joint positions in Euclidean space; local joint rotations on \SO through a continuous 6D embedding; and a redundant full kinematic encoding. For each, we train VQ-VAE tokenizers whose codebooks lie on the unit hypersphere, varying codebook size, temporal stride, and code dimension. Beyond reconstruction, we measure codebook utilization and the effective vocabulary size, i.e., the number of equiprobable spherical Voronoi cells with the same entropy as the observed token distribution. Rotation and position encodings keep converting additional codewords into effective vocabulary, whereas the full kinematic encoding saturates at a few hundred effective cells and heatmaps at fewer than sixty. The smallest code dimension gives the highest utilization for three of the four encodings, and the effect of temporal stride depends on the encoding. Encodings of the same pose family can thus yield effective vocabularies differing by an order of magnitude, a difference that reconstruction loss alone does not reveal, which points to input geometry as a first-order design choice in motion tokenization.
Figures & tables
Figure 1: Overview, drawn from one synthetic walking sequence using the definitions of Sect. 3 . Left: the four encodings, (i) max-merged Gaussian heatmaps on a 72×72 lattice (the zoom shows the ≈3×3 support of one joint), (ii) pelvis-relative positions, (iii) local rotations from minimal-twist IK, drawn as joint frames, and (iv) the T×263 full kinematic feature matrix. Right: normalized latents hi (orange) are assigned to the nearest codeword, i.e., to a cell of the spherical Voronoi tessellation, and the cell index is the token. Shading shows how often each cell is used. Because the latents concentrate near a curve, most cells stay empty: here 23 of 120 cells are used and Keff=11.1 ( Sect. 4 ).
Encoding
Ambient
Underlying geometric space
dof
(i) Heatmap
R72×72
lattice samples of exp(ιX⊕qσ) , X⊂R2 , ∣X∣=22
≤44
(ii) Positions
R66
(S2)21 per subject (fixed bone lengths)
42
(iii) Rotations
R130
V2(R3)21≅SO(3)21 ; IK image ≅(S2)21
42+4
(iv) Full kin.
R263
as (iii), plus redundant positions, velocities, contacts
42+4
Table 1: Geometry of the four encodings: per-frame ambient space, the space in which the pose actually varies, and its per-frame degrees of freedom (dof). The root features of (iii) and (iv) are counted separately.
Figure 2: Validation loss during training. Columns: encodings (i)–(iv). Rows: codebook size K ( s=2 , D=32 ), temporal stride s ( K=1024 , D=32 ), and code dimension D ( K=1024 , s=2 ). Heatmap losses are scaled by 103 .
(i) Heatmap
(ii) Positions
(iii) Rotations
(iv) Full kin.
H^
U
Keff
H^
U
Keff
H^
U
Keff
H^
U
Keff
K
256
.48
60.16
14
.97
92.96
217
.86
52.34
118
.95
77.78
194
512
.40
44.92
12
.95
83.78
375
.96
87.89
399
.90
54.18
274
1024
.30
22.75
8
.95
85.44
724
.98
100.00
891
.84
35.14
338
2048
.30
14.70
10
.94
78.90
1296
.98
100.00
1758
.79
21.75
413
4096
.49
24.76
59
.92
67.57
2106
.97
99.97
3191
.66
6.36
242
Table 2: Codebook capacity: normalized entropy H^ , utilization U (%), and effective vocabulary size Keff=KH^ ( Eq. 6 ). Each block varies one factor from the default K=1024 , s=2 , D=32 . Bold: largest Keff per encoding within a block; –: not run.
Figure 3: Effective vocabulary size Keff against codebook size K (data of Table 2 , s=2 , D=32 ). The dashed line marks the bound Keff=K , reached when all Voronoi cells are used equally often.
Effective discrete representation learning with VQ-VAE serves as a fundamental component of modern autoregressive generative models. However, it is widely known to suffer from codebook collapse and inefficient code utilization. To address this issue, we propose FlowVQ, a simple yet effective alternative to standard vector quantization by formulating neural quantization as a joint modeling and clustering process. Specifically, we learn the latent space using a lightweight flow matching model alongside training, corresponding to a probability flow ODE that transports an often fixed well-behaved source distribution while preserving probability mass. Based on this, we define the codebook C_1 by transporting even-quantile partitioned prior codes C_0 from the source space through the learned flow, and obtaining the Voronoi centroids under the induced distribution, leading to a structured discretization. We evaluated on image tokenization and generation, as well as on two tasks requiring high-precision tokenization, demonstrating scalability and improved expressivity of the codebook with full utilization.
Robot policies increasingly act through a small vocabulary of discrete latent actions learned from unlabeled video, yet training leaves open which motion receives which code. We find this choice highly unstable: two independently trained runs of Moto's latent motion tokenizer assign the same robot clips to nearly unrelated codes (adjusted Rand index 0.001), and evaluating one tokenizer in bf16 instead of fp32 reassigns 74% of its tokens. We use this freedom to impose the representational geometry of macaque ventral premotor cortex (PMv), recorded while a monkey drew shapes, on the tokenizer's code assignment, with a straight-through similarity loss trained alongside its reconstruction objective on robot video. In one training run per target, on five shape categories the loss never saw, the PMv-trained tokenizer reaches a representational similarity of 0.52 with PMv, against 0.20-0.26 for released and no-prior tokenizers and 0.42 for the 95th percentile of random kinematic encoders. Kinematic, language, and random targets do not transfer (-0.17 to -0.03). Two results limit the claim: a hand-picked kinematic descriptor, the stroke's initial direction, matches PMv better (0.65), and a label-shuffled PMv control scores 0.41. We specify the downstream manipulation test, on a CALVIN pipeline that reproduces Moto's published results, before running it.
Most action tokenizers for autoregressive vision language action (VLA) models compress motion into chunks that must be decoded in full before their actions become executable. We introduce ActionCodec2, a non-neural streaming tokenizer that releases actions incrementally while keeping trajectory reconstruction error within a prescribed tolerance. Its encoder quantizes each step relative to the decoder's reconstructed state, so the selected motion primitive corrects residual error rather than allowing errors from independently quantized increments to accumulate. Finding the smallest such primitive set is a geometric covering problem; we derive the exact minimum for translation and constant-factor-tight bounds for rotation. Set-BPE then losslessly merges co-occurring primitive symbols across coordinates and neighboring steps, without a fixed coordinate ordering. A universal version fitted on 20k hours of data from 24 embodiments averages 8.8 tokens per second for 15 Hz end-effector motion on this corpus. With the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on three simulation benchmarks and in zero-shot real-robot evaluation. Its streaming representation allows policies to incorporate state feedback within an action chunk, substantially improving performance over open-loop execution under randomized dynamics, external disturbances, and moving objects. The same tokens also can serve as a direct control interface for VLM agents, which compose successful action sequences into reusable macros. These agent-built macros reduce control decisions and thinking-token usage across attempts and transfer to a held-out task. More results are available at https://zibindong.github.io/actioncodec2-page/
Zibin Dong, Yicheng Liu, Shiduo Zhang +4
Tsinghua University · Galaxea AI · Fudan University