Organizations: Sony Computer Science Laboratories - Rome. Joint Initiative CREF-SONY, Centro Ricerche Enrico Fermi. Via Panisperna 89/A, 00184, Rome, Italy · Department of Computer, Automatic and Management Engineering. Sapienza University, Via Ariosto 25, Rome, Italy
Vector-quantized representations enable powerful discrete generative models but lack semantic structure in token space, limiting interpretable human control. We introduce SOM-VQ, a tokenization method that combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit low-dimensional topology. Unlike standard VQ-VAE, SOM-VQ uses topology-aware updates that preserve neighborhood structure: nearby tokens on a learned grid correspond to semantically similar states, enabling direct geometric manipulation of the latent space. We demonstrate that SOM-VQ produces more learnable token sequences in the evaluated domains while providing an explicit navigable geometry in code space. Critically, the topological organization enables intuitive human-in-the-loop control: users can steer generation by manipulating distances in token space, achieving semantic alignment without frame-level constraints. We focus on human motion generation - a domain where kinematic structure, smooth temporal continuity, and interactive use cases (choreography, rehabilitation, HCI) make topology-aware control especially natural - demonstrating controlled divergence and convergence from reference sequences through simple grid-based sampling. SOM-VQ provides a general framework for interpretable discrete representations applicable to music, gesture, and other interactive generative domains.
Figures & tables
Domain
Method
Act. (%)
Trust ↑
Cont. ↑
MSE ↓
Seq-PPL ↓
Dist. ↓
VQ
99
0.999
0.960
0.005
678
—
SOM-hard
95
0.998
0.942
0.006
679 ×
0.041
Lorenz
VQ-VAE
99
0.999
0.961
0.003
672
—
SOM-VQ
71
0.998
0.942
0.007
617
0.062
VQ
61
0.940
0.898
0.294
719
—
SOM-hard
72
0.932
0.897
0.342
742 ×
0.093
Table 1: Cross-domain comparison at 32 × 32 ( K=1024 ). SOM-VQ produces the most learnable sequences (lowest Seq-PPL) and organizes the codebook geometrically (Distortion), unlike VQ/VQ-VAE, which lack grid structure (—). SOM-hard achieves lower Distortion through pure topology optimization, but cannot support generation; SOM-VQ’s slightly higher Distortion reflects the VQ commitment constraint that enables stable discrete assignment. VQ-VAE uses larger architecture (hidden=512 vs. 256) and dead-code reset. Full statistics in Appendix Table 2 .
Figure 1: Human-in-the-loop control dynamics. Smoothed SOM grid distance (left) and prototype MSE (right), both normalised by their respective prompt-phase means, across the three interaction phases. Both metrics rise during divergence and return toward baseline during convergence, confirming that the topological steering operates consistently in grid space and in pose space simultaneously.
Figure 2: Generated poses under topology-guided control. Top: reference. Bottom: model generation. The grid enables semantic control through geometric operations in token space.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Method
Act. (%)
Trust ↑
Cont. ↑
MSE ↓
Seq-PPL ↓
Dist. ↓
VQ
99
0.999 ± 0.000
0.960 ± 0.001
0.005 ± 0.000
678 ± 12
—
SOM-hard
95
0.998 ± 0.000
0.942 ± 0.001
0.006 ± 0.000
679 ×
0.041 ± 0.004
Lorenz
VQ-VAE
99
0.999 ± 0.000
0.961 ± 0.001
0.003 ± 0.000
672 ± 9
—
SOM-VQ
71
0.998 ± 0.000
0.942 ± 0.001
0.007 ± 0.001
617 ± 8
0.062 ± 0.006
VQ
61
0.940 ± 0.006
0.898 ± 0.004
0.294 ± 0.039
719 ± 18
—
SOM-hard
72
0.932 ± 0.011
0.897 ± 0.003
0.342 ± 0.050
742 ×
0.093 ± 0.003
Appendix
Table 2: Full method comparison at 32 × 32 ( K=1024 ), mean ± std over 5 seeds.
Domain
Method
Act. (%)
Trust ↑
Cont. ↑
MSE ↓
Seq-PPL ↓
Dist. ↓
Lorenz
VQ-EMA
100
0.999 ± 0.000
0.999 ± 0.000
0.041 ± 0.001
1014 ± 4
—
VQ-EMA + reset
100
0.999 ± 0.000
0.999 ± 0.000
0.046 ± 0.003
1017 ± 3
—
SOM-VQ
76
0.999 ± 0.000
0.999 ± 0.000
0.074 ± 0.016
829 ± 23
0.210 ± 0.005
AIST++
VQ-EMA
90
0.936 ± 0.003
0.941 ± 0.003
0.948 ± 0.535
28 ± 3
—
VQ-EMA + reset
99
0.932 ± 0.003
0.941 ± 0.004
1.179 ± 0.501
36 ± 4
—
SOM-VQ
98
0.906 ± 0.004
0.916 ± 0.004
1.149 ± 0.470
61 ± 5
0.350 ± 0.007
Appendix
Table 3: Capacity-matched ablation at 32 × 32 ( K=1024 ), mean ± std over 5 seeds. All variants share identical architecture (hidden=256). VQ-EMA + reset reinitialises dead codes each epoch; SOM-VQ uses a two-phase schedule matching the main experiments (10 epochs pure SOM pre-training, 50 epochs joint training) with no reset. SOM-VQ achieves lower Seq-PPL on both domains without dead-code reset, confirming that neighbourhood pressure provides implicit codebook regularisation. Distortion (—) is undefined for unstructured codebooks.
Grid
K
Act. (%)
Trust
Cont.
MSE
Dist.
8 × 8
64
89
0.983 ± 0.002
0.941 ± 0.004
0.047 ± 0.004
0.202 ± 0.013
16 × 16
256
81
0.996 ± 0.001
0.943 ± 0.002
0.016 ± 0.002
0.117 ± 0.004
32 × 32
1024
71
0.998 ± 0.000
0.942 ± 0.001
0.007 ± 0.001
0.062 ± 0.006
Appendix
Table 4: SOM-VQ grid size ablation on Lorenz attractor, mean ± std over 5 seeds. Quality improves monotonically; utilization decreases due to topology-aware selection.
Grid
K
Act. (%)
Trust
Cont.
MSE
Dist.
16 × 16
256
89
0.915 ± 0.007
0.876 ± 0.007
0.351 ± 0.047
0.263 ± 0.008
32 × 32
1024
61
0.937 ± 0.008
0.895 ± 0.004
0.319 ± 0.041
0.111 ± 0.005
48 × 48
2304
38
0.945 ± 0.008
0.904 ± 0.006
0.324 ± 0.043
0.057 ± 0.001
64 × 64
4096
23
0.945 ± 0.006
0.905 ± 0.006
0.321 ± 0.039
0.035 ± 0.001
Appendix
Table 5: SOM-VQ grid size ablation on AIST++, mean ± std over 5 seeds. Quality plateaus at 32 × 32; utilization drops sharply beyond, indicating dataset complexity saturation.