Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?

Varsha Suresh; Divij Jain; Jia Liu; M. Hamza Mughal; Vera Demberg

cs.CVSep 28, 2026

Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?

Authors: Varsha Suresh, Divij Jain, Jia Liu, M. Hamza Mughal, Vera Demberg

Organizations: Saarland University · Max Planck Institute for Informatics, Saarland Informatics Campus · IIT Delhi

Abstract

Discrete motion tokenizers encode motion as atomic units and are widely used for co-speech gesture generation. It remains unclear which motion properties, especially those relevant to gesture semantics, are recoverable from these codebooks. We probe a reconstruction-trained codebook using 19 co-speech gesture descriptors spanning from raw motion to abstract communicative function. Results show that geometry and handedness are readily decodable from token embeddings, while motion category is only weakly decoded despite showing systematic differences in discrete code usage. This gap between reconstruction quality and descriptor decodability suggests that reconstruction objectives alone do not guarantee that gesture semantics are captured, and that evaluating codebooks on such properties can guide the design of more semantic motion tokenizers.

Figures & tables

Descriptor group	N	Linear $\uparrow$	Non-Linear $\uparrow$	MI $\uparrow$	Sig.	$r$ (F1, MPJPE) $\downarrow$
		Probing F1		Code usage		Reconstruction
Motion dynamics	2	0.49 $\pm$ 0.07	0.52 $\pm$ 0.08	0.35	1/2	$-0.26$
Spatial geometry	9	0.77 $\pm$ 0.01	0.74 $\pm$ 0.03	1.07	9/9	$-0.69$
Handedness	1	0.73 $\pm$ 0.03	0.66 $\pm$ 0.05	0.77	1/1	$\mathbf{-0.89}$
Annotated spatial form	4	0.30 $\pm$ 0.01	0.31 $\pm$ 0.02	1.06	3/4	$-0.48$
Handshape	1	0.16 $\pm$ 0.03	0.20 $\pm$ 0.07	1.50	0/1	$-0.67$

Table 1: Group-level descriptor analysis. Linear and MLP probes use mean-pooled token embeddings; F1 is reported as mean

\pm

SD across descriptors in each group. MI measures information in discrete code activations, and Sig. reports the number of descriptors with significant JS segregation. More negative correlations indicate that lower MPJPE is associated with higher probe performance. ∗ This group-level MI is driven primarily by motion category (MI = 3.8,

q

= 0.006)

Group	Descriptor
Motion dynamics	Gesture size, Kinematics
Spatial geometry	Left/Right vertical, horizontal, depth,
	Left/Right elbow bend, Hand reach
Handedness	Handedness
Annotated spatial form	Hand orientation, Spatial radial,
	Spatial height, Spatial distance

Table 2: Descriptor panel used in the main analysis.

Figure 1: Probe macro-F1 for each descriptor

Figure 2: Probing macro-F1 by descriptor group, for EMAGE.

Figure 3: Qualitative comparison showing that reconstruction-only VQ-VAEs preserve upper-body geometry but average out fine-grained hand poses

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Descriptor	Annotation	$C$	Labels
Motion dynamics	Gesture size	Manual	2	normal; big
Motion dynamics	Kinematics	Manual	3	slow; normal; fast
Spatial geometry	Left vertical position	Automatic	5	below-waist; waist-level; torso-level; chest-level; above-head
Spatial geometry	Left horizontal position	Automatic	3	body-center; torso-side; extended-outward
Spatial geometry	Left depth	Automatic	2	at-torso; in-front-of-torso
Spatial geometry	Left elbow bend	Automatic	3	straight; slightly-bent; bent

Appendix

Table 3: The 19-descriptor panel used for probing. Automatic descriptors are computed and discretized from skeleton coordinates, whereas manual descriptors are human annotated.

C

denotes the number of retained classes after removing labels represented by fewer than five clips.

Group	Descriptor	Cls.	Maj-F1	Rand-F1	Embed.	One-hot
Motion dynamics	Gesture size	2	0.457	0.500	0.619	0.562
Motion dynamics	Kinematics	3	0.320	0.333	0.367	0.340
Spatial geometry	Left vertical position	5	0.113	0.200	0.639	0.563
Spatial geometry	Left horizontal position	3	0.248	0.333	0.749	0.611
Spatial geometry	Left depth	2	0.486	0.500	0.898	0.884
Spatial geometry	Left elbow bend	3	0.231	0.333	0.787	0.669

Appendix

Table 4: Descriptor-level linear probing macro-F1 with chance baselines. Cls. is the number of retained classes (

\geq 5

samples). Maj-F1 is the majority-class macro-F1 and Rand-F1 is the random-assignment macro-F1 (

=1/C

in expectation). For near-balanced binary labels such as depth and handedness, Rand-F1 exceeds Maj-F1, so the higher of the two is the relevant chance level. The best of the two probe representations is shown in bold. Spatial geometry and handedness clear chance by large margins, while motion category, communicative intent, and handshape exceed chance only marginally.

Group	Descriptor	Cls.	Stream	MI	$q$
Motion dynamics	Gesture size	2	hand	0.397	0.006
Motion dynamics	Kinematics	3	hand	0.311	0.327
Spatial geometry	Left vertical position	5	hand	1.849	0.006
Spatial geometry	Left horizontal position	3	hand	1.056	0.006
Spatial geometry	Left depth	2	hand	0.493	0.006
Spatial geometry	Left elbow bend	3	hand	1.039	0.006

Appendix

Table 5: Descriptor-level code activation results using the stream with the highest mutual information for each descriptor. Bold

q

-values indicate FDR-corrected

q<0.05

. Spatial geometry is consistently significant. Motion category is the main higher-level descriptor that shows a reliable code-level trace despite weak linear probing performance.

Co-Speech Gesture Generation

Co-speech gesture generation creates body movements that express and synchronize with spoken language.

See all 50 papers

Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures
May 28, 2026Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman +5Cross-Modal Alignment Human Motion Generation
1Saarland University · University of British Columbia · 4Vector Institute +2
SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling
Aug 25, 2026Tianlv Huang, Hetian Guo, Ziyi Cai +6Text-to-Motion Generation Human Motion Generation
Jilin University · Frontier Robotics · Harbin Institute of Technology, Shenzhen
PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation
May 8, 2026Junchuan Zhao, Qifan Liang, Ye WangRepresentation Learning Human Motion Generation
School of Computing, National University of Singapore

May 28, 2026J/K move · Enter open · S save

Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures

Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman, M. Hamza Mughal+4

1Saarland University · University of British Columbia · 4Vector Institute+2

Cross-Modal Alignment Human Motion Generation Cross-Modal Retrieval Co-Speech Gesture Generation

Open paper AskPDF arXiv

Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.

Abstract

Co-Speech Gesture Generation

Explore similar work

Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling

PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation