Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer

Chenyang Gu; Mingyuan Zhang; Haozhe Xie; Zhongang Cai; Lei Yang; Ziwei Liu

cs.CVMar 19, 2026

Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer

Authors: Chenyang Gu, Mingyuan Zhang, Haozhe Xie, Zhongang Cai, Lei Yang, Ziwei Liu

Organizations: S-Lab, Nanyang Technological University · The Chinese University of Hong Kong

Abstract

Prior motion generation largely follows two paradigms: continuous diffusion models that excel at kinematic control, and discrete token-based generators that are effective for semantic conditioning. To combine their strengths, we propose a three-stage framework comprising condition feature extraction (Perception), discrete token generation (Planning), and diffusion-based motion synthesis (Control). Central to this framework is MoTok, a diffusion-based discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction by delegating motion recovery to a diffusion decoder, enabling compact single-layer tokens while preserving motion fidelity. For kinematic conditions, coarse constraints guide token generation during planning, while fine-grained constraints are enforced during control through diffusion-based optimization. This design prevents kinematic details from disrupting semantic token planning. On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens, reducing trajectory error from 0.72 cm to 0.08 cm and FID from 0.083 to 0.029. Unlike prior methods that degrade under stronger kinematic constraints, ours improves fidelity, reducing FID from 0.033 to 0.014.

Figures & tables

Figure 1: (a) A unified Perception–Planning–Control pipeline for conditional motion generation. (b) MoTok enables compact motion tokenization with fewer tokens while maintaining competitive performance. (c) Bridging semantics and kinematics by applying coarse constraints in planning and fine constraints in motion control. MoTok maintains and improves fidelity as more joints are controlled, rather than compromising between controllability and realism. (d) Left-wrist XY trajectories with sparse control (red triangles). MoTok yields the most natural trajectory and best alignment.

Figure 2: Overview of MoTok and the unified motion generation framework. (a) MoTok factorizes motion representation into compact discrete tokens and diffusion-based reconstruction by decoding tokens into per-frame conditioning for conditional diffusion. (b) A unified conditional generation framework built on MoTok supports both discrete diffusion and autoregressive planners, integrating global and local conditions in a generator-agnostic manner.

	Method	FID $\downarrow$	R-Precision $\uparrow$ (Top-3)	Diversity $\rightarrow$	Foot Skating Ratio $\downarrow$	Traj. Err. $\downarrow$ (50cm)	Loc. Err. $\downarrow$ (50cm)	Avg. Err. $\downarrow$ (m)
	Real Motion	0.002	0.797	9.503	0.000	$\mathbf{0.0000}$	$\mathbf{0.0000}$	$\mathbf{0.0000}$
Pelvis	PriorMDM [ 34 ]	0.475	0.583	9.156	0.0897	0.3457	0.2132	0.4417
	GMD [ 17 ]	0.576	0.665	9.206	0.109	0.0931	0.0321	0.1439
	OmniControl [ 43 ]	0.218	0.687	9.422	0.0547	0.0387	0.0096	0.0338
	InterControl [ 38 ]	0.159	0.671	9.482	0.0729	$\mathbf{0.0132}$	$\mathbf{0.0004}$	0.0496
	CrowdMoGen [ 4 ]	0.132	0.784	9.109	0.0762	$\mathbf{0.0000}$	$\mathbf{0.0000}$	0.0196

Table 1: Controllable motion generation quantitative results on HumanML3D test set. Note that “Traj. Err.”, “Loc. Err.”, and “Avg. Err.” denote “Trajectory Error”, “Localization Error”, and “Average Error”, respectively.

\uparrow

(

\downarrow

) indicates that the values are better if the metric is larger (smaller). ‘

\rightarrow

’ means closer to real data is better. Red and Blue highlight the best and second-best results, respectively.

Figure 3: Visual comparison with state-of-the-art methods for any-joint any-frame control. The right panels show trajectory views of two of the controlled joints. Red indicates the control signal and blue indicates the generated motion.

	Method	R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
	Method	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
HumanML3D	Real Motion	$0.511^{\pm.003}$	$0.703^{\pm.003}$	$0.797^{\pm.002}$	$0.002^{\pm.000}$	$2.974^{\pm.008}$	$9.503^{\pm.065}$	-
	MotionDiffuse [ 46 ]	$0.491^{\pm.001}$	$0.681^{\pm.001}$	$0.782^{\pm.001}$	$0.630^{\pm.001}$	$3.113^{\pm.001}$	$\mathbf{9.410}^{\pm.049}$	$1.553^{\pm.042}$
	ReMoDiffuse [ 47 ]	$0.510^{\pm.005}$	$0.698^{\pm.006}$	$0.795^{\pm.004}$	$0.103^{\pm.004}$	$2.974^{\pm.016}$	$9.018^{\pm.075}$	$1.795^{\pm.043}$
	T2M-GPT [ 45 ]	$0.492^{\pm.003}$	$0.679^{\pm.002}$	$0.775^{\pm.002}$	$0.141^{\pm.005}$	$3.121^{\pm.009}$	$9.722^{\pm.082}$	$1.831^{\pm.048}$
	MoMask [ 9 ]	$\mathbf{0.521}^{\pm.002}$	$\mathbf{0.713}^{\pm.002}$	$\mathbf{0.807}^{\pm.002}$	$0.045^{\pm.002}$	$\mathbf{2.958}^{\pm.008}$	-	$1.241^{\pm.040}$

Table 2: Quantitative results on the HumanML3D and KIT-ML test sets. For each metric, we conduct the evaluation 20 times and report the average with a 95% confidence interval. The right arrow (

\rightarrow

) signifies that results closer to real motion are better. Red and Blue highlight the best and second-best results, respectively. Notably, MoTok-DDM operates with a token budget that is only one-sixth of that used by MoMask, while maintaining comparable or improved performance across most metrics.

#	Latent Dim	Decoder	DR	Kernel Size	Recon. FID $\downarrow$	DDM Planner			AR Planner
#	Latent Dim	Decoder	DR	Kernel Size	Recon. FID $\downarrow$	T2M FID $\downarrow$	Ctrl. FID $\downarrow$	Ctrl. Err. $\downarrow$	T2M FID $\downarrow$	Ctrl. FID $\downarrow$	Ctrl. Err. $\downarrow$
1	768	Conv	4	3	0.0704	0.0640	0.0601	0.2027	0.0935	0.0874	0.2604
2	768	DH	4	N/A	0.0396	0.0678	0.0373	0.0051	0.0686	0.0739	0.0053
3	768	DC	4	3	0.0259	0.0500	0.0379	$\mathbf{0.0047}$	0.0671	0.0469	$\mathbf{0.0049}$
4	768			5	0.0244	$\mathbf{0.0394}$	$\mathbf{0.0292}$	$\mathbf{0.0049}$	$\mathbf{0.0535}$	$\mathbf{0.0461}$	0.0051
5	768			7	0.0288	0.0458	$\mathbf{0.0279}$	$\mathbf{0.0047}$	0.0690	$\mathbf{0.0454}$	$\mathbf{0.0050}$

Table 3: Ablation study of tokenizer configurations across DDM and AR planners. Latent Dim is the codebook dimension, and Downrate denotes the temporal compression ratio (frames per latent). DiffusionHead (DH) applies independent, per-frame diffusion decoding using AdaIN-conditioned MLP blocks, while DiffusionConv (DC) augments it with residual 1D convolutions inserted between MLP blocks to model local temporal dependencies (kernel size

\in

{3, 5, 7, 9}). “DR”, “Recon.”, “Ctrl.”, and “Err.” stand for downsample rate, reconstruction, control, and error, respectively. Red and Blue highlight the best and second-best results, respectively.

Low-level Condition		DDM Planner		AR Planner
Generator	Tok. Decoder	Ctrl. FID $\downarrow$	Ctrl. Err. $\downarrow$	Ctrl. FID $\downarrow$	Ctrl. Err. $\downarrow$
✓	✓	$\mathbf{0.029}$	$\mathbf{0.0049}$	$\mathbf{0.046}$	$\mathbf{0.0051}$
✓	✗	$\mathbf{0.028}$	0.2170	$\mathbf{0.037}$	0.2770
✗	✓	0.365	$\mathbf{0.0056}$	0.447	$\mathbf{0.0063}$
✗	✗	0.039	0.6239	0.053	0.6009

Table 4: Effect of low-level control injection location. Comparison of injecting low-level control signals in the generator (token-space planning), the tokenizer decoder (diffusion decoding), evaluated with DDM and AR planners under “Pelvis” setting.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Stage-1 Decoder	Stage-2 Decoder	Reconstruction	T2M	M2T
		FID $\downarrow$	FID $\downarrow$	R@1 $\uparrow$	R@3 $\uparrow$	BLEU@1 $\uparrow$	BLEU@4 $\uparrow$
Conv (VQ)	Conv (VQ)	0.0954	0.1271	$\mathbf{0.473}$	$\mathbf{0.732}$	$\mathbf{50.1}$	$\mathbf{14.6}$
Conv (VQ)	Diffusion (MoTok)	$\mathbf{0.0544}$	0.1098
Diffusion (MoTok)	Conv (VQ)	0.1347	$\mathbf{0.0718}$	$\mathbf{0.488}$	$\mathbf{0.743}$	$\mathbf{51.3}$	$\mathbf{15.5}$
Diffusion (MoTok)	Diffusion (MoTok)	$\mathbf{0.0433}$	$\mathbf{0.0642}$

Appendix

Table 5: Two-stage training ablation of tokenizer decoders on HumanML3D. Stage-1 trains the tokenizer end-to-end. Stage-2 freezes the encoder and codebook and trains the decoder only. All variants share the same discrete tokens in Stage-2, enabling a fair comparison of reconstruction capability under identical token information.

	Method	FID $\downarrow$	R-Precision $\uparrow$ (Top-3)	Diversity $\rightarrow$	Foot Skating Ratio $\downarrow$	Traj. Err. $\downarrow$ (50cm)	Loc. Err. $\downarrow$ (50cm)	Avg. Err. $\downarrow$ (m)
	Real Motion	0.002	0.797	9.503	0.000	0.0000	0.0000	0.0000
One	OmniControl [ 43 ]	0.310	0.693	$\mathbf{9.502}$	0.0608	0.0617	0.0107	0.0404
	InterControl [ 38 ]	0.178	0.669	$\mathbf{9.498}$	0.0968	$\mathbf{0.0403}$	$\mathbf{0.0031}$	0.0741
	CrowdMoGen [ 4 ]	0.147	0.781	9.461	0.0829	$\mathbf{0.0000}$	$\mathbf{0.0000}$	$\mathbf{0.0028}$
	MaskControl [ 29 ]	0.083	$\mathbf{0.805}$	9.395	0.0545	$\mathbf{0.0000}$	$\mathbf{0.0000}$	0.0072
	MoTok-DDM-2	$\mathbf{0.025}$	0.779	9.756	0.0436	$\mathbf{0.0000}$	$\mathbf{0.0000}$	$\mathbf{0.0008}$

Appendix

Table 6: Controllable motion generation quantitative results on HumanML3D test set. Note that “Traj. Err.”, “Loc. Err.”, and “Avg. Err.” denote “Trajectory Error”, “Localization Error”, and “Average Error”, respectively.

\uparrow

(

\downarrow

) indicates that the values are better if the metric is larger (smaller). ‘

\rightarrow

’ means closer to real data is better. Red and Blue highlight the best and second-best results, respectively.

Variant	1.6	1.8	2.0	2.2	2.4	2.6	2.8	3.0	3.2	3.4	3.6
MoTok-DDM-1	0.5551	0.4139	0.3251	0.3130	0.2669	0.2093	0.2519	0.1730	0.1756	0.1769	0.1551
MoTok-DDM-2	0.2792	0.2077	0.1741	0.1373	0.1069	0.0835	0.0630	0.0519	0.0332	0.0403	0.0349
MoTok-DDM-4	0.1430	0.1023	0.0753	0.0466	0.0396	0.0394	0.0423	0.0653	0.0798	0.0984	0.1272
MoTok-DDM-8	0.1031	0.0772	0.0707	0.0887	0.0993	0.1149	0.1477	0.1607	0.1949	0.2173	0.2755
MoTok-DDM-16	0.1036	0.0875	0.0975	0.1104	0.1148	0.1282	0.1335	0.1418	0.1550	0.1604	0.1878
MoTok-AR-2	0.0857	0.0852	0.0803	0.0799	0.0783	0.0710	0.0699	0.0696	0.0617	0.0692	0.0710

Appendix

Table 7: Ablation on CFG scale. We vary the CFG scale from 1.6 to 3.6 with a step size of 0.2. Each entry reports the evaluation result (e.g., FID; lower is better).

	Method	R@1 $\uparrow$	R@3 $\uparrow$	BLEU@1 $\uparrow$	BLEU@4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$	BERTScore $\uparrow$
	Real Motion	0.523	0.828	-	-	-	-	-
Sep.	TM2T [ 11 ]	0.516	0.823	48.9	7.0	38.1	16.8	32.2
	LaMP [ 19 ]	0.547	0.831	47.8	13.0	37.1	28.9	-
	MG-MotionLLM [ 40 ]	$\mathbf{0.592}$	$\mathbf{0.866}$	-	8.1	-	-	$\mathbf{36.7}$
Uni.	MotionGPT [ 48 ]	0.543	0.827	48.2	12.5	37.4	29.2	32.4
	MotionGPT2 [ 37 ]	0.558	0.838	48.7	13.8	37.6	29.8	32.6

Appendix

Table 8: Quantitative results of Motion-to-Text (M2T) on HumanML3D. Note that “Sep.” and “Uni.” denote “Seperated Model” and “Unified Model”, respectively.

\uparrow

indicates higher is better. Red and Blue highlight the best and second-best results, respectively.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation
May 28, 2026Hequan Wang, Xuean Chen, Jiaxu Zhang +2Text-to-Motion Generation Human Motion Generation
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430072, China · Geophysical Exploration Brigade of Hubei Provincial Geological Bureau, Wuhan 430058, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore
Language-Guided Transformer Tokenizer for Human Motion Generation
Feb 9, 2026Sheng Yan, Yong Wang, Xin Du +2Text-to-Motion Generation Human Motion Generation
1Transsion Ltd. · 2Chongqing University of Technology · 3Afari Intelligence Drive +2
Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control
May 14, 2026Nhat Le, Daochang Liu, Anh Nguyen +1Text-to-Motion Generation Test-Time Optimization
The University of Western Australia, Crawley WA 6009, Australia · The University of Liverpool, Liverpool L69 3DR, United Kingdom

May 28, 2026J/K move · Enter open · S save

DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation

Hequan Wang, Xuean Chen, Jiaxu Zhang, Zhengbo Zhang+1

State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430072, China · Geophysical Exploration Brigade of Hubei Provincial Geological Bureau, Wuhan 430058, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore

Text-to-Motion Generation Human Motion Generation Generative Modeling Diffusion Models for Motion Generation Conditional Diffusion Models

Open paper AskPDF arXiv

Text-to-motion generation requires modeling both global action structure and fine-grained motion dynamics from natural language. Existing approaches typically rely on either continuous diffusion models or vector-quantized discrete representations. Diffusion models generate smooth motions but lack explicit compositional structure for temporal planning, while discrete token-based methods improve controllability but compress motion into finite codebooks, losing fine-grained dynamics. We argue that this limitation stems from a representation mismatch: action semantics such as intent, phase transitions, and temporal layout are inherently discrete and compositional, whereas joint trajectories and motion dynamics are continuous and locally correlated. To address this, we propose DC-Motion, a discrete-continuous factorized framework for human motion generation. DC-Motion decomposes motion into discrete structural tokens capturing global action layout and continuous residual latents modeling fine-grained dynamics. A text-conditioned structure generator predicts discrete tokens via iterative masked modeling, and a diffusion-based residual generator produces continuous motion conditioned on the structure. Experiments on HumanML3D and KIT-ML demonstrate that DC-Motion achieves strong performance in both FID and R-Precision, outperforming representative diffusion-based and discrete-token baselines.

Abstract

Human Motion Generation

Explore similar work

DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation

Language-Guided Transformer Tokenizer for Human Motion Generation

Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control