ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models

Yijin Wei; Yawen Zou; Wenqi Cai; Guang Li; Hangli Ge; Katsuya Hotta; Chunzhi Gu; Chao Zhang

cs.CVOct 12, 2026

ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models

Authors: Yijin Wei, Yawen Zou, Wenqi Cai, Guang Li, Hangli Ge, Katsuya Hotta, Chunzhi Gu, Chao Zhang

Organizations: Graduate School of Science and Engineering, University of Toyama, Toyama, Japan · Education and Research Center for Mathematical and Data Science, Hokkaido University, Sapporo, Japan · Interfaculty Initiative in Information Studies, The University of Tokyo, Tokyo, Japan · Faculty of Science and Engineering, Iwate University, Iwate, Japan · Faculty of Engineering, University of Fukui, Fukui, Japan · Faculty of Engineering, University of Toyama, Toyama, Japan

Abstract

Dataset distillation (DD) aims to retain the training utility of a large dataset in a much smaller set of examples. For skeleton-based action recognition, this requires capturing discriminative motion variation without disrupting the geometry and temporal continuity of articulated bodies. Directly optimizing joint coordinates can violate these constraints, while plausible motion generation alone does not ensure useful training examples. We propose Prototype-Conditioned Motion Dataset Distillation (ProMoDD), which conditions a motion diffusion model on class-wise prototypes of temporally resolved recognition features. We further tailor diffusion training to motion data, enabling the generator to model both skeletal configurations and their temporal evolution. We then construct each compact training set through budget-aware hybrid selection: representative real sequences cover densely populated regions of the pooled recognition space, while selected generated candidates complement them in less-populated regions. This couples prototype-guided generation with feature-space selection, balancing the reliability of observed motions with the variation of synthesis under a fixed per-class sequence budget. Across five difficulty-stratified subsets of the single-person classes in NTU RGB+D 60, ProMoDD achieves its largest gains under tight distillation budgets, reaching 40.32% top-1 accuracy on Full49 with three sequences per class, 4.57 percentage points above class-wise k-means. These gains reflect the complete framework. The same distilled sets train ST-GCN, MS-G3D, and CTR-GCN without regeneration.

Figures & tables

Fig. 1: Coordinate-space distribution matching (DM) breaks bone-length consistency, while ProMoDD preserves it. In the illustrated sequence, the normalized right-shin length ranges from $0.50$ to $1.39$ across 60 frames under DM, but remains stable in the real and ProMoDD sequences.

Fig. 2: Overview of ProMoDD. A frozen recognizer extracts temporally resolved features from real sequences, which are clustered into class-wise prototypes to condition the motion diffusion model. The generated candidates and real sequences are then selected in pooled feature space under a per-class budget. Real sequences represent densely populated regions, while generated candidates are selected for less-populated regions.

Fig. 3: Training and generation of the conditional motion generator. During training, the forward process adds noise to a real motion $\bm{x}_{i}$ to obtain $\bm{x}_{t}$ , while the frozen recognizer extracts a temporal condition $\bm{z}_{i}$ . Given $(\bm{x}_{t},t,y_{i},\bm{z}_{i})$ , the generator predicts the clean motion. During generation, the model starts from Gaussian noise, and the class-wise prototype $\bm{p}_{c,k}$ replaces $\bm{z}_{i}$ as the temporal condition. The temporal and global conditions guide denoising.

	Easy10	Medium20	Hard20	Mixed30	Full49
# Classes	10	20	20	30	49
# Training sequences	6,682	13,394	13,372	20,045	32,772
# Test sequences	2,751	5,497	5,490	8,237	13,459

TABLE I: Composition of our NTU RGB+D 60 evaluation subsets.

Student		Generator
Optimizer	Nesterov SGD	Optimizer	AdamW
LR	$0.05$	LR	$10^{-4}$
Batch	16	Batch	256
Training	80 epochs	Training	160k steps
Momentum	$0.9$	Layers / width	8 / 512
Weight decay	$4\times 10^{-4}$	CFG dropout	$0.1$

TABLE II: Default experimental settings.

Subset (Full data)	Method	SPC ${=}1$	SPC ${=}3$	SPC ${=}5$	SPC ${=}10$	SPC ${=}20$	SPC ${=}30$	SPC ${=}50$	SPC ${=}100$
Easy10 98.98%	Random	11.78 $\pm$ 2.62	71.44 $\pm$ 4.35	87.55 $\pm$ 1.42	92.13 $\pm$ 0.55	95.33 $\pm$ 0.49	96.29 $\pm$ 0.27	97.16 $\pm$ 0.06	98.10 $\pm$ 0.20
	$k$ -means	11.78 $\pm$ 2.62	70.79 $\pm$ 1.96	88.11 $\pm$ 1.19	93.06 $\pm$ 0.85	95.33 $\pm$ 0.42	96.21 $\pm$ 0.28	97.29 $\pm$ 0.27	98.31 $\pm$ 0.23
	DM	11.78 $\pm$ 2.62	14.09 $\pm$ 3.85	24.02 $\pm$ 7.52	45.65 $\pm$ 8.55	79.60 $\pm$ 4.07	84.85 $\pm$ 4.03	85.83 $\pm$ 3.93	83.25 $\pm$ 3.22
	ProMoDD	11.78 $\pm$ 2.62	72.72 $\pm$ 3.99	87.46 $\pm$ 0.71	92.48 $\pm$ 0.40	95.06 $\pm$ 0.31	96.36 $\pm$ 0.41	97.33 $\pm$ 0.20	98.18 $\pm$ 0.18
Medium20 87.05%	Random	9.12 $\pm$ 1.27	22.89 $\pm$ 2.20	35.97 $\pm$ 1.78	55.76 $\pm$ 2.81	69.27 $\pm$ 0.30	75.05 $\pm$ 0.67	79.24 $\pm$ 0.36	83.48 $\pm$ 0.27
	$k$ -means	12.88 $\pm$ 0.96	29.54 $\pm$ 2.33	45.47 $\pm$ 2.29	63.57 $\pm$ 1.07	72.09 $\pm$ 1.32	76.04 $\pm$ 0.70	79.56 $\pm$ 0.67	82.96 $\pm$ 0.39

TABLE III: Main results on the NTU RGB+D 60 single-person subsets under the cross-subject (CS) protocol. Entries report top-1 accuracy (%) as mean

\pm

standard deviation over five evaluation seeds. The trained generator is held fixed.

Method	Offline	Student	Storage	Top-1
Method	(GPU-h)	(GPU-h)	(GB)	(%)
Full real data	N/A	2.89	0.5320	76.06
Random	$<0.01$	0.10	0.0080	51.56 $\pm$ 2.12
$k$ -means	$<0.01$	0.10	0.0080	56.31 $\pm$ 0.72
DM	0.98	0.10	0.0080	12.49 $\pm$ 1.57
ProMoDD	4.56	0.10	0.0080	56.43 $\pm$ 1.75

TABLE IV: Construction cost, repeated student cost, storage, and accuracy on Full49 at SPC

=10

(one RTX 5090).

Architecture

Easy10

Medium20

Hard20

Mixed30

Full49

ST-GCN (3.08M)

70.59

\pm

3.61

92.38

\pm

1.35

96.50

\pm

0.38

37.51

\pm

2.76

59.72

\pm

2.17

74.84

\pm

0.19

21.23

\pm

2.18

38.49

\pm

1.15

52.82

\pm

0.85

46.30

\pm

2.21

61.92

\pm

0.81

70.83

\pm

0.71

39.69

\pm

1.64

56.60

\pm

1.01

67.72

\pm

0.53

MS-G3D (3.04M)

72.08

\pm

4.59

92.91

\pm

0.53

95.67

\pm

2.25

38.97

\pm

2.03

62.48

\pm

0.82

74.87

\pm

0.59

24.75

\pm

0.96

42.19

\pm

1.12

54.44

\pm

0.51

48.37

\pm

0.77

62.83

\pm

0.39

72.40

\pm

0.74

40.41

\pm

0.78

57.81

\pm

1.13

69.07

\pm

0.44

CTR-GCN (1.43M)

72.72

\pm

3.99

92.48

\pm

0.40

96.36

\pm

0.41

35.71

\pm

1.89

64.02

\pm

0.99

75.73

\pm

0.74

27.61

\pm

3.29

42.96

\pm

0.51

55.12

\pm

0.92

48.55

\pm

1.22

62.68

\pm

1.23

71.69

\pm

0.30

40.32

\pm

0.82

56.43

\pm

1.75

68.59

\pm

0.56

TABLE V: Cross-architecture generalization of ProMoDD on the NTU RGB+D 60 single-person evaluation subsets (top-1 accuracy, %, mean

\pm

standard deviation over five seeds). Seeds follow the protocol in Section IV-C .

Cond.	TT	CF	SPC ${=}3$	SPC ${=}10$	SPC ${=}30$
$\times$	$\times$	$\times$	33.42 $\pm$ 1.87	56.26 $\pm$ 1.20	67.81 $\pm$ 0.52
fc	$\times$	$\times$	34.08 $\pm$ 1.13	54.54 $\pm$ 1.37	67.82 $\pm$ 0.49
l8	$\times$	$\times$	37.05 $\pm$ 0.93	55.84 $\pm$ 0.92	68.10 $\pm$ 0.63
l8	$\checkmark$	$\times$	40.22 $\pm$ 1.97	55.83 $\pm$ 0.95	68.16 $\pm$ 0.84
l8	$\times$	$\checkmark$	37.22 $\pm$ 2.39	54.99 $\pm$ 1.56	68.44 $\pm$ 0.29
l8	$\checkmark$	$\checkmark$	40.32 $\pm$ 0.82	56.43 $\pm$ 1.75	68.59 $\pm$ 0.56

TABLE VI: Component ablation on Full49.

Layer-8 condition	Top-1 (%)
Time-resolved	37.67 $\pm$ 1.00
Time-pooled (fixed assignments)	36.23 $\pm$ 1.90

TABLE VII: Same-layer conditioning control on Full49 at SPC

=3

. Each variant uses a separately trained generator.

	Motion diagnostics		Student top-1 (%)
Configuration	BLV (%)	NJ (real ${=}1$ )	SPC ${=}3$	SPC ${=}10$	SPC ${=}30$
No regularization	7.55	2.925	37.58 $\pm$ 2.20	55.29 $\pm$ 0.99	67.89 $\pm$ 0.48
Temporal losses	6.74	2.059	37.54 $\pm$ 1.03	55.17 $\pm$ 1.34	68.22 $\pm$ 0.22
Stability weighting	6.70	2.446	38.15 $\pm$ 1.40	54.78 $\pm$ 1.32	67.76 $\pm$ 0.64
Full model	5.50	1.672	40.32 $\pm$ 0.82	56.43 $\pm$ 1.75	68.59 $\pm$ 0.56
Full model $+\mathcal{L}_{\mathrm{bone}}$	5.36	1.761	39.27 $\pm$ 1.22	55.77 $\pm$ 1.19	68.18 $\pm$ 0.41

TABLE VIII: Motion-regularization ablation on Full49.

Subset	Setting	SPC ${=}3$	SPC ${=}5$	SPC ${=}10$	SPC ${=}20$	SPC ${=}30$
Easy10	dense-first (ours)	72.72 $\pm$ 3.99	87.46 $\pm$ 0.71	92.48 $\pm$ 0.40	95.06 $\pm$ 0.31	96.36 $\pm$ 0.41
Easy10	sparse-first (reverse)	73.72 $\pm$ 2.96	87.70 $\pm$ 1.66	92.69 $\pm$ 0.90	95.29 $\pm$ 0.40	96.61 $\pm$ 0.28
Medium20	dense-first (ours)	35.71 $\pm$ 1.89	50.32 $\pm$ 1.31	64.02 $\pm$ 0.99	72.26 $\pm$ 0.58	75.73 $\pm$ 0.74
Medium20	sparse-first (reverse)	36.68 $\pm$ 0.69	47.58 $\pm$ 1.75	62.78 $\pm$ 1.82	70.83 $\pm$ 1.44	74.62 $\pm$ 0.75
Hard20	dense-first (ours)	27.61 $\pm$ 3.29	35.03 $\pm$ 1.53	42.96 $\pm$ 0.51	51.14 $\pm$ 0.80	55.12 $\pm$ 0.92
Hard20	sparse-first (reverse)	26.41 $\pm$ 1.49	32.30 $\pm$ 1.84	41.45 $\pm$ 1.46	50.41 $\pm$ 0.63	53.64 $\pm$ 1.61

TABLE IX: Real–synthetic assignment ablation.

Fig. 4: Shared t-SNE visualization of three action classes: (a) all real training sequences, (b) real-only selection at SPC

=10

, and (c)–(e) hybrid sets with six real and four generated sequences per class. The hybrid sets use (c) static pooled conditioning, (d) temporal conditioning, and (e) temporal conditioning with confidence filtering (CF). Small dots show real sequences, open circles mark selected real sequences, and triangles mark generated sequences. Blue, orange, and green denote shake head, brushing hair, and wipe face, respectively. All panels use the same projection of globally pooled 256-D classifier features.

Explore similar work

Latent Dataset Distillation for Human Motion Prediction
Sep 22, 2026Ge Tian, Guang Li, Takahiro Ogawa +1Dataset Distillation Human Motion Prediction
Hokkaido University
DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition
Sep 28, 2026Shanaka Ramesh Gunasekara, Wanqing Li, Nikalal Kaldera +2Skeleton-Based Action Recognition Masked Diffusion Models
Advanced Multimedia Research Lab, University of Wollongong, Australia
Image Diffusion Models Are Good 3D Human Motion Generators
Oct 9, 2026Siyuan Yang, Chong Wang, Hao Cheng +4Text-to-Motion Generation Diffusion Models for Motion Generation
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3

Sep 22, 2026J/K move · Enter open · S save

Latent Dataset Distillation for Human Motion Prediction

Ge Tian, Guang Li, Takahiro Ogawa, Miki Haseyama

Hokkaido University

Dataset Distillation Human Motion Prediction

Open paper AskPDF arXiv

Dataset distillation (DD) compresses a large training set into a compact synthetic set while preserving downstream training utility. Although DD has been widely studied for images and recently extended to time-series forecasting, its application to human motion prediction remains largely unexplored. Human motion is high-dimensional and structurally coupled, and gradient matching (GM) in the original motion space optimizes many correlated variables without a prior on pose plausibility or temporal dynamics, which frequently yields implausible and unstable synthetic motions. To address this limitation, we propose a latent DD framework that regularizes distillation with a learned motion prior. Motions are first compressed by a residual-quantized variational autoencoder (RVQ-VAE), and distillation then updates only a learnable latent bank through the frozen quantizer and decoder. The pretrained decoder restricts synthetic motions to its output space, while residual quantization progressively refines the latent approximation across multiple codebooks and alleviates the representational bottleneck of single-stage vector quantization. Experiments on Human3.6M, CMU, and 3DPW with two prediction backbones show that the proposed framework outperforms direct GM in 27 of 30 evaluated settings and random subsets in every setting, and produces visibly more plausible synthetic motions in qualitative comparisons.

Abstract

Explore similar work

Latent Dataset Distillation for Human Motion Prediction

DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition

Image Diffusion Models Are Good 3D Human Motion Generators