ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models

Yijin Wei; Yawen Zou; Wenqi Cai; Guang Li; Hangli Ge; Katsuya Hotta; Chunzhi Gu; Chao Zhang

cs.CVOct 12, 2026

ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models

Authors: Yijin Wei, Yawen Zou, Wenqi Cai, Guang Li, Hangli Ge, Katsuya Hotta, Chunzhi Gu, Chao Zhang

Organizations: Graduate School of Science and Engineering, University of Toyama, Toyama, Japan · Education and Research Center for Mathematical and Data Science, Hokkaido University, Sapporo, Japan · Interfaculty Initiative in Information Studies, The University of Tokyo, Tokyo, Japan · Faculty of Science and Engineering, Iwate University, Iwate, Japan · Faculty of Engineering, University of Fukui, Fukui, Japan · Faculty of Engineering, University of Toyama, Toyama, Japan

Abstract

Dataset distillation (DD) aims to retain the training utility of a large dataset in a much smaller set of examples. For skeleton-based action recognition, this requires capturing discriminative motion variation without disrupting the geometry and temporal continuity of articulated bodies. Directly optimizing joint coordinates can violate these constraints, while plausible motion generation alone does not ensure useful training examples. We propose Prototype-Conditioned Motion Dataset Distillation (ProMoDD), which conditions a motion diffusion model on class-wise prototypes of temporally resolved recognition features. We further tailor diffusion training to motion data, enabling the generator to model both skeletal configurations and their temporal evolution. We then construct each compact training set through budget-aware hybrid selection: representative real sequences cover densely populated regions of the pooled recognition space, while selected generated candidates complement them in less-populated regions. This couples prototype-guided generation with feature-space selection, balancing the reliability of observed motions with the variation of synthesis under a fixed per-class sequence budget. Across five difficulty-stratified subsets of the single-person classes in NTU RGB+D 60, ProMoDD achieves its largest gains under tight distillation budgets, reaching 40.32% top-1 accuracy on Full49 with three sequences per class, 4.57 percentage points above class-wise k-means. These gains reflect the complete framework. The same distilled sets train ST-GCN, MS-G3D, and CTR-GCN without regeneration.

Figures & tables

Fig. 1: Coordinate-space distribution matching (DM) breaks bone-length consistency, while ProMoDD preserves it. In the illustrated sequence, the normalized right-shin length ranges from $0.50$ to $1.39$ across 60 frames under DM, but remains stable in the real and ProMoDD sequences.

Fig. 2: Overview of ProMoDD. A frozen recognizer extracts temporally resolved features from real sequences, which are clustered into class-wise prototypes to condition the motion diffusion model. The generated candidates and real sequences are then selected in pooled feature space under a per-class budget. Real sequences represent densely populated regions, while generated candidates are selected for less-populated regions.

Fig. 3: Training and generation of the conditional motion generator. During training, the forward process adds noise to a real motion $\bm{x}_{i}$ to obtain $\bm{x}_{t}$ , while the frozen recognizer extracts a temporal condition $\bm{z}_{i}$ . Given $(\bm{x}_{t},t,y_{i},\bm{z}_{i})$ , the generator predicts the clean motion. During generation, the model starts from Gaussian noise, and the class-wise prototype $\bm{p}_{c,k}$ replaces $\bm{z}_{i}$ as the temporal condition. The temporal and global conditions guide denoising.

	Easy10	Medium20	Hard20	Mixed30	Full49
# Classes	10	20	20	30	49
# Training sequences	6,682	13,394	13,372	20,045	32,772
# Test sequences	2,751	5,497	5,490	8,237	13,459

TABLE I: Composition of our NTU RGB+D 60 evaluation subsets.

Student		Generator
Optimizer	Nesterov SGD	Optimizer	AdamW
LR	$0.05$	LR	$10^{-4}$
Batch	16	Batch	256
Training	80 epochs	Training	160k steps
Momentum	$0.9$	Layers / width	8 / 512
Weight decay	$4\times 10^{-4}$	CFG dropout	$0.1$

TABLE II: Default experimental settings.

Subset (Full data)	Method	SPC ${=}1$	SPC ${=}3$	SPC ${=}5$	SPC ${=}10$	SPC ${=}20$	SPC ${=}30$	SPC ${=}50$	SPC ${=}100$
Easy10 98.98%	Random	11.78 $\pm$ 2.62	71.44 $\pm$ 4.35	87.55 $\pm$ 1.42	92.13 $\pm$ 0.55	95.33 $\pm$ 0.49	96.29 $\pm$ 0.27	97.16 $\pm$ 0.06	98.10 $\pm$ 0.20
	$k$ -means	11.78 $\pm$ 2.62	70.79 $\pm$ 1.96	88.11 $\pm$ 1.19	93.06 $\pm$ 0.85	95.33 $\pm$ 0.42	96.21 $\pm$ 0.28	97.29 $\pm$ 0.27	98.31 $\pm$ 0.23
	DM	11.78 $\pm$ 2.62	14.09 $\pm$ 3.85	24.02 $\pm$ 7.52	45.65 $\pm$ 8.55	79.60 $\pm$ 4.07	84.85 $\pm$ 4.03	85.83 $\pm$ 3.93	83.25 $\pm$ 3.22
	ProMoDD	11.78 $\pm$ 2.62	72.72 $\pm$ 3.99	87.46 $\pm$ 0.71	92.48 $\pm$ 0.40	95.06 $\pm$ 0.31	96.36 $\pm$ 0.41	97.33 $\pm$ 0.20	98.18 $\pm$ 0.18
Medium20 87.05%	Random	9.12 $\pm$ 1.27	22.89 $\pm$ 2.20	35.97 $\pm$ 1.78	55.76 $\pm$ 2.81	69.27 $\pm$ 0.30	75.05 $\pm$ 0.67	79.24 $\pm$ 0.36	83.48 $\pm$ 0.27
	$k$ -means	12.88 $\pm$ 0.96	29.54 $\pm$ 2.33	45.47 $\pm$ 2.29	63.57 $\pm$ 1.07	72.09 $\pm$ 1.32	76.04 $\pm$ 0.70	79.56 $\pm$ 0.67	82.96 $\pm$ 0.39

TABLE III: Main results on the NTU RGB+D 60 single-person subsets under the cross-subject (CS) protocol. Entries report top-1 accuracy (%) as mean

\pm

standard deviation over five evaluation seeds. The trained generator is held fixed.

Method	Offline	Student	Storage	Top-1
Method	(GPU-h)	(GPU-h)	(GB)	(%)
Full real data	N/A	2.89	0.5320	76.06
Random	$<0.01$	0.10	0.0080	51.56 $\pm$ 2.12
$k$ -means	$<0.01$	0.10	0.0080	56.31 $\pm$ 0.72
DM	0.98	0.10	0.0080	12.49 $\pm$ 1.57
ProMoDD	4.56	0.10	0.0080	56.43 $\pm$ 1.75

TABLE IV: Construction cost, repeated student cost, storage, and accuracy on Full49 at SPC

=10

(one RTX 5090).

Architecture

Easy10

Medium20

Hard20

Mixed30

Full49

ST-GCN (3.08M)

70.59

\pm

3.61

92.38

\pm

1.35

96.50

\pm

0.38

37.51

\pm

2.76

59.72

\pm

2.17

74.84

\pm

0.19

21.23

\pm

2.18

38.49

\pm

1.15

52.82

\pm

0.85

46.30

\pm

2.21

61.92

\pm

0.81

70.83

\pm

0.71

39.69

\pm

1.64

56.60

\pm

1.01

67.72

\pm

0.53

MS-G3D (3.04M)

72.08

\pm

4.59

92.91

\pm

0.53

95.67

\pm

2.25

38.97

\pm

2.03

62.48

\pm

0.82

74.87

\pm

0.59

24.75

\pm

0.96

42.19

\pm

1.12

54.44

\pm

0.51

48.37

\pm

0.77

62.83

\pm

0.39

72.40

\pm

0.74

40.41

\pm

0.78

57.81

\pm

1.13

69.07

\pm

0.44

CTR-GCN (1.43M)

72.72

\pm

3.99

92.48

\pm

0.40

96.36

\pm

0.41

35.71

\pm

1.89

64.02

\pm

0.99

75.73

\pm

0.74

27.61

\pm

3.29

42.96

\pm

0.51

55.12

\pm

0.92

48.55

\pm

1.22

62.68

\pm

1.23

71.69

\pm

0.30

40.32

\pm

0.82

56.43

\pm

1.75

68.59

\pm

0.56

TABLE V: Cross-architecture generalization of ProMoDD on the NTU RGB+D 60 single-person evaluation subsets (top-1 accuracy, %, mean

\pm

standard deviation over five seeds). Seeds follow the protocol in Section IV-C .

Cond.	TT	CF	SPC ${=}3$	SPC ${=}10$	SPC ${=}30$
$\times$	$\times$	$\times$	33.42 $\pm$ 1.87	56.26 $\pm$ 1.20	67.81 $\pm$ 0.52
fc	$\times$	$\times$	34.08 $\pm$ 1.13	54.54 $\pm$ 1.37	67.82 $\pm$ 0.49
l8	$\times$	$\times$	37.05 $\pm$ 0.93	55.84 $\pm$ 0.92	68.10 $\pm$ 0.63
l8	$\checkmark$	$\times$	40.22 $\pm$ 1.97	55.83 $\pm$ 0.95	68.16 $\pm$ 0.84
l8	$\times$	$\checkmark$	37.22 $\pm$ 2.39	54.99 $\pm$ 1.56	68.44 $\pm$ 0.29
l8	$\checkmark$	$\checkmark$	40.32 $\pm$ 0.82	56.43 $\pm$ 1.75	68.59 $\pm$ 0.56

TABLE VI: Component ablation on Full49.

Layer-8 condition	Top-1 (%)
Time-resolved	37.67 $\pm$ 1.00
Time-pooled (fixed assignments)	36.23 $\pm$ 1.90

TABLE VII: Same-layer conditioning control on Full49 at SPC

=3

. Each variant uses a separately trained generator.

	Motion diagnostics		Student top-1 (%)
Configuration	BLV (%)	NJ (real ${=}1$ )	SPC ${=}3$	SPC ${=}10$	SPC ${=}30$
No regularization	7.55	2.925	37.58 $\pm$ 2.20	55.29 $\pm$ 0.99	67.89 $\pm$ 0.48
Temporal losses	6.74	2.059	37.54 $\pm$ 1.03	55.17 $\pm$ 1.34	68.22 $\pm$ 0.22
Stability weighting	6.70	2.446	38.15 $\pm$ 1.40	54.78 $\pm$ 1.32	67.76 $\pm$ 0.64
Full model	5.50	1.672	40.32 $\pm$ 0.82	56.43 $\pm$ 1.75	68.59 $\pm$ 0.56
Full model $+\mathcal{L}_{\mathrm{bone}}$	5.36	1.761	39.27 $\pm$ 1.22	55.77 $\pm$ 1.19	68.18 $\pm$ 0.41

TABLE VIII: Motion-regularization ablation on Full49.

Subset	Setting	SPC ${=}3$	SPC ${=}5$	SPC ${=}10$	SPC ${=}20$	SPC ${=}30$
Easy10	dense-first (ours)	72.72 $\pm$ 3.99	87.46 $\pm$ 0.71	92.48 $\pm$ 0.40	95.06 $\pm$ 0.31	96.36 $\pm$ 0.41
Easy10	sparse-first (reverse)	73.72 $\pm$ 2.96	87.70 $\pm$ 1.66	92.69 $\pm$ 0.90	95.29 $\pm$ 0.40	96.61 $\pm$ 0.28
Medium20	dense-first (ours)	35.71 $\pm$ 1.89	50.32 $\pm$ 1.31	64.02 $\pm$ 0.99	72.26 $\pm$ 0.58	75.73 $\pm$ 0.74
Medium20	sparse-first (reverse)	36.68 $\pm$ 0.69	47.58 $\pm$ 1.75	62.78 $\pm$ 1.82	70.83 $\pm$ 1.44	74.62 $\pm$ 0.75
Hard20	dense-first (ours)	27.61 $\pm$ 3.29	35.03 $\pm$ 1.53	42.96 $\pm$ 0.51	51.14 $\pm$ 0.80	55.12 $\pm$ 0.92
Hard20	sparse-first (reverse)	26.41 $\pm$ 1.49	32.30 $\pm$ 1.84	41.45 $\pm$ 1.46	50.41 $\pm$ 0.63	53.64 $\pm$ 1.61

TABLE IX: Real–synthetic assignment ablation.

Fig. 4: Shared t-SNE visualization of three action classes: (a) all real training sequences, (b) real-only selection at SPC

=10

, and (c)–(e) hybrid sets with six real and four generated sequences per class. The hybrid sets use (c) static pooled conditioning, (d) temporal conditioning, and (e) temporal conditioning with confidence filtering (CF). Small dots show real sequences, open circles mark selected real sequences, and triangles mark generated sequences. Blue, orange, and green denote shake head, brushing hair, and wipe face, respectively. All panels use the same projection of globally pooled 256-D classifier features.

Explore similar work

Sep 22, 2026

Latent Dataset Distillation for Human Motion Prediction

Dataset distillation (DD) compresses a large training set into a compact synthetic set while preserving downstream training utility. Although DD has been widely studied for images and recently extended to time-series forecasting, its application to human motion prediction remains largely unexplored. Human motion is high-dimensional and structurally coupled, and gradient matching (GM) in the original motion space optimizes many correlated variables without a prior on pose plausibility or temporal dynamics, which frequently yields implausible and unstable synthetic motions. To address this limitation, we propose a latent DD framework that regularizes distillation with a learned motion prior. Motions are first compressed by a residual-quantized variational autoencoder (RVQ-VAE), and distillation then updates only a learnable latent bank through the frozen quantizer and decoder. The pretrained decoder restricts synthetic motions to its output space, while residual quantization progressively refines the latent approximation across multiple codebooks and alleviates the representational bottleneck of single-stage vector quantization. Experiments on Human3.6M, CMU, and 3DPW with two prediction backbones show that the proposed framework outperforms direct GM in 27 of 30 evaluated settings and random subsets in every setting, and produces visibly more plausible synthetic motions in qualitative comparisons.

Ge Tian, Guang Li, Takahiro Ogawa +1

Hokkaido University

Dataset Distillation Human Motion Prediction

Ask PDF

Sep 28, 2026

DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition

Robust skeleton-based action recognition requires representations that capture a wide spectrum of motions, from subtle to moderate and strong ones. Existing methods often focus on strong motions. This paper introduces DiMoP, a masking- and diffusion-driven motion representation learning method with frame-level pseudo-classification to explicitly learn the distribution of joint motions rather than regressing deterministic coordinates, as existing methods often do. By diffusing masked joints with progressive noise and denoising them conditioned on visible joints, DiMoP learns through controllable noising and denoising processes, enabling uniform learning of weak, moderate, and strong dynamics. To enable the masking-based generative diffusion learning with a discriminative capability, a pseudo-frame classifier is proposed that enforces the learning towards sequence-consistent and temporally coherent pseudo-labels without manual annotations. Together, these strategies provide a principled mechanism for joint generative and discriminative motion modeling. DiMoP achieves state-of-the-art performance across NTU RGB+D 60/120, and PKUMMD, including a 1.1 percentage point gain over prior works on NTU RGB+D 120 with the cross-subject protocol.

Shanaka Ramesh Gunasekara, Wanqing Li, Nikalal Kaldera +2

Advanced Multimedia Research Lab, University of Wollongong, Australia

Skeleton-Based Action Recognition Masked Diffusion Models Motion Representation Learning

Ask PDF

Oct 9, 2026

Image Diffusion Models Are Good 3D Human Motion Generators

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.

Siyuan Yang, Chong Wang, Hao Cheng +4

Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3

Text-to-Motion Generation Diffusion Models for Motion Generation Motion Representation Learning Diffusion Transformer

Ask PDF