Latent Dataset Distillation for Human Motion Prediction

Ge Tian; Guang Li; Takahiro Ogawa; Miki Haseyama

cs.CVSep 22, 2026

Latent Dataset Distillation for Human Motion Prediction

Authors: Ge Tian, Guang Li, Takahiro Ogawa, Miki Haseyama

Organizations: Hokkaido University

Abstract

Dataset distillation (DD) compresses a large training set into a compact synthetic set while preserving downstream training utility. Although DD has been widely studied for images and recently extended to time-series forecasting, its application to human motion prediction remains largely unexplored. Human motion is high-dimensional and structurally coupled, and gradient matching (GM) in the original motion space optimizes many correlated variables without a prior on pose plausibility or temporal dynamics, which frequently yields implausible and unstable synthetic motions. To address this limitation, we propose a latent DD framework that regularizes distillation with a learned motion prior. Motions are first compressed by a residual-quantized variational autoencoder (RVQ-VAE), and distillation then updates only a learnable latent bank through the frozen quantizer and decoder. The pretrained decoder restricts synthetic motions to its output space, while residual quantization progressively refines the latent approximation across multiple codebooks and alleviates the representational bottleneck of single-stage vector quantization. Experiments on Human3.6M, CMU, and 3DPW with two prediction backbones show that the proposed framework outperforms direct GM in 27 of 30 evaluated settings and random subsets in every setting, and produces visibly more plausible synthetic motions in qualitative comparisons.

Figures & tables

Figure 1: An illustration of the proposed latent dataset distillation framework. Stage I learns a compact motion representation with an RVQ-VAE. In Stage II, real segments are encoded to initialize a learnable latent bank $\mathcal{H}$ , which is the only quantity optimized while the RVQ-VAE remains frozen.

Dataset	Method	#2	#4	#8	#10	#14	Mean5
(a) Simple
H36M	Full	$7.48{\pm}0.37$	$15.14{\pm}0.67$	$31.72{\pm}1.08$	$40.72{\pm}1.19$	$58.98{\pm}1.86$	30.81
	Random	$17.72{\pm}1.22$	$34.88{\pm}3.17$	$68.34{\pm}4.71$	$82.98{\pm}6.00$	$107.94{\pm}7.66$	62.37
	GM	$45.84{\pm}10.42$	$56.12{\pm}16.20$	$92.16{\pm}18.85$	$114.08{\pm}12.39$	$119.52{\pm}14.32$	85.54
	Ours	$\bm{13.78{\pm}0.60}$	$\bm{26.50{\pm}0.89}$	$\bm{47.38{\pm}1.27}$	$\bm{57.04{\pm}1.81}$	$\bm{73.94{\pm}1.13}$	43.73
CMU	Full	$11.82{\pm}1.27$	$23.09{\pm}2.26$	$47.62{\pm}3.08$	$60.56{\pm}2.80$	$87.90{\pm}2.93$	46.20

Table 1: Comparisons at the

1\times

synthetic-data budget on Human3.6M, CMU, and 3DPW. Results are MPJPE (mm) at frames #2/#4/#8/#10/#14, given as mean

\pm

std over five runs, and Mean5 is the average of the five horizon-wise means. Best compressed-set results are in bold, and Full is trained under the same fixed budget and serves as a reference.

Figure 2: Synthetic training trajectories (not predictions) initialized from two Human3.6M sequences; the left column shows the initialization, and frame indices #0–#30 index the 50-frame trajectory rather than prediction horizons. The distilled motions need not retain the initializing action. In the examples shown, direct GM produces implausible joint configurations and abrupt transitions, whereas motions decoded from the latent bank stay within the decoder output space.

Figure 3: Ablation studies. (a) Mean5 MPJPE on Human3.6M with DLinear as the budget grows, where Full is budget-independent and trained for the same 4000 updates. (b) Mean5 and Mean8 MPJPE on CMU with Simple at the $2\times$ budget. GM+Reg is direct GM trained with the objective of Eq. ( 4 ), $K=1$ is a single-stage VQ variant, and error bars are standard deviations over five runs.

Transfer	GM	Ours
CMU S $\rightarrow$ D	$70.40{\pm}3.48$ (68.37)	$\mathbf{58.45{\pm}2.61}$ (63.32)
CMU D $\rightarrow$ S	$100.34{\pm}7.41$ (80.21)	$\mathbf{90.98{\pm}4.35}$ (67.51)
H36M S $\rightarrow$ D	$66.12{\pm}14.37$ (57.22)	$\mathbf{47.30{\pm}4.05}$ (43.82)
H36M D $\rightarrow$ S	$102.16{\pm}14.05$ (70.97)	$\mathbf{78.41{\pm}9.18}$ (43.89)

Table 2: Cross-backbone transfer at the

2\times

budget. Each cell is the Mean5 MPJPE (mm) of the target backbone trained on the set distilled with the source backbone (S: Simple, D: DLinear). Parentheses give the target backbone trained on a set distilled with the target backbone itself.

Explore similar work

Oct 12, 2026

ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models

Dataset distillation (DD) aims to retain the training utility of a large dataset in a much smaller set of examples. For skeleton-based action recognition, this requires capturing discriminative motion variation without disrupting the geometry and temporal continuity of articulated bodies. Directly optimizing joint coordinates can violate these constraints, while plausible motion generation alone does not ensure useful training examples. We propose Prototype-Conditioned Motion Dataset Distillation (ProMoDD), which conditions a motion diffusion model on class-wise prototypes of temporally resolved recognition features. We further tailor diffusion training to motion data, enabling the generator to model both skeletal configurations and their temporal evolution. We then construct each compact training set through budget-aware hybrid selection: representative real sequences cover densely populated regions of the pooled recognition space, while selected generated candidates complement them in less-populated regions. This couples prototype-guided generation with feature-space selection, balancing the reliability of observed motions with the variation of synthesis under a fixed per-class sequence budget. Across five difficulty-stratified subsets of the single-person classes in NTU RGB+D 60, ProMoDD achieves its largest gains under tight distillation budgets, reaching 40.32% top-1 accuracy on Full49 with three sequences per class, 4.57 percentage points above class-wise k-means. These gains reflect the complete framework. The same distilled sets train ST-GCN, MS-G3D, and CTR-GCN without regeneration.

Yijin Wei, Yawen Zou, Wenqi Cai +5

Graduate School of Science and Engineering, University of Toyama, Toyama, Japan · Education and Research Center for Mathematical and Data Science, Hokkaido University, Sapporo, Japan · Interfaculty Initiative in Information Studies, The University of Tokyo, Tokyo, Japan +3

Dataset Distillation Skeleton-Based Action Recognition Diffusion Models for Motion Generation

Ask PDF

Aug 21, 2026

Gaussian-Mixture Latent Flow for Stochastic 3D Human Motion Prediction

Stochastic human motion prediction aims to forecast future motion distributions. Although recent studies have achieved strong performance in terms of accuracy and diversity, they often overlook plausibility (e.g., resulting in physically unrealistic predictions) and uncertainty quantification, both of which are essential for real-world applications and downstream tasks. To address these issues, we propose a latent flow-based model equipped with a data-driven Gaussian mixture prior that more effectively disentangles diverse human behaviors than conventional single-modal priors. This prior is derived from patterns in the training data without requiring additional annotations. Furthermore, the fully invertible nature of our model enables natural uncertainty quantification through tractable likelihood computation. Experiments on the Human3.6M and AMASS datasets demonstrate that our approach achieves state-of-the-art performance in both accuracy and plausibility.

Yue Ma, Frederick W. B. Li, Xiaohui Liang

Beihang University · Durham University · Zhongguancun Laboratory

Human Motion Prediction Flow-Based Generative Modeling Uncertainty Quantification

Ask PDF

Jun 13, 2026

MotionVLA: Vision-Language-Action Model for Humanoid Motion

Generating realistic humanoid motion from scene images and text involves both low-frequency pose semantics and high-frequency physical dynamics. However, many existing methods tokenize motion with a single shared codebook, forcing heterogeneous motion signals into the same quantization space. Our frequency-domain analysis of human motion data reveals a clear mismatch between single-codebook quantization and motion statistics: five DCT coefficients capture 93% of joint-position energy but only 37% of joint-velocity energy, which can bias quantization toward pose statistics and under-represent high-frequency velocity components. A second challenge lies in adapting a standard autoregressive model to effectively model high-frequency physical signals in motion sequences. Therefore, we propose DSFT, a dual-stream frequency tokenizer that separates motion into Base and physical streams and compresses them independently with DCT truncation and BPE. Furthermore, we present MotionVLA, a Qwen3.5-based model that arranges Base and physical tokens in a unified sequence, where Phys tokens are predicted after Base tokens. Experiments on HumanML3D and MBench show that, despite using a lightweight 2B backbone, MotionVLA reduces the Diversity gap to real data by over 50% on HumanML3D and improves Motion-Condition Consistency by 3.8% on MBench, supporting frequency-aware dual-stream decoupling as an effective formulation for autoregressive motion generation. Code: https://github.com/AIGeeksGroup/MotionVLA. Website: https://aigeeksgroup.github.io/MotionVLA.

Nonghai Zhang, Siyu Zhai, Yanjun Li +5

School of Computer Science, Peking University · AI2 Robotics

Text-to-Motion Generation Human Motion Generation Vision-Language-Action Models Motion Representation Learning

Ask PDF