Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptors

Pengfei Zhou; Xiangyue Zhang; Xukun Shen; Yong Hu

cs.CVMar 31, 2026

Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptors

Authors: Pengfei Zhou, Xiangyue Zhang, Xukun Shen, Yong Hu

Organizations: Wuhan University, Wuhan, China

Abstract

Masked generative models have become a strong paradigm for text-to-motion synthesis, but they still treat motion frames too uniformly during masking, attention, and decoding. This is a poor match for motion, where local dynamic complexity varies sharply over time. We show that current masked motion generators degrade disproportionately on dynamically complex motions, and that frame-wise generation error is strongly correlated with motion dynamics. Motivated by this mismatch, we introduce the Motion Spectral Descriptor (MSD), a simple and parameter-free measure of local dynamic complexity computed from the short-time spectrum of motion velocity. Unlike learned difficulty predictors, MSD is deterministic, interpretable, and derived directly from the motion signal itself. We use MSD to make masked motion generation complexity-aware. In particular, MSD guides content-focused masking during training, provides a spectral similarity prior for self-attention, and can additionally modulate token-level sampling during iterative decoding. Built on top of masked motion generators, our method, DynMask, improves motion generation most clearly on dynamically complex motions while also yielding stronger overall FID on HumanML3D and KIT-ML. These results suggest that respecting local motion complexity is a useful design principle for masked motion generation. Project page: https://xiangyue-zhang.github.io/DynMask

Figures & tables

Figure 1. Motivation. Left : standard masked motion generation allocates uniform modeling budget across all frames, applying the same masking, attention, and sampling regardless of local motion difficulty—this leads to degraded quality on dynamically complex motion. Right : DynMask uses the Motion Spectral Descriptor (MSD), a DCT-based per-frame complexity signal, to make masked generation complexity-aware: more supervision, stronger attention exchange, and broader decoding exploration are directed toward dynamically difficult frames, yielding better motion quality.

Figure 2. Overview of DynMask. (a) Core and full framework. Given motion tokens from a VQ-VAE and a text condition from a frozen CLIP encoder, DynMask computes one motion-grounded complexity signal, the Motion Spectral Descriptor (MSD), and reuses it throughout masked generation. In the core model , MSD guides content-focused mask selection and motion-aware attention inside the masked transformer. In the full model , the same signal is further used by complexity-aware decoding at inference time. (b) MSD computation. For each frame, we compute token-embedding velocity, apply a sliding-window Type-II DCT, and obtain both a frame-level spectral descriptor $\bm{\phi}_{t}$ and a scalar complexity summary $\Omega(t)$ . (c) Component details. Motion-aware attention blends learned attention logits with MSD-derived spectral similarity using a layer-decayed coefficient, while complexity-aware decoding assigns higher temperature and noise to dynamically harder frames.

Diffusion-based Methods
Method	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\uparrow$	MModality $\uparrow$
MotionDiffuse ( Zhang et al., 2024a )	0.491 $\pm$ .001	0.681 $\pm$ .001	0.782 $\pm$ .001	0.630 $\pm$ .001	3.113 $\pm$ .001	9.410 $\pm$ .049	1.553 $\pm$ .042
MLD ( Chen et al., 2023 )	0.481 $\pm$ .003	0.673 $\pm$ .003	0.772 $\pm$ .002	0.473 $\pm$ .013	3.196 $\pm$ .010	9.724 $\pm$ .082	2.413 $\pm$ .079
ReMoDiffuse ( Zhang et al., 2023a )	0.510 $\pm$ .003	0.698 $\pm$ .006	0.795 $\pm$ .004	0.103 $\pm$ .004	2.974 $\pm$ .016	9.018 $\pm$ .075	—
DiverseMotion ( Lou et al., 2023 )	0.496 $\pm$ .004	0.687 $\pm$ .004	0.783 $\pm$ .004	0.070 $\pm$ .004	3.063 $\pm$ .011	9.551 $\pm$ .068	—
Other Token-based Methods

Table 1. Comparison on HumanML3D. Methods are grouped by generation paradigm. Our main scope is improvement within the masked motion generation family, while other paradigms are reported for broader context. Blue background indicates the best and second-best generated results.

Diffusion-based Methods
Method	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\uparrow$	MModality $\uparrow$
MotionDiffuse ( Zhang et al., 2024a )	0.417 $\pm$ .004	0.621 $\pm$ .004	0.739 $\pm$ .004	1.954 $\pm$ .064	2.958 $\pm$ .005	11.10 $\pm$ .143	0.730 $\pm$ .013
MLD ( Chen et al., 2023 )	0.390 $\pm$ .008	0.609 $\pm$ .008	0.734 $\pm$ .007	0.404 $\pm$ .027	3.204 $\pm$ .027	10.80 $\pm$ .117	2.192 $\pm$ .071
ReMoDiffuse ( Zhang et al., 2023a )	0.427 $\pm$ .014	0.641 $\pm$ .004	0.765 $\pm$ .005	0.155 $\pm$ .006	2.814 $\pm$ .012	10.80 $\pm$ .105	—
DiverseMotion ( Lou et al., 2023 )	0.416 $\pm$ .005	0.637 $\pm$ .008	0.760 $\pm$ .011	0.468 $\pm$ .098	2.892 $\pm$ .041	10.873 $\pm$ .101	—
Other Token-based Methods

Table 2. Comparison on KIT-ML. Methods are grouped by generation paradigm. Our main scope is masked motion generation, and other paradigms are included for reference. Blue background indicates the best and second-best generated results.

Complexity Signal	FID $\downarrow$	Top-3 $\uparrow$	MM-Dist $\downarrow$
None (uniform)	.045 $\pm$ .003	.807 $\pm$ .004	2.958 $\pm$ .021
Velocity mag.	.043 $\pm$ .003	.798 $\pm$ .004	3.016 $\pm$ .020
Loss-based (HGM3)	.042 $\pm$ .002	.792 $\pm$ .004	3.049 $\pm$ .019
MSD + Loss	.031 $\pm$ .002	.800 $\pm$ .003	3.013 $\pm$ .017
MSD (ours)	.028 $\pm$ .005	.795 $\pm$ .003	2.879 $\pm$ .006

Table 3. Complexity signal comparison on HumanML3D (mean

\pm

95% CI). MSD gives the strongest FID among the tested signals.

Figure 3. MSD spectral fingerprints. Representative MSD heatmaps for different motion types show distinct temporal-frequency patterns across time.

Configuration	FID $\downarrow$	Top-3 $\uparrow$	MM-Dist $\downarrow$
DynMask (baseline)	.045 $\pm$ .003	.807 $\pm$ .004	2.958 $\pm$ .021
+ Spectral Attn (I)	.039 $\pm$ .002	.802 $\pm$ .004	2.975 $\pm$ .019
+ CFS Masking (II)	.032 $\pm$ .002	.798 $\pm$ .003	3.010 $\pm$ .018
+ DAS Sampling (III)	.028 $\pm$ .005	.795 $\pm$ .003	2.879 $\pm$ .006

Table 4. Additive ablation on HumanML3D (mean

\pm

95% CI over 20 runs). Each component provides a non-redundant gain.

Category	MoMask FID $\downarrow$	DynMask FID $\downarrow$
Static (walk, stand, wave, sit)	0.077 $\pm$ .002	0.089 $\pm$ .003
Dynamic (kick, spin, jump, dance)	0.227 $\pm$ .006	0.158 $\pm$ .004
Ratio (dynamic / static)	2.96 $\times$	1.77 $\times$

Table 5. Per-category FID on HumanML3D. Dynamic actions are substantially harder for the masked baseline, while DynMask reduces the gap. Motions are classified by keyword matching into static (3030 samples) and dynamic (1168 samples) groups.

Complexity	MoMask	DynMask	$\Delta$ FID
Simple	0.440 $\pm$ .007	0.560 $\pm$ .010	$+$ 27%
Medium	0.221 $\pm$ .006	0.186 $\pm$ .003	$-$ 16%
Complex	0.604 $\pm$ .010	0.462 $\pm$ .010	$-$ 24%

Table 6. Per-complexity FID on HumanML3D. Subgroup FID is higher than full-set FID because each group contains fewer samples.

Model	FID $\downarrow$	$\Delta$
MoMask	14.889 $\pm$ .038	—
DynMask	11.543 $\pm$ .040	$-$ 22.5%

Table 7. Cross-dataset transfer (HumanML3D

\to

BABEL, zero-shot).

Figure 4. Qualitative comparison on challenging dynamic prompts. Compared with representative masked baselines, DynMask better preserves dynamic timing, side-specific limb actions, kick extension, running gait, jump takeoff, and airborne phases. Green circles highlight locally correct dynamic details, while red circles mark failure cases in the baselines.

Figure 5. Attention visualization on a multi-phase turning-and-spinning sequence. (a) The MoMask baseline produces comparatively diffuse attention and weak phase separation. (b) DynMask yields clearer phase-structured attention aligned with the underlying motion segments. (c) The MSD spectral similarity matrix shows strong within-phase consistency, explaining why motion-aware attention connects dynamically compatible regions more effectively.

Criterion	Overall			Complex only
Criterion	DynMask	MoMask	BAMM	DynMask	MoMask	BAMM
Motion Quality	55.4	26.8	21.4	45.8	31.2	25.0
Text Faithfulness	67.9	23.2	15.2	64.6	27.1	10.4
Temporal Naturalness	53.6	25.9	25.0	45.8	29.2	25.0

Table 8. User study (% ranked 1st). DynMask is preferred across across all criteria.

Explore similar work

MotionMaestro: Masked Tokenization for Unified Motion Generation
Sep 26, 2026Yun Chen, Munchurl Kim, Jeonghyeok DoHuman Motion Generation Conditional Generative Modeling
Korea Advanced Institute of Science and Technology (KAIST)
ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
Sep 8, 2026Yiran Wang, Zeyu Zhang, Ling Shao +1Text-to-Motion Generation Human Motion Generation
University of Sydney, Camperdown 2006, Australia · School of Computer Science, Peking University, Beijing 100871, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 101408, China
CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration
May 21, 2026Adil Meric, Lin Geng Foo, Mert Kiray +3HOI Generation Image-to-Video Generation
Technical University of Munich · Max Planck Institute for Informatics, Saarland Informatics Campus · Munich Center for Machine Learning (MCML) +1

Sep 26, 2026J/K move · Enter open · S save

MotionMaestro: Masked Tokenization for Unified Motion Generation

Yun Chen, Munchurl Kim, Jeonghyeok Do

Korea Advanced Institute of Science and Technology (KAIST)

Human Motion Generation Conditional Generative Modeling Text-to-Motion Generation

Open paper AskPDF arXiv

Human motion generation plays an important role in applications such as character animation, virtual environments, and embodied interaction. While existing approaches have achieved remarkable progress, many of them are developed for individual tasks, including text-to-motion, pose-conditioned generation, and trajectory control. Although these tasks involve different types of conditions, a unified framework capable of handling them within a common representation would greatly simplify motion generation systems. We observe that diverse motion conditions can be naturally formulated as different observation patterns over motion sequences, where each task corresponds to a specific masking strategy. Based on this insight, we introduce MotionMaestro, a unified motion generation framework that learns a shared representation for complete motions and heterogeneous partial observations through masked motion tokenization. MotionMaestro employs a three-stage training strategy that first learns a masked motion tokenizer, then refines its reconstruction ability on clean motions, and finally trains a conditional flow-matching generator in the learned latent space. Furthermore, we introduce an observation map and an observation loss to explicitly preserve provided motion conditions during generation. With this unified representation and conditioning mechanism, MotionMaestro supports text-guided and unconditional synthesis, pose conditioning and partial completion, temporal interpolation, trajectory control, and motion continuation. Experiments on the large-scale RoMo and MotionMillion datasets show state-of-the-art performance across diverse motion generation tasks.

Abstract

Explore similar work

MotionMaestro: Masked Tokenization for Unified Motion Generation

ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation

CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration