Vision-based motion capture solutions often struggle with occlusions, which result in the loss of critical joint information and hinder accurate 3D motion reconstruction. Other wearable alternatives also suffer from noisy or unstable data, often requiring extensive manual cleaning and correction to achieve reliable results. To address these challenges, we introduce the Masked Motion Diffusion Model (MMDM), a diffusion-based generative reconstruction framework that enhances incomplete or low-confidence motion data using partially available high-quality reconstructions within a Masked Autoencoder architecture. Central to our design is the Kinematic Attention Aggregation (KAA) mechanism, which enables efficient, deep, and iterative encoding of both joint-level and pose-level features, capturing structural and temporal motion patterns essential for task-specific reconstruction. We focus on learning context-adaptive motion priors, specialized structural and temporal features extracted by the same reusable architecture, where each learned prior emphasizes different aspects of motion dynamics and is specifically efficient for its corresponding task. This enables the architecture to adaptively specialize without altering its structure. Such versatility allows MMDM to efficiently learn motion priors tailored to scenarios such as motion refinement, completion, and in-betweening. Extensive evaluations on public benchmarks demonstrate that MMDM achieves strong performance across diverse masking strategies and task settings. The source code is available at https://github.com/jjkislele/MMDM.
Figures & tables
Fig. 1: Architecture comparison of the proposed Masked Motion Diffusion Model (MMDM) against other methods. (a) Masked Autoencoders (MAEs) [ 4 , 29 , 30 ] reconstruct masked (low-confidence) joints from unmasked (visible/high-confidence) joints, but they are not designed for noisy input. (b) Motion diffusion models [ 27 , 31 ] denoise pose sequences to generate high-quality motions, which typically require complete input tokens. (c) Our MMDM combines both paradigms, taking partial, noisy inputs and fusing joint- and pose-level representations via the proposed Kinematic Attention Aggregation (KAA) to output complete, high-quality motions.
Method
Pub. Venue
Joint lvl.
Pose lvl.
Objective
HPE
MIB
MG
D-MAE [ 4 ]
ACMMM’22
✓
✓
StridedTran. [ 28 ]
TMM’22
✓
✓
DiffPose [ 27 ]
CVPR’23
✓
✓
MDM [ 17 ]
ICLR’23
✓
✓
✓
GMD [ 18 ]
ICCV’23
✓
✓
✓
TABLE I: Comparisons of motion representation in state-of-the-art studies, with their objectives: human pose estimation (HPE), motion in-betweening (MIB), and motion generation (MG).
Fig. 2: Illustration of the reverse diffusion process in the proposed Masked Motion Diffusion Model (MMDM). It begins at iteration k=K and proceeds sequentially to k=0 , reconstructing the masked motion sequence dkm from Gaussian noise into high‐quality data, conditioned on the unmasked motion sequence dm . Specifically, at each iteration k , (a) the Kinematic Encoder encodes the unmasked joints to latent tokens hm and yields the kinematic condition c , and (b) the Motion Decoder decodes the concatenated tokens [hm;zkm] conditioned on c . To preserve motion context, the unmasked output dkm is replaced by the input dm at every step. Positional encoding involves joint indices, frame numbers, and diffusion step indices are incorporated into the hidden features before each encoding and decoding stage.
Fig. 3: Qualitative comparisons of motion capture performance. Both 2D projections and new perspectives of 3D renderings are shown. In other methods, green and blue skeletons represent the ground truth and motion capture results, respectively. For our approach, applied in the motion completion setting, green, blue, and red skeletons differentiate the ground truth, motion capture results, and completion results. Red dotted boxes highlight the failure cases, indicating that our method yields more accurate results.
Method
Shelf
Campus
A-1
A-2
A-3
AVG
A-1
A-2
A-3
AVG
3DPS [ 32 ]
75.3
69.7
87.6
77.5
93.5
75.7
84.4
84.5
4DAG [ 1 ]
99.0
96.2
97.6
97.6
64.8
82.0
96.6
81.1
MVPose [ 3 ]
98.8
94.1
97.8
96.9
97.6
93.3
98.0
96.3
TesseTrack [ 2 ]
99.1
96.3
98.3
97.9 †
97.9
95.2
99.1
97.4
PlaneSweep [ 33 ]
99.3
96.5
98.0
97.9
98.4
93.7
99.0
97.0
TABLE II: Motion capture performance on Shelf and Campus [ 51 ] datasets using the PCP (%) metric. “A-n” corresponds to the n -th actor, while ‘AVG’ denotes the average PCP. ‘ † ’ indicates the corrected value. Bold indicates the best performance; Underlined indicates the second-best.
Fig. 4: Qualitative comparisons for motion in-betweening task. Motion sequences are sampled into key poses at a fixed ratio. Grey segments illustrate preceding and succeeding parts, while the transitioning part is color-coded from yellow to purple in a rainbow gradient, indicating the chronological order. We emphasize the joint trajectories of the pelvis, elbows, shoulders, knees, and four end-effectors. Our model generates trajectories that are closest to the ground truth, whereas other methods suffer from issues such as over-smoothing and jitter.
Dataset
Method
PCP ↑
Precision ↑
Recall ↑
MPJPE ↓
BU
4DAG [ 1 ]
82.1
95.1
94.4
70.8
D-MAE [ 4 ]
94.3
97.5
97.5
56.3
TEMPO [ 5 ]
88.7
86.4
86.4
96.5
JCSAT [ 6 ]
95.5
97.9
97.9
52.2
MMDM (Ours)
96.2 ±.52
98.1 ±.14
98.1 ±.14
52.2 ±.20
BU-X
4DAG [ 1 ]
73.1
95.1
95.1
56.5
TABLE III: Motion capture performance on the BUMocap [ 4 ] (BU) and BUMocap-X [ 6 ] (BU-X) datasets. Average PCP (%), Precision (%), Recall (%), and MPJPE (mm) are reported.
Input
Method
PCP
MPJPE
Accel
Before
After
Δ↑
Before
After
Δ↑
Before
After
Δ↑
Noise (5 cm)
SmoothNet [ 58 ]
95.4
95.6
0.2
79.3
67.4
15.0
305.8
131.3
57.0
VPoser-t [ 35 ]
97.1
1.7
70.9
10.5
165.9
45.7
HuMoR [ 35 ]
97.8
2.5
63.4
20.0
134.4
56.0
MMDM (Ours)
98.3 ±.21
3.0
60.2 ±.31
24.0
132.1 ±.28
56.8
Noise (10 cm)
SmoothNet [ 58 ]
86.4
88.2
2.0
107.5
94.2
12.3
418.4
183.8
56.0
TABLE IV: Quantitative comparisons for the motion refinement task on the Shelf dataset [ 51 ] . We report the results before and after refinement, with the incremental change Δ (%).
Method
L2-P ↓
L2-Q ↓
NPSS ↓
Interp.
0.0764
0.0497
0.5501
RMIB [ 46 ]
0.3830
0.2345
2.2503
CMIB [ 47 ]
0.0963
0.0490
0.4687
MDM [ 17 ]
0.2223 ±.17
0.1419 ±.09
1.1515 ±.10
GMD [ 18 ]
0.2499 ±.21
0.1109 ±.18
1.0820 ±.16
MDM ⋆
0.2051 ±.20
0.1361 ±.18
1.2511 ±.08
TABLE V: Motion in-betweening results on the BABEL-TEACH [ 55 ] dataset. The performance of the 30-frame transition is reported. MDM ⋆ and MDM ⋆⋆ represent two MDM variants. 10× , 50× , 100× denote results under DDIM sampling [ 42 ] .
Fig. 5: Ablation study for the motion refinement task. We report the average PCP on the Shelf dataset [ 51 ] . The line graphs depict: (a) horizontal axis for the fine-tune masking patterns, with lines showing pre-train masking patterns; (b) horizontal axis for fine-tune masking ratios, with lines showing pre-train masking ratios; (c) horizontal axis for diffusion steps.
Method
PCP ↑
MPJPE ↓
Accel ↓
FPS
SmoothNet [ 58 ]
97.4
48.7
33.3
1257.2
HuMoR [ 35 ]
98.8
43.0
35.0
31.9
MMDM (Ours)
S
98.2 ±.08
49.2 ±.11
35.6 ±.15
154.1
T
98.3 ±.11
49.0 ±.13
35.1 ±.14
148.2
S&T
98.8 ±.13
45.6 ±.18
33.6 ±.18
50.8
KAA †
98.6 ±.25
44.9 ±.19
33.1 ±.23
107.9
TABLE VI: Motion refinements results on the Shelf dataset [ 51 ] using “Mocap” data. 5× and 10× denote the speed-up ratios under DDIM sampling [ 42 ] .
Fig. 6: Computational complexity and motion in-betweening performance of each model are evaluated on the BABEL-TEACH dataset [ 55 ] using the L2P metric. The size of the circle indicates the number of parameters, with a larger area representing a greater number.
Normalization
Augmentation
Avg. PCP ↑
Hip
Centroid
Rot.
Flip
✓
94.9
✓
✓
95.7
✓
✓
96.1
✓
✓
✓
97.8
✓
96.2
TABLE I: Impact of combinations of the data preparation processes. “Rot.” denotes the random rotation augmentation. We provide the average PCP (Avg. PCP) (%) for motion completion conducted on the Shelf dataset.
Encoder
Num.
Decoder
Num.
In Dim.
3
In Dim.
512
Feat. Dim.
512
Feat. Dim.
512
Depth
9
Depth
3
Head
4
Head
4
Head Dim.
32
Head Dim.
32
FFN Dim.
512
FFN Dim.
512
TABLE II: The detailed structure of the Masked Motion Diffusion Model, for motion completion and motion refinement tasks.
Fig. 1: Demonstration for the reserve diffusion process at k time step. Green and blue skeletons denote the ground truth and the prediction, respectively. The masked joints are first sampled from a normal distribution and iteratively denoised.
Objective
Version
Avg. PCP ↑
ϵ
d0
✓
Normal
98.47
✓
Normal
98.48
✓
Lite
95.41
✓
Small
96.90
✓
Large
98.52
TABLE III: Motion completion performance on the Shelf dataset, using the average PCP metric. We evaluate the MMDM under different objective and network structure settings.
Estimator
Avg. PCP ↑
MPJPE ↓
Pub. Venue
AlphaPose [ 61 ]
98.5
44.7
TPAMI’22
SimCC [ 62 ]
98.6
44.8
ECCV’22
OpenPose [ 52 ]
98.5
45.1
TPAMI’19
TABLE IV: Motion completion performance on the Shelf dataset, using the average PCP and MPJPE metrics. We evaluate the MMDM under different 2D pose inputs.
Encoder
Num.
In Dim.
12
Feat. Dim.
512
Depth
8
Head
8
Head Dim.
64
FFN Dim.
1024
TABLE V: The detailed structure of the proposed Masked Motion Diffusion Model, for motion in-betweening.
Method
Pre-process
L2-P ↓
L2-Q ↓
NPSS ↓
Rand. Proj.
Scale
Interp.
-
-
0.0764
0.0497
0.5501
MDM [ 17 ]
✗
-
0.2236
0.1422
1.1508
MDM [ 17 ]
✓
-
0.1952
0.1264
1.1509
GMD [ 18 ]
✓
0.3615
0.1410
1.4928
GMD [ 18 ]
✓
0.2
0.2917
0.1240
1.2498
TABLE VI: Impact of GMD [ 18 ] ’s pre-processing in the motion in-betweening task. The performance of the 30-frame transition on the BABEL-TEACH dataset is reported.
We present DuoMo, a generative method that recovers human motion in world-space coordinates from unconstrained videos with noisy or incomplete observations. Reconstructing such motion requires solving a fundamental trade-off: generalizing from diverse and noisy video inputs while maintaining global motion consistency. Our approach addresses this problem by factorizing motion learning into two diffusion models. The camera-space model first estimates motion from videos in camera coordinates. The world-space model then lifts this initial estimate into world coordinates and refines it to be globally consistent. Together, the two models can reconstruct motion across diverse scenes and trajectories, even from highly noisy or incomplete observations. Moreover, our formulation is general, generating the motion of mesh vertices directly and bypassing parametric models. DuoMo achieves state-of-the-art performance. On EMDB, our method obtains a 16% reduction in world-space reconstruction error while maintaining low foot skating. On RICH, it obtains a 30% reduction in world-space error. Project page: https://yufu-wang.github.io/duomo/
Yufu Wang, Evonne Ng, Soyong Shin +8
Meta Reality Labs · University of Pennsylvania · Carnegie Mellon University
Masked generative models have become a strong paradigm for text-to-motion synthesis, but they still treat motion frames too uniformly during masking, attention, and decoding. This is a poor match for motion, where local dynamic complexity varies sharply over time. We show that current masked motion generators degrade disproportionately on dynamically complex motions, and that frame-wise generation error is strongly correlated with motion dynamics. Motivated by this mismatch, we introduce the Motion Spectral Descriptor (MSD), a simple and parameter-free measure of local dynamic complexity computed from the short-time spectrum of motion velocity. Unlike learned difficulty predictors, MSD is deterministic, interpretable, and derived directly from the motion signal itself. We use MSD to make masked motion generation complexity-aware. In particular, MSD guides content-focused masking during training, provides a spectral similarity prior for self-attention, and can additionally modulate token-level sampling during iterative decoding. Built on top of masked motion generators, our method, DynMask, improves motion generation most clearly on dynamically complex motions while also yielding stronger overall FID on HumanML3D and KIT-ML. These results suggest that respecting local motion complexity is a useful design principle for masked motion generation. Project page: https://xiangyue-zhang.github.io/DynMask
Motion capture data is rarely directly usable, as they typically exhibit missing segments, jitter, drift and contact artifacts. Traditionally, corrupted motions are cleaned by animators through the manual identification of keyframes from noisy motion, subsequent keyframe correction, and interpolation between corrected keyframes to reconstruct coherent motion. While the rise of generative motion models has made automatic cleanup feasible, most approaches operate as black box denoisers with limited controllability, making it difficult to preserve reliable segments or enforce specific user intents. Inspired by animation workflows, we present CleanMDM, a unified multimodal motion cleanup framework that formulates cleanup as masked conditional generation with plug-and-play conditions. This single model supports arbitrary combinations of noisy 3D motion, sparse 2D keyframes, sparse 3D keyframes, and text. This design enables both automatic cleanup without additional user annotation and controllable cleanup under multimodal guidance. To further improve motion realism, we incorporate the Latent Motion Quality Discriminator (LMQD) to better match kinematic distributions and reduce skating, jitter, and interpenetration artifacts, and we apply Mesh-Aware Contact Projection as a test-time optimization step to enhance contact and physical consistency. Experiments across multiple datasets demonstrate that CleanMDM consistently outperforms prior cleanup and generation baselines, and that low cost conditions (text and 2D keyframes) provide reliable controllability gains in multimodal cleanup scenarios.