MotionMaestro: Masked Tokenization for Unified Motion Generation

Yun Chen; Munchurl Kim; Jeonghyeok Do

cs.CVSep 26, 2026

MotionMaestro: Masked Tokenization for Unified Motion Generation

Authors: Yun Chen, Munchurl Kim, Jeonghyeok Do

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Human motion generation plays an important role in applications such as character animation, virtual environments, and embodied interaction. While existing approaches have achieved remarkable progress, many of them are developed for individual tasks, including text-to-motion, pose-conditioned generation, and trajectory control. Although these tasks involve different types of conditions, a unified framework capable of handling them within a common representation would greatly simplify motion generation systems. We observe that diverse motion conditions can be naturally formulated as different observation patterns over motion sequences, where each task corresponds to a specific masking strategy. Based on this insight, we introduce MotionMaestro, a unified motion generation framework that learns a shared representation for complete motions and heterogeneous partial observations through masked motion tokenization. MotionMaestro employs a three-stage training strategy that first learns a masked motion tokenizer, then refines its reconstruction ability on clean motions, and finally trains a conditional flow-matching generator in the learned latent space. Furthermore, we introduce an observation map and an observation loss to explicitly preserve provided motion conditions during generation. With this unified representation and conditioning mechanism, MotionMaestro supports text-guided and unconditional synthesis, pose conditioning and partial completion, temporal interpolation, trajectory control, and motion continuation. Experiments on the large-scale RoMo and MotionMillion datasets show state-of-the-art performance across diverse motion generation tasks.

Figures & tables

Figure 2: Motion observation patterns. Temporal, spatial, and feature-group masks describe the conditions for different motion generation tasks. Orange marks observed features, while gray denotes masked regions to be generated.

Figure 3: Overview of MotionMaestro. Stage 1 trains the tokenizer’s encoder and decoder through masked reconstruction. Stage 2 refines only the decoder on clean sequences with the encoder frozen. Stage 3 freezes the entire tokenizer and trains a conditional flow-matching generator. Complete targets and masked conditions share the same encoder, while an observation map identifies available motion information. The observation loss adds motion-space supervision on observed features.

Figure 4: Task-aligned masking for motion tokenization. Anatomical tube, temporal stride, feature-group, and short-window masking expose the tokenizer to the motion conditions associated with different tasks. Pink regions denote observed inputs, while hatched regions are masked.

Method	Venue	Uncond. FID $\downarrow$	T2M FID $\downarrow$	P2M FID $\downarrow$	TP2M FID $\downarrow$	FLF FID $\downarrow$	FI MPJPE $\downarrow$	Extrap. FID $\downarrow$	Partial MPJPE $\downarrow$	Trajectory FID $\downarrow$	Long FID $\downarrow$
MDM ( Tevet et al., 2022 )	ICLR2023	507.6	450.7	518.4	456.1	523.3	117.6	342.8	278.9	–	510.9
OmniControl ( Xie et al., 2024 )	ICLR2024	429.1	199.4	771.0	312.8	753.2	875.9	740.3	627.3	283.7	398.6
MotionLab ( Guo et al., 2025 )	ICCV2025	76.7	101.5	53.5	46.5	30.0	12.7	28.1	71.8	89.3	123.5
MotionMaestro (Ours)	–	20.4	14.9	34.8	28.7	20.8	6.7	19.4	61.0	59.2	26.2

Table 1: Unified motion generation on RoMo. Each method is evaluated on its supported tasks. FI and partial completion report MPJPE (mm) over unobserved frames and joints, respectively; the remaining tasks report FID. Lower is better. Bold and underlined values mark the best and second-best results per task. Results for MotionMaestro-5B, scaled up from our default 1.3B model, are provided in the Appendix .

Method	Uncond. FID $\downarrow$	T2M FID $\downarrow$	P2M FID $\downarrow$	TP2M FID $\downarrow$	FLF FID $\downarrow$	FI MPJPE $\downarrow$	Extrap. FID $\downarrow$	Partial MPJPE $\downarrow$	Trajectory FID $\downarrow$	Long FID $\downarrow$	Inference Time (s)	Memory GB	Params Billion (B)
MDM	427.7	297.6	523.8	334.4	530.6	122.8	370.4	206.8	–	336.2	4.69	0.4	0.081
OmniControl	376.3	153.9	366.8	193.8	365.5	375.4	551.3	349.6	276.6	177.9	61.58	0.5	0.102
MotionLab	135.1	61.4	74.4	49.4	50.7	12.3	38.8	71.1	94.4	108.6	0.97	2.6	0.367
MotionMaestro	67.2	36.2	74.3	52.8	44.5	3.8	36.9	61.3	118.2	46.4	1.28	11.7	1.285

Table 2: Unified motion generation and computational cost on MotionMillion. Evaluation settings, metrics, and notation follow Table 1 . We additionally report computational cost.

Figure 5: Qualitative comparison and zero-shot motion editing. We compare MotionMaestro with MDM and MotionLab on frame interpolation, partial completion, long sequence generation, and text-guided partial editing. The partial editing example demonstrates MotionMaestro’s editing capability without editing-specific training or fine-tuning. See the Appendix for additional results.

Table 7

Method	Uncond. FID $\downarrow$	T2M FID $\downarrow$	P2M FID $\downarrow$	TP2M FID $\downarrow$	FLF FID $\downarrow$	FI MPJPE $\downarrow$	Extrap. FID $\downarrow$	Partial MPJPE $\downarrow$	Trajectory FID $\downarrow$	Long FID $\downarrow$
Tokenizer training (Stage 1)
Clean sequences only	35.5	18.3	38.3	31.0	24.2	7.8	22.1	61.8	72.9	41.2
Random masking	22.3	15.6	37.8	30.6	21.0	7.5	21.3	63.7	59.7	36.2
Generator training (Stage 3)
w/o observation map $\mathbf{O}^{c}$	18.6	15.0	36.9	29.6	20.9	6.2	19.6	61.3	57.5	30.9
w/o $\mathcal{L}_{\mathrm{obs}}$	21.8	15.2	35.9	29.8	20.1	6.8	19.9	61.1	58.9	28.7

Table 5: Ablations of tokenizer training and observation conditioning on RoMo. The upper block compares training on clean sequences only, random masking, and structured masking (MotionMaestro, bottom row). The lower block uses the same tokenizer trained with structured masking and separately removes the generator’s observation map input

\mathbf{O}^{c}

or observation loss

\mathcal{L}_{\mathrm{obs}}

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Contents
Section A	Additional ablations and discussion
Section B	Additional experiments
Section C	Data and preprocessing
Section D	Architecture and training details
Section E	Evaluation protocol and baseline adaptations

Appendix

Table 6: Overview of the Appendix .

Tokenizer	MPJPE local (mm) $\downarrow$	MPJPE world (mm) $\downarrow$	Rotation (deg) $\downarrow$	Root path (mm) $\downarrow$	Final pos. (mm) $\downarrow$	Velocity (mm) $\downarrow$
Clean-only AE	2.339	3.887	0.407	1.872	4.500	0.216
Random-mask AE	2.440	5.392	0.423	3.339	7.469	0.924
Structured-mask AE (Ours)	3.392	6.199	0.455	3.807	8.899	0.740

Appendix

Table 7: Clean-motion reconstruction after Stage 1 on RoMo. We evaluate complete-motion reconstruction from unmasked inputs before decoder refinement. Lower is better for all metrics. Bold and underlined values denote the best and second-best results in each column, respectively.

Tokenizer	MPJPE local (mm) $\downarrow$	MPJPE world (mm) $\downarrow$	Rotation (deg) $\downarrow$	Root path (mm) $\downarrow$	Final pos. (mm) $\downarrow$	Velocity (mm) $\downarrow$
Clean-only AE	1.876	2.400	0.388	0.800	1.706	0.207
Random-mask AE	1.897	3.392	0.360	1.811	4.134	0.833
Structured-mask AE (Ours)	2.148	3.127	0.394	1.284	2.822	0.642

Appendix

Table 8: Clean-motion reconstruction after Stage 2 on RoMo. We evaluate the refined tokenizers used for downstream generation on unmasked inputs. Metrics and notation follow Table 7 .

Tokenizer	Anatomical tube $\downarrow$	Temporal stride $\downarrow$	Feature-group $\downarrow$	Mixed $\downarrow$
Clean-only AE	257.8	207.4	273.1	249.4
Random-mask AE	201.7	9.3	306.7	199.8
Structured-mask AE (Ours)	46.2	7.0	158.7	43.8

Appendix

Table 9: Masked-motion reconstruction after Stage 2. All tokenizers are evaluated on the same four masking families using their refined decoders, without the latent generator. We report MPJPE (mm) over hidden regions only. Lower is better; bold and underlined values denote the best and second-best results, respectively.

	RoMo			MotionMillion
Stage	Local MPJPE (mm) $\downarrow$	Rotation error ( ∘ ) $\downarrow$	Path error (mm) $\downarrow$	Local MPJPE (mm) $\downarrow$	Rotation error ( ∘ ) $\downarrow$	Path error (mm) $\downarrow$
Stage 1	3.392	0.455	3.807	3.160	0.460	4.358
Stage 2	2.148	0.394	1.284	2.063	0.363	1.909

Appendix

Table 10: Effect of decoder refinement. MPJPE uses all valid joints and frames of clean inputs, and path error uses all valid frames. All metrics are lower-is-better; bold marks the better stage within each dataset.

Variant	FLF $\downarrow$	FI $\downarrow$	Extrap. $\downarrow$	Partial $\downarrow$	P2M $\downarrow$	TP2M $\downarrow$	Trajectory $\downarrow$
w/o observation map	5.50	4.32	2.92	3.56	6.25	5.92	8.06
w/o $\mathcal{L}_{\mathrm{obs}}$	5.78	3.78	2.98	3.74	6.40	5.94	7.93
MotionMaestro (Ours)	4.85	3.70	2.91	3.49	5.06	4.63	7.51

Appendix

Table 11: Observation preservation on RoMo. The first six columns report observed-joint MPJPE; Trajectory reports observed root-

xz

error. All values are in millimeters on 512 clips per task. All variants use the same frozen tokenizer. Bold and underlined values mark the lowest and second-lowest point estimates, not statistical significance.

Variant	FID $\downarrow$	Paired $\Delta$ vs. full model
Clean sequences only	17.910 $\pm$ 0.360	+3.335 $\pm$ 0.161
Random masking	15.928 $\pm$ 0.576	+1.352 $\pm$ 0.749
w/o observation map	14.918 $\pm$ 0.366	+0.342 $\pm$ 0.496
w/o $\mathcal{L}_{\mathrm{obs}}$	15.056 $\pm$ 0.047	+0.480 $\pm$ 0.324
MotionMaestro (Ours)	14.575 $\pm$ 0.283	–

Appendix

Table 12: RoMo T2M sampling variability. Mean and sample standard deviation over three sampling seeds. Paired

\Delta

subtracts the full-model FID at the same seed; positive values indicate higher FID.

Method	Venue	Uncond. FID $\downarrow$	T2M FID $\downarrow$	P2M FID $\downarrow$	TP2M FID $\downarrow$	FLF FID $\downarrow$	FI MPJPE $\downarrow$	Extrap. FID $\downarrow$	Partial MPJPE $\downarrow$	Trajectory FID $\downarrow$	Long FID $\downarrow$
MDM ( Tevet et al., 2022 )	ICLR2023	507.6	450.7	518.4	456.1	523.3	117.6	342.8	278.9	–	510.9
OmniControl ( Xie et al., 2024 )	ICLR2024	429.1	199.4	771.0	312.8	753.2	875.9	740.3	627.3	283.7	398.6
MotionLab ( Guo et al., 2025 )	ICCV2025	76.7	101.5	53.5	46.5	30.0	12.7	28.1	71.8	89.3	123.5
MotionMaestro (Ours)	–	20.4	14.9	34.8	28.7	20.8	6.7	19.4	61.0	59.2	26.2
MotionMaestro-5B (Ours)	–	20.3	17.2	31.3	28.6	18.4	5.8	17.9	47.2	59.0	19.0

Appendix

Table 13: Unified motion generation including MotionMaestro-5B on RoMo. FI and Partial report MPJPE (mm) over unobserved frames and joints, respectively; the remaining tasks report FID. Lower is better; bold and underlined mark the best and second-best results per task.

Method	Uncond. FID $\downarrow$	T2M FID $\downarrow$	P2M FID $\downarrow$	TP2M FID $\downarrow$	FLF FID $\downarrow$	FI MPJPE $\downarrow$	Extrap. FID $\downarrow$	Partial MPJPE $\downarrow$	Trajectory FID $\downarrow$	Long FID $\downarrow$	Inference Time (s)	Memory GB	Params Billion (B)
MDM	427.7	297.6	523.8	334.4	530.6	122.8	370.4	206.8	–	336.2	4.69	0.4	0.081
OmniControl	376.3	153.9	366.8	193.8	365.5	375.4	551.3	349.6	276.6	177.9	61.58	0.5	0.102
MotionLab	135.1	61.4	74.4	49.4	50.7	12.3	38.8	71.1	94.4	108.6	0.97	2.6	0.367
MotionMaestro	67.2	36.2	74.3	52.8	44.5	3.8	36.9	61.3	118.2	46.4	1.28	11.7	1.285
MotionMaestro-5B	67.9	34.7	51.3	42.8	32.5	3.2	24.3	45.2	109.5	35.8	1.31	30.6	5.096

Appendix

Table 14: Unified motion generation and computational cost on MotionMillion. Evaluation settings, metrics, and notation follow Table 13 . We additionally report computational cost.

Method	RoMo								MotionMillion
	$s{=}2$		$s{=}8$		$s{=}16$		$s{=}32$		$s{=}2$		$s{=}8$		$s{=}16$		$s{=}32$
	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$
MDM	59.3	61.9	252.2	282.5	375.2	403.6	431.3	464.0	75.4	128.7	203.5	301.0	285.9	401.1	330.4	454.8
OmniControl	803.2	747.8	929.2	782.7	943.3	776.1	930.6	745.8	383.9	503.6	372.0	473.5	373.9	457.7	373.6	426.0
MotionLab	11.1	10.7	18.9	14.1	35.4	21.4	57.5	30.4	11.6	9.4	15.7	11.4	26.1	16.9	47.0	27.8
MotionMaestro	4.8	3.9	12.7	4.7	30.1	12.9	54.1	18.6	2.8	1.5	7.5	1.8	20.3	5.9	43.1	15.2

Appendix

Table 15: Frame interpolation with varying observation stride on RoMo and MotionMillion. MPJPE (mm) is computed over unobserved frames; FID uses complete outputs. Lower is better; bold and underlined mark the best and second-best results per column.

Method	RoMo						MotionMillion
	torso		head $+$ hands		one arm		upper		head $+$ hands		one arm
	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$
MDM	301.4	553.8	346.0	516.8	313.5	561.4	206.8	401.9	247.9	469.3	230.6	470.0
OmniControl	535.3	477.5	627.3	631.5	297.4	293.1	349.6	448.5	349.6	448.5	252.5	287.0
MotionLab	378.1	379.3	73.8	35.3	81.2	32.7	71.1	24.3	68.6	47.7	74.7	40.5
MotionMaestro	109.3	44.7	68.5	31.8	79.5	39.0	61.3	23.5	73.7	67.0	79.7	62.2

Appendix

Table 16: Partial completion under different observed body parts on RoMo and MotionMillion. We report MPJPE (mm) over unobserved joints and FID. Observed body parts are specified separately for each dataset. Lower is better; bold and underlined mark the best and second-best results, respectively.

	RoMo						MotionMillion
Method	Foot skate ratio	GMD skate (mm/s)	Penetration (%)	Float (%)	Peak jerk	Accel. mean	Foot skate ratio	GMD skate (mm/s)	Penetration (%)	Float (%)	Peak jerk	Accel. mean
Ground Truth	0.611	76.5	0.77	25.31	561.5	4.83	0.509	63.1	0.66	25.42	225.9	2.17
MDM ( Tevet et al., 2022 )	0.961	163.5	1.59	40.17	2306.8	17.76	0.959	351.9	1.48	40.04	1675.7	11.23
OmniControl ( Xie et al., 2024 )	0.877	117.4	1.23	37.12	1023.3	7.98	0.877	130.4	0.91	30.73	622.6	5.18
MotionLab ( Guo et al., 2025 )	0.932	121.9	1.68	38.60	760.9	6.78	0.915	123.2	1.82	39.53	388.1	3.87
MotionMaestro (Ours)	0.841	82.4	0.75	26.97	571.4	6.55	0.760	69.9	0.59	25.71	289.9	3.60

Appendix

Table 17: Physical plausibility on RoMo and MotionMillion. Metrics are computed directly from joint positions without a learned evaluator. Foot skate measures the fraction of contact frames with foot movement. Penetration and float measure the percentage of frames whose lowest joint is more than 10 mm below or 15 mm above the floor, respectively. Bold and underlined mark the closest and second-closest generated results to ground truth per metric and dataset. Mean acceleration is displayed with a

10^{-3}

scale factor.

		RoMo					MotionMillion
Method	$w$	T2M FID $\downarrow$	T2M R@1 $\uparrow$	TP2M FID $\downarrow$	Long FID $\downarrow$	Long R@1 $\uparrow$	T2M FID $\downarrow$	T2M R@1 $\uparrow$	TP2M FID $\downarrow$	Long FID $\downarrow$	Long R@1 $\uparrow$
Ground truth	–	–	0.363	–	–	0.370	–	0.854	–	–	0.897
MDM	2.5	450.7	0.205	456.1	510.9	0.220	297.6	0.323	334.4	336.2	0.281
OmniControl	2.5	199.4	0.338	312.8	398.6	0.278	153.9	0.484	193.8	177.9	0.493
MotionLab	5.75	101.5	0.424	46.5	123.5	0.380	61.4	0.706	49.4	108.6	0.662
MotionMaestro	1.0 †	14.9	0.293	28.7	26.2	0.258	36.2	0.401	52.8	46.4	0.436

Appendix

Table 18: Effect of text guidance scale

w

on RoMo and MotionMillion. The null branch removes text while retaining all motion conditions.

w=1

( † ) is used for all our main comparisons; baselines retain their published scales. Bold marks each model’s lowest FID per task and dataset across the grid, including ties. These test-set minima are not used to select the main-table setting. Caption-free tasks are unaffected by

w

Setting	RoMo	MotionMillion
Training clips (train + validation)	773,240	744,906
Official test clips	40,677	131,812
Motion features per frame	272	272
Stage-3 maximum training length	384	300
Standard evaluation lengths (frames)	24–128	56–256
Long-sequence evaluation lengths (frames)	129–300	129–300

Appendix

Table 19: Data and sequence lengths. Counts refer to the cleaned splits used in our experiments, not the complete source releases. Standard evaluation lengths are the actual exported lengths after deterministic bucketing and center cropping.

Setting	Stage 1	Stage 2	Stage 3
Trainable components	Encoder + decoder	Decoder	Generator
Updates	100,000	20,000	200,000
Peak learning rate	$10^{-4}$	$2\times 10^{-5}$	$10^{-4}$
Learning-rate warmup	2,000	500	2,000
Final / peak learning rate	0.1	0.1	0.1
Global padded-frame budget	49,152	49,152	8,192

Appendix

Table 20: Three-stage optimization settings. Settings for the default MotionMaestro model are shared across datasets unless specified in the text. Frame and token budgets are global upper budgets; the realized batch size depends on sequence length.

Task	Motion observation	Text	Main metric
Uncond.	None	No	FID
T2M	None	Yes	FID
P2M	First pose	No	FID
TP2M	First pose	Yes	FID
FLF	First and last poses	No	FID
FI	Periodic poses at spacing $s=4$	No	Hidden MPJPE

Appendix

Table 21: The ten main-paper evaluation tasks. A pose includes joint positions and rotations together with root placement and heading. Text is provided only for T2M, TP2M, and Long. MPJPE is evaluated on unobserved joint positions; FID uses the whole output sequence.

Figure 6: Additional qualitative comparison (1/5). We compare MDM, MotionLab, and the 1.3B and 5B variants of MotionMaestro. Rows show frame interpolation, partial completion, long-sequence generation, and text-guided partial editing, respectively. Orange denotes supplied motion observations. The rightmost column shows ground-truth motions for the first three rows and the source motion for partial editing. The editing prompt requests placing both hands on the hips.

Figure 7: Additional qualitative comparison (2/5). Methods, row order, and color conventions follow Figure 6 . The editing prompt requests crossing both arms over the chest while retaining the supplied lower-body motion.

Figure 8: Additional qualitative comparison (3/5). Methods, row order, and color conventions follow Figure 6 . The editing prompt requests placing both hands on the hips while retaining the supplied lower-body motion.

Figure 9: Additional qualitative comparison (4/5). Methods, row order, and color conventions follow Figure 6 . The editing prompt requests crossing both arms over the chest while retaining the supplied lower-body motion.

Figure 10: Additional qualitative comparison (5/5). Methods, row order, and color conventions follow Figure 6 . The editing prompt requests raising both arms overhead while retaining the supplied lower-body motion.

Explore similar work

Unified Number-Free Text-to-Motion Generation Via Flow Matching
Mar 27, 2026Guanhe Huang, Oya CeliktutanText-to-Motion Generation Human Motion Generation
King’s College London
FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling
Apr 13, 2026Dawei Guan, Di Yang, Chengjie Jin +1Text-to-Motion Generation Human Motion Generation
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China
ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation
May 12, 2026Inwoo Hwang, Hojun Jang, Bing Zhou +3Text-to-Motion Generation Human Motion Generation
Seoul National University · Snap Inc. · Meta Reality Labs

Mar 27, 2026J/K move · Enter open · S save

Unified Number-Free Text-to-Motion Generation Via Flow Matching

Guanhe Huang, Oya Celiktutan

King’s College London

Text-to-Motion Generation Human Motion Generation Flow-Based Generative Modeling Flow Matching

Open paper AskPDF arXiv

Generative models excel at motion synthesis for a fixed number of agents but struggle to generalize with variable agents. Based on limited, domain-specific data, existing methods employ autoregressive models to generate motion recursively, which suffer from inefficiency and error accumulation. We propose Unified Motion Flow (UMF), which consists of Pyramid Motion Flow (P-Flow) and Semi-Noise Motion Flow (S-Flow). UMF decomposes the number-free motion generation into a single-pass motion prior generation stage and multi-pass reaction generation stages. Specifically, UMF utilizes a unified latent space to bridge the distribution gap between heterogeneous motion datasets, enabling effective unified training. For motion prior generation, P-Flow operates on hierarchical resolutions conditioned on different noise levels, thereby mitigating computational overheads. For reaction generation, S-Flow learns a joint probabilistic path that adaptively performs reaction transformation and context reconstruction, alleviating error accumulation. Extensive results and user studies demonstrate UMF' s effectiveness as a generalist model for multi-person motion generation from text. Project page: https://githubhgh.github.io/umf/.

Abstract

Explore similar work

Unified Number-Free Text-to-Motion Generation Via Flow Matching

FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation