MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Space

Qing Yu; Kent Fujiwara

cs.CVSep 28, 2026

MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Space

Authors: Qing Yu, Kent Fujiwara

Organizations: LY Corporation, Tokyo, Japan

Abstract

Recent advances in diffusion and flow models have substantially improved text-driven human motion generation. Yet most methods generate in low-dimensional, temporally downsampled latent spaces learned primarily for reconstruction, a bottleneck that can limit generation quality and preclude direct manipulation of individual frames and joints. We introduce MotionSpaceFlow (MSFlow), a representation-aware flow-matching framework that predicts clean motion directly in continuous motion space without a learned encoder or decoder. To account for the anisotropic structure of direct motion representations, we propose representation-aware noise scaling and show how the initial Gaussian source scale governs the covariance of intermediate probability-path marginals. We further introduce a Representation-Aware Multimodal Diffusion Transformer (RA-MMDiT), which jointly updates token-level language and full-resolution motion features through joint attention while adapting temporal information flow to the motion representation: causal attention for incremental features defined by frame-to-frame changes, and bidirectional attention for global features such as absolute joint coordinates. Across different datasets and motion representations, MSFlow achieves state-of-the-art text-to-motion performance. Its global representation variant additionally enables zero-shot, inference-time control over any joint or frame through projection sampling without control-conditioned training, delivering leading motion quality with exact constraint satisfaction.

Figures & tables

Figure 1: Latent-space versus direct motion-space generation. Existing methods encode a $T$ -frame sequence into a shorter latent sequence, apply a diffusion transformer in the latent space, and decode the result back to motion. MSFlow removes the motion encoder and decoder and applies a flow-matching transformer directly to the full sequence. This end-to-end design avoids autoencoder reconstruction error and preserves direct access to frames and joints for zero-shot spatial control.

Figure 2: Framework of MSFlow . Left: motion-space flow matching interpolates a scaled Gaussian source $\mathbf{x}_{0}=s\bm{\epsilon}$ with a clean $T$ -frame motion $\mathbf{x}_{1}$ in either an incremental or global representation. Right: RA-MMDiT jointly processes the noised motion $\mathbf{x}_{t}$ and time-refined text tokens to predict the clean endpoint $\hat{\mathbf{x}}_{1}$ . Its attention mask follows the motion representation: a motion query attends only to its prefix and all text tokens in the causal variant, whereas the bidirectional variant uses the complete motion–text context.

Method	Representation	R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	MModality $\uparrow$	CLIP $\uparrow$
Method	Representation	Top 1	Top 2	Top 3	FID $\downarrow$	MM-Dist $\downarrow$	MModality $\uparrow$	CLIP $\uparrow$
T2M-GPT ( Zhang et al., 2023a )	Discrete latent	$.470^{\pm.003}$	$.659^{\pm.002}$	$.758^{\pm.002}$	$.335^{\pm.003}$	$3.505^{\pm.017}$	$2.018^{\pm.053}$	$.607^{\pm.005}$
MMM ( Pinyoanuntapong et al., 2024 )		$.487^{\pm.003}$	$.683^{\pm.002}$	$.782^{\pm.002}$	$.132^{\pm.004}$	$3.359^{\pm.019}$	$2.241^{\pm.073}$	$.635^{\pm.003}$
MoMask ( Guo et al., 2024 )		$.490^{\pm.004}$	$.687^{\pm.003}$	$.786^{\pm.003}$	$.116^{\pm.006}$	$3.353^{\pm.010}$	$1.263^{\pm.079}$	$.637^{\pm.003}$
MLD ( Chen et al., 2023 )	Continuous latent	$.461^{\pm.004}$	$.651^{\pm.004}$	$.750^{\pm.003}$	$.431^{\pm.014}$	$3.445^{\pm.019}$	$\underline{3.506^{\pm.031}}$	$.615^{\pm.003}$
SALAD ( Hong et al., 2025 )		$.552^{\pm.003}$	$.748^{\pm.003}$	$.839^{\pm.002}$	$.124^{\pm.005}$	$2.990^{\pm.010}$	$1.833^{\pm.081}$	$.671^{\pm.001}$

Table 1: Results of text-to-motion performance on HumanML3D. We use the 67D evaluator following MARDM. MSFlow (263D) denotes the incremental-263D model, while MSFlow (XYZ) denotes the global-XYZ variant. The average is reported over 10 runs with 95% confidence intervals. Bold indicates the best result, and underline denotes the second-best result.

Controlling Joint	Methods	Zero-shot?	FID $\downarrow$	R-Precision	Diversity $\rightarrow$	Foot Skating	Traj. err. $\downarrow$	Loc. err. $\downarrow$	Avg. err. $\downarrow$
Controlling Joint	Methods	Zero-shot?	FID $\downarrow$	Top 3	Diversity $\rightarrow$	Ratio. $\downarrow$	Traj. err. $\downarrow$	Loc. err. $\downarrow$	Avg. err. $\downarrow$
	GT	-	$0.000$	$0.795$	$10.455$	-	$0.000$	$0.000$	$0.000$
Pelvis	MDM ( Tevet et al., 2023 )	✓	$1.792$	$0.673$	$9.131$	$0.1019$	$0.4022$	$0.3076$	$0.5959$
	PriorMDM ( Shafir et al., 2024 )	✗	$0.393$	$0.707$	$9.847$	$0.0897$	$0.3457$	$0.2132$	$0.4417$
	GMD ( Karunratanakul et al., 2023 )	✓	$0.238$	$0.763$	$10.011$	$0.1009$	$0.0931$	$0.0321$	$0.1439$
	OmniControl ( Xie et al., 2024 )	✗	$0.081$	$0.789$	$10.323$	$\underline{0.0547}$	$0.0387$	$0.0096$	$0.0338$

Table 4: Results of text-conditioned motion generation with spatial controls on HumanML3D. The first block trains and evaluates on pelvis controls. The second trains on all joints and reports the average over separately controlled joints; complete per-joint results appear in Table 11 . Bold and underline denote the best and second-best values, respectively.

Figure 3: Qualitative inference-time spatial control. Colored markers indicate spatial constraints on selected joints and frames. MSFlow follows the text prompts while satisfying these constraints.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qualitative text-to-motion comparisons on HumanML3D. We compare ACMDM, SALAD, and the global-XYZ and incremental-263D variants of MSFlow across prompts requiring fine-grained action semantics, directional motion, and contact fidelity. Red annotations identify visible failure modes in the baselines; both MSFlow variants more faithfully follow the highlighted prompt details. For the clearest visualization of these dynamic behaviors, please view the accompanying supplementary videos on our project website.

Configuration					R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	CLIP $\uparrow$
Factor	Rep.	Attention	Blocks/width	Token Refiner	Top 1	Top 2	Top 3	FID $\downarrow$	MM-Dist $\downarrow$	CLIP $\uparrow$
Capacity	263D	MM	$4/256$	✓	$.555^{\pm.003}$	$.756^{\pm.003}$	$.848^{\pm.003}$	$.206^{\pm.008}$	$2.940^{\pm.011}$	$.678^{\pm.001}$
		MM	$8/512$	✓	$\mathbf{.571}^{\pm.005}$	$\mathbf{.764}^{\pm.003}$	$\mathbf{.853}^{\pm.003}$	$\mathbf{.046}^{\pm.004}$	$\mathbf{2.890}^{\pm.010}$	$\mathbf{.686}^{\pm.000}$
		MM	$12/768$	✓	$.561^{\pm.004}$	$.758^{\pm.003}$	$.847^{\pm.003}$	$.076^{\pm.006}$	$2.925^{\pm.012}$	$.681^{\pm.001}$
	XYZ	MM	$4/256$	✓	$.549^{\pm.004}$	$.750^{\pm.003}$	$.845^{\pm.003}$	$.098^{\pm.006}$	$3.000^{\pm.014}$	$.658^{\pm.001}$
		MM	$8/512$	✓	$\mathbf{.566}^{\pm.004}$	$.759^{\pm.003}$	$\mathbf{.849}^{\pm.003}$	$\mathbf{.038}^{\pm.004}$	$\mathbf{2.916}^{\pm.008}$	$.676^{\pm.001}$

Appendix

Table 5: Model-capacity and text–motion attention ablations on HumanML3D. All models use clean-motion prediction, source scale

s=5

, classifier-free guidance weight 3, and the representation-aware temporal mask (causal for 263D and bidirectional for XYZ). MM denotes RA-MMDiT blocks that update motion and text through multimodal-attention; Cross denotes conventional motion self-attention followed by text cross-attention. Token Refiner denotes the two-layer, flow-time-conditioned module applied to text tokens. Each interval summarizes 10 stochastic evaluations. Bold denotes the best point estimate for each metric within each representation and ablation group. Gray cells mark the proposed setting.

Factor	Representation	Configuration	R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$
Factor	Representation	Configuration	Top 1	Top 2	Top 3	FID $\downarrow$	MM-Dist $\downarrow$
Attention	263D	Causal	$\mathbf{.571}^{\pm.005}$	$\mathbf{.764}^{\pm.003}$	$\mathbf{.853}^{\pm.003}$	$\mathbf{.046}^{\pm.004}$	$\mathbf{2.890}^{\pm.010}$
	263D	Bidirectional	$.558^{\pm.004}$	$.756^{\pm.003}$	$.846^{\pm.002}$	$.067^{\pm.002}$	$2.973^{\pm.005}$
	XYZ	Causal	$.506^{\pm.004}$	$.701^{\pm.002}$	$.800^{\pm.002}$	$1.563^{\pm.025}$	$3.243^{\pm.008}$
	XYZ	Bidirectional	$\mathbf{.566}^{\pm.004}$	$\mathbf{.759}^{\pm.003}$	$\mathbf{.849}^{\pm.003}$	$\mathbf{.038}^{\pm.004}$	$\mathbf{2.916}^{\pm.008}$
Source scale	263D	$s=1$ , causal	$\mathbf{.580}^{\pm.004}$	$\mathbf{.775}^{\pm.003}$	$\mathbf{.861}^{\pm.002}$	$.111^{\pm.005}$	$\mathbf{2.851}^{\pm.007}$

Appendix

Table 6: Complete analysis of representation-aware design choices. All configurations use

s=5

and clean-motion prediction unless stated otherwise. Each interval summarizes 10 stochastic evaluations. Bold denotes the best result for each metric within each group, and gray cells mark the proposed settings.

Method	Representation	R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	MModality $\uparrow$
Method	Representation	Top 1	Top 2	Top 3	FID $\downarrow$	MM-Dist $\downarrow$	MModality $\uparrow$
GT	–	$.511^{\pm.003}$	$.703^{\pm.003}$	$.797^{\pm.002}$	$.002^{\pm.000}$	$2.974^{\pm.008}$	–
T2M-GPT ( Zhang et al., 2023a )	Discrete latent	$.492^{\pm.003}$	$.679^{\pm.002}$	$.775^{\pm.002}$	$.141^{\pm.005}$	$3.121^{\pm.009}$	$1.831^{\pm.048}$
MMM ( Pinyoanuntapong et al., 2024 )		$.515^{\pm.002}$	$.708^{\pm.002}$	$.804^{\pm.002}$	$.089^{\pm.005}$	$2.926^{\pm.007}$	$1.226^{\pm.035}$
MoMask ( Guo et al., 2024 )		$.521^{\pm.002}$	$.713^{\pm.002}$	$.807^{\pm.002}$	$\mathbf{.045^{\pm.002}}$	$2.958^{\pm.008}$	$1.241^{\pm.040}$
IRG-MotionLLM ( Li et al., 2026 )		$.564^{\pm.002}$	$.754^{\pm.002}$	$.841^{\pm.002}$	$.208^{\pm.003}$	$2.628^{\pm.005}$	–

Appendix

Table 7: Text-to-motion results on HumanML3D using the standard 263D evaluator. MSFlow (263D) denotes the causal incremental-263D variant, while MSFlow (XYZ) denotes the bidirectional global-XYZ variant; both use clean-motion prediction and

s=5

. All intervals are 95% confidence intervals. Bold indicates the best result, and underline denotes the second-best result.

Method	Representation	FID $\downarrow$	R-Precision $\uparrow$			MM-Dist $\downarrow$
Method	Representation	FID $\downarrow$	Top 1	Top 2	Top 3	MM-Dist $\downarrow$
Real motion	–	$.002$	$.702$	$.864$	$.914$	$15.151$
T2M-GPT ( Zhang et al., 2023a )	Discrete latent	$12.475$	$.606$	$.774$	$.838$	$16.812$
MotionGPT ( Jiang et al., 2023 )		$14.375$	$.456$	$.598$	$.628$	$17.892$
MoMask ( Guo et al., 2024 )		$12.232$	$.621$	$.784$	$.846$	$16.138$
AttT2M ( Zhong et al., 2023 )		$15.428$	$.592$	$.765$	$.834$	$15.726$

Appendix

Table 8: Text-to-motion results on HumanML3D using the MotionStreamer 272D evaluator. We include all baselines reported by MotionStreamer and UMO. UMO converts the 201D outputs of HY-Motion and its own models to 272D for evaluation. Our causal 272D model is trained from scratch with clean-motion prediction and

s=5

. Its intervals are 95% confidence intervals. Bold indicates the best generated result.

Method	Representation	R-Precision $\uparrow$			FID $\downarrow$	MModality $\uparrow$
Method	Representation	Top 1	Top 2	Top 3	FID $\downarrow$	MModality $\uparrow$
GT	–	$.940^{\pm.001}$	$.976^{\pm.001}$	$.985^{\pm.001}$	$.001^{\pm.000}$	–
T2M-GPT ( Zhang et al., 2023a )	Discrete latent	$.618^{\pm.002}$	$.773^{\pm.002}$	$.812^{\pm.002}$	$32.629^{\pm.087}$	$9.172^{\pm.181}$
MoMask ( Guo et al., 2024 )		$.777^{\pm.002}$	$.888^{\pm.002}$	$.927^{\pm.002}$	$17.404^{\pm.051}$	$8.183^{\pm.184}$
MoMask ++ ( Guo et al., 2025 )		$.802^{\pm.001}$	$.905^{\pm.002}$	$.938^{\pm.001}$	$\underline{15.061^{\pm.065}}$	$7.259^{\pm.180}$
ScaleMoGen ( Hwang et al., 2026 )		$.807^{\pm.004}$	$.908^{\pm.003}$	$.941^{\pm.003}$	$16.350^{\pm.086}$	$9.399^{\pm.669}$

Appendix

Table 9: Text-to-motion results on SnapMoGen. Our result uses a model trained from scratch on the native 296D SnapMoGen representation with clean-motion prediction and

s=10

. All intervals are 95% confidence intervals over 10 runs. Bold and underline indicate the best and second-best generated results.

Method	Params (M) $\downarrow$	FLOPs (TF) $\downarrow$	Time (s) $\downarrow$	FID $\downarrow$	R-Top3 $\uparrow$
MARDM ( Meng et al., 2025b )	309.65	75.477	9.579	0.116	0.805
SALAD ( Hong et al., 2025 )	10.06	0.467	0.477	0.076	0.857
MotionStreamer ( Xiao et al., 2025 )	318.51	1.959	9.023	0.201	0.793
CMDM ( Yu et al., 2026 )	115.01	0.572	1.809	0.068	0.860
FloodDiffusion ( Cai et al., 2026 )	131.22	7.060	3.537	0.057	0.810
MSFlow	68.77	1.502	1.742	0.057	0.863

Appendix

Table 10: Quality and compute efficiency for generating a 196-frame motion. Parameters include the complete learned generation stack and motion encoder, where applicable, but exclude frozen text encoders. FLOPs count a multiply–add as two operations. Time is the median of five runs on one NVIDIA A100-SXM4-80GB GPU. FID and R-Top3 are point estimates from the standard 263D evaluation in Table 7 . Bold and underline denote the best and second-best available results among the methods shown.

Controlling Joint	Methods	Zero-shot?	FID $\downarrow$	R-Precision	Diversity $\rightarrow$	Foot Skating	Traj. err. $\downarrow$	Loc. err. $\downarrow$	Avg. err. $\downarrow$
Controlling Joint	Methods	Zero-shot?	FID $\downarrow$	Top 3	Diversity $\rightarrow$	Ratio. $\downarrow$	Traj. err. $\downarrow$	Loc. err. $\downarrow$	Avg. err. $\downarrow$
	GT	$-$	$0.000$	$0.795$	$10.455$	-	$0.000$	$0.000$	$0.000$
Train On Pelvis	MDM ( Tevet et al., 2023 )	✓	$1.792$	$0.673$	$9.131$	$0.1019$	$0.4022$	$0.3076$	$0.5959$
	PriorMDM ( Shafir et al., 2024 )	✗	$0.393$	$0.707$	$9.847$	$0.0897$	$0.3457$	$0.2132$	$0.4417$
	GMD ( Karunratanakul et al., 2023 )	✓	$0.238$	$0.763$	$10.011$	$0.1009$	$0.0931$	$0.0321$	$0.1439$
	OmniControl ( Xie et al., 2024 )	✗	$\underline{0.081}$	$\underline{0.789}$	$\underline{10.323}$	$\mathbf{0.0547}$	$\underline{0.0387}$	$\underline{0.0096}$	$\underline{0.0338}$

Appendix

Table 11: Complete results for text-conditioned spatial control and upper-body editing on HumanML3D. The first block trains and evaluates on pelvis controls. The per-joint blocks use methods trained on all joints. The final block follows the upper-body editing format of ProjFlow ( Watanabe et al., 2026 ) . Bold and underline denote the best and second-best values, respectively; diversity is ranked by absolute deviation from GT, ties share the same formatting, and gray cells mark MSFlow .

Item	Selected value
Model variant	causal 263D, $s=5$
Training representation	HumanML3D, 263 dimensions
Evaluation representation	root/RIC subset, 67 dimensions
Backbone	8 joint blocks, width 512, 4 heads
Motion block size	1 frame
Training / inference mask	causal / causal

Appendix

Table 12: Configuration of the causal 263D model used for the primary HumanML3D result. The evaluated EMA checkpoint was not retained with an immutable run and epoch identifier.

Item	Selected value
Model variant	bidirectional XYZ, $s=5$
Training representation	shared-axis-normalized XYZ, 66 dimensions
Evaluation representation	root/RIC subset, 67 dimensions
Backbone	8 joint blocks, width 512, 4 heads
Motion block size	1 frame
Training / inference mask	bidirectional / bidirectional

Appendix

Table 13: Configuration of the bidirectional XYZ model used for the primary HumanML3D result.

Item	Control-evaluation setting
Training attention	bidirectional, fixed for the XYZ checkpoint
Inference attention	bidirectional
Control training	none
Raw controls	any valid frames, selected joint IDs, and XYZ axes
CFG weight / projected steps	3.0 / 100
Source noise	scale $s$ of the training path

Appendix

Table 14: Inference configuration for bidirectional XYZ endpoint projection. The model is trained for text-to-motion generation without masks or target coordinates.

Representation	$T_{0}$	$N$	$C_{\mathrm{off}}$	$A_{\mathrm{spec}}$	$\overline{r}_{\mathrm{eff}}$
263D	32	22,326	$.166/.007$	$1088.4/1.0$	$.009/1.000$
	64	20,850	$.148/.007$	$2116.0/1.0$	$.007/1.000$
	128	13,328	$.137/.009$	$4336.5/1.0$	$.004/1.000$
XYZ	32	22,326	$.485/.007$	$1027.0/1.0$	$.002/1.000$
	64	20,850	$.444/.007$	$1808.7/1.0$	$.001/1.000$
	128	13,328	$.390/.009$	$3346.3/1.0$	$.001/1.000$

Appendix

Table 15: Spatiotemporal covariance diagnostics across window lengths. Each statistic is reported as motion / a matched isotropic Gaussian with the same sample count

N

, dimension

p

, and covariance estimator. Lower

C_{\mathrm{off}}

, lower

A_{\mathrm{spec}}

, and higher

\overline{r}_{\mathrm{eff}}

indicate greater isotropy.

Method	Latent shape	$C_{\mathrm{off}}$	$A_{\mathrm{spec}}$	$\overline{r}_{\mathrm{eff}}$
CMDM	$16\times 64$	$.181/.007$	$118.9/1.0$	$.077/1.000$
SALAD	$16\times 7\times 32$	$.076/.007$	$129.3/1.0$	$.238/1.000$

Appendix

Table 16: Covariance diagnostics of pretrained latent representations. All methods use the same 20,850 HumanML3D training motions and one 64-frame window per motion. Latent shape excludes the batch dimension. Each diagnostic is reported as latent / a matched isotropic Gaussian with the same

N

p

, and covariance estimator.

Representation	Scale $s$	$t=0.1$	$t=0.5$	$t=0.9$	FID $\downarrow$
263D	1	$1.516$	$3.410$	$5.241$	$.111^{\pm.005}$
263D	5	$\mathbf{.356}$	$\mathbf{2.017}$	$\mathbf{3.918}$	$\mathbf{.046}^{\pm.004}$
XYZ	1	$1.253$	$3.136$	$5.023$	$.144^{\pm.009}$
XYZ	5	$\mathbf{.224}$	$\mathbf{1.746}$	$\mathbf{3.646}$	$\mathbf{.038}^{\pm.004}$

Appendix

Table 17: Source scale changes path conditioning and generation quality. The columns headed by

t

report

\log_{10}\kappa_{t}(s)

, computed at

T_{0}=64

from the extreme eigenvalues of the estimated shrinkage covariance; lower is better. FID uses the matched clean-prediction configurations from Table 3 : causal attention for 263D and bidirectional attention for XYZ.

Representation	Scale	Mask	Future	Local $\pm 4$	Span	Text	Text $\rightarrow$ motion
263D	$s=1$	Causal	$0.0$	$43.6$	$13.6$	$48.9$	$0.0$
	$s=1$	Bidirectional	$47.4$	$50.3$	$9.7$	$31.6$	$76.4$
	$s=5$	Causal	$0.0$	$46.6$	$10.4$	$27.2$	$0.0$
	$s=5$	Bidirectional	$49.1$	$38.1$	$12.7$	$17.5$	$86.4$
XYZ	$s=1$	Causal	$0.0$	$24.2$	$20.6$	$61.6$	$0.0$
	$s=1$	Bidirectional	$49.6$	$25.4$	$17.5$	$24.2$	$86.1$

Appendix

Table 18: Midpoint attention routing for all eight selected checkpoints. Representation specifies the motion input format, Scale is the Gaussian source scale

s

, and Mask is the temporal attention graph. The remaining columns report percentages at

t=0.5

, averaged over 512 paired test motions. Future is the fraction of motion-to-motion attention from query frame

i

assigned to later key frames

j>i

; Local

\pm 4

is the fraction assigned within four frames of the query; and Span is the attention-weighted expected absolute offset

|j-i|

, normalized by sequence length. Text is the fraction of a motion query’s total attention assigned to text keys, while Text

\rightarrow

motion is the fraction of a text query’s total attention assigned to motion keys.

Figure 5: Representation-dependent attention routing for all eight selected checkpoints. Rows compare 263D and XYZ; columns compare causal and bidirectional masks at source scales $s=1$ and $s=5$ . Each map averages 512 paired test motions, four heads, and eight layers at $t=0.5$ after conditioning each motion-query row on its mass assigned to valid motion keys. The causal mask removes the suffix exactly. Within the bidirectional graph, XYZ distributes attention over a broader temporal range than 263D at both source scales.

Explore similar work

MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation
Jul 31, 2026Yifei Zhu, Mingyi Shi, Yangyang Cai +3Text-To-Motion Generation Latent Flow
1Tohoku University, Sendai, Japan · 2The University of Hong Kong, Hong Kong
MotionHiFlow: Text-to-motion via hierarchical flow matching
Apr 25, 2026Heng Li, Xiaotong Lin, Ling-An Zeng +3Text-To-Motion Generation Contrastive Flow Matching
Sun Yat-sen University · Shandong University · Guangdong Province Key Laboratory of Information Security Technology, China +1
MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
Dec 15, 2025Yannan He, Garvita Tiwari, Xiaohan Zhang +4Text-To-Motion Generation Motion-Language Alignment
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2

Jul 31, 2026, cs.CVJ/K move · Enter open · S save

MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation

Yifei Zhu, Mingyi Shi, Yangyang Cai, Miao Cheng+2

1Tohoku University, Sendai, Japan · 2The University of Hong Kong, Hong Kong

Text-To-Motion Generation Latent Flow Autoencoder Architectures S-Jepa Bottlenecks

Open paper AskPDF arXiv

Text-to-motion generation must produce motions that are semantically correct, temporally coherent, and physically plausible. A natural approach is to first project motion data into a structured semantic space and then train a generative model within that space. Such a paradigm has been highly successful in image generation through Representation Autoencoders (RAEs), where a frozen self-supervised encoder provides semantic features for diffusion or flow models to learn from. However, direct transfer of such a paradigm to motion space using Motion-JEPA as the frozen encoder fails dramatically. We diagnose this failure geometrically and identify two motion-specific bottlenecks: (1) the JEPA feature space is spectrally ill-conditioned, making the Gaussian-to-data transport unstable; and (2) even with a well-conditioned spectrum, flow residuals tend to align with decoder-sensitive directions, where small latent errors are amplified into large motion artifacts after decoding. Based on these insights, we propose MoRAE. MoRAE addresses the two bottlenecks separately. A compact bottleneck distills the structured JEPA representation while removing weak and redundant directions, bringing the latent spectrum into a transport-stable regime. Motion-coupled training then aligns the retained latent geometry with the decoder, making characteristic flow errors less costly after decoding. With this flow-friendly latent, a standard non-autoregressive Flow-Matching DiT achieves state-of-the-art performance.

Abstract

Explore similar work

MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation

MotionHiFlow: Text-to-motion via hierarchical flow matching

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation