IKMo: Image-Keyframed Motion Generation with Trajectory-Pose Conditioned Motion Diffusion Model

Yang Zhao; Yan Zhang; Zixuan Guo; Hangyu Zhou; Haotian Mao; Hui Wang; Xubo Yang

cs.GRMay 27, 2025

IKMo: Image-Keyframed Motion Generation with Trajectory-Pose Conditioned Motion Diffusion Model

Authors: Yang Zhao, Yan Zhang, Zixuan Guo, Hangyu Zhou, Haotian Mao, Hui Wang, Xubo Yang

Organizations: Shanghai Jiao Tong University China

Abstract

While text-to-motion generation is mainstream, integrating explicit spatial constraints has become a standard solution for precise animation. Existing state-of-the-art models are tailored for professional workflows, effectively processing high-quality, pre-aligned 3D inputs via unified conditioning mechanisms. However, this strictly aligned setting restricts accessibility for general users who rely on unaligned, in-the-wild 2D reference images. This accessible scenario introduces a fundamental conflict: heterogeneous geometric constraints, where user-defined global trajectories clash with local image-derived poses. To resolve this coordinate mismatch, we propose TPMDM, a diffusion model featuring a decoupled parallel control strategy that explicitly isolates and aligns these divergent signals. Furthermore, we introduce IKMo, a holistic framework that synergizes TPMDM with a VLM-powered multi-agent system. By bridging the semantic-geometric gap, IKMo translates user-friendly inputs into explicit motion constraints to drive the diffusion backbone. Experiments on HumanML3D and KIT-ML demonstrate that our method establishes a new state-of-the-art in the task of generating motions under heterogeneous trajectory and keyframe pose constraints.

Figures & tables

Fig. 1: Our diffusion model jointly controls trajectories and keyframe poses, while IKMo adds a multi-agent MLLM interface for image-guided generation.

Fig. 2: ( a ) IKMo pipeline. A multi-agent MLLM extracts keyframes and trajectories from image–text input. ( b ) Diffusion model. Generates motion from noise under these constraints. ( c ) Perturbation. Iteratively refines the result. ( d ) Control. Trajectory and keyframe features are fused via ControlNet.

Method	Condition	FID $\downarrow$	R-precision (Top-3) $\uparrow$	Diversity $\rightarrow$	Foot skating ratio $\downarrow$	Traj.err. (50 cm) $\downarrow$	Loc.err. (50 cm) $\downarrow$	Avg.err. $\downarrow$
Real	-	0.002	0.797	9.503	0.000	0.000	0.000	0.000
MDM	Pelvis	0.698	0.602	9.197	0.1019	0.4022	0.3076	0.5959
PriorMDM	Pelvis	0.475	0.583	9.156	0.0897	0.3457	0.2132	0.4417
GMD	Pelvis	0.576	0.665	9.206	0.1009	0.0931	0.0321	0.1439
OmniControl	Pelvis+Keyframes	1.420	0.652	9.268	0.1039	0.4457	0.0670	0.1371
CondMDI	Pelvis+Keyframes	0.266	0.678	9.198	0.0900	0.4671	0.2093	0.1661

TABLE I: Quantitative results on the HumanML3D test set. Bold indicates the best result, underline the second-best.

\uparrow

means higher is better,

\downarrow

lower is better, and

\rightarrow

closer to real data is preferred.

Method	Condition	FID $\downarrow$	R-precision (Top-3) $\uparrow$	Diversity $\rightarrow$	Traj.err. (50 cm) $\downarrow$	Loc.err. (50 cm) $\downarrow$	Avg.err. $\downarrow$
Real	-	0.031	0.779	11.08	0.000	0.000	0.000
PriorMDM	Pelvis	0.851	0.397	10.518	0.3310	0.1400	0.2305
GMD	Pelvis	1.565	0.382	9.664	0.5443	0.3003	0.4070
OmniControl	Pelvis+Keyframes	0.690	0.405	10.597	0.1226	0.0123	0.0673
Ours	Pelvis+Keyframes	0.531	0.413	10.748	0.0341	0.0084	0.0333

TABLE II: Quantitative results on the KIT-ML test set.

Method	Condition	Pose Dist $\downarrow$
Real	-	0.000
OmniControl	Pelvis+Keyframes	0.0373
Ours(undecoupled)	Pelvis+Keyframes	0.0249
Ours	Pelvis+Keyframes	0.0210

TABLE III: Pose Dist Evaluation on the HumanML3D test set.

Fig. 3: Qualitative Results. Colored frames are keyframes and gray frames are non-keyframes, with transparency indicating temporal order. Green curves denote reference trajectories, and Origin shows source motions. Some results are rigidly aligned for clearer visualization.

Fig. 4: Qualitative results using video/text inputs. Both methods are given the same textual prompt.

Method	Traj.err. (50 cm) $\downarrow$	Loc.err. (50 cm) $\downarrow$	Avg.err. $\downarrow$	Pose Dist $\downarrow$
w/o Perturbation	0.3006	0.1982	0.3491	0.0618
w/o ControlNet	0.1383	0.0336	0.0754	0.1622
Ours	0.0246	0.0076	0.0250	0.0210

TABLE IV: Ablation study on the control modules.

Method	Eval Rotations	FID $\downarrow$	R-precision (Top-3) $\uparrow$	Diversity $\rightarrow$
Ours(w/ rotations)	w/o Rotations	0.239	0.678	9.686
Ours(w/o rotations)	w/o Rotations	0.177	0.672	9.616
Ours(w/ rotations)	w/ Rotations	0.238	0.671	9.642
Ours(w/o rotations)	w/ Rotations	0.594	0.635	9.032

TABLE V: Ablation studies on rotation. Eval Rotations indicates whether random rotations are applied during evaluation.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Image Diffusion Models Are Good 3D Human Motion Generators
Oct 9, 2026Siyuan Yang, Chong Wang, Hao Cheng +4Text-to-Motion Generation Diffusion Models for Motion Generation
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3
Kimodo: Scaling Controllable Human Motion Generation
Mar 16, 2026Davis Rempe, Mathis Petrovich, Ye Yuan +21Human Motion Generation Diffusion Models for Motion Generation
NVIDIA
Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement
Apr 20, 2026Jiakun Zheng, Ting Xiao, Shiqin Cao +3Text-to-Motion Generation Human Motion Generation
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom

Oct 9, 2026J/K move · Enter open · S save

Image Diffusion Models Are Good 3D Human Motion Generators

Siyuan Yang, Chong Wang, Hao Cheng, Jun Liu+3

Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China+3

Text-to-Motion Generation Diffusion Models for Motion Generation Motion Representation Learning Diffusion Transformer

Open paper AskPDF arXiv

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.

Abstract

Human Motion Generation

Explore similar work

Image Diffusion Models Are Good 3D Human Motion Generators

Kimodo: Scaling Controllable Human Motion Generation

Re2^22MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement