Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer

Yuze Li; Dong Gong; Xiao Cao; Junchao Yuan; Dongsheng Li; Lei Zhou; Yun Sing Koh; Cheng Yan

cs.CVMar 1, 2026

Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer

Authors: Yuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, Yun Sing Koh, Cheng Yan,

Organizations: Tianjin University · University of New South Wales · University of Electronic Science and Technology of China · Hainan University ·

Abstract

Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/

Figures & tables

Figure 2: Overview of FlexiMMT. (a) Training: Given one reference video, insert trainable motion tokens into text and video tokens. Get the object mask through a simple $QK$ multiplication method, then mask out M2M, M2V and T2M parts in attention map. (b) Inference: Given a multi-object conditional image, we first segment each object’s mask with semantic segmentation model [ 35 ] . Concatenate pre-trained motion tokens into text and video tokens for inference. Extract each object’s latent-space mask in subsequent frames via Dynamic Regressive Mask Propagation Mechanism (Dynamic RMPM), and apply it to Motion parts and Text parts in attention map.

Figure 3: Overview of the simple method for training and inference. Single object can be extracted through a simple $QK$ multiplication method. However, different objects cannot be extracted through this simple method due to feature entanglement.

Figure 4: Illustration of the propagate feature and mask changes during denoising steps. We found that the transfer of motion can be completed in the early stage of denoising steps.

Method	Automatic Evaluations					Human Evaluations (%)
Method	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	MF $\uparrow$
AnyV2V [ 19 ]	0.749	0.890	0.275	0.422	0.562	0.000	0.000	0.000	0.000
Flexiact [ 53 ]	0.939	0.954	0.281	0.279	0.609	15.750	11.550	6.475	6.500
I2VEdit [ 29 ]	0.915	0.932	0.267	0.395	0.521	2.000	0.575	0.000	0.000
Go-with-the-Flow [ 4 ]	0.774	0.881	0.277	0.488	0.648	0.000	0.000	0.500	0.525
CogVideoX-5B-I2V [ 49 ]	0.948	0.956	0.293	0.292	0.607	5.500	4.000	3.475	3.500

Table 1: Quantitative comparison. We compared the effectiveness of different methods with our method. There are five metrics for Automatic Evaluations and four for Human Evaluations, among which Human Evaluations score are in percentage.

Figure 5: Qualitative comparison. We compared the effects of different methods on transferring multi-object motions. The first row is the referenced motion, and the next six rows are the results of different methods.

Method	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$
w/o M2X	0.910	0.940	0.284	0.381	0.618
w/o T2X	0.926	0.946	0.284	0.461	0.665
Our FlexiMMT	0.904	0.932	0.286	0.577	0.723

Table 2: Impact of M2X and T2X masks in MDMA. “w/o M2X” denotes removing all Motion-to-[X] masks and “w/o T2X” denotes removing all Text-to-[X] masks during inference.

Figure 6: Qualitative comparison of each component in FlexiMMT. The caption for the generated videos is: “A beer bows head. A hedgehog stands up.”

Method	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$
w/o Train	0.925	0.945	0.285	0.440	0.656
w/o Infer	0.884	0.924	0.285	0.373	0.602
w/o RMPM	0.895	0.924	0.285	0.377	0.607
Our FlexiMMT	0.904	0.932	0.286	0.577	0.723

Table 3: Impact of DMEM and RMPM. “w/o Train” and “w/o Infer” indicate disabling mask extraction during training and inference, respectively. “w/o RMPM” denotes removing the regressive mask extraction mechanism (RMPM) at inference.

Figure 7: Comparison of RMPM (w/o Dynamic) and Dynamic RMPM (w/ Dynamic). Dynamic RMPM substantially accelerates inference without compromising performance.

Figure 8: Visulization of human evaluation interface

Panel of Figure 9 — Figure 9: Detail of four indicators.

Method	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$
w/o M2V	0.907	0.938	0.285	0.383	0.616
w/o M2M	0.902	0.931	0.286	0.576	0.723
w/o T2V	0.900	0.931	0.287	0.572	0.720
w/o T2T	0.903	0.931	0.286	0.573	0.722
w/o T2M	0.928	0.948	0.284	0.476	0.663
Ours	0.904	0.932	0.286	0.577	0.723

Table 4: Impact of each mask part in MDMA.

$W$	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$	Time
1	0.901	0.931	0.288	0.575	0.718	17.39s
3	0.904	0.932	0.286	0.577	0.723	17.81s
5	0.903	0.932	0.287	0.582	0.724	18.60s

Table 5: Anchor frames

W

in RMPM. Time represents the time required for each step of denoising.

$\alpha$	AC $\uparrow$	TC $\uparrow$	TS $\uparrow$	TF $\uparrow$	FF $\uparrow$	Time
w/o	0.902	0.931	0.286	0.579	0.725	977s
5%	0.904	0.932	0.286	0.577	0.723	564s
10%	0.902	0.930	0.286	0.577	0.722	530s

Table 6: Effect of

\alpha

in Dynamic RMPM.

Figure 10: Limitation. Visualization of failure cases in our FlexiMMT. The caption for the correctly generated video is: “An elephant bows its head. A tiger jumps up while barking.” The caption for the incorrectly generated video is: “An elephant bows its head. A tiger is doing yoga.”

Figure 11: More qualitative comparisons. The left row shows reference videos. The right row shows generated videos.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 309 papers

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer
Apr 1, 2026Samuel Teodoro, Yun Chen, Agus Gunawan +3Controllable Video Generation Video Diffusion Models
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea
Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations
Aug 3, 2026Zhixue Fang, Zhimin Zhang, Bi'an Du +6Representation Learning Motion Representation Learning
Kling Team · Peking University
FlowMotion: Training-Free Flow Guidance for Video Motion Transfer
Mar 6, 2026Zhen Wang, Youcan Xu, Jun Xiao +1Controllable Video Generation Temporal Consistency in Video Generation
The Hong Kong University of Science and Technology · State key lab of CAD&CG, Zhejiang University

Apr 1, 2026J/K move · Enter open · S save

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer

Samuel Teodoro, Yun Chen, Agus Gunawan, Soo Ye Kim+2

School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea

Controllable Video Generation Video Diffusion Models Diffusion Transformer

Open paper AskPDF arXiv

Motion transfer enables controllable video generation by transferring temporal dynamics from a reference video to synthesize a new video conditioned on a target caption. However, existing Diffusion Transformer (DiT)-based methods are limited to single-object videos, restricting fine-grained control in real-world scenes with multiple objects. In this work, we introduce MotionGrounder, a DiT-based framework that firstly handles motion transfer with multi-object controllability. Our Flow-based Motion Signal (FMS) in MotionGrounder provides a stable motion prior for target video generation, while our Object-Caption Alignment Loss (OCAL) grounds object captions to their corresponding spatial regions. We further propose a new Object Grounding Score (OGS), which jointly evaluates (i) spatial alignment between source video objects and their generated counterparts and (ii) semantic consistency between each generated object and its target caption. Our experiments show that MotionGrounder consistently outperforms recent baselines across quantitative, qualitative, and human evaluations.

Abstract

Controllable Video Generation

Explore similar work

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer

Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations

FlowMotion: Training-Free Flow Guidance for Video Motion Transfer