Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer
Authors: Yuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, Yun Sing Koh, Cheng Yan, Xinyu Zhang
Organizations: Tianjin University · University of New South Wales · University of Electronic Science and Technology of China · Hainan University · University of Auckland
Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/
Figures & tables
Figure 2: Overview of FlexiMMT. (a) Training: Given one reference video, insert trainable motion tokens into text and video tokens. Get the object mask through a simple QK multiplication method, then mask out M2M, M2V and T2M parts in attention map. (b) Inference: Given a multi-object conditional image, we first segment each object’s mask with semantic segmentation model [ 35 ] . Concatenate pre-trained motion tokens into text and video tokens for inference. Extract each object’s latent-space mask in subsequent frames via Dynamic Regressive Mask Propagation Mechanism (Dynamic RMPM), and apply it to Motion parts and Text parts in attention map.
Figure 3: Overview of the simple method for training and inference. Single object can be extracted through a simple QK multiplication method. However, different objects cannot be extracted through this simple method due to feature entanglement.
Figure 4: Illustration of the propagate feature and mask changes during denoising steps. We found that the transfer of motion can be completed in the early stage of denoising steps.
Method
Automatic Evaluations
Human Evaluations (%)
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
AC ↑
TC ↑
TS ↑
MF ↑
AnyV2V [ 19 ]
0.749
0.890
0.275
0.422
0.562
0.000
0.000
0.000
0.000
Flexiact [ 53 ]
0.939
0.954
0.281
0.279
0.609
15.750
11.550
6.475
6.500
I2VEdit [ 29 ]
0.915
0.932
0.267
0.395
0.521
2.000
0.575
0.000
0.000
Go-with-the-Flow [ 4 ]
0.774
0.881
0.277
0.488
0.648
0.000
0.000
0.500
0.525
CogVideoX-5B-I2V [ 49 ]
0.948
0.956
0.293
0.292
0.607
5.500
4.000
3.475
3.500
Table 1: Quantitative comparison. We compared the effectiveness of different methods with our method. There are five metrics for Automatic Evaluations and four for Human Evaluations, among which Human Evaluations score are in percentage.
Figure 5: Qualitative comparison. We compared the effects of different methods on transferring multi-object motions. The first row is the referenced motion, and the next six rows are the results of different methods.
Method
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
w/o M2X
0.910
0.940
0.284
0.381
0.618
w/o T2X
0.926
0.946
0.284
0.461
0.665
Our FlexiMMT
0.904
0.932
0.286
0.577
0.723
Table 2: Impact of M2X and T2X masks in MDMA. “w/o M2X” denotes removing all Motion-to-[X] masks and “w/o T2X” denotes removing all Text-to-[X] masks during inference.
Figure 6: Qualitative comparison of each component in FlexiMMT. The caption for the generated videos is: “A beer bows head. A hedgehog stands up.”
Method
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
w/o Train
0.925
0.945
0.285
0.440
0.656
w/o Infer
0.884
0.924
0.285
0.373
0.602
w/o RMPM
0.895
0.924
0.285
0.377
0.607
Our FlexiMMT
0.904
0.932
0.286
0.577
0.723
Table 3: Impact of DMEM and RMPM. “w/o Train” and “w/o Infer” indicate disabling mask extraction during training and inference, respectively. “w/o RMPM” denotes removing the regressive mask extraction mechanism (RMPM) at inference.
Figure 7: Comparison of RMPM (w/o Dynamic) and Dynamic RMPM (w/ Dynamic). Dynamic RMPM substantially accelerates inference without compromising performance.
Figure 8: Visulization of human evaluation interface
Figure 9: Detail of four indicators.
Method
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
w/o M2V
0.907
0.938
0.285
0.383
0.616
w/o M2M
0.902
0.931
0.286
0.576
0.723
w/o T2V
0.900
0.931
0.287
0.572
0.720
w/o T2T
0.903
0.931
0.286
0.573
0.722
w/o T2M
0.928
0.948
0.284
0.476
0.663
Ours
0.904
0.932
0.286
0.577
0.723
Table 4: Impact of each mask part in MDMA.
W
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
Time
1
0.901
0.931
0.288
0.575
0.718
17.39s
3
0.904
0.932
0.286
0.577
0.723
17.81s
5
0.903
0.932
0.287
0.582
0.724
18.60s
Table 5: Anchor frames W in RMPM. Time represents the time required for each step of denoising.
α
AC ↑
TC ↑
TS ↑
TF ↑
FF ↑
Time
w/o
0.902
0.931
0.286
0.579
0.725
977s
5%
0.904
0.932
0.286
0.577
0.723
564s
10%
0.902
0.930
0.286
0.577
0.722
530s
Table 6: Effect of α in Dynamic RMPM.
Figure 10: Limitation. Visualization of failure cases in our FlexiMMT. The caption for the correctly generated video is: “An elephant bows its head. A tiger jumps up while barking.” The caption for the incorrectly generated video is: “An elephant bows its head. A tiger is doing yoga.”
Figure 11: More qualitative comparisons. The left row shows reference videos. The right row shows generated videos.
Motion transfer enables controllable video generation by transferring temporal dynamics from a reference video to synthesize a new video conditioned on a target caption. However, existing Diffusion Transformer (DiT)-based methods are limited to single-object videos, restricting fine-grained control in real-world scenes with multiple objects. In this work, we introduce MotionGrounder, a DiT-based framework that firstly handles motion transfer with multi-object controllability. Our Flow-based Motion Signal (FMS) in MotionGrounder provides a stable motion prior for target video generation, while our Object-Caption Alignment Loss (OCAL) grounds object captions to their corresponding spatial regions. We further propose a new Object Grounding Score (OGS), which jointly evaluates (i) spatial alignment between source video objects and their generated counterparts and (ii) semantic consistency between each generated object and its target caption. Our experiments show that MotionGrounder consistently outperforms recent baselines across quantitative, qualitative, and human evaluations.
Samuel Teodoro, Yun Chen, Agus Gunawan +3
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea
Video motion transfer aims to animate a target object using dynamics from a reference video. Existing formulations largely rely on fixed structural correspondence, which becomes ill-defined when reference and target objects differ substantially in morphology, articulation, or deformation mechanisms. We introduce Motion Beyond Morphology, a perspective that seeks to transfer motion beyond fixed structural correspondence, by preserving dynamics that remain meaningful across different target morphologies. To realize this, we propose a two-stage framework. StageI learns complementary multi-granularity abstract motion views and uses them to bootstrap cross-category video pairs that preserve transferable dynamics across diverse morphologies. StageII internalizes this supervision into direct reference-video-conditioned generation, removing the need for explicit motion extraction at inference. We further introduce OpenVMT-Dataset and OpenVMT-Bench for training and evaluating image- and text-conditioned motion transfer across Same, Near, and Far category gaps. Extensive experiments demonstrate state-of-the-art motion fidelity and target preservation. Project page: https://miniz233.github.io/MotionBeyondMorphology/
Video motion transfer aims to generate a target video that inherits motion patterns from a source video while rendering new scenes. Existing training-free approaches focus on constructing motion guidance based on the intermediate outputs of pre-trained T2V models, which results in heavy computational overhead and limited flexibility. In this paper, we present FlowMotion, a novel training-free framework that enables efficient and flexible motion transfer by directly leveraging the predicted outputs of flow-based T2V models. Our key insight is that early latent predictions inherently encode rich temporal information. Motivated by this, we propose flow guidance, which extracts motion representations based on latent predictions to align motion patterns between source and generated videos. We further introduce a velocity regularization strategy to stabilize optimization and ensure smooth motion evolution. By operating purely on model predictions, FlowMotion achieves superior time and resource efficiency as well as competitive performance compared with state-of-the-art methods.
Zhen Wang, Youcan Xu, Jun Xiao +1
The Hong Kong University of Science and Technology · State key lab of CAD&CG, Zhejiang University