Compositional motion transfer requires a model to extract motions from multiple reference videos, and apply each motion to a user-specified object in one target scene. Existing methods normally struggle to reliably assign different reference motions to the correct target objects, thus a motion intended for one object may affect another. Meanwhile, some motion-specific tuning approaches are still facing another issue: they must run a new optimization whenever the reference video changes, making repeated transfers inefficient. To address these limitations, we propose RART, a framework that unifies object-wise motion extraction, target-region anchoring, and frame-aligned injection into a frozen video Diffusion Transformer for multi-object motion transfer. For each reference motion, RART extracts temporally ordered features from the masked reference video, and then conditions them on the target first-frame mask to form mask-anchored motion tokens. Frame-aligned motion cross-attention allows each generated frame to read only the motion tokens at the corresponding time slots, composing all object motions in one denoising process. During training, a motion-attention routing loss encourages each target region to attend to its corresponding motion tokens and reduces interference between objects. Therefore, RART can handle novel reference-target combinations without per-video optimization. Experiments show that RART achieves accurate motion preservation, efficient inference, and strong compositional generalization.
Figures & tables
Figure 1: Compositional motion transfer with RART. Masked reference videos provide object-wise motions, which are assigned to colored target regions and jointly rendered in the generated video. The examples show (a) and (b) motion reassignment, (c) cross-species transfer, and (d) multi-object composition.
Figure 2: Overview of RART. (a) The motion encoder extracts temporally ordered features from each masked reference video and injects the corresponding target-mask anchor to produce object-wise motion tokens. (b) Frame-aligned motion cross-attention injects the motion tokens of each latent frame into the frozen DiT.
Figure 3: Motion-Attention Routing Loss. For each object, RART encourages its target region to attend to the corresponding motion tokens while discouraging other object regions from attending to the same token block.
Automatic Metrics
Human Evaluations (%)
Method
TS ↑
TC ↑
AC ↑
FF ↑
MF ↑
App. ↑
Temp. ↑
Motion ↑
Time ↓
AnyV2V
0.261
0.879
0.742
0.571
0.338
0.0
0.0
0.0
176s/ref + 50s
DisMo
0.266
0.944
0.913
0.652
0.451
3.0
3.0
2.6
113s
SCAIL-2
0.266
0.922
0.906
0.648
0.428
15.8
9.0
24.1
233s
FlexiMMT
0.267
0.925
0.899
0.671
0.462
14.7
27.1
21.8
2h/ref + 432s
RART
0.265
0.928
0.903
0.713
0.480
66.5
60.9
51.5
41s
Table 1: Quantitative and human evaluations. TS, TC, AC, FF, and MF denote text similarity, temporal consistency, appearance consistency, flow fidelity, and motion fidelity, respectively. Human evaluation columns report five-way selection rates (%), while Time denotes runtime. For AnyV2V and FlexiMMT, the first time term denotes per-reference preparation.
Figure 4: Qualitative comparison. The top row shows the reference motions, target image, and motion-to-object assignments; the remaining rows show frames generated by the baselines and RART. RART transfers each reference motion to its designated target object while preserving target appearance.
Figure 5: Qualitative ablation. The top row shows reference motions and their assignments to target regions; the bottom panels compare RART with variants without frame alignment, target-mask anchoring, or routing loss.
Figure 6: Motion-attention heat maps. Columns correspond to the two reference motions, while rows compare the model without the routing loss and RART. Warmer colors indicate stronger attention to each motion-token block.
Variant
FF ↑
MF ↑
w/o Frame Alignment
0.678
0.400
w/o Mask Anchor
0.690
0.413
w/o Routing Loss
0.727
0.448
RART
0.713
0.480
Table 2: Quantitative ablations. We replace frame-aligned motion attention with global attention, use all-ones target masks, or remove the motion-attention routing loss.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Video backbone
Wan2.2 TI2V-5B
Motion extractor
DisMo Encoder
Training steps
200K
Learning rate
1×10−5
Hardware
8 × NVIDIA H200 GPUs
Batch size per GPU
1
Appendix
Table 3: Training configuration of the reported RART model.
Figure 7: Additional qualitative results for compositional motion transfer. Each example shows color-coded reference sequences at the top, followed by the target-object assignments and generated frames at the bottom. Blue and red indicate the correspondence between each reference motion and its target object.
Figure 8: Further qualitative results for compositional motion transfer. RART jointly transfers independently specified reference motions to their color-matched target objects while preserving the appearance and scene of the target image.
Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/
Yuze Li, Dong Gong, Xiao Cao +6
Tianjin University · University of New South Wales · University of Electronic Science and Technology of China +2
Motion transfer enables controllable video generation by transferring temporal dynamics from a reference video to synthesize a new video conditioned on a target caption. However, existing Diffusion Transformer (DiT)-based methods are limited to single-object videos, restricting fine-grained control in real-world scenes with multiple objects. In this work, we introduce MotionGrounder, a DiT-based framework that firstly handles motion transfer with multi-object controllability. Our Flow-based Motion Signal (FMS) in MotionGrounder provides a stable motion prior for target video generation, while our Object-Caption Alignment Loss (OCAL) grounds object captions to their corresponding spatial regions. We further propose a new Object Grounding Score (OGS), which jointly evaluates (i) spatial alignment between source video objects and their generated counterparts and (ii) semantic consistency between each generated object and its target caption. Our experiments show that MotionGrounder consistently outperforms recent baselines across quantitative, qualitative, and human evaluations.
Samuel Teodoro, Yun Chen, Agus Gunawan +3
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea
Video motion transfer aims to generate a target video that inherits motion patterns from a source video while rendering new scenes. Existing training-free approaches focus on constructing motion guidance based on the intermediate outputs of pre-trained T2V models, which results in heavy computational overhead and limited flexibility. In this paper, we present FlowMotion, a novel training-free framework that enables efficient and flexible motion transfer by directly leveraging the predicted outputs of flow-based T2V models. Our key insight is that early latent predictions inherently encode rich temporal information. Motivated by this, we propose flow guidance, which extracts motion representations based on latent predictions to align motion patterns between source and generated videos. We further introduce a velocity regularization strategy to stabilize optimization and ensure smooth motion evolution. By operating purely on model predictions, FlowMotion achieves superior time and resource efficiency as well as competitive performance compared with state-of-the-art methods.
Zhen Wang, Youcan Xu, Jun Xiao +1
The Hong Kong University of Science and Technology · State key lab of CAD&CG, Zhejiang University