MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer

Samuel Teodoro; Yun Chen; Agus Gunawan; Soo Ye Kim; Jihyong Oh; Munchurl Kim

cs.CVApr 1, 2026

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer

Authors: Samuel Teodoro, Yun Chen, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim

Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea

Abstract

Motion transfer enables controllable video generation by transferring temporal dynamics from a reference video to synthesize a new video conditioned on a target caption. However, existing Diffusion Transformer (DiT)-based methods are limited to single-object videos, restricting fine-grained control in real-world scenes with multiple objects. In this work, we introduce MotionGrounder, a DiT-based framework that firstly handles motion transfer with multi-object controllability. Our Flow-based Motion Signal (FMS) in MotionGrounder provides a stable motion prior for target video generation, while our Object-Caption Alignment Loss (OCAL) grounds object captions to their corresponding spatial regions. We further propose a new Object Grounding Score (OGS), which jointly evaluates (i) spatial alignment between source video objects and their generated counterparts and (ii) semantic consistency between each generated object and its target caption. Our experiments show that MotionGrounder consistently outperforms recent baselines across quantitative, qualitative, and human evaluations.

Figures & tables

Methods	DiT-	Without	Multi-	Object-	PTBM
Methods	Based	Inversion	Object	Grounding	PTBM
DMT ( Yatim et al., 2024 )	✗	✗	Limited	✗	✗
MOFT ( Xiao et al., 2024 )	✗	✓	Limited	✗	✗
MotionClone ( Ling et al., 2025 )	✗	✓	Limited	✗	✗
ConMo ( Gao et al., 2025 )	✗	✗	✓	✗	✗
MultiMotion ( Liu et al., 2025 )	✓	✓	✓	✗	✓(Noisy)

Table 1: Comparison of motion transfer methods highlighting key limitations of prior works. Motion signal quality is reported only for patch trajectory-based methods (PTBM).

Figure 2: Overall framework of MotionGrounder. Given a source video $V_{S}$ with $N$ objects, global caption $c_{g}$ , object captions $\{c_{i}\}_{i=1}^{N}$ , and corresponding object masks $\{m_{i}^{1:F}\}_{i=1}^{N}$ , MotionGrounder transfers motion dynamics to a text-defined target video $V_{T}$ . A Flow-based Motion Signal (FMS, Sec. 3.3 ) provides stable motion guidance, while the Object-Caption Alignment Loss (OCAL, Sec. 3.4 ) enforces spatial grounding between each object caption and its designated object region, enabling training-free multi-object motion transfer.

Figure 3: Flow-based Motion Signal (FMS, Sec. 3.3 ). FMS constructs stable latent space patch trajectories from optical flows estimated on sampled source video frames, and uses the resulting displacements to supervise motion transfer during denoising.

Figure 4: Object-Caption Alignment Loss (OCAL, Sec. 3.4 ). OCAL aggregates object-specific attention maps and aligns them with their corresponding object masks to enforce precise spatial grounding of each object caption during generation.

Figure 5: Qualitative comparison. For clarity, we show color-coded bounding boxes inferred from object masks, where each color corresponds to an object in the caption. While all other methods suffer from spatial misalignment and motion misattribution , our MotionGrounder generates correct objects, preserves spatial alignment, and maintains object-specific motion across all scenarios.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
DMT ( Yatim et al., 2024 )	0.7069	0.2172	0.2754	0.0627	0.3186	0.9358	0.9172	0.6840	0.2158	0.2610	0.0588	0.3183	0.9337	0.9016
MOFT ( Xiao et al., 2024 )	0.5682	0.2548	0.2767	0.0734	0.3165	0.9390	0.9119	0.5331	0.1899	0.2553	0.0514	0.3116	0.9379	0.8749
MotionClone ( Ling et al., 2025 )	0.5419	0.2353	0.2786	0.0690	0.3156	0.9420	0.9451	0.5125	0.1951	0.2572	0.0532	0.3145	0.9390	0.9340
ConMo ( Gao et al., 2025 )	0.6495	0.2336	0.2738	0.0680	0.3154	0.9381	0.9084	0.6377	0.2014	0.2578	0.0553	0.3154	0.9354	0.8900
DiTFlow ( Pondaven et al., 2025 )	0.6762	0.2134	0.2781	0.0625	0.3151	0.9414	0.9342	0.6483	0.1917	0.2587	0.0514	0.3155	0.9375	0.9146

Table 2: Quantitative comparison. We compare MotionGrounder across three settings, Caption , Subject , and Scene , against five baselines. We achieve the strongest performance across all scenarios, attaining the best and second best results on the majority of metrics. Best results are in bold and second best are underlined .

Method	MA $\downarrow$	GTF $\downarrow$	OG $\downarrow$	OGS Rank $\downarrow$
DMT ( Yatim et al., 2024 )	3.65	3.70	3.72	3.68
MOFT ( Xiao et al., 2024 )	3.81	3.78	3.81	3.74
MotionClone ( Ling et al., 2025 )	3.59	3.56	3.59	3.61
ConMo ( Gao et al., 2025 )	3.56	3.59	3.58	3.52
DiTFlow ( Pondaven et al., 2025 )	3.40	3.40	3.46	3.75
MotionGrounder (Ours)	2.99	2.97	2.83	2.70

Table 3: User study. Average human ranking (lower is better) on motion adherence (MA), global textual faithfulness (GTF), and object grounding (OG). OGS ranks are reported for reference.

Figure 6: Ablation of essential components. (b) FMS improves motion fidelity while (c) OCAL enhances object grounding. Their combination in (d) yields faithful motion transfer with accurate object localization in multi-object scenarios.

Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) CogVideoX-5B	0.4257	0.1593	0.2662	0.0446	0.3177	0.9430	0.9393
(b) w/ FMS only	0.6714	0.1652	0.2634	0.0462	0.3125	0.9419	0.9339
(c) w/ OCAL only	0.4860	0.3099	0.2684	0.0868	0.3223	0.9430	0.9422
(d) Ours, Full	0.6813	0.2371	0.2677	0.0671	0.3182	0.9417	0.9379

Table 4: Ablation of essential components. FMS improves MF, OCAL enhances object grounding, and their combination provides the best trade-off between accurate motion transfer (best MF) and object grounding (second best IoU, LTA, OGS).

Figure 7: FMS ablation. (b) No trajectory duplication suppresses motion while (c) allowing it better preserves original motion.

Figure 8: OCAL ablation. (a) Removing OCAL leads to weak object grounding, (b)-(d) optimizing attention maps improves alignment, (e) repetition enhances motion fidelity, and (f) size regularization enables complete and balanced object generation.

Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) no OCAL	0.6714	0.1652	0.2634	0.0462	0.3125	0.9419	0.9339
(b) text-to-vid only	0.6749	0.2328	0.2668	0.0659	0.3172	0.9416	0.9363
(c) vid-to-text only	0.6675	0.2157	0.2662	0.0609	0.3151	0.9420	0.9368
(d) text-to-vid +	0.6791	0.2375	0.2654	0.0667	0.3172	0.9420	0.9361
vid-to-text	0.6791	0.2375	0.2654	0.0667	0.3172	0.9420	0.9361
(e) w/ repetition	0.6800	0.2345	0.2644	0.0656	0.3176	0.9416	0.9349

Table 5: OCAL ablation. OCAL consistently improves object grounding and semantic alignment, while also boosting MF.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Dataset generation pipeline. Long captions are first generated using CogVLM2-Caption ( Hong et al., 2024 ) and summarized into structured prompts with GPT-5 ( OpenAI, 2025 ) , followed by object–mask association, and target prompt construction.

Figure 10: Sample frames and masks from videos in our dataset, shown with their corresponding Caption , Subject , and Scene prompts.

Metric	Global Alignment	Object Localization	Object–Caption Association
Global CLIP Score	✓	✗	✗
Intersection-over-Union	✗	✓	✗
Local CLIP Score	✗	✗	✓
OGS (Ours)	✗	✓	✓

Appendix

Table 6: Motivation for Object Grounding Score (OGS). Comparison of commonly used evaluation metrics and the aspects of alignment they capture.

Figure 11: Comparison between bounding boxes inferred from ground-truth masks and Grounding DINO predictions. We show the bounding box inferred from the ground-truth mask in (b), the Grounding DINO prediction in (c), and the overlaid frames and bounding boxes in (d), where it is evident that the ground-truth mask–based bounding box no longer accurately localizes the generated object.

Figure 12: Effect of SAM2 ( Ravi et al., 2025 ) segmentation masks on CLIP scores. We report the (a) local CLIP scores computed without applying the SAM2 segmentation mask and the (b) local CLIP scores computed with the SAM2 segmentation mask.

Figure 13: Comparison of patch trajectories obtained using Attention Motion Flow (AMF) ( Pondaven et al., 2025 ) and our Flow-based Motion Signal (FMS, 3.3 ). The red dots indicate the tracked patch locations over time. The visualization reflects the $4\times$ temporal downsampling of the VAE encoder, with trajectories fused every four input frames. AMF exhibits several failure modes, including (i) inducing motion in stationary background regions due to texture similarity, (ii) confusing visually similar objects that degrades motion segregation and object-level motion preservation, and (iii) instability under large or fast motions. In contrast, FMS provides a stable and object-consistent motion signal, yielding accurate and robust trajectory estimation even in the presence of large motions.

Figure 14: Additional qualitative comparisons.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) CogVideoX-5B	0.4288	0.1749	0.2770	0.0504	0.3158	0.9434	0.9463	0.4306	0.1527	0.2608	0.0415	0.3204	0.9431	0.9400
(b) w/ FMS only	0.6846	0.1863	0.2763	0.0541	0.3140	0.9426	0.9408	0.6730	0.1652	0.2532	0.0450	0.3082	0.9414	0.9312
(c) w/ OCAL only	0.5011	0.3278	0.2789	0.0945	0.3202	0.9430	0.9444	0.4964	0.2993	0.2633	0.0827	0.3262	0.9419	0.9374
(d) Ours, Full	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
Method	Scene							All

Appendix

Table 7: Ablation of essential components. FMS improves MF, OCAL enhances object grounding, and their combination provides the best trade-off between accurate motion transfer (best MF) and object grounding (second best IoU, LTA, OGS).

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) no FMS	0.5011	0.3278	0.2789	0.0945	0.3202	0.9430	0.9444	0.4964	0.2993	0.2633	0.0827	0.3262	0.9419	0.9374
(b) w/o traj dup	0.6814	0.2574	0.2781	0.0740	0.3194	0.9417	0.9484	0.6737	0.2244	0.2604	0.0621	0.3187	0.9404	0.9369
(c) w/ traj dup (Ours, Full)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
Method	Scene							All
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$

Appendix

Table 8: FMS ablation. FMS improves MF overall. No trajectory duplication in (b) suppresses motion while allowing it in (c) better preserves original motion.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) no OCAL	0.6846	0.1863	0.2763	0.0541	0.3140	0.9426	0.9408	0.6730	0.1652	0.2532	0.0450	0.3082	0.9414	0.9312
(b) text-to-vid only	0.6815	0.2601	0.2796	0.0752	0.3206	0.9426	0.9432	0.6718	0.2332	0.2586	0.0643	0.3134	0.9401	0.9315
(c) vid-to-text only	0.6744	0.2441	0.2752	0.0704	0.3160	0.9430	0.9437	0.6680	0.2239	0.2613	0.0623	0.3122	0.9409	0.9322
(d) text-to-vid + vid-to-text	0.6800	0.2672	0.2779	0.0766	0.3209	0.9431	0.9424	0.6812	0.2367	0.2586	0.0652	0.3144	0.9408	0.9311
(e) w/ repetition	0.6832	0.2628	0.2770	0.0752	0.3219	0.9425	0.9399	0.6784	0.2367	0.2584	0.0651	0.3151	0.9401	0.9284

Appendix

Table 9: OCAL ablation. OCAL consistently improves object grounding and semantic alignment, while also boosting MF.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) $\lambda_{\text{FMS}}=1.00$	0.6681	0.2533	0.2772	0.0732	0.3157	0.9419	0.9433	0.6498	0.2232	0.2582	0.0620	0.3123	0.9410	0.9358
(b) $\lambda_{\text{FMS}}=1.25$ (Ours)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
(c) $\lambda_{\text{FMS}}=1.50$	0.6708	0.2535	0.2747	0.0728	0.3145	0.9427	0.9466	0.6538	0.2239	0.2592	0.0616	0.3126	0.9406	0.9345
Method	Scene							All
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$

Appendix

Table 10: Ablation study on

\lambda_{\text{FMS}}

. Setting

\lambda_{\text{FMS}}=1.25

achieves the best overall performance, improving motion fidelity (MF) while preserving object grounding (highest IoU, LTA, OGS) and temporal consistency across different prompt settings.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) $\lambda_{\text{OCAL}}=0.75$	0.6757	0.2576	0.2775	0.0744	0.3194	0.9422	0.9425	0.6712	0.2210	0.2581	0.0608	0.3124	0.9409	0.9321
(b) $\lambda_{\text{OCAL}}=1.00$ (Ours)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
(c) $\lambda_{\text{OCAL}}=1.25$	0.6748	0.2655	0.2754	0.0759	0.3211	0.9427	0.9423	0.6769	0.2437	0.2613	0.0673	0.3169	0.9402	0.9331
Method	Scene							All
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$

Appendix

Table 11: Ablation study on

\lambda_{\text{OCAL}}

. IoU, LTA, and OGS change only marginally with

\lambda_{\text{OCAL}}

and show no consistent improvements across prompt settings. Lower weighting (

\lambda_{\text{OCAL}}=0.75

) slightly weakens object grounding, while higher weighting (

\lambda_{\text{OCAL}}=1.25

) yields minor but variable gains. We therefore choose

\lambda_{\text{OCAL}}=1.0

, which provides stable grounding and achieves the highest MF across Caption , Subject , and All settings.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) $T_{opt,\text{FMS}}=5$	0.5751	0.2823	0.2827	0.0820	0.3231	0.9425	0.9422	0.5473	0.2712	0.2604	0.0744	0.3222	0.9405	0.9397
(b) $T_{opt,\text{FMS}}=10$	0.6690	0.2757	0.2798	0.0794	0.3212	0.9419	0.9418	0.6604	0.2362	0.2603	0.0651	0.3160	0.9400	0.9310
(c) $T_{opt,\text{FMS}}=15$ (Ours)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
(d) $T_{opt,\text{FMS}}=20$	0.6805	0.2652	0.2781	0.0763	0.3196	0.9415	0.9435	0.6779	0.2348	0.2592	0.0643	0.3149	0.9404	0.9316
Method	Scene							All

Appendix

Table 12: Ablation study on number of FMS optimization steps. Increasing

T_{opt,\text{FMS}}

improves MF up to 15 steps, after which performance saturates, while grounding metrics and GTA remain stable with minor trade-offs, making 15 steps the best overall balance.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) $T_{opt,\text{OCAL}}=5$	0.6730	0.2657	0.2790	0.0766	0.3213	0.9424	0.9396	0.6771	0.2440	0.2607	0.0674	0.3179	0.9404	0.9283
(b) $T_{opt,\text{OCAL}}=10$	0.6770	0.2655	0.2790	0.0764	0.3201	0.9422	0.9407	0.6781	0.2400	0.2607	0.0665	0.3167	0.9407	0.9290
(c) $T_{opt,\text{OCAL}}=15$ (Ours)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
(d) $T_{opt,\text{OCAL}}=20$	0.6757	0.2695	0.2779	0.0772	0.3222	0.9417	0.9423	0.6762	0.2444	0.2578	0.0673	0.3146	0.9406	0.9285
Method	Scene							All

Appendix

Table 13: Ablation study on number of OCAL optimization steps. We report performance as a function of

T_{opt,\text{OCAL}}

, showing that 15 steps provide the best overall trade-off, achieving the highest motion fidelity while maintaining stable object grounding and alignment.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
(a) $B=5$	0.6785	0.2305	0.2768	0.0669	0.3164	0.9422	0.9419	0.6662	0.2166	0.2604	0.0601	0.3144	0.9408	0.9308
(b) $B=10$ (Ours)	0.6875	0.2638	0.2776	0.0763	0.3224	0.9426	0.9439	0.6818	0.2403	0.2617	0.0667	0.3150	0.9410	0.9335
(c) $B=15$	0.6739	0.2560	0.2795	0.0744	0.3196	0.9422	0.9439	0.6558	0.2369	0.2626	0.0658	0.3130	0.9414	0.9356
(d) $B=20$	0.6746	0.2532	0.2751	0.0732	0.3140	0.9428	0.9453	0.6502	0.2231	0.2598	0.0617	0.3140	0.9411	0.9358
Method	Scene							All

Appendix

Table 14: Ablation study on OCAL block position. Setting

B=10

achieves the best overall trade-off, yielding the highest motion fidelity while preserving strong object grounding and stable temporal consistency across different prompt settings.

Method	Caption							Subject
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
Wan2.1	0.4661	0.1867	0.2790	0.0543	0.3230	0.9443	0.9615	0.4632	0.1642	0.2670	0.0453	0.3267	0.9459	0.9662
Wan2.1 + MotionGrounder	0.6050	0.2045	0.2786	0.0592	0.3191	0.9406	0.9565	0.5829	0.1972	0.2704	0.0555	0.3305	0.9398	0.9467
Method	Scene							All
Method	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$	MF $\uparrow$	IoU $\uparrow$	LTA $\uparrow$	OGS $\uparrow$	GTA $\uparrow$	CTC $\uparrow$	DTC $\uparrow$
Wan2.1	0.4574	0.1476	0.2640	0.0404	0.3257	0.9453	0.9658	0.4623	0.1662	0.2700	0.0466	0.3251	0.9452	0.9645

Appendix

Table 15: Quantitative evaluation on the Wan2.1 backbone. We integrate MotionGrounder into the Wan2.1-1.3B architecture ( Wan et al., 2025 ) to evaluate its zero-shot generalization capabilities. Results across Caption, Subject, and Scene prompts demonstrate that our framework significantly boosts MF and grounding metrics (IoU, LTA, OGS) while maintaining the semantic integrity (GTA) and inherent temporal stability (CTC, DTC) of the base model.

Figure 15: Limitation. Excessive number of objects leads to degraded object consistency and weaker motion–object alignment in MotionGrounder, a challenge also observed in existing motion transfer methods. The starting points at which failures emerge are indicated by red dashed lines.

Figure 16: Controllability demo. MotionGrounder demonstrates flexible, mask-driven object-level control, enabling seamless object swapping by interchanging masks and object removal by removing masks and corresponding object captions during inference.

Method	1 object		2 objects		3 objects
Method	Runtime (s) $\downarrow$	Memory (GB) $\downarrow$	Runtime (s) $\downarrow$	Memory (GB) $\downarrow$	Runtime (s) $\downarrow$	Memory (GB) $\downarrow$
DMT ( Yatim et al., 2024 )	475.39	22.64	475.99	22.64	475.50	22.64
MOFT ( Xiao et al., 2024 )	477.19	22.64	477.31	22.64	476.21	22.64
MotionClone ( Ling et al., 2025 )	481.17	32.42	481.59	32.42	480.45	32.42
ConMo ( Gao et al., 2025 )	480.01	22.64	483.74	22.64	480.54	22.64
DiTFlow ( Pondaven et al., 2025 )	486.35	31.92	486.09	31.92	485.92	31.92

Appendix

Table 16: Runtime and GPU Memory Comparison . Our MotionGrounder incurs higher runtime and memory overhead due to the additional object-level grounding optimization (OCAL) on top of our FMS, but provides finer-grained control and improved motion fidelity. The cost remains constant as the number of objects increases, indicating stable scalability in multi-object settings.

Figure 17: Our user study interface and questionnaire form.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 309 papers

Mar 1, 2026

Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer

Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/

Yuze Li, Dong Gong, Xiao Cao +6

Tianjin University · University of New South Wales · University of Electronic Science and Technology of China +2

Controllable Video Generation Image-to-Video Generation Video Diffusion Models Representation Disentanglement

Ask PDF

Jul 13, 2026

Controlling Motion Transfer in Diffusion Transformers via Attention Heads

Diffusion Transformers (DiTs) have advanced video generation with high-quality, temporally coherent results. However, extending them to motion transfer, which requires following reference motion while aligning with a target prompt, remains challenging due to limited understanding of motion and structure representations within DiTs. We analyze video DiTs at the attention-head level and identify distinct heads specialized for motion and spatial structure. Based on this insight, we propose a head-aware controllable motion transfer framework that requires no parameter updates. Our method refines motion cues from motion-specialized heads via semantic correspondence guidance and preserves structure through selective feature injection. This head-level control not only enables accurate motion transfer but also provides an interpretable foundation for controllable video generation with DiTs.

Sunyoung Jung, Jiwoo Park, Yoonseok Choi +3

Yonsei University · LG Electronics · University of California, Merced

Diffusion Model Guidance Diffusion Transformer Motion Retargeting Attention Mechanisms Controllable Video Generation

Ask PDF

Mar 6, 2026

FlowMotion: Training-Free Flow Guidance for Video Motion Transfer

Video motion transfer aims to generate a target video that inherits motion patterns from a source video while rendering new scenes. Existing training-free approaches focus on constructing motion guidance based on the intermediate outputs of pre-trained T2V models, which results in heavy computational overhead and limited flexibility. In this paper, we present FlowMotion, a novel training-free framework that enables efficient and flexible motion transfer by directly leveraging the predicted outputs of flow-based T2V models. Our key insight is that early latent predictions inherently encode rich temporal information. Motivated by this, we propose flow guidance, which extracts motion representations based on latent predictions to align motion patterns between source and generated videos. We further introduce a velocity regularization strategy to stabilize optimization and ensure smooth motion evolution. By operating purely on model predictions, FlowMotion achieves superior time and resource efficiency as well as competitive performance compared with state-of-the-art methods.

Zhen Wang, Youcan Xu, Jun Xiao +1

The Hong Kong University of Science and Technology · State key lab of CAD&CG, Zhejiang University

Controllable Video Generation Temporal Consistency in Video Generation

Ask PDF

Abstract

Controllable Video Generation

Explore similar work

Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer

Controlling Motion Transfer in Diffusion Transformers via Attention Heads

FlowMotion: Training-Free Flow Guidance for Video Motion Transfer