Organizations: School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Tongji University, Shanghai, China · the Hong Kong University of Science and Technology, Hong Kong, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Existing motion transfer methods excel at exact, rigid motion replication from reference videos to target scenes, whereas they often fail to account for the compatibility between the transferred motion and the target scene environment in accordance with physical laws. In this work, we propose MotionShift, a novel motion transfer framework that achieves high-fidelity motion transfer while ensuring physical compatibility with the target scene. In particular, we design a planner--evaluator iterative motion adaptation mechanism, in which a Large Language Model (LLM) acts as the planner to generate motion trajectories by leveraging its strong semantic reasoning capabilities, while a Vision-Language Model (VLM) serves as the evaluator to rectify the motion based on its superior visual perception and scene understanding. Through this planner--evaluator synergistic reasoning process, the motion is adapted to the target scene in a training-free manner. The adapted motion is then used as a generative condition for an image-to-video (I2V) model to render videos in the target scene. Extensive experiments demonstrate that MotionShift achieves high-fidelity and physically compatible motion transfer across diverse and complex scenes. Moreover, we show that MotionShift can be readily integrated into existing trajectory-conditioned I2V models, endowing them with realistic motion transfer capabilities.
Figures & tables
Fig. 2: Overview of MotionShift. Given a reference video and a target image, we first initialize a motion trajectory for the target scene. Then, through planner–evaluator iterative adaptation, we refine the trajectory to produce an environment-compatible trajectory and a corresponding motion prompt. Finally, these are progressively rendered into the output video: a motion generation model converts the trajectory into a semantic mask sequence, and an appearance rendering model synthesizes the final video from the masks and target image.
Fig. 3: Illustration of environment-compatible motion transfer, where identical reference motions adapt naturally to different scenes, yielding contextually consistent behaviors.
Fig. 4: Illustration of Planner–Evaluator Iterative Adaptation. At the beginning of each iteration, we leverage the semantic reasoning capability of the planner (LLM) to infer object–environment interactions and perform detailed motion planning along the given trajectory, producing a corrected motion trajectory. We then construct a pseudo video based on the target image and the corrected trajectory, and feed it to the evaluator (VLM) to assess visual consistency and physical plausibility, which then outputs revision suggestions. These suggestions are then appended to the planner’s instruction prompt, and we iteratively refine the motion trajectory until the evaluator detects no major issues.
Fig. 5: Visual comparison with baseline methods. MotionShift exhibits superior motion consistency and environmental adaptability, accurately capturing perspective variations, realistic reflections, and synchronized hand movements compared to baseline approaches.
Method
Automatic Evaluation ↑
User Study ↑
Appearance
Temporal
Motion
Motion
Appearance
Temporal
Motion
Motion
Consistency
Consistency
Accuracy
Appropriateness
Consistency
Consistency
Accuracy
Appropriateness
VideoComposer [ 7 ]
73.44
92.77
18.60
26.70
2.15
2.23
2.08
2.19
MotionClone [ 9 ]
71.25
94.57
11.60
32.20
2.31
2.38
2.24
2.27
MotionDirector [ 34 ]
79.78
98.56
10.90
52.80
2.18
2.26
2.14
2.11
FlexiAct [ 12 ]
85.68
97.77
30.10
77.60
3.52
3.61
3.48
3.57
TABLE I: Quantitative comparison. Our method significantly outperforms the other leading methods.
Fig. 6: Visualizations of MotionShift in scenes with complex interactions.
Fig. 7: Ablation on planner–evaluator adaptation. We compare variants by disabling adaptation, removing the evaluator, or replacing the planner.
Fig. 8: Ablation on motion-to-appearance rendering: w/o mask vs. w/ mask (ours).
Fig. 9: Visual results of Go-with-the-Flow using the initial trajectory vs. the environment-compatible trajectory produced by MotionShift as a plugin.
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
Video motion transfer aims to generate a target video that inherits motion patterns from a source video while rendering new scenes. Existing training-free approaches focus on constructing motion guidance based on the intermediate outputs of pre-trained T2V models, which results in heavy computational overhead and limited flexibility. In this paper, we present FlowMotion, a novel training-free framework that enables efficient and flexible motion transfer by directly leveraging the predicted outputs of flow-based T2V models. Our key insight is that early latent predictions inherently encode rich temporal information. Motivated by this, we propose flow guidance, which extracts motion representations based on latent predictions to align motion patterns between source and generated videos. We further introduce a velocity regularization strategy to stabilize optimization and ensure smooth motion evolution. By operating purely on model predictions, FlowMotion achieves superior time and resource efficiency as well as competitive performance compared with state-of-the-art methods.
Zhen Wang, Youcan Xu, Jun Xiao +1
The Hong Kong University of Science and Technology · State key lab of CAD&CG, Zhejiang University
Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/
Yuze Li, Dong Gong, Xiao Cao +6
Tianjin University · University of New South Wales · University of Electronic Science and Technology of China +2
Video motion transfer aims to animate a target object using dynamics from a reference video. Existing formulations largely rely on fixed structural correspondence, which becomes ill-defined when reference and target objects differ substantially in morphology, articulation, or deformation mechanisms. We introduce Motion Beyond Morphology, a perspective that seeks to transfer motion beyond fixed structural correspondence, by preserving dynamics that remain meaningful across different target morphologies. To realize this, we propose a two-stage framework. StageI learns complementary multi-granularity abstract motion views and uses them to bootstrap cross-category video pairs that preserve transferable dynamics across diverse morphologies. StageII internalizes this supervision into direct reference-video-conditioned generation, removing the need for explicit motion extraction at inference. We further introduce OpenVMT-Dataset and OpenVMT-Bench for training and evaluating image- and text-conditioned motion transfer across Same, Near, and Far category gaps. Extensive experiments demonstrate state-of-the-art motion fidelity and target preservation. Project page: https://miniz233.github.io/MotionBeyondMorphology/