Diffusion models for video generation have achieved remarkable success, yet precise motion control under explicit user guidance remains challenging. Recent noise-warping methods introduce temporal correlations by warping the noise according to the user-provided motion signal. However, standard spatial warping generally disrupts the noise's i.i.d. Gaussian structure, requiring distribution correction, such as Jacobian-based correction, but at a substantial computational overhead. We propose Householder Motion Embedding (HME), which encodes the motion signal into the noise through a sequence of Householder reflections that together form an orthogonal transformation applied to each location's channel vector. HME exactly preserves the spatial Gaussian structure within each frame while introducing cross-frame correlations that promote temporally consistent video. HME is agnostic to the network architecture and integrates directly into standard video diffusion fine-tuning pipelines. Experiments on different scenarios show that HME matches or outperforms the state of the art, while achieving a 2.7x speedup over noise warping. These results demonstrate that channel-space Householder transformations offer a principled and efficient alternative to noise warping for motion-controllable video generation.
Figures & tables
Figure 2: HME generates motion controlled videos in 3 stages: 1) optical flow estimation and spatial resampling, 2) temporally correlated noise construction via Householder Reflectors (see Section 3.1 for details), 3) diffusion model fine-tuning and inference. At inference time, our method enables motion- and camera-controlled video generation given a text prompt, an initial frame, and a reference video as the motion signal.
Figure 3: Qualitative comparison of (a) motion-transfer, (b) fine-grained motion control, and (c) local object motion control I2V generation, evaluated on DAVIS and MC-Bench. We present the intermediate frame and last frame of generated videos by each method. Our method demonstrates strong appearance consistency and high generation fidelity while faithfully adhering to the provided motion control signals. Best viewed zoomed in; videos are provided in the Supplementary Material.
Method
Subject
BG
Temporal
Motion
FVD
Optical Flow
CoTracker
PSNR
CLIP
Consist. ↑
Consist. ↑
Flicker. ↑
Smooth. ↑
FID ↓
( ×103 ) ↓
(EPE) ↓
(mIoU) ↑
(dB) ↑
Image ↑
MotionClone [ 32 ]
0.754
0.862
0.874
0.928
133.852
1.771
26.451
0.373
10.032
0.814
ImageConductor [ 30 ]
0.791
0.891
0.913
0.939
128.087
1.830
20.382
0.372
12.117
0.811
GWTF γ=0.5 [ 7 ]
0.888
0.920
0.942
0.978
68.385
0.987
5.644
0.464
14.748
0.912
MotionPro [ 65 ]
0.873
0.913
0.932
0.964
103.859
1.253
19.692
0.473
14.257
0.879
DisMo [ 43 ]
0.851
0.903
0.920
0.956
85.201
1.156
11.189
0.428
11.475
0.882
Table 1: Quantitative comparison of motion controlled I2V generation on DAVIS. Best performance in Bold , best baseline performance in Underline .
Method
Subject Cons. ↑
Background Cons. ↑
Motion Smooth. ↑
Aesthetic Qual. ↑
Imaging Qual. ↑
CTD ↓
Bg-Obj CTD ↑
MotionPro [ 65 ]
0.980
0.976
0.995
0.586
0.610
8.019
23.30
TTM [ 46 ]
0.984
0.981
0.995
0.601
0.645
8.432
36.32
DisMo [ 43 ]
0.985
0.973
0.991
0.608
0.601
8.954
36.16
Noise-based methods
GWTF γ=0.5 [ 7 ]
0.975
0.974
0.993
0.589
0.645
15.747
42.44
Ours γ=0.3,K=2
0.976
0.977
0.995
0.594
0.645
13.879
39.53
Table 2: Quantitative comparison of Fine-grained and Object-level motion control on MC-Bench.
Figure 4: Qualitative Comparison of camera motion controlled I2V generation, evaluated on DL3DV. We present the intermediate frames and last frames of generated videos by each method. Best viewed zoomed in; videos are provided in the Supplementary Material.
Method
FID ↓
FVD ↓
PSNR (dB) ↑
CLIP Image ↑
LPIPS ↓
SSIM ↑
ATE ↓
MotionPro [ 65 ]
37.195
291.510
12.701
0.814
0.545
0.470
0.825
DisMo [ 43 ]
22.458
187.134
12.255
0.876
0.506
0.446
1.155
TTM [ 46 ]
21.367
196.533
12.764
0.881
0.443
0.467
0.464
GWTF γ=0.5 [ 7 ]
23.394
235.260
12.626
0.880
0.431
0.469
0.473
Ours γ=0.3,K=2
20.338
175.103
13.656
0.885
0.402
0.496
0.421
Table 3: Quantitative comparison of DL3DV camera motion control.
Method
Subject
BG
Temporal
Motion
FVD
Optical Flow
CoTracker
PSNR
CLIP
Consist. ↑
Consist. ↑
Flicker. ↑
Smooth. ↑
FID ↓
( ×103 ) ↓
(EPE) ↓
(mIoU) ↑
(dB) ↑
Image ↑
GWTF γ=0.5 [ 7 ]
0.882
0.918
0.945
0.979
73.348
1.180
5.857
0.472
14.437
0.901
Ours γ=0.3,K=2
0.895
0.926
0.948
0.979
63.619
0.837
4.707
0.541
15.418
0.915
Table 4: Robustness to the flow estimator: DAVIS results with UniMatch [ 56 ] flows at evaluation time.
Method
Subject ↑
Background ↑
Temporal ↑
Motion ↑
FID ↓
FVD ↓
Flow EPE ↓
CoTracker mIoU ↑
PSNR ↑
CLIP Image ↑
Ours K=1
0.898
0.925
0.946
0.978
63.992
861.121
4.868
0.543
15.300
0.918
Ours K=2
0.898
0.925
0.946
0.979
61.780
861.740
4.433
0.546
15.399
0.923
Ours K=4
0.898
0.926
0.948
0.979
64.579
821.749
4.358
0.555
15.424
0.922
Ours K=8
0.898
0.927
0.948
0.979
63.314
827.664
4.144
0.550
15.512
0.923
Ours K=12
0.898
0.927
0.950
0.979
64.699
831.586
4.197
0.558
15.581
0.923
Table 5: Performance comparison across different numbers of Householder reflections K (all with γ=0.3 ), on DAVIS.
Method
HIWYN [ 8 ]
GWTF [ 7 ]
InfRes [ 10 ]
Ours K=1
Ours K=2
Ours K=4
Ours K=6
Ours K=8
Ours K=12
GPU Time (ms/Latent) ↓
38.00
7.63
23.94
2.27
2.80
5.18
7.59
10.01
14.78
Memory Overhead (MB) ↓
49,384
76
285
0.5
1
2
3
4
6
Table 6: Computational overhead of noise-based methods: GPU time and approximate peak additional GPU memory on an NVIDIA RTX PRO 6000 Blackwell GPU (1024 × 1024 resolution, C=16 ; HIWYN with L=3 ).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Moran’s I
Moran’s I
K-S Test
K-S Test
Std.
Skewness
Ex. Kurtosis
GPU Time
( ≈0 )
p-value ↑
(index) ↓
p-value ↑
( ≈1.0 )
( ≈0 )
( ≈0 )
ms/Latent ↓
Warping + Interpolation Operation
Nearest
0.1150
0.012
0.0107
0.115
1.0015
0.0016
-0.0116
-
Bilinear
0.6218
0.008
0.3318
0.000
0.2179
0.0720
0.4753
-
Bicubic
-0.0671
0.020
0.0670
0.000
1.0489
0.0075
0.2194
-
PYoCo [ 13 ]
0.0002
0.478
0.0033
0.512
1.000
0.0005
0.0000
-
Appendix
Table 7: Comparison of Gaussianity preservation and computational efficiency between methods that produce temporally correlated noise (1024 × 1024 resolution, C=16 ) driven by optical flow; statistics for our method are computed on Z~ (i.e., γ=0 ), and all GPU times are measured on an NVIDIA RTX PRO 6000 Blackwell GPU.
Figure 5: Motion transfer example: we demonstrate strong motion adherence to the reference video while maintaining high generation quality and fidelity. Prompt used: "Cyberpunk rooftop at night, wet reflective ground, neon city skyline, three humanoid robots standing slightly left of center in a triangular group pose, aligned like three girls in an open field. White-blue robot, black-orange robot, white-red-green robot. Wide composition, large empty foreground and right-side space, glowing Japanese signs, holographic billboards, photorealistic, cinematic, ultra detailed."
Figure 6: Example from MC-Bench: we highlight the disentanglement between the moving subject (masked planet) and the background in our method.
Figure 7: Example from MC-Bench: we highlight the disentanglement between the moving astronaut and the background in our method.
Figure 8: Example from DAVIS: our method shows better motion adherence.
Figure 9: Example from DAVIS: our method shows better motion adherence.
Figure 10: Example from DL3DV: our method shows better camera motion adherence under optical flow control.
Figure 11: Example from DL3DV: our method shows better camera motion adherence under optical flow control.
Figure 12: The effect of noise degradation level γ on generated videos. As degradation decreases, the video adheres more to the input flow.
Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.
Long Vu, Tan Ngo, Animesh Karnewar +5
Qualcomm AI Research · Trinity College Dublin, Ireland
Latent video diffusion models generate videos by progressively transforming Gaussian noise into realistic samples conditioned on text or visual inputs. However, existing conditioning methods often require additional training and computational overhead. Motivated by recent findings on the importance of frequency components in generative models, we propose a simple, training-free approach for motion-conditioned video generation by injecting low-frequency phase information from a reference video directly into the diffusion noise latents. Our method transfers motion cues without modifying the model architecture or inference pipeline. Using several applications, we demonstrate effective control over both appearance and dynamics in generated videos, while achieving competitive or superior results compared to more complex conditioning approaches.
Ofir Abramovich, Nadav Z. Cohen, Adi Rosenthal +1
Canvas-Lab Department of Computer Science Reichman University
Despite tremendous recent progress in human video generation, generative video diffusion models still struggle to capture the dynamics and physics of human motions faithfully. In this paper, we propose a new framework for human video generation, HumANDiff, which enhances the human motion control with three key designs: 1) Articulated motion-consistent noise sampling that correlates the spatiotemporal distribution of latent noise and replaces the unstructured random Gaussian noise with 3D articulated noise sampled on the dense surface manifold of a statistical human body template. It inherits body topology priors for spatially and temporally consistent noise sampling. 2) Joint appearance-motion learning that enhances the standard training objective of video diffusion models by jointly predicting pixel appearances and corresponding physical motions from the articulated noises. It enables high-fidelity human video synthesis, e.g., capturing motion-dependent clothing wrinkles. 3) Geometric motion consistency learning that enforces physical motion consistency across frames via a novel geometric motion consistency loss defined in the articulated noise space. HumANDiff enables scalable controllable human video generation by fine-tuning video diffusion models with articulated noise sampling. Consequently, our method is agnostic to diffusion model design, and requires no modifications to the model architecture. During inference, HumANDiff enables image-to-video generation within a single framework, achieving intrinsic motion control without requiring additional motion modules. Extensive experiments demonstrate that our method achieves state-of-the-art performance in rendering motion-consistent, high-fidelity humans with diverse clothing styles. Project page: https://taohuumd.github.io/projects/HumANDiff/