cs.CVApr 7, 2026

HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation

Authors: Tao Hu, Varun Jampani

Organizations: Stability AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite tremendous recent progress in human video generation, generative video diffusion models still struggle to capture the dynamics and physics of human motions faithfully. In this paper, we propose a new framework for human video generation, HumANDiff, which enhances the human motion control with three key designs: 1) Articulated motion-consistent noise sampling that correlates the spatiotemporal distribution of latent noise and replaces the unstructured random Gaussian noise with 3D articulated noise sampled on the dense surface manifold of a statistical human body template. It inherits body topology priors for spatially and temporally consistent noise sampling. 2) Joint appearance-motion learning that enhances the standard training objective of video diffusion models by jointly predicting pixel appearances and corresponding physical motions from the articulated noises. It enables high-fidelity human video synthesis, e.g., capturing motion-dependent clothing wrinkles. 3) Geometric motion consistency learning that enforces physical motion consistency across frames via a novel geometric motion consistency loss defined in the articulated noise space. HumANDiff enables scalable controllable human video generation by fine-tuning video diffusion models with articulated noise sampling. Consequently, our method is agnostic to diffusion model design, and requires no modifications to the model architecture. During inference, HumANDiff enables image-to-video generation within a single framework, achieving intrinsic motion control without requiring additional motion modules. Extensive experiments demonstrate that our method achieves state-of-the-art performance in rendering motion-consistent, high-fidelity humans with diverse clothing styles. Project page: https://taohuumd.github.io/projects/HumANDiff/

Figures & tables

Figure 1: HumANDiff generates temporally consistent human videos from a single image by sampling motion-consistent noise on the UV manifold of a parametric body model ( e.g . , SMPL [ 36 ] ), enabling intrinsic motion control without explicit motion adapters.
Figure 2: Training framework. During training, HumANDiff learns to generate motion-controllable human videos using a latent video diffusion model with four components: 1) Video Encoding : we employ a 3D VAE to encode video segments x0\bm{x_{0}} into latent space. 2) Articulated Motion Consistent Noise Sampling (Sec. 3.2 ): We model motions on the surface UV manifold of SMPL [ 36 ] and integrate motion control m0\bm{m_{0}} as a warping field to sample noise ϵ\bm{\epsilon} from a noise texture UV map. 3) Joint Appearance-Motion Learning (Sec. 3.3 ): The diffusion model is enhanced to learn both video and motion for latent denoising to improve motion coherence. 4) Geometric Motion Consistency Learning (Sec. 3.3 ): We further propose a motion consistency loss to enforce motion consistency across frames between the predicted noise ϵuv′\bm{\epsilon^{\prime}_{uv}} and sampled noise ϵuv\bm{\epsilon_{uv}} in UV space.
Figure 3
Figure 4: Comparisons against Animate Anyone [ 21 ] and Wan2.1 based UniAnimate [ 69 ] . Our method can generate high-quality clothing details ( \raisebox{-0.9pt}{1}⃝ , \raisebox{-0.9pt}{3}⃝ ), and face details ( \raisebox{-0.9pt}{2}⃝ ).
Figure 6: Generalization results. Our method trained on fashion videos generalizes well to unseen out-of-domain data, and synthesizes human novel pose videos from a single image in diverse clothing styles (a), and diverse in-the-wild backgrounds (b).
Figure 7: Ablation study of Joint Appearance-Motion Learning (JAML). JAML improves motion coherence for faithful motion control, while the variant without JAML failed to generate accurate animation videos.
Figure 8: Ablation study of Geometric Motion Consistency Learning (GMCL), where two frames at different time steps are shown. GMCL improves appearance consistency, i.e. , it preserves the face identity details better ( \raisebox{-0.9pt}{2}⃝ vs. \raisebox{-0.9pt}{1}⃝ , and \raisebox{-0.9pt}{4}⃝ vs. \raisebox{-0.9pt}{3}⃝ ).
 L1 ↓\downarrowSSIM ↑\uparrowLPIPS ↓\downarrowFID ↓\downarrowFVD ↓\downarrowFID-VID ↓\downarrow
Base + Articulated Noise Sampling.0378.903.07110.4712.03131.30
+ JAML.0371.905.0659.0210.00117.68
+ GMCL.0364.906.0577.944.6884.10
Table 2: Ablation study results for JAML and GMCL. Note that perceptual metrics ( e.g . , LPIPS, FID, FVD) are generally more reliable than pixel-level metrics ( e.g . , SSIM and L1) for evaluating human generation tasks [ 32 ] .
Figure 9: Generation results of model variants trained on Wan2.1-I2V-14B and CogVideoX-I2V-5B. Our method is model agnostic, and the model trained on Wan2.1-I2V-14B generates higher quality appearance details ( \raisebox{-0.9pt}{1}⃝ \raisebox{-0.9pt}{2}⃝ \raisebox{-0.9pt}{3}⃝ ).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Ablation study of noise degradation. We set γ=0.1\gamma=0.1 to achieve a balance between motion control and video quality.
 LPIPS ↓\downarrowSSIM ↑\uparrow
γ=0\gamma=0.0857.858
γ=0.1\gamma=0.1.0821.860
γ=0.2\gamma=0.2.0859.860
γ=0.3\gamma=0.3.0898.858
γ=0.4\gamma=0.4.1088.841
γ=0.5\gamma=0.5.1184.834
Appendix
Table 3: Ablation study of noise degradation. γ\gamma measures how strictly the resulting video adheres to the motion input.
Figure 11: Illustration of video VAE encoding in CogVideoX. Fine details such as faces and hands are often lost during video encoding, which makes it challenging for our model to generate high-quality face and hand details.

Explore similar work

CardsList

  1. ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis

    Apr 21, 2026Zhengwentai Sun, Keru Zheng, Chenghong Li +7Camera-Controlled Video GenerationVideo Generation

    School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Future Network of Intelligence Institute, CUHK-Shenzhen, China

  2. Human Video Generation from a Single Image with 3D Pose and View Control

    Feb 24, 2026Tiantian Wang, Chun-Han Yao, Tao Hu +3Image-to-Video GenerationVideo Generation

    Electrical Engineering and Computer Science, University of California, Merced, CA 95343, United States. · Stability AI, Los Angeles, CA 90067, United States.

  3. RDM: Recurrent Diffusion Model for Human Motion Generation

    Jun 11, 2024Mirgahney Mohamed, Harry Jake Cunningham, Riccardo Barbano +2Diffusion Models for Motion GenerationDiffusion Models

    Department of Computer Science, University College London