cs.CVApr 7, 2026

HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation

Authors: Tao Hu, Varun Jampani

Organizations: Stability AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite tremendous recent progress in human video generation, generative video diffusion models still struggle to capture the dynamics and physics of human motions faithfully. In this paper, we propose a new framework for human video generation, HumANDiff, which enhances the human motion control with three key designs: 1) Articulated motion-consistent noise sampling that correlates the spatiotemporal distribution of latent noise and replaces the unstructured random Gaussian noise with 3D articulated noise sampled on the dense surface manifold of a statistical human body template. It inherits body topology priors for spatially and temporally consistent noise sampling. 2) Joint appearance-motion learning that enhances the standard training objective of video diffusion models by jointly predicting pixel appearances and corresponding physical motions from the articulated noises. It enables high-fidelity human video synthesis, e.g., capturing motion-dependent clothing wrinkles. 3) Geometric motion consistency learning that enforces physical motion consistency across frames via a novel geometric motion consistency loss defined in the articulated noise space. HumANDiff enables scalable controllable human video generation by fine-tuning video diffusion models with articulated noise sampling. Consequently, our method is agnostic to diffusion model design, and requires no modifications to the model architecture. During inference, HumANDiff enables image-to-video generation within a single framework, achieving intrinsic motion control without requiring additional motion modules. Extensive experiments demonstrate that our method achieves state-of-the-art performance in rendering motion-consistent, high-fidelity humans with diverse clothing styles. Project page: https://taohuumd.github.io/projects/HumANDiff/

Figures & tables

Figure 1: HumANDiff generates temporally consistent human videos from a single image by sampling motion-consistent noise on the UV manifold of a parametric body model ( e.g . , SMPL [ 36 ] ), enabling intrinsic motion control without explicit motion adapters.
Figure 2: Training framework. During training, HumANDiff learns to generate motion-controllable human videos using a latent video diffusion model with four components: 1) Video Encoding : we employ a 3D VAE to encode video segments x0\bm{x_{0}} into latent space. 2) Articulated Motion Consistent Noise Sampling (Sec. 3.2 ): We model motions on the surface UV manifold of SMPL [ 36 ] and integrate motion control m0\bm{m_{0}} as a warping field to sample noise ϵ\bm{\epsilon} from a noise texture UV map. 3) Joint Appearance-Motion Learning (Sec. 3.3 ): The diffusion model is enhanced to learn both video and motion for latent denoising to improve motion coherence. 4) Geometric Motion Consistency Learning (Sec. 3.3 ): We further propose a motion consistency loss to enforce motion consistency across frames between the predicted noise ϵuv′\bm{\epsilon^{\prime}_{uv}} and sampled noise ϵuv\bm{\epsilon_{uv}} in UV space.
Figure 3
Figure 4: Comparisons against Animate Anyone [ 21 ] and Wan2.1 based UniAnimate [ 69 ] . Our method can generate high-quality clothing details ( \raisebox{-0.9pt}{1}⃝ , \raisebox{-0.9pt}{3}⃝ ), and face details ( \raisebox{-0.9pt}{2}⃝ ).
Figure 6: Generalization results. Our method trained on fashion videos generalizes well to unseen out-of-domain data, and synthesizes human novel pose videos from a single image in diverse clothing styles (a), and diverse in-the-wild backgrounds (b).
Figure 7: Ablation study of Joint Appearance-Motion Learning (JAML). JAML improves motion coherence for faithful motion control, while the variant without JAML failed to generate accurate animation videos.
Figure 8: Ablation study of Geometric Motion Consistency Learning (GMCL), where two frames at different time steps are shown. GMCL improves appearance consistency, i.e. , it preserves the face identity details better ( \raisebox{-0.9pt}{2}⃝ vs. \raisebox{-0.9pt}{1}⃝ , and \raisebox{-0.9pt}{4}⃝ vs. \raisebox{-0.9pt}{3}⃝ ).
 L1 ↓\downarrowSSIM ↑\uparrowLPIPS ↓\downarrowFID ↓\downarrowFVD ↓\downarrowFID-VID ↓\downarrow
Base + Articulated Noise Sampling.0378.903.07110.4712.03131.30
+ JAML.0371.905.0659.0210.00117.68
+ GMCL.0364.906.0577.944.6884.10
Table 2: Ablation study results for JAML and GMCL. Note that perceptual metrics ( e.g . , LPIPS, FID, FVD) are generally more reliable than pixel-level metrics ( e.g . , SSIM and L1) for evaluating human generation tasks [ 32 ] .
Figure 9: Generation results of model variants trained on Wan2.1-I2V-14B and CogVideoX-I2V-5B. Our method is model agnostic, and the model trained on Wan2.1-I2V-14B generates higher quality appearance details ( \raisebox{-0.9pt}{1}⃝ \raisebox{-0.9pt}{2}⃝ \raisebox{-0.9pt}{3}⃝ ).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Ablation study of noise degradation. We set γ=0.1\gamma=0.1 to achieve a balance between motion control and video quality.
 LPIPS ↓\downarrowSSIM ↑\uparrow
γ=0\gamma=0.0857.858
γ=0.1\gamma=0.1.0821.860
γ=0.2\gamma=0.2.0859.860
γ=0.3\gamma=0.3.0898.858
γ=0.4\gamma=0.4.1088.841
γ=0.5\gamma=0.5.1184.834
Appendix
Table 3: Ablation study of noise degradation. γ\gamma measures how strictly the resulting video adheres to the motion input.
Figure 11: Illustration of video VAE encoding in CogVideoX. Fine details such as faces and hands are often lost during video encoding, which makes it challenging for our model to generate high-quality face and hand details.

Explore similar work

CardsList
Apr 21, 2026

ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis

Human video generation remains challenging due to the difficulty of jointly modeling human appearance, motion, and camera viewpoint under limited multi-view data. Existing methods often address these factors separately, resulting in limited controllability or reduced visual quality. We revisit this problem from an image-first perspective, where high-quality human appearance is learned via image generation and used as a prior for video synthesis, decoupling appearance modeling from temporal consistency. We propose a pose- and viewpoint-controllable pipeline that combines a pretrained image backbone with SMPL-X-based motion guidance, together with a training-free temporal refinement stage based on a pretrained video diffusion model. Our method produces high-quality, temporally consistent videos under diverse poses and viewpoints. We also release a canonical human dataset and an auxiliary model for compositional human image synthesis. Code and data are publicly available at https://github.com/Taited/ReImagine.

Zhengwentai Sun, Keru Zheng, Chenghong Li +7

School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Future Network of Intelligence Institute, CUHK-Shenzhen, China

Camera-Controlled Video GenerationVideo GenerationPose-Conditioned Video GenerationControllable Video GenerationTemporal Consistency in Video Generation
AskPDF
Feb 24, 2026

Human Video Generation from a Single Image with 3D Pose and View Control

Recent diffusion methods have made significant progress in generating videos from single images due to their powerful visual generation capabilities. However, challenges persist in image-to-video synthesis, particularly in human video generation, where inferring view-consistent, motion-dependent clothing wrinkles from a single image remains a formidable problem. In this paper, we present Human Video Generation in 4D (HVG), a latent video diffusion model capable of generating high-quality, multi-view, spatiotemporally coherent human videos from a single image with 3D pose and view control. HVG achieves this through three key designs: (i) Articulated Pose Modulation, which captures the anatomical relationships of 3D joints via a novel dual-dimensional bone map and resolves self-occlusions across views by introducing 3D information; (ii) View and Temporal Alignment, which ensures multi-view consistency and alignment between a reference image and pose sequences for frame-to-frame stability; and (iii) Progressive Spatio-Temporal Sampling with temporal alignment to maintain smooth transitions in long multi-view animations. Extensive experiments on image-to-video tasks demonstrate that HVG outperforms existing methods in generating high-quality 4D human videos from diverse human images and pose inputs.

Tiantian Wang, Chun-Han Yao, Tao Hu +3

Electrical Engineering and Computer Science, University of California, Merced, CA 95343, United States. · Stability AI, Los Angeles, CA 90067, United States.

Image-to-Video GenerationVideo GenerationPose-Conditioned Video GenerationMulti-View Video GenerationVideo Diffusion ModelsCamera-Controlled Video Generation
AskPDF
Jun 11, 2024

RDM: Recurrent Diffusion Model for Human Motion Generation

Human motion generation is a challenging task due to its high dimensionality and the difficulty of generating fine-grained motions. Diffusion methods have been proposed due to their high sample quality and expressiveness. Early approaches treat the entire sequence as a whole, which is computationally expensive and restricts sequence length. In contrast, autoregressive diffusion models generate longer sequences. However, their reliance on fully denoising previous frames complicates training and inference. Consequently, we propose RDM, a new recurrent diffusion formulation inspired by Recurrent Neural Networks (RNNs). RDM explicitly conditions the diffusion process on preceding noisy frames, avoiding the cost of full denoising. To overcome the challenge of maintaining its probabilistic nature, we employ Normalizing Flows to model recurrent connections. Our evaluations demonstrate RDM's effectiveness: it achieves comparable performance to autoregressive baselines and generates long sequences that remain aligned with the text. RDM also skips diffusion steps during inference, reducing computational cost by up to 18x over autoregressive baselines. Project page: https://www.mirgahney.com/rfdm.io/.

Mirgahney Mohamed, Harry Jake Cunningham, Riccardo Barbano +2

Department of Computer Science, University College London

Diffusion Models for Motion GenerationDiffusion ModelsText-to-Motion GenerationDiffusion Model Inference Acceleration
AskPDF