Organizations: Electrical Engineering and Computer Science, University of California, Merced, CA 95343, United States. · Stability AI, Los Angeles, CA 90067, United States.
Recent diffusion methods have made significant progress in generating videos from single images due to their powerful visual generation capabilities. However, challenges persist in image-to-video synthesis, particularly in human video generation, where inferring view-consistent, motion-dependent clothing wrinkles from a single image remains a formidable problem. In this paper, we present Human Video Generation in 4D (HVG), a latent video diffusion model capable of generating high-quality, multi-view, spatiotemporally coherent human videos from a single image with 3D pose and view control. HVG achieves this through three key designs: (i) Articulated Pose Modulation, which captures the anatomical relationships of 3D joints via a novel dual-dimensional bone map and resolves self-occlusions across views by introducing 3D information; (ii) View and Temporal Alignment, which ensures multi-view consistency and alignment between a reference image and pose sequences for frame-to-frame stability; and (iii) Progressive Spatio-Temporal Sampling with temporal alignment to maintain smooth transitions in long multi-view animations. Extensive experiments on image-to-video tasks demonstrate that HVG outperforms existing methods in generating high-quality 4D human videos from diverse human images and pose inputs.
Figures & tables
Figure 1: HVG Overview. HVG is capable of generating consistent multi-view human videos from a single image, conditioned on the given multi-view pose sequences and camera poses.
Figure 2: Framework of HVG. The bone map sequence is processed by the pose modulator, followed by the denoising process of DenoisingNet, which integrates camera parameters via camera embedding and time steps to generate a multi-view human video. DenoisingNet consists of convolutional blocks, spatial attention, view attention, and temporal attention to capture temporal and spatial correspondences. The reference image contributes in three key ways: First, ReferenceNet extracts fine-grained details to enhance spatial attention. Second, semantic features are captured through the CLIP Encoder for convolutional blocks and view attention and are fused with multi-frame noise. Third, the VAE Encoder processes reference image features for temporal attention.
Figure 3: Human position alignment. The left and right figures display the human subject across different views, both without and with alignment. After alignment, the human subject is positioned consistently in the same location across views.
Figure 4: Illustration of spatio-temporal sampling. To generate a multi-view long video, we divide the sequence into overlapping segments along both the temporal and view dimensions, denoted as { STti∣i=1,2,…} and { SVtj∣j=1,2,…} . These segments are independently aggregated to form long-range latent representations ztST and ztSV in the temporal and view dimensions at each timestep t . At each timestep t , the denoised temporal latent ztST and view latent ztSV are combined through a learned weighting strategy to produce the updated latent feature zt−1 . Repeating this denoising process until t=1 yields z0 , which is then decoded to synthesize the final long multi-view human video.
Method
FID ↓
SSIM ↑
PSNR ↑
LPIPS ↓
L1 ↓
FID-VID ↓
FVD ↓
MagicAnimate [ 22 ]
363.1
0.756
8.227
0.455
2.8e-04
161.3
AnimateAnyone [ 21 ]
117.5
0.868
17.48
0.179
4.6e-05
81.15
660.7
Champ [ 3 ]
126.1
0.886
17.74
0.108
2.8e-05
34.00
410.2
MimicMotion [ 28 ]
223.6
0.905
19.59
0.082
3.2e-05
24.55
271.1
AniGS [ 44 ]
101.5
0.900
18.97
0.084
3.0e-05
28.62
262.0
LHM [ 45 ]
81.60
0.907
19.82
0.075
1.8e-05
25.80
248.4
Table 1: Comparison for novel view synthesis on 3D scans with image & video metrics. ∗ indicates that we fine-tuned the model on our dataset. Our method outperforms prior work and demonstrates clear effectiveness in learning accurate 3D viewpoint transformations.
Figure 5: Novel-view results from single-view images. HVG (ours) generates novel-view images with higher anatomic fidelity, faithfulness to input images, and consistency of geometry and texture.
Method
FID ↓
SSIM ↑
PSNR ↑
LPIPS ↓
L1 ↓
FID-VID ↓
FVD ↓
MagicAnimate [ 22 ]
219.1
0.722
8.173
0.478
1.8e-04
211.0
1197
AnimateAnyone [ 21 ]
154.1
0.843
16.71
0.218
3.8e-05
96.12
718.4
Champ [ 3 ]
158.2
0.858
16.29
0.145
2.7e-05
53.88
468.0
MimicMotion [ 28 ]
120.5
0.876
18.26
0.119
2.8e-05
34.98
291.0
AniGS [ 44 ]
105.3
0.871
17.36
0.127
2.9e-05
36.31
299.3
LHM [ 45 ]
103.7
0.884
18.43
0.110
1.8e-05
28.86
258.8
Table 2: Comparison for novel view and novel pose synthesis on multi-view videos. ∗ indicates that we fine-tuned the model on our dataset. Our approach HVG surpasses existing methods by effectively addressing both camera viewpoint differences and human motion variations at the same time.
Figure 6: Novel-view novel-pose results from single-view images. HVG generates novel-view images with higher anatomic fidelity, faithfulness to inputs, and consistency of geometry and texture.
Method
PSNR ↑
LPIPS ↓
FVD ↓
HVG (w/o normal map)
20.34
0.068
205.7
HVG (w/o depth map)
20.69
0.063
186.2
HVG (w/ skeleton map)
18.52
0.088
277.5
HVG (w/ SMPL body normal map)
19.61
0.073
230.6
HVG
22.13
0.057
152.1
Table 3: Ablation study on bone maps. Removing normal and depth maps leads to clear performance degradation, while combining these complementary cues yields the best results. Bone maps also outperform using only the skeleton map or the SMPL body normal map, underscoring their effectiveness in addressing occlusion and reducing shape-leakage artifacts.
Figure 7: Ablation study on bone maps. Without either the normal or depth input, the generated human exhibits clear clothing artifacts, highlighting the importance of these geometric cues. Skeleton-only input causes limb misplacement, and SMPL-normal–only input leads to exaggerated clothing contours. Our full model HVG resolves both artifacts by introducing the depth and normal bone maps.
Figure 8: Effectiveness of our spatio-temporal sampling strategy. We compare our results for two adjacent views with only temporal sampling in the left image. The right image shows results for two adjacent frames when only spatial window sampling is applied.
Figure 9: Failure case. In generating full-body 4D human videos, HVG may introduce artifacts in facial regions due to the inherent trade-off between capturing global structure and preserving local fidelity.
Figure 10: Novel-view results from single-view images. HVG avoids the head distortions produced by MagicAnimate and predicts textures closer to the ground truth.
Figure 11: Novel-view results from single-view images. HVG avoids the head distortions produced by AnimateAnyone and predicts textures closer to the ground truth.
Figure 12: Novel-view results from single-view images. Compared with Champ, HVG eliminates head distortions and produces sharper, more detailed textures.
Figure 13: Novel-view results from single-view images. Compared with MimicMotion, HVG produces sharper, more detailed textures.
Figure 14: Novel-view results from single-view images. AniGS struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.
Figure 15: Novel-view results from single-view images. LHM struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation.
Figure 16: Novel-view novel-pose results from single-view images. HVG avoids the head distortions produced by MagicAnimate and predicts textures closer to the ground truth.
Figure 17: Novel-view novel-pose results from single-view images. HVG avoids the head distortions produced by AnimateAnyone and predicts textures closer to the ground truth.
Figure 18: Novel-view novel-pose results from single-view images. Compared with Champ, HVG produces sharper, more detailed textures.
Figure 19: Novel-view novel-pose results from single-view images. Compared with MimicMotion, HVG produces sharper, more detailed textures.
Figure 20: Novel-view novel-pose results from single-view images. AniGS struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.
Figure 21: Novel-view novel-pose results from single-view images. LHM struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.
Human video generation remains challenging due to the difficulty of jointly modeling human appearance, motion, and camera viewpoint under limited multi-view data. Existing methods often address these factors separately, resulting in limited controllability or reduced visual quality. We revisit this problem from an image-first perspective, where high-quality human appearance is learned via image generation and used as a prior for video synthesis, decoupling appearance modeling from temporal consistency. We propose a pose- and viewpoint-controllable pipeline that combines a pretrained image backbone with SMPL-X-based motion guidance, together with a training-free temporal refinement stage based on a pretrained video diffusion model. Our method produces high-quality, temporally consistent videos under diverse poses and viewpoints. We also release a canonical human dataset and an auxiliary model for compositional human image synthesis. Code and data are publicly available at https://github.com/Taited/ReImagine.
Zhengwentai Sun, Keru Zheng, Chenghong Li +7
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Future Network of Intelligence Institute, CUHK-Shenzhen, China
We present Flex4DHuman, a multi-view video diffusion model that transforms a monocular or sparse multi-view video of a dynamic subject into synchronized dense multi-view videos using only relative camera-pose conditioning. Unlike prior human-centric methods that rely on skeletons, depth maps, normals, or rendered target-view geometry, Flex4DHuman requires no explicit geometry priors and instead conditions generation through relative camera-pose positional encoding. The generated videos can be directly ingested by downstream reconstruction pipelines to create dynamic 4D Gaussian splats. Built on the Wan 2.1 1.3B text-to-video model, Flex4DHuman preserves the backbone architecture and encodes camera and view information through a five-axis positional encoding that extends spatio-temporal RoPE with view indices and continuous SE(3) relative camera geometry. A three-stage curriculum progressively trains the model for pose following, flexible reference-to-target view generation, and temporal rollout. To support temporal rollout, we train with clean historical target-view tokens. We also add multi-view captions to enable test-time text control. Combined with an off-the-shelf 4D Gaussian Splatting stage, our framework lifts monocular static-camera videos into dynamic 4D Gaussian splats. Experiments on DNA-Rendering and ActorsHQ show that Flex4DHuman surpasses prior state-of-the-art methods, while the same formulation generalizes to animal categories after mixed human-animal training. These capabilities make Flex4DHuman a practical step toward scalable 4D content creation from casual monocular videos for simulation, gaming, AR/VR, and video re-shooting.
Despite tremendous recent progress in human video generation, generative video diffusion models still struggle to capture the dynamics and physics of human motions faithfully. In this paper, we propose a new framework for human video generation, HumANDiff, which enhances the human motion control with three key designs: 1) Articulated motion-consistent noise sampling that correlates the spatiotemporal distribution of latent noise and replaces the unstructured random Gaussian noise with 3D articulated noise sampled on the dense surface manifold of a statistical human body template. It inherits body topology priors for spatially and temporally consistent noise sampling. 2) Joint appearance-motion learning that enhances the standard training objective of video diffusion models by jointly predicting pixel appearances and corresponding physical motions from the articulated noises. It enables high-fidelity human video synthesis, e.g., capturing motion-dependent clothing wrinkles. 3) Geometric motion consistency learning that enforces physical motion consistency across frames via a novel geometric motion consistency loss defined in the articulated noise space. HumANDiff enables scalable controllable human video generation by fine-tuning video diffusion models with articulated noise sampling. Consequently, our method is agnostic to diffusion model design, and requires no modifications to the model architecture. During inference, HumANDiff enables image-to-video generation within a single framework, achieving intrinsic motion control without requiring additional motion modules. Extensive experiments demonstrate that our method achieves state-of-the-art performance in rendering motion-consistent, high-fidelity humans with diverse clothing styles. Project page: https://taohuumd.github.io/projects/HumANDiff/