Human Video Generation from a Single Image with 3D Pose and View Control

Tiantian Wang; Chun-Han Yao; Tao Hu; Mallikarjun Byrasandra Ramalinga Reddy; Ming-Hsuan Yang; Varun Jampani

Figure 1: HVG Overview. HVG is capable of generating consistent multi-view human videos from a single image, conditioned on the given multi-view pose sequences and camera poses.

Figure 2: Framework of HVG. The bone map sequence is processed by the pose modulator, followed by the denoising process of DenoisingNet, which integrates camera parameters via camera embedding and time steps to generate a multi-view human video. DenoisingNet consists of convolutional blocks, spatial attention, view attention, and temporal attention to capture temporal and spatial correspondences. The reference image contributes in three key ways: First, ReferenceNet extracts fine-grained details to enhance spatial attention. Second, semantic features are captured through the CLIP Encoder for convolutional blocks and view attention and are fused with multi-frame noise. Third, the VAE Encoder processes reference image features for temporal attention.

Figure 3: Human position alignment. The left and right figures display the human subject across different views, both without and with alignment. After alignment, the human subject is positioned consistently in the same location across views.

Figure 4: Illustration of spatio-temporal sampling. To generate a multi-view long video, we divide the sequence into overlapping segments along both the temporal and view dimensions, denoted as {

\mathcal{ST}_{t}^{i}

|i=1,2,\ldots\}

and {

\mathcal{SV}_{t}^{j}

|j=1,2,\ldots\}

. These segments are independently aggregated to form long-range latent representations

\mathbf{z}_{t}^{\text{ST}}

and

\mathbf{z}_{t}^{\text{SV}}

in the temporal and view dimensions at each timestep

t

. At each timestep

t

, the denoised temporal latent

\mathbf{z}_{t}^{\text{ST}}

and view latent

\mathbf{z}_{t}^{\text{SV}}

are combined through a learned weighting strategy to produce the updated latent feature

\mathbf{z}_{t-1}

. Repeating this denoising process until

t=1

yields

\mathbf{z}_{0}

, which is then decoded to synthesize the final long multi-view human video.

Method	FID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	L1 $\downarrow$	FID-VID $\downarrow$	FVD $\downarrow$
MagicAnimate [ 22 ]	363.1	0.756	8.227	0.455	2.8e-04	161.3
AnimateAnyone [ 21 ]	117.5	0.868	17.48	0.179	4.6e-05	81.15	660.7
Champ [ 3 ]	126.1	0.886	17.74	0.108	2.8e-05	34.00	410.2
MimicMotion [ 28 ]	223.6	0.905	19.59	0.082	3.2e-05	24.55	271.1
AniGS [ 44 ]	101.5	0.900	18.97	0.084	3.0e-05	28.62	262.0
LHM [ 45 ]	81.60	0.907	19.82	0.075	1.8e-05	25.80	248.4

Table 1: Comparison for novel view synthesis on 3D scans with image & video metrics.

*

indicates that we fine-tuned the model on our dataset. Our method outperforms prior work and demonstrates clear effectiveness in learning accurate 3D viewpoint transformations.

Figure 5: Novel-view results from single-view images. HVG (ours) generates novel-view images with higher anatomic fidelity, faithfulness to input images, and consistency of geometry and texture.

Method	FID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	L1 $\downarrow$	FID-VID $\downarrow$	FVD $\downarrow$
MagicAnimate [ 22 ]	219.1	0.722	8.173	0.478	1.8e-04	211.0	1197
AnimateAnyone [ 21 ]	154.1	0.843	16.71	0.218	3.8e-05	96.12	718.4
Champ [ 3 ]	158.2	0.858	16.29	0.145	2.7e-05	53.88	468.0
MimicMotion [ 28 ]	120.5	0.876	18.26	0.119	2.8e-05	34.98	291.0
AniGS [ 44 ]	105.3	0.871	17.36	0.127	2.9e-05	36.31	299.3
LHM [ 45 ]	103.7	0.884	18.43	0.110	1.8e-05	28.86	258.8

Table 2: Comparison for novel view and novel pose synthesis on multi-view videos.

*

indicates that we fine-tuned the model on our dataset. Our approach HVG surpasses existing methods by effectively addressing both camera viewpoint differences and human motion variations at the same time.

Figure 6: Novel-view novel-pose results from single-view images. HVG generates novel-view images with higher anatomic fidelity, faithfulness to inputs, and consistency of geometry and texture.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
HVG (w/o normal map)	20.34	0.068	205.7
HVG (w/o depth map)	20.69	0.063	186.2
HVG (w/ skeleton map)	18.52	0.088	277.5
HVG (w/ SMPL body normal map)	19.61	0.073	230.6
HVG	22.13	0.057	152.1

Table 3: Ablation study on bone maps. Removing normal and depth maps leads to clear performance degradation, while combining these complementary cues yields the best results. Bone maps also outperform using only the skeleton map or the SMPL body normal map, underscoring their effectiveness in addressing occlusion and reducing shape-leakage artifacts.

Figure 7: Ablation study on bone maps. Without either the normal or depth input, the generated human exhibits clear clothing artifacts, highlighting the importance of these geometric cues. Skeleton-only input causes limb misplacement, and SMPL-normal–only input leads to exaggerated clothing contours. Our full model HVG resolves both artifacts by introducing the depth and normal bone maps.

Figure 8: Effectiveness of our spatio-temporal sampling strategy. We compare our results for two adjacent views with only temporal sampling in the left image. The right image shows results for two adjacent frames when only spatial window sampling is applied.

Figure 9: Failure case. In generating full-body 4D human videos, HVG may introduce artifacts in facial regions due to the inherent trade-off between capturing global structure and preserving local fidelity.

Figure 10: Novel-view results from single-view images. HVG avoids the head distortions produced by MagicAnimate and predicts textures closer to the ground truth.

Figure 11: Novel-view results from single-view images. HVG avoids the head distortions produced by AnimateAnyone and predicts textures closer to the ground truth.

Figure 12: Novel-view results from single-view images. Compared with Champ, HVG eliminates head distortions and produces sharper, more detailed textures.

Figure 13: Novel-view results from single-view images. Compared with MimicMotion, HVG produces sharper, more detailed textures.

Figure 14: Novel-view results from single-view images. AniGS struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.

Figure 15: Novel-view results from single-view images. LHM struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation.

Figure 16: Novel-view novel-pose results from single-view images. HVG avoids the head distortions produced by MagicAnimate and predicts textures closer to the ground truth.

Figure 17: Novel-view novel-pose results from single-view images. HVG avoids the head distortions produced by AnimateAnyone and predicts textures closer to the ground truth.

Figure 18: Novel-view novel-pose results from single-view images. Compared with Champ, HVG produces sharper, more detailed textures.

Figure 19: Novel-view novel-pose results from single-view images. Compared with MimicMotion, HVG produces sharper, more detailed textures.

Figure 20: Novel-view novel-pose results from single-view images. AniGS struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.

Figure 21: Novel-view novel-pose results from single-view images. LHM struggles to generate motion-dependent textures, whereas HVG effectively addresses this limitation and generates more clearer textures on the clothes.

Human Video Generation from a Single Image with 3D Pose and View Control

Abstract

Figures & tables

Explore similar work

ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis

Flex4DHuman: Flexible Multi-view Video Diffusion for 4D Human Reconstruction

HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation