Ani3DHuman: Photorealistic 3D Human Animation with Self-guided Stochastic Sampling

Qi Sun; Can Wang; Jiaxiang Shang; Yingchun Liu; Jing Liao

cs.CVFeb 22, 2026

Ani3DHuman: Photorealistic 3D Human Animation with Self-guided Stochastic Sampling

Authors: Qi Sun, Can Wang, Jiaxiang Shang, Yingchun Liu, Jing Liao

Organizations: City University of Hong Kong

Abstract

Current 3D human animation methods struggle to achieve photorealism: kinematics-based approaches lack non-rigid dynamics (e.g., clothing dynamics), while methods that leverage video diffusion priors can synthesize non-rigid motion but suffer from quality artifacts and identity loss. To overcome these limitations, we present Ani3DHuman, a framework that marries kinematics-based animation with video diffusion priors. We first introduce a layered motion representation that disentangles rigid motion from residual non-rigid motion. Rigid motion is generated by a kinematic method, which then produces a coarse rendering to guide the video diffusion model in generating video sequences that restore the residual non-rigid motion. However, this restoration task, based on diffusion sampling, is highly challenging, as the initial renderings are out-of-distribution, causing standard deterministic ODE samplers to fail. Therefore, we propose a novel self-guided stochastic sampling method, which effectively addresses the out-of-distribution problem by combining stochastic sampling (for photorealistic quality) with self-guidance (for identity fidelity). These restored videos provide high-quality supervision, enabling the optimization of the residual non-rigid motion field. Extensive experiments demonstrate that \MethodName can generate photorealistic 3D human animation, outperforming existing methods. Code is available in https://github.com/qiisun/ani3dhuman.

Figures & tables

Figure 1: Given a reference human image and a target SMPL mesh sequence, our method synthesizes photorealistic 3D human animation. Unlike the previous state-of-the-art (SOTA) methods (e.g., LHM [ 58 ] (top-right)) that are limited to rigid motion, our Ani3DHuman (bottom) can further generate high-fidelity nonrigid dynamics, capturing the natural flow of the dress.

Figure 2: Pipeline overview. Our Ani3DHuman animates a 3D Gaussian $\mathcal{G}$ (reconstructed with LHM [ 58 ] from the reference image) with a mesh sequence. Our layered motion combines a mesh-rigged motion with a residual field for non-rigid dynamics. A coarse rendering ${\bm{y}}$ from the rigid motion is restored to a high-quality video ${\bm{x}}^{*}$ using our self-guided stochastic sampling. This restored video ${\bm{x}}^{*}$ then provides supervision to progressively optimize the residual motion field.

Figure 3: Distribution mismatch in deterministic flow matching. Our degraded input

{\bm{y}}

(out-of-distribution, OOD) creates a noisy latent

{\bm{x}}_{t}

that is off the marginal distribution

p_{t}({\bm{x}})

. A deterministic Flow-ODE ( orange path ) follows an incorrect trajectory as its velocity predictions are inaccurate for OOD samples, resulting in a low-quality sample. This motivates our use of an SDE sampler, which can actively correct the path by driving the sample back toward the marginal distribution.

Figure 4: Diagonal view-time sampling . (a) Illustration of diagonal sampling in a view-time matrix ( $N_{\text{traj}}=3$ ). This method simultaneously evolves the camera view and time, distinct from fixed-time ( bullet-time ) or fixed-camera ( independent-view ) sampling. (b) An example trajectory shows the camera orbiting 360° as time progresses.

Figure 5: Comparison with state-of-the-art methods. Our method (Ours) is the only one to simultaneously achieve high quality, identity preservation, and realistic non-rigid motion. Existing methods fail in key areas: Disco4D [ 53 ] and SV4D 2.0 [ 87 ] suffers from low quality (due to SDS and multi-view video diffusion); PERSONA loses identity (due to direct reconstruction from pose-driven video diffusion); and LHM [ 58 ] captures identity but fails to model clothing dynamics. (* self-implementation)

Figure 6: Comparison on other video re-rendering methods. (a) original rendering ${\bm{x}}$ ; (b-f) competitive sampling methods; (g) our results ${\bm{x}}^{*}$ . Only our self-guided stochastic sampling can generate sharp details while preserving the original identity well.

Figure 7: Ablative experiment on self-guided stochastic sampling. We compare full sampling method (e) and the generation results of our model with a set of ablations. (a) Original rendering with mesh-rigged animation; (b) replacing personalized diffusion prior with general diffusion prior introduces slight performance degradation and artifacts; (c) We observe that our method produces significant quality drop when removing stochastic sampling; (d) removing self-guided sampling greatly reduces the identity preservation.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	FID $\downarrow$	FVD $\downarrow$
Disco4D [ 53 ]	12.05	0.5590	0.5019	0.6439	613.9	622.1
SV4D 2.0 [ 87 ]	15.25	0.7708	0.3773	0.7640	364.9	478.7
PERSONA [ 66 ]	17.01	0.8219	0.2602	0.8779	199.1	367.0
LHM [ 58 ]	19.51	0.8382	0.2169	0.9009	124.1	339.9
Ours	20.08	0.8312	0.2125	0.9160	105.3	295.2

Table 1: Quantitative results with the state-of-the-art methods of human animation in ActorsHQ [ 23 ] dataset.

Methods ( $\%$ )	Identity Preservation	Frame Quality	Motion Realism	(Non-rigid) Physical Plausibility	Overall Preference
Disco4D [ 53 ]	0	1.47	4.41	0	0
SV4D 2.0 [ 87 ]	0	2.94	16.2	4.34	5.89
PERSONA [ 66 ]	20.6	30.9	14.7	19.1	22.1
LHM [ 58 ]	39.2	29.4	29.4	14.7	17.6
Ours	40.1	35.3	35.3	61.8	54.4

Table 2: User study on human animation with novel motion.

Figure 8: Ablation study on motion field. Our layered motion (right) captures intricate hand details, while the single-layer baseline (left) fails.

Figure 9: Ablation study on sampling method. Baseline methods (left) suffer from significant floaters and spikes, while our diagonal sampling (right) reconstructs sharp details.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training Objective	Single-view Image Input	Skeleton-controllable	Identity Preservation	Non-rigid Motion	High-quality Rendering
Disco4D [ 53 ]	MSE+SDS	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
AKD [ 38 ]	SDS	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
PhysAvatar [ 91 ]	MSE	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
SV4D/SV4D 2.0 [ 83 , 87 ]	MSE	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$
CharacterShot [ 13 ]	MSE	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$
Human4DiT [ 65 ]	MSE	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$

Appendix

Table 3: Difference among the other human (character) animation methods. “-” means there is no optimization process in the animation.

Metrics	Coarse Model	w/o Stochastic	w/o Self-Guidance	w/o Personalized	Full Model
FID $\downarrow$	199.1	187.4	104.1	125.3	105.3
FVD $\downarrow$	367.0	349.7	298.8	301.4	295.2
CLIP-Identity $\uparrow$	0.8847	0.8804	0.8220	0.8645	0.8838

Appendix

Table 4: Quantitative ablation study of the proposed components. The results demonstrate that the full model configuration achieves the most robust performance across all evaluation metrics.

Figure 10: Sensitivity analysis of the initial noise strength $t_{0}$ . We visualize restoration results across varying noise strengths. Low noise levels ( $t_{0}=0.2,0.4$ ) fail to deviate sufficiently from the source, leaving artifacts from the coarse mesh rendering intact. Higher noise levels ( $t_{0}=0.6,0.8$ ) effectively hallucinate plausible details and correct non-rigid dynamics. Notably, thanks to our self-guidance mechanism, the identity is preserved even at high noise strengths ( $t_{0}=0.8$ ), overcoming the traditional quality-fidelity trade-off.

Panel of Figure 11 — Figure 11: Ablations on optimization strategies. (a) Adaptive densification is crucial for capturing high-frequency texture dynamics. (b) Mask loss regularization is essential to constrain the geometry. (c) Dataset update mitigates over-smoothing caused by inconsistent supervision, allowing the model to converge on sharp, clear details.

Figure 12: Comparison with image-based animation.

Figure 13: Visual comparison of sampling strategies. Cases I: Two girls are walking (Row2/4) and running (Row1/3). The Mesh-Rigged Animation (Input) exhibits unrealistic artifacts, such as unnatural cloth dynamics and blurry edges. Direct Generation suffers from severe identity shift, introducing hallucinations like a bag (Row 2) or a watch (Row 3). ODE Sampling (represented by MCS [ 76 ] ) fails to recover high-frequency details, leaving garment edges blurry due to the OOD nature of the input. In contrast, Ours successfully restores high-fidelity details and realistic motion while maintaining strict identity consistency.

Figure 14: Visual comparison of sampling strategies. Case II: Two girls are walking (Row1/3), running (Row2), dancing (Row4).

Figure 15: Qualitative evaluation on the ActorsHQ [ 23 ] dataset (I). We show single person with difference motions. The asterisk (*) denotes renderings at a specific viewpoint (elevation $10^{\circ}$ , azimuth $0^{\circ}$ ). Note that slight spatial misalignments between the generation and ground truth are due to inherent errors in the SMPL estimation derived from the source video. Despite relying on a single-view input, our method faithfully preserves human identity and captures complex non-rigid deformations (e.g., dress dynamics), even during extreme poses such as high leg raises.

Figure 16: Human reconstruction results in ActorsHQ [ 23 ] dataset (II). We show different person with diverse motions.

Figure 17: Additional human animation results. We visualize diverse subjects performing various motions, rendered with dynamic 360-degree camera trajectories.

Metrics	Champ	Uni3C	Personalized Diffusion	Ours
FID $\downarrow$	196.3	132.3	138.5	112.8
FVD $\downarrow$	467.2	284.4	330.6	289.0
CLIP-Identity $\uparrow$	0.7633	0.8357	0.8001	0.8844

Appendix

Table 5: Quantitative comparison with state-of-the-art image-driven animation methods. Our method achieves a superior balance between motion fidelity and identity preservation.

Human Avatar Animation

Human avatar animation builds digital people that can be reconstructed, driven by motion or speech, and rendered consistently across poses and views.

See all 91 papers

LUNA: Learning Universal 3D Human Animation Beyond Skinning
Jun 30, 2026Peng Li, Rawal Khirodkar, Junxuan Li +6Human Motion Generation Human Avatar Animation
The Hong Kong University of Science and Technology · Work was done during an internship at Meta. · Codec Avatars Lab, Meta
3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement
Jun 29, 2026Deyin Liu, Jicheng Xu, Lin Yuanbo Wu +4Camera-Controlled Video Generation Motion Retargeting
Engineering Research Center of Autonomous Unmanned System Technology, Ministry of Education; Anhui Provincial Key Laboratory of Security Artificial Intelligence; and School of Artificial Intelligence, Anhui University, Hefei, China · University of Surrey, Guildford, United Kingdom · University of Warwick, Coventry, United Kingdom +1
DiGS-Avatar: Single-Image Animatable 3D Human Reconstruction via UV-Space Diffusion
Aug 21, 2026Jiakun Li, Li Fang, Hao Zhu +43D Avatar Generation 3D Human Reconstruction
Key Laboratory of Media Audio and Video (Communication University of China), Ministry of Education, Beijing 100024, China · School of Intelligence Science and Technology, Nanjing University, Nanjing 210023, China

Jun 30, 2026J/K move · Enter open · S save

LUNA: Learning Universal 3D Human Animation Beyond Skinning

Peng Li, Rawal Khirodkar, Junxuan Li, Yuan Dong+5

The Hong Kong University of Science and Technology · Work was done during an internship at Meta. · Codec Avatars Lab, Meta

Human Motion Generation Human Avatar Animation

Open paper AskPDF arXiv

Creating photorealistic, animatable 3D human avatars from monocular images still largely depends on Linear Blend Skinning (LBS) and parametric body models, which constrain expressivity and often introduce artifacts due to imperfect fitting. We propose LUNA, an LBS-free universal neural animation model that directly maps multiple 2D controls like images, keypoints, sketches, and unseen characters into 3D Gaussian deformations, bypassing explicit body fitting. At its core, a transformer-based motion regressor disentangles global rigid motion from fine-grained local dynamics to capture both coherent movement and subtle non-rigid effects. To resolve the inherent ambiguity of 2D-to-3D lifting while scaling beyond fitted datasets, we introduce hybrid supervision that distills soft structural priors from an LBS teacher and a loss that supports training on both limited fitted data and large in-the-wild unlabeled videos. Extensive experiments show LUNA achieves competitive visual fidelity compared to LBS-based approaches, while delivering realistic human motion and zero-shot cross-identity generalization across diverse driving modalities. To the best of our knowledge, LUNA is the first end-to-end 3D animatable model that supports implicit 2D driving.

Abstract

Human Avatar Animation

Explore similar work

LUNA: Learning Universal 3D Human Animation Beyond Skinning

3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement

DiGS-Avatar: Single-Image Animatable 3D Human Reconstruction via UV-Space Diffusion