Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

Simon Giebenhain; Tobias Kirschstein; Liam Schoneveld; Davide Davoli; Zhe Chen; Matthias Nießner

cs.CVOct 16, 2026

Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

Authors: Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen, Matthias Nießner

Organizations: Technical University of Munich · Woven by Toyota · Toyota Motor Europe

Abstract

Video diffusion models (VDMs) trained on internet-scale data encode a rich prior on appearance, motion, and physics that is difficult to replicate with explicit 3D rendering and costly 3D data capture. We introduce Video Diffusion Avatars (VDAs), a method that creates photo-realistic, multi-view consistent avatars from a single image by fine-tuning a VDM for precise control over camera viewpoint, facial expression, and upper-body pose. Departing from 3D morphable models (3DMMs), VDAs use a learned expression space that lifts the expressiveness ceiling of 3DMMs, e.g., for the tongue and mouth interior. However, learned expression codes exhibit heavy viewpoint bias, as our DISTRACT benchmark reveals. We show that this bias can be removed through disentanglement, enabling their use as driving signals for multi-view consistent avatars. Camera and upper-body pose are controlled jointly via proxy-mesh renderings, giving independent control over expression, pose, and viewpoint. A camera-triplet training strategy makes the approach efficient: our models train for 34 hours on a single A100 using only public multi-view data. VDAs outperform state-of-the-art single-image 3D avatars on novel-view reenactment, reducing face LPIPS by 29% and improving perceptual quality (JOD) by 66% on BecomingLit. A static 3DGS reconstruction probe further confirms that our generations are the most multi-view consistent. Built on a video prior, VDAs produce effects beyond the reach of explicit renderers, including physically plausible hair dynamics, shadows, reflections, and refractions.

Figures & tables

Figure 2: Video Diffusion Avatar architecture : Camera, upper-body pose, and facial expression are injected as independent control signals into WAN-Animate-14B. Disentangled X-NeMo expression codes are translated into WAN-Animate’s native conditioning format with a viewpoint-neutral camera code, while a proxy-mesh rendering encodes camera and pose. Camera triplets from multi-view data teach multi-view consistency.

	Self-reenactment ( $n_{\text{BecomingLit}}$ =25 / $n_{\text{Multiface}}$ =22)						Cross-reenact. ( $n$ =13)
Method	PSNR $\uparrow$	LPIPS $\downarrow$	JOD $\uparrow$	ID $\uparrow$	Expr $\uparrow$	AKD $\downarrow$	ID $\uparrow$	Expr $\uparrow$	AKD $\downarrow$
LAM	18.95/19.83	0.408/0.415	2.00/1.92	0.295/0.334	0.670/0.528	0.103/0.114	0.314	0.623	0.091
FlexAvatar	20.55/19.45	0.327/0.364	2.28/2.33	0.624 / 0.645	0.890 / 0.850	0.086/0.092	0.570	0.700	0.130
UIKA	20.37/20.76	0.327/0.332	2.34 / 2.41	0.532/0.571	0.865 /0.805	0.076/0.072	0.493	0.775	0.084
MMDM	20.85 /21.02	0.283 / 0.270	1.56/1.35	0.555/ 0.608	0.853/ 0.840	0.058 /0.063	0.522	0.780	0.069
MVP4D	20.16/ 21.64	0.282 / 0.273	1.85/1.64	0.490/0.531	0.831/0.828	0.063/ 0.060	0.480	0.755	0.074

Table 2: Quantitative results (BecomingLit/Multiface per cell). MMDM is CAP4D’s diffusion component. Cross-reenactment on BecomingLit only; colours rank within each dataset.

Figure 4: Self-reenactment on BecomingLit and Multiface.

Table 5

Figure 5: Cross-reenactment on BecomingLit. The reference identity is driven by a different person’s expression and rendered to the target camera; the last column shows the driver from that camera.

Figure 6: Multi-view consistency probe: each method’s generation along a novel-view ellipse is fit into a static 3DGS and re-rendered at a rig camera (vs. GT). Quantitative results in Table 4 .

Figure 7: In-the-wild novel-view reenactment: the camera sweeps a wide arc, isolating viewpoint consistency from expression transfer.

Panel of Figure 8 — Figure 8: 360° novel-view synthesis from a single reference image: the camera orbits from the front ( $0^{\circ}$ ) to the back of the head ( $180^{\circ}$ ). Rows: FlexAvatar, FaceCam, MVP4D, and ours.

	Ablated knob				Self-reenactment, BecomingLit v3 ( $n$ =25)
Method	Fine-tune	Expr. code	Pose cond.	Disent.	PSNR $\uparrow$	LPIPS $\downarrow$	JOD $\uparrow$	ID $\uparrow$	Expr. $\uparrow$	AKD $\downarrow$	APD $\downarrow$
WAN-A.	none	WAN native	2D skel.	–	16.87	0.2582	3.897	0.5825	0.8279	0.0813	6.06
WAN-A. disent.	none	WAN native	2D skel.	✓	17.03	0.2584	3.934	0.6044	0.8386	0.0791	5.72
Ours, WAN-A. expr.	all but enc.	WAN native	skel. $+$ mesh	–	21.52	0.2098	4.438	0.7012	0.9115	0.0532	4.06
$+$ frozen fusers	all but enc., fusers	WAN native	skel. $+$ mesh	–	19.67	0.2806	4.052	0.6534	0.8175	0.0808	6.03
Ours, no disent. $\dagger$	full	X-NeMo	skel. $+$ mesh	$\times$	22.06	0.2048	4.608	0.6927	0.9099	0.0550	3.94

Table 5: Self-reenactment ablations on BecomingLit v3 (

n

=25), the only ablation dataset.

Figure 9: Qualitative ablation on BecomingLit v3; columns match Table 5 .

Figure 10: Comparison to FactorPortrait on its released examples.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Trainable module	Parameters	Learning rate
Face fusers $\mathnet{B}_{\text{face}}$	839.0 M	$5\times 10^{-6}$
Face encoder $\mathnet{P}_{\text{face}}$	17.8 M	$10^{-5}$
Translator ( $\mathnet{T}$ , $\mathnet{E}^{\text{WAN}}$ merged)	3.1 M	$10^{-5}$
Expression head of $\mathnet{D}^{\text{XN}}$	0.8 M	$10^{-5}$
$\mathnet{P}_{\text{skel}}$ , $\mathnet{P}_{\text{mesh}}$	0.3 M each	$5\times 10^{-5}$
	Stage 1	Stage 2

Appendix

Table 7: Fine-tuning configuration. Wall-clock times include periodic validation.

Panel of Figure 11 — Figure 11: Pelvis-anchored kinematic fitting stabilizes the proxy-mesh body. Proxy-mesh conditioning renders ( Section F.2 ) overlaid on the ground-truth frames of three NeRSemble EXP-1 sequences with large head rotations. For each subject, the top row aligns every per-frame SAM-3D estimate independently to the tracked FLAME head; the bottom row uses pelvis-anchored kinematic fitting. Both rows share the same head. The crosshair marks the shoulder midpoint’s median position over the sequence, the yellow dot its position in the shown frame; red boxes mark frames whose shoulder midpoint lies more than 45 px (at $720{\times}1280$ ) from that median. Under per-frame alignment the torso drifts, although the subject is seated; pelvis-anchored fitting holds it in place while the head, and with it the head’s color, follows the performance. The three camera views show how the ray-direction colors depend on the viewpoint.

	X-NeMo	WAN-Animate
	(deployed)	( epoch_0850 )
$d$ / $d_{\text{ex}}$ / $d_{\text{cam}}$	512 / 512 / 32	512 / 512 / 51
$d_{\text{id}}$ / $d_{\text{res}}$	– / –	128 / 32
MLP width (3 layers)	512	512
$\lambda_{\text{rec}}$	1	100
$\lambda_{\text{ex}}$ / $\lambda_{\text{cam}}$ / $\lambda_{\text{id}}$	1 / 1 / –	1 / 1 / 1

Appendix

Table 8: Disentangler configurations. X-NeMo : the checkpoint that produces

\mathbf{z}_{\text{ex}}

for all VDA training and inference. WAN-Animate : the disentangler of WAN-Animate’s native expression code used for training-free viewpoint control.

Figure 12: Training-free head-pose control in X-NeMo. The disentangled expression code is kept fixed while the camera code is swept through other views, (a) with a constant expression and (b) with an extreme driving expression (top row: driving frames). The head rotates and the expression is preserved; X-NeMo does not separate camera from neck motion, so the camera code moves the head rather than the viewpoint.

Figure 13: Frozen WAN-Animate rendering five views of a camera orbit, driven by its raw expression codes or by our disentangled codes (expression latent re-entangled with the target view’s camera code). Insets magnify the mouth. Top: the expression source is a profile view of another person. Bottom: the expression is taken from the reference image itself. Raw codes carry the source viewpoint, which shows up as expression artifacts once the target view differs; the disentangled codes preserve the source expression.

Figure 14: Cross-view supervision creates view invariance; single-view supervision does not. DISTRACT viewpoint-invariance rank-1 of the expression code at the face-encoder output, over 500 triplets, for two runs that differ only in the camera the driving expression code is read from. Both begin at 0.244, WAN-Animate’s pre-trained face encoder on the shared translator initialisation. Cross-view training reaches 0.79, while single-view training stalls below 0.49. The final gap is 0.288.

Figure 15: Our full model (20 steps) versus the distilled four-step student on BecomingLit self-reenactment. Insets magnify facial details.

	View Inv.		Identity Inv.
Encoder	R-1 $\uparrow$	Err A/B/C/D $\downarrow$	R-1 $\uparrow$	Err A/B/C/D $\downarrow$
DECA ( Feng et al., 2021 )	61	33 / 18 / 18 / 11	34	51 / 37 / 25 / 24
EMOCA ( Danecek et al., 2022 )	42	47 / 22 / 30 / 16	32	56 / 34 / 43 / 21
SHeaP ( Schoneveld et al., 2025 )	54	41 / 17 / 07 / 03	17	69 /52/ 31 / 21
EMOPortraits ( Drobyshev et al., 2024 )	23	70/37/51/28	14	76/42/58/36
PD-FGC ( Wang et al., 2023 )	08	90/42/79/43	07	91/48/81/43

Appendix

Table 10: All 8 evaluated encoders on the DISTRACT benchmark (

n

=500 for Task 1,

n

=126 for Task 2; entries in %), without the MLP probe. Columns as in Table 1 . Classical 3DMM regressors (top) are consistently less viewpoint-biased than the learned reenactment/portrait-animation encoders below the rule.

Figure 16: BecomingLit self-reenactment, qualitative comparison of our method against LAM ( He et al., 2025 ) , FlexAvatar ( Kirschstein et al., 2026 ) , UIKA ( Wu et al., 2026 ) , and MMDM (the morphable multi-view diffusion component of CAP4D ( Taubner et al., 2025b ) , rendered directly at the target camera – not the full CAP4D avatar pipeline). One row per identity (8 of the benchmark’s 13), each frame chosen to maximize expression diversity within that identity’s own driving clips (farthest-point sampling over X-NeMo expression embeddings ( Zhao et al., 2025 ) ), then narrowed from an initial per-identity diverse set down to these 8 through several rounds of manual review.

Figure 17: Multiface self-reenactment, qualitative comparison of our method against LAM ( He et al., 2025 ) , FlexAvatar ( Kirschstein et al., 2026 ) , UIKA ( Wu et al., 2026 ) , and MMDM (the morphable multi-view diffusion component of CAP4D ( Taubner et al., 2025b ) , rendered directly at the target camera – not the full CAP4D avatar pipeline). One row per identity (8 of the benchmark’s 11), each frame chosen to maximize expression diversity within that identity’s own driving clips (farthest-point sampling over X-NeMo expression embeddings ( Zhao et al., 2025 ) ), then narrowed from an initial per-identity diverse set down to these 8 through two rounds of manual review.

Figure 18: Qualitative comparisons on the in-the-wild VFHQ test set ( Xie et al., 2022 ) , against SotA 3D avatars.

3D Avatar Generation

3D avatar generation creates realistic, controllable digital humans from images, video, text, or other inputs.

See all 90 papers

GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos
Apr 8, 2026Yiqian Wu, Rawal Khirodkar, Egor Zakharov +63D Avatar Generation 3D Diffusion Models
State Key Laboratory of CAD&CG, Zhejiang University, China · Codec Avatars Lab, Meta, USA · State Key Lab of CAD&CG, Zhejiang University, China
LiftAvatar: Kinematic-Space Completion for Expression-Controlled 3D Gaussian Avatar Animation
Mar 2, 2026Hualiang Wei, Shunran Jia, Jialun Liu +1Facial Expression Synthesis Human Avatar Animation
College of Computer Science and Technology, Jilin University, No. 2699 Qianjin Street, Changchun, 130012, Jilin, China · Impressed Inc DBA SocialBook, 950 Tower Ln, Foster City, 94404, California, USA · Institute of Artificial Intelligence of China Telecom, Zhonghui Building, Dongcheng District, 100007, Beijing, China
Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo
May 31, 2026Hao Liang, Zhixuan Ge, Soumendu Majee +33D Gaussian Splatting 3D Asset Generation
Rice University Houston, TX 77005 · Samsung Research America Plano, TX 75023

Apr 8, 2026J/K move · Enter open · S save

GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos

Yiqian Wu, Rawal Khirodkar, Egor Zakharov, Timur Bagautdinov+5

State Key Laboratory of CAD&CG, Zhejiang University, China · Codec Avatars Lab, Meta, USA · State Key Lab of CAD&CG, Zhejiang University, China

3D Avatar Generation 3D Diffusion Models Human Avatar Animation

Open paper AskPDF arXiv

We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.

Abstract

3D Avatar Generation

Explore similar work

GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos

LiftAvatar: Kinematic-Space Completion for Expression-Controlled 3D Gaussian Avatar Animation

Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo