While diffusion models have shown great potential in portrait generation, generating expressive, coherent, and controllable cinematic portrait videos remains a significant challenge. Existing intermediate signals for portrait generation, such as 2D landmarks and parametric models, have limited disentanglement capabilities and cannot express personalized details due to their sparse or low-rank representation. Therefore, existing methods based on these models struggle to accurately preserve subject identity and expressions, hindering the generation of highly expressive portrait videos. To overcome these limitations, we propose a high-fidelity personalized head representation that more effectively disentangles expression and identity. This representation captures both static, subject-specific global geometry and dynamic, expression-related details. Furthermore, we introduce an expression transfer module to achieve personalized transfer of head pose and expression details between different identities. We use this sophisticated and highly expressive head model as a conditional signal to train a diffusion transformer (DiT)-based generator to synthesize richly detailed portrait videos. Extensive experiments on self- and cross-reenactment tasks demonstrate that our method outperforms previous models in terms of identity preservation, expression accuracy, and temporal stability, particularly in capturing fine-grained details of complex motion.
Figures & tables
Figure 2: Our framework. To address the limited decoupling capability and insufficient expressiveness of current parametric head representations, we propose a personalized head representation. Starting from the SMPL-X base mesh, we perform joint optimization learning of two complementary static and dynamic offset fields. We then construct an identity-adaptive expression transfer module to achieve cross-identity expression transfer. Using our head representation as a control signal, we guide a diffusion model for highly consistent and expressive portrait video generation.
Figure 3: An illustration of our optimization pipeline for transforming a generic SMPL-X mesh into our highly detailed, personalized head representation.
Figure 4: Design of our Expression Transfer Module.
Figure 5: Qualitative results in self-reenactment. Compared to other methods, our method can reveal more details about identity and facial expressions.
Figure 6: Qualitative results in cross-reenactment. Our method can effectively transfer body posture and facial expressions while ensuring identity consistency.
Method
Self Reenactment
Cross Reenactment
PSNR ↑
SSIM ↑
LPIPS ↓
L1 ↓
AED ↓
APD ↓
CSIM ↑
AED ↓
APD ↓
CSIM ↑
LivePortrait [ 12 ]
23.2897
0.82985
0.37339
0.04619
0.12924
0.02144
0.82972
0.28641
0.23048
0.72869
AniPortrait [ 40 ]
22.2527
0.80838
0.30018
0.04078
0.17581
0.01631
0.76038
0.25256
0.02161
0.62998
Follow-Your-Emoji [ 25 ]
25.6872
0.84146
0.23645
0.02905
0.14717
0.01485
0.80284
0.22069
0.02338
0.68373
Hunyuan Portrait [ 46 ]
22.9453
0.79333
0.27573
0.03962
0.15593
0.02329
0.81150
0.22753
0.08784
0.67451
X-NeMo [ 53 ]
21.5618
0.78051
0.32411
0.04760
0.13671
0.01789
0.83018
0.17092
0.02101
0.72164
Table 1: Quantitative comparison. Our method achieves superior numerical results to all the baselines in self- and cross-reenactment tasks, evaluated on an image resolution of 512×512 .
Figure 7: Ablation study between SMPL-X and our head representation.
Figure 8: Ablation study between directly performing deformation with offsets and using our expression transfer module.
Identity-preserving video generation aims to maintain a subject's identity while synthesizing realistic videos. Yet a single reference portrait captures the subject's appearance under only one facial configuration. As expressions change, facial appearance can vary in highly identity-specific ways, leaving the subject's appearance under unseen expressions underdetermined by the reference alone. This expression-dependent variation also complicates evaluation: similarity to a neutral reference may decrease under strong expressions even for real images of the same person. We investigate this limitation from both generation and evaluation perspectives. First, we quantify how face-recognition similarity varies with expression intensity using controlled photographs and MEAD videos. We then construct a compact yet expressive reference gallery that captures diverse expression-dependent facial configurations. Matching against this gallery provides a more robust measure of identity similarity under expressive motion. To further expose performance degradation with expression intensity, we report identity similarity separately for mild, intense, and extreme expressions. For generation, we extend Stand-In to condition on our expression-diverse reference sets and develop a data-curation pipeline that extracts consistent yet diverse face crops from training videos. In practical settings where only a single portrait is available, we construct the reference set by synthesizing additional expressions with a pretrained facial reenactment model. On our controlled benchmark, both real and synthesized reference sets outperform the evaluated baselines in identity similarity across all three expression-intensity regimes, with the largest improvements for extreme expressions.
Tianwen Fu, Wenbin Teng, Gonglin Chen +3
Institute for Creative Technologies, University of Southern California
We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/
Wenbo Ji, Davide Davoli, Zhe Chen +3
Technical University of Munich · Toyota Motor Europe NV/SA · Woven by Toyota
Reconstructing photorealistic and animatable 4D head avatars from a single portrait image remains a fundamental challenge in computer vision. While diffusion models have enabled remarkable progress in image and video generation for avatar reconstruction, existing methods primarily rely on 2D priors and struggle to achieve consistent 3D geometry. We propose a novel framework that leverages geometry-aware diffusion to learn strong geometry priors for high-fidelity head avatar reconstruction. Our approach jointly synthesizes portrait images and corresponding surface normals, while a pose-free expression encoder captures implicit expression representations. Both synthesized images and expression latents are incorporated into 3D Gaussian-based avatars, enabling photorealistic rendering with accurate geometry. Extensive experiments demonstrate that our method substantially outperforms state-of-the-art approaches in visual quality, expression fidelity, and cross-identity generalization, while supporting real-time rendering.