Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China
Recent video generative models have shown strong capability in synthesizing high-quality and temporally coherent videos. Yet, adapting them for controllable portrait animation remains nontrivial, as existing methods typically rely on architectural changes or extra conditioning modules, increasing complexity and reducing compatibility with pretrained models. We present CatPortrait, a simple and effective framework that enables controllable portrait animation via 3D face guidance as in-context conditioning. CatPortrait preserves the original video generation backbone and injects structured 3D facial guidance into the input space through spatial concatenation, allowing pretrained image-to-video models to animate portraits without sacrificing their pretrained priors. CatPortrait supports both video-driven and audio-driven portrait animation within a unified framework. Extensive experiments demonstrate that CatPortrait achieves competitive or superior results across both settings.
Figures & tables
Figure 1: Overview of the proposed framework. During inference, the model takes a reference image IR and driving signals as inputs. A motion parameter estimator converts the driving signals into FLAME parameters (β,ψ,θ) . The rendered FLAME sequence is then encoded by E and spatially concatenated with the noisy latent zt to form a conditioned latent, which, together with the identity information from IR , is fed into the video generator to synthesize the output video. During training, paired sequences are used to construct the input, and the generator is finetuned.
Method
FID
FVD
LSE-D
LSE-C
M-LMD
Hallo [ hallo ]
82.7286
132.09
8.0047
7.0980
8.0245
Hallo2 [ cui2024hallo2 ]
147.1788
130.65
8.2012
7.067
6.5618
SadTalker [ sadtalker ]
685.7087
305.59
7.9159
6.8427
3.6368
AniPortrait [ aniportrait ]
138.636
414.77
12.6616
4.1945
2.3688
Ours
81.1032
113.53
8.2837
6.8823
1.632
Table 1: Quantitative comparison on audio-driven talking head generation.
Method
Self-Reenactment
Cross-Reenactment
FID
FVD
LMD
LPIPS
AED
AKD
ID Sim.
Image Qual.
AniPortrait [ aniportrait ]
155.190
413.48
2.8264
0.2105
1.1301
13.5388
0.5753
30.5514
X-Portrait [ xportrait ]
89.47
149.61
11.599
0.0985
0.9661
7.087
0.8445
35.4013
HunyuanPortrait [ hunyuanportrait ]
82.609
108.02
1.003
0.0666
0.9585
6.9574
0.8290
36.4791
Ours
88.79
104.99
1.503
0.0757
0.9563
6.9923
0.8025
37.5668
Table 2: Quantitative comparison on video-driven portrait reenactment under self-reenactment and cross-reenactment settings.
Figure 2: Qualitative comparison on audio-driven talking head generation.
Figure 3: Qualitative comparison on video-driven portrait reenactment, including self-reenactment and cross-reenactment.
Method Variant
Self-Reenactment
Cross-Reenactment
FID ↓
FVD ↓
LMD ↓
LPIPS ↓
AED ↓
AKD ↓
ID Sim. ↑
IQ ↑
Left RGB + Right 3DMM (Full FT)
88.79
104.99
1.503
0.0757
0.9563
6.9923
0.8025
37.5668
Left 3DMM + Right RGB
88.84
105.35
1.497
0.0769
0.9688
7.0831
0.7989
37.4215
Top RGB + Bottom 3DMM
90.21
107.43
1.524
0.0774
0.9736
7.1205
0.7967
37.3084
Top 3DMM + Bottom RGB
90.87
107.96
1.533
0.0782
0.9798
7.1886
0.7941
37.1862
2D Landmark guidance
95.84
114.72
1.621
0.0829
1.0476
7.5429
0.7794
36.4812
Table 3: Ablation study on video-driven portrait reenactment. All variants are evaluated under the same protocol. Lower is better for FID, FVD, LMD, LPIPS, AED, and AKD. Higher is better for ID Sim. and Image Quality.
Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate poorly with human perception. We propose FlowPortrait, a reinforcement-learning framework for audio-driven portrait animation built on a multimodal backbone for autoregressive audio-to-video generation. FlowPortrait introduces a human-aligned evaluation system based on Multimodal Large Language Models (MLLMs) to assess lip-sync accuracy, expressiveness, and motion quality. These signals are combined with perceptual and temporal consistency regularizers to form a stable composite reward, which is used to post-train the generator via Group Relative Policy Optimization (GRPO). Extensive experiments, including both automatic evaluations and human preference studies, demonstrate that FlowPortrait consistently produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation.
We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/
Wenbo Ji, Davide Davoli, Zhe Chen +3
Technical University of Munich · Toyota Motor Europe NV/SA · Woven by Toyota
We introduce FaceCam, a system that generates video under customizable camera trajectories for monocular human portrait video input. Recent camera control approaches based on large video-generation models have shown promising progress but often exhibit geometric distortions and visual artifacts on portrait videos due to scale-ambiguous camera representations or 3D reconstruction errors. To overcome these limitations, we propose a face-tailored scale-aware representation for camera transformations that provides deterministic conditioning without relying on 3D priors. We train a video generation model on both multi-view studio captures and in-the-wild monocular videos, and introduce two camera-control data generation strategies: synthetic camera motion and multi-shot stitching, to exploit stationary training cameras while generalizing to dynamic, continuous camera trajectories at inference time. Experiments on Ava-256 dataset and diverse in-the-wild videos demonstrate that FaceCam achieves superior performance in camera controllability, visual quality, identity and motion preservation.