Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China
Recent video generative models have shown strong capability in synthesizing high-quality and temporally coherent videos. Yet, adapting them for controllable portrait animation remains nontrivial, as existing methods typically rely on architectural changes or extra conditioning modules, increasing complexity and reducing compatibility with pretrained models. We present CatPortrait, a simple and effective framework that enables controllable portrait animation via 3D face guidance as in-context conditioning. CatPortrait preserves the original video generation backbone and injects structured 3D facial guidance into the input space through spatial concatenation, allowing pretrained image-to-video models to animate portraits without sacrificing their pretrained priors. CatPortrait supports both video-driven and audio-driven portrait animation within a unified framework. Extensive experiments demonstrate that CatPortrait achieves competitive or superior results across both settings.
Figures & tables
Figure 1: Overview of the proposed framework. During inference, the model takes a reference image IR and driving signals as inputs. A motion parameter estimator converts the driving signals into FLAME parameters (β,ψ,θ) . The rendered FLAME sequence is then encoded by E and spatially concatenated with the noisy latent zt to form a conditioned latent, which, together with the identity information from IR , is fed into the video generator to synthesize the output video. During training, paired sequences are used to construct the input, and the generator is finetuned.
Method
FID
FVD
LSE-D
LSE-C
M-LMD
Hallo [ hallo ]
82.7286
132.09
8.0047
7.0980
8.0245
Hallo2 [ cui2024hallo2 ]
147.1788
130.65
8.2012
7.067
6.5618
SadTalker [ sadtalker ]
685.7087
305.59
7.9159
6.8427
3.6368
AniPortrait [ aniportrait ]
138.636
414.77
12.6616
4.1945
2.3688
Ours
81.1032
113.53
8.2837
6.8823
1.632
Table 1: Quantitative comparison on audio-driven talking head generation.
Method
Self-Reenactment
Cross-Reenactment
FID
FVD
LMD
LPIPS
AED
AKD
ID Sim.
Image Qual.
AniPortrait [ aniportrait ]
155.190
413.48
2.8264
0.2105
1.1301
13.5388
0.5753
30.5514
X-Portrait [ xportrait ]
89.47
149.61
11.599
0.0985
0.9661
7.087
0.8445
35.4013
HunyuanPortrait [ hunyuanportrait ]
82.609
108.02
1.003
0.0666
0.9585
6.9574
0.8290
36.4791
Ours
88.79
104.99
1.503
0.0757
0.9563
6.9923
0.8025
37.5668
Table 2: Quantitative comparison on video-driven portrait reenactment under self-reenactment and cross-reenactment settings.
Figure 2: Qualitative comparison on audio-driven talking head generation.
Figure 3: Qualitative comparison on video-driven portrait reenactment, including self-reenactment and cross-reenactment.
Method Variant
Self-Reenactment
Cross-Reenactment
FID ↓
FVD ↓
LMD ↓
LPIPS ↓
AED ↓
AKD ↓
ID Sim. ↑
IQ ↑
Left RGB + Right 3DMM (Full FT)
88.79
104.99
1.503
0.0757
0.9563
6.9923
0.8025
37.5668
Left 3DMM + Right RGB
88.84
105.35
1.497
0.0769
0.9688
7.0831
0.7989
37.4215
Top RGB + Bottom 3DMM
90.21
107.43
1.524
0.0774
0.9736
7.1205
0.7967
37.3084
Top 3DMM + Bottom RGB
90.87
107.96
1.533
0.0782
0.9798
7.1886
0.7941
37.1862
2D Landmark guidance
95.84
114.72
1.621
0.0829
1.0476
7.5429
0.7794
36.4812
Table 3: Ablation study on video-driven portrait reenactment. All variants are evaluated under the same protocol. Lower is better for FID, FVD, LMD, LPIPS, AED, and AKD. Higher is better for ID Sim. and Image Quality.