This paper primarily investigates the task of expression-only portrait video performance editing based on a driving video, which plays a crucial role in animation and film industries. Most existing research mainly focuses on portrait animation, which aims to animate a static portrait image according to the facial motion from the driving video. As a consequence, it remains challenging for them to disentangle the facial expression from head pose rotation and thus lack the ability to edit facial expression independently. In this paper, we propose PerformRecast, a versatile expression-only video editing method which is dedicated to recast the performance in existing film and animation. The key insight of our method comes from the characteristics of 3D Morphable Face Model (3DMM), which models the face identity, facial expression and head pose of 3D face mesh with separate parameters. Therefore, we improve the keypoints transformation formula in previous methods to make it more consistent with 3DMM model, which achieves a better disentanglement and provides users with much more fine-grained control. Furthermore, to avoid the misalignment around the boundary of face in generated results, we decouple the facial and non-facial regions of input portrait images and pre-train a teacher model to provide separate supervision for them. Extensive experiments show that our method produces high-quality results which are more faithful to the driving video, outperforming existing methods in both controllability and efficiency. Our code, data and trained models are available at https://youku-aigc.github.io/PerformRecast.
Figures & tables
Figure 2: An overview of our PerformRecast framework. The motion extractor extracts canonical keypoints, head pose, expression deformation, scale factor and translation from source and driving frames. The facial keypoints are obtained via our improved keypoints transformation formula and compared with the tracking results from Pixel3DMM [ 17 ] to calculate the FLAME loss. Finally, the appearance feature volume, source and driving keypoints are sent to the warping module, followed by the decoder to reconstruct the driving frame.
Figure 3: Top: we adopt the boundary alignment module to alleviate the misalignment between the facial and non-facial regions. Bottom: we provide some visual comparisons to demonstrate the necessity of using boundary alignment module (denoted as “BAM”). The red dashed circles highlight the misalignment between the facial and non-facial regions. The bottom-right insets show the driving frames. Please zoom in for better inspection.
Figure 4: The typical usage scenarios for our proposed replacement mode and enhancement mode. The bottom-right insets exhibit driving frames. Please zoom in for better inspection.
Figure 5: Qualitative comparison of portrait video expression editing on replacement mode. The top of the figure shows editing results of different methods. The bottom presents our ablation studies and analysis. The bottom-right insets exhibit driving frames. The red dashed circles highlight the misalignment between the facial and non-facial regions. Please zoom in for better inspection.
Method
Replacement Mode
Enhancement Mode
PSNR ↑
SSIM ↑
LPIPS ↓
L1↓
CSIM ↑
MAE(°) ↓
AED ↓
APD ↓
FID ↓
FVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
L1↓
CSIM ↑
MAE(°) ↓
AED ↓
APD ↓
FID ↓
FVD ↓
SkyReels-A1 [ 42 ]
24.9051
0.8586
0.1622
0.0418
0.7163
13.0791
0.7157
0.0155
50.2007
1249.7021
24.9206
0.8584
0.1612
0.0417
0.7176
11.3797
0.676
0.016
51.47
1196.1767
Hunyuan-Portrait [ 70 ]
22.4287
0.7924
0.1691
0.0423
0.7364
10.4001
0.6608
0.0348
38.5997
1925.0654
22.7968
0.7984
0.1649
0.0406
0.736
9.6267
0.6073
0.0332
39.3112
1861.9558
FantasyPortrait [ 60 ]
23.9456
0.8204
0.1883
0.0349
0.7338
16.7765
0.7953
0.0168
60.4381
606.6303
24.2032
0.8239
0.1848
0.0339
0.7387
12.2154
0.6614
0.0155
62.0027
551.448
Wan-Animate [ 7 ]
22.8196
0.8021
0.1319
0.0467
0.614
11.9722
0.7002
0.0238
28.1003
849.2188
22.6417
0.8006
0.1402
0.0491
0.6062
11.2752
0.6847
0.0225
31.2186
885.4313
Act-Two [ 46 ]
20.8344
0.7913
0.1634
0.0459
0.6819
15.9618
0.7901
0.0723
36.1933
322.0726
20.8066
0.7916
0.165
0.0458
0.685
15.4109
0.8019
0.0727
38.0579
330.5131
Table 1: Quantitative results of facial expression editing on our test benchmark on both replacement and enhancement modes. The top of the table shows the comparison with other methods while the bottom presents our ablation studies and analysis.
Figure 6: Qualitative comparison of self-reenactment portrait animation. We show the source frame, driving frame and generative results. These source-driving paired images are from official test split of VFHQ dataset [ 68 ] . Please zoom in for better inspection.
Method
Self-reenactment
Cross-reenactment
PSNR ↑
SSIM ↑
LPIPS ↓
L1↓
CSIM ↑
MAE(°) ↓
AED ↓
APD ↓
FID ↓
FVD ↓
CSIM ↑
MAE(°) ↓
AED ↓
APD ↓
GAGAvatar [ 8 ]
-
-
-
-
0.7878
8.1317
0.4057
0.014
-
-
0.6871
12.2879
0.8144
0.0256
Portrait4D-v2 [ 11 ]
14.3781
0.5601
0.4877
0.1268
0.7525
8.4331
0.4434
0.0262
55.0744
506.1298
0.6702
13.4384
0.8385
0.0321
PD-FGC [ 58 ]
16.4477
0.62
0.4252
0.0936
0.3282
11.9368
0.6345
0.0322
76.2318
628.5279
0.2181
15.0314
0.8988
0.0417
EMOPortrait [ 13 ]
18.7677
0.6979
0.2711
0.0648
0.6297
7.7967
0.4676
0.0179
43.2991
444.5448
0.3483
10.3905
0.8012
0.0245
EDTalk [ 53 ]
20.0771
0.7272
0.3054
0.0648
0.674
16.3315
0.572
0.0252
60.5285
369.9713
0.5435
22.5909
1.018
0.0531
Table 2: Quantitative comparisons of different methods on portrait animation task. The top of the table shows the results of non-diffusion-based methods while the bottom presents diffusion-based methods.
Figure 7: Qualitative comparison of cross-reenactment portrait animation. We show the source image, driving image and the generative results. The source images are from FFHQ dataset and driving frames are from VFHQ dataset.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The facial mask calculation process of each frame in training dataset.
Figure 9: The specific location of each keypoint used in our method. Please zoom in for better inspection.
Figure 10: Construction pipeline of our proposed test benchmark.
Figure 11: Qualitative comparison of portrait video expression editing on enhancement mode. The top of the figure shows editing results of different methods. The bottom presents our ablation studies and analysis. The bottom-right insets exhibit driving frames. The light green circles highlight the misalignment between the facial and non-facial regions. Please zoom in for better inspection.
Figure 12: Full qualitative comparison with all the methods mentioned in the main manuscript on our proposed test benchmark. The bottom-right insets exhibit driving frames. Please zoom in for better inspection.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
L1↓
CSIM ↑
MAE(°) ↓
AED ↓
APD ↓
FID ↓
FVD ↓
GAGAvatar [ 8 ]
-
-
-
-
0.8946
5.1074
0.3781
0.0101
-
-
Portrait4D-v2 [ 11 ]
20.0907
0.7746
0.3358
0.0617
0.8793
5.4329
0.4353
0.0149
85.3589
460.7595
PD-FGC [ 58 ]
20.8419
0.7824
0.341
0.0573
0.3256
8.3772
0.7156
0.0202
92.8886
1276.3855
EMOPortrait [ 13 ]
26.1748
0.8729
0.1544
0.0287
0.5959
6.6994
0.4992
0.0128
37.5216
443.7428
EDTalk [ 53 ]
26.9246
0.8964
0.1443
0.0319
0.8592
6.218
0.4333
0.0077
43.4199
343.9973
LIA-X [ 63 ]
22.6439
0.8232
0.1816
0.0386
0.8957
5.3919
0.4633
0.0636
32.4902
323.2831
Appendix
Table 3: Quantitative comparisons of self-reenactment portrait animation on MEAD [ 59 ] dataset. The top of the table shows the results of non-diffusion-based methods, while the bottom presents diffusion-based methods.
Figure 13: More generated results on self-reenactment task of different methods. The first two source-driving paired images are from VFHQ dataset [ 68 ] and the last two source-driving paired images are from MEAD dataset [ 59 ] .
Figure 14: More generated results on cross-reenactment task of different methods. The source images are from FFHQ dataset and the driving frames are from famous films and television clips.
Figure 15: A typical failure case when our method generating teeth while the mouth is closed in source video.
Motion transfer from the driving to the source portrait remains a key challenge in the portrait animation. Current diffusion-based approaches condition only on the driving motion, which fails to capture source-to-driving correspondences and consequently yields suboptimal motion transfer. Although flow estimation provides an alternative, predicting dense correspondences from 2D input is ill-posed and often yields inaccurate animation. We address this problem by introducing 3D flows, a learning-free and geometry-driven motion correspondence directly computed from parametric 3D head models. To integrate this 3D prior into diffusion model, we introduce 3D flow encoding to query potential 3D flows for each target pixel to indicate its displacement back to the source location. To obtain 3D flows aligned with 2D motion changes, we further propose depth-guided sampling to accurately locate the corresponding 3D points for each pixel. Beyond high-fidelity portrait animation, our model further supports user-specified editing of facial expression and head pose. Extensive experiments demonstrate the superiority of our method on consistent driving motion transfer as well as faithful source identity preservation.
Yating Xu, Yunqi Miao, Evangelos Ververas +2
National University of Singapore · University of Warwick · Imperial College London +1
We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/
Wenbo Ji, Davide Davoli, Zhe Chen +3
Technical University of Munich · Toyota Motor Europe NV/SA · Woven by Toyota
While diffusion models have shown great potential in portrait generation, generating expressive, coherent, and controllable cinematic portrait videos remains a significant challenge. Existing intermediate signals for portrait generation, such as 2D landmarks and parametric models, have limited disentanglement capabilities and cannot express personalized details due to their sparse or low-rank representation. Therefore, existing methods based on these models struggle to accurately preserve subject identity and expressions, hindering the generation of highly expressive portrait videos. To overcome these limitations, we propose a high-fidelity personalized head representation that more effectively disentangles expression and identity. This representation captures both static, subject-specific global geometry and dynamic, expression-related details. Furthermore, we introduce an expression transfer module to achieve personalized transfer of head pose and expression details between different identities. We use this sophisticated and highly expressive head model as a conditional signal to train a diffusion transformer (DiT)-based generator to synthesize richly detailed portrait videos. Extensive experiments on self- and cross-reenactment tasks demonstrate that our method outperforms previous models in terms of identity preservation, expression accuracy, and temporal stability, particularly in capturing fine-grained details of complex motion.