Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/
Figures & tables
Figure 1: Comparison between our method and baseline approaches. Identity, lip, and pose are taken from a neutral video, while the emotion source is provided from MELD [ 38 ] (dialogue 5, utterance 8). From top to bottom: ours (C-MET), the label-based method (EAT [ 19 ] ), the existing audio-based method (FLOAT [ 25 ] ), and the image-based method (EDTalk [ 49 ] ). Our results better reflect the target emotional speech ( sarcastic ), exhibiting a more pronounced widening of the lip corners compared to the baselines.
Figure 2: The comparison of emotion condition modules in the pipeline of emotion editing in talking face video at inference.
Figure 3: Overview of the proposed Cross-Modal Emotion Transfer (C-MET). (a) We extract input and target embeddings using pretrained audio and visual encoders, and compute the semantic vectors by subtracting the target embeddings from the inputs. (b) During training, we apply contrastive learning between multimodal tokens—both from visual to audio and audio to visual—to align the representation spaces. (c) A multimodal transformer encoder is used to regress the target expression vectors, guided by the speech vectors. The predicted vectors are then added to the input visual embeddings, which are decoded by a pretrained visual decoder to reconstruct the target video from the neutral video.
Figure 4: Qualitative results for angry (left) and sarcastic (right), respectively.
Loss
Metric
Lrecon
Lcnt
Ldir
Accemo↑
✓
49.43
✓
✓
53.46
✓
✓
✓
55.91
Table 2: We evaluate the impact of ablation on training loss.
Disentanglement
Metric
Network
AITV ↓
Accemo↑
PD-FGC [ 57 ]
1.247
33.36
w/ Ours
1.180
36.82
EDTalk [ 49 ]
2.827
41.99
w/ Ours
2.643
55.91
Table 3: Effect of integrating C-MET into disentanglement networks on MEAD.
Figure 5: Qualitative analysis of ablation in the training loss.
Figure 6: Qualitative analysis of integrating C-MET into disentanglement networks.
Metric
Ours
EAMM
Tie
Ours
EAT
Tie
Ours
EDTalk
Tie
Ours
FLOAT
Tie
Basic Emotion
Emotional Expression (%)
77.8
10.4
11.8
61.6
21.4
17.1
42.4
22.0
35.7
84.5
14.9
0.6
Visual Quality (%)
77.8
10.6
11.6
61.4
22.4
16.3
40.6
23.5
35.9
81.4
18.2
0.4
Lip Synchronization (%)
71.4
14.5
14.1
58.2
24.7
17.1
40.4
23.3
36.3
79.0
20.4
0.6
Extended Emotion
Emotional Expression (%)
91.0
6.7
2.2
80.4
17.1
2.4
51.2
36.5
12.2
86.9
11.8
1.2
Visual Quality (%)
90.8
8.8
0.4
77.8
19.4
2.9
48.0
39.8
12.2
87.1
11.4
1.4
Lip Synchronization (%)
87.6
9.8
2.7
78.4
19.4
2.9
45.3
39.6
15.1
86.7
11.8
1.4
Table 4: User study results across basic and extended emotions. We report the percentage of participants who preferred our method, a baseline, or rated them equally (tie), in terms of emotional expression, visual quality, and lip synchronization. Our method consistently outperforms all baselines across both emotion categories.
Figure 7: The results of continuous emotion editing.
Figure 8: Trend of emotion accuracy as the number of emotional speech samples (“speech-shots”) increases. “Ours (smoothed)” refers to a fitted saturating exponential curve that approximates the overall trend of emotion accuracy, removing local fluctuations in raw measurements for clearer visualization..
Figure 9: Emotion editing results on basic emotions.
Figure 10: Emotion editing results on extended emotions.
Figure 11: Confusion matrices of input (x-axis) and predicted emotions (y-axis) on MEAD across models.
Metric
Audio encoder
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
emotion2vec+large
2.643
90.804
329.862
7.9996
55.91
Qwen2.5-Omni
3.358
88.320
333.695
7.9985
52.06
Qwen2.5-Omni w/o Lcnt
3.358
87.226
332.618
7.9592
51.18
Table 5: Ablation study on audio encoder on MEAD.
Loss
Metric
Lrecon
Lcnt
Ldir
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
✓
2.643
88.951
325.926
7.9892
49.43
✓
✓
2.643
88.082
321.961
8.0018
53.46
✓
✓
✓
2.643
90.804
329.862
7.9996
55.91
Table 6: Quantitative results of ablation in the training loss on MEAD.
Disentanglement
Metric
Network
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
PD-FGC [ 57 ]
1.247
171.464
937.870
6.7265
33.36
w/ Ours
1.180
173.097
453.436
6.7743
36.82
EDTalk [ 49 ]
2.827
76.423
293.904
8.0529
41.99
w/ Ours
2.643
90.804
329.862
7.9996
55.91
Table 7: Quantitative results of integrating C-MET into disentanglement networks on MEAD.
Figure 12: Example interface used in our user study comparing our method with a baseline.
MEAD
CREMA-D
Method
Accemo↑
Accemo↑
Ang
Con
Dis
Fea
Hap
Sad
Sur
Avg
Ang
Dis
Fea
Hap
Sad
Avg
EAMM
20.13
9.04
0.62
0.00
1.19
0.61
99.39
18.81
27.51
1.03
41.03
0.32
23.38
19.15
EAT
94.34
6.02
62.11
17.61
17.26
3.64
93.94
41.82
47.90
14.04
75.08
11.69
47.40
39.97
EDTalk
78.62
1.81
41.61
8.18
77.98
29.09
56.36
41.99
18.77
1.03
20.06
30.19
77.60
29.69
FLOAT
28.30
0.00
0.00
0.00
33.33
0.61
29.70
13.21
45.31
3.77
13.68
3.57
78.90
29.11
Table 8: Emotion-wise accuracy comparison on MEAD and CREMA-D.
Talking Face Generation (TFG) strives to create realistic and emotionally expressive digital faces. While previous TFG works have mastered the creation of naturalistic facial movements, they typically express a fixed target emotion in synthetic videos and lack the ability to exhibit continuously changing and natural expressions like humans do when conveying information. To synthesize realistic videos, we propose a novel task called Emotionally Continuous Talking Face Generation (EC-TFG), which takes a text segment and an emotion description with varying emotions as driving data, aiming to generate a video where the person speaks the text while reflecting the emotional changes within the description. Alongside this, we introduce a customized model, i.e., Temporal-Intensive Emotion Modulated Talking Face Generation (TIE-TFG), which innovatively manages dynamic emotional variations by employing Temporal-Intensive Emotion Fluctuation Modeling, allowing it to provide emotion variation sequences corresponding to the input text to drive continuous facial expression changes in synthesized videos. Extensive evaluations demonstrate our method's exceptional ability to produce smooth emotion transitions and uphold high-quality visuals and motion authenticity across diverse emotional states.
Audio-driven emotional talking face generation aims to synthesize realistic videos with expressive facial dynamics. However, existing methods struggle to balance controllability and visual fidelity. Although implicit representations capture rich semantics, they lack structural guidance, often resulting in averaged emotional expressions. In contrast, explicit geometric methods offer better control over facial expressions but tend to sacrifice high-frequency texture details. To address it, we propose GemTalk, a diffusion-based framework that combines the semantic richness of implicit representations with the structural precision of explicit geometric priors. We introduce a Vision-guided Audio Emotion Projection (V-AEP) module to extract implicit emotional lip and expression features. At the same time, a Diffusion-based Geometric Priors Generator (D-GPG) generates identity-aware blendshape coefficients as explicit structural priors. Crucially, our Geometry-guided Emotion Modulation (GEM) module leverages these geometric priors to recalibrate the magnitude of implicit features, enabling precise, continuous control over emotional expressions, especially emotion intensity, without sacrificing visual quality. Extensive experiments show GemTalk achieves superior performance in photo-realism, and facial emotional dynamics.
Chenggong Hu, Shaoyin Ma, Yi Wang +3
School of Software Technology, Zhejiang University Ningbo, China
Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.
Peng Jia, Li Dai, Zhen Xiao +2
Hefei University of Technology, Hefei 230009, Anhui, China