Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/
Figures & tables
Figure 1: Comparison between our method and baseline approaches. Identity, lip, and pose are taken from a neutral video, while the emotion source is provided from MELD [ 38 ] (dialogue 5, utterance 8). From top to bottom: ours (C-MET), the label-based method (EAT [ 19 ] ), the existing audio-based method (FLOAT [ 25 ] ), and the image-based method (EDTalk [ 49 ] ). Our results better reflect the target emotional speech ( sarcastic ), exhibiting a more pronounced widening of the lip corners compared to the baselines.
Figure 2: The comparison of emotion condition modules in the pipeline of emotion editing in talking face video at inference.
Figure 3: Overview of the proposed Cross-Modal Emotion Transfer (C-MET). (a) We extract input and target embeddings using pretrained audio and visual encoders, and compute the semantic vectors by subtracting the target embeddings from the inputs. (b) During training, we apply contrastive learning between multimodal tokens—both from visual to audio and audio to visual—to align the representation spaces. (c) A multimodal transformer encoder is used to regress the target expression vectors, guided by the speech vectors. The predicted vectors are then added to the input visual embeddings, which are decoded by a pretrained visual decoder to reconstruct the target video from the neutral video.
Figure 4: Qualitative results for angry (left) and sarcastic (right), respectively.
Loss
Metric
Lrecon
Lcnt
Ldir
Accemo↑
✓
49.43
✓
✓
53.46
✓
✓
✓
55.91
Table 2: We evaluate the impact of ablation on training loss.
Disentanglement
Metric
Network
AITV ↓
Accemo↑
PD-FGC [ 57 ]
1.247
33.36
w/ Ours
1.180
36.82
EDTalk [ 49 ]
2.827
41.99
w/ Ours
2.643
55.91
Table 3: Effect of integrating C-MET into disentanglement networks on MEAD.
Figure 5: Qualitative analysis of ablation in the training loss.
Figure 6: Qualitative analysis of integrating C-MET into disentanglement networks.
Metric
Ours
EAMM
Tie
Ours
EAT
Tie
Ours
EDTalk
Tie
Ours
FLOAT
Tie
Basic Emotion
Emotional Expression (%)
77.8
10.4
11.8
61.6
21.4
17.1
42.4
22.0
35.7
84.5
14.9
0.6
Visual Quality (%)
77.8
10.6
11.6
61.4
22.4
16.3
40.6
23.5
35.9
81.4
18.2
0.4
Lip Synchronization (%)
71.4
14.5
14.1
58.2
24.7
17.1
40.4
23.3
36.3
79.0
20.4
0.6
Extended Emotion
Emotional Expression (%)
91.0
6.7
2.2
80.4
17.1
2.4
51.2
36.5
12.2
86.9
11.8
1.2
Visual Quality (%)
90.8
8.8
0.4
77.8
19.4
2.9
48.0
39.8
12.2
87.1
11.4
1.4
Lip Synchronization (%)
87.6
9.8
2.7
78.4
19.4
2.9
45.3
39.6
15.1
86.7
11.8
1.4
Table 4: User study results across basic and extended emotions. We report the percentage of participants who preferred our method, a baseline, or rated them equally (tie), in terms of emotional expression, visual quality, and lip synchronization. Our method consistently outperforms all baselines across both emotion categories.
Figure 7: The results of continuous emotion editing.
Figure 8: Trend of emotion accuracy as the number of emotional speech samples (“speech-shots”) increases. “Ours (smoothed)” refers to a fitted saturating exponential curve that approximates the overall trend of emotion accuracy, removing local fluctuations in raw measurements for clearer visualization..
Figure 9: Emotion editing results on basic emotions.
Figure 10: Emotion editing results on extended emotions.
Figure 11: Confusion matrices of input (x-axis) and predicted emotions (y-axis) on MEAD across models.
Metric
Audio encoder
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
emotion2vec+large
2.643
90.804
329.862
7.9996
55.91
Qwen2.5-Omni
3.358
88.320
333.695
7.9985
52.06
Qwen2.5-Omni w/o Lcnt
3.358
87.226
332.618
7.9592
51.18
Table 5: Ablation study on audio encoder on MEAD.
Loss
Metric
Lrecon
Lcnt
Ldir
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
✓
2.643
88.951
325.926
7.9892
49.43
✓
✓
2.643
88.082
321.961
8.0018
53.46
✓
✓
✓
2.643
90.804
329.862
7.9996
55.91
Table 6: Quantitative results of ablation in the training loss on MEAD.
Disentanglement
Metric
Network
AITV ↓
FID ↓
FVD ↓
Syncconf↑
Accemo↑
PD-FGC [ 57 ]
1.247
171.464
937.870
6.7265
33.36
w/ Ours
1.180
173.097
453.436
6.7743
36.82
EDTalk [ 49 ]
2.827
76.423
293.904
8.0529
41.99
w/ Ours
2.643
90.804
329.862
7.9996
55.91
Table 7: Quantitative results of integrating C-MET into disentanglement networks on MEAD.
Figure 12: Example interface used in our user study comparing our method with a baseline.
MEAD
CREMA-D
Method
Accemo↑
Accemo↑
Ang
Con
Dis
Fea
Hap
Sad
Sur
Avg
Ang
Dis
Fea
Hap
Sad
Avg
EAMM
20.13
9.04
0.62
0.00
1.19
0.61
99.39
18.81
27.51
1.03
41.03
0.32
23.38
19.15
EAT
94.34
6.02
62.11
17.61
17.26
3.64
93.94
41.82
47.90
14.04
75.08
11.69
47.40
39.97
EDTalk
78.62
1.81
41.61
8.18
77.98
29.09
56.36
41.99
18.77
1.03
20.06
30.19
77.60
29.69
FLOAT
28.30
0.00
0.00
0.00
33.33
0.61
29.70
13.21
45.31
3.77
13.68
3.57
78.90
29.11
Table 8: Emotion-wise accuracy comparison on MEAD and CREMA-D.