We introduce SEDTalker, an emotion-aware framework for speech-driven 3D facial animation that leverages frame-level speech emotion diarization to achieve fine-grained expressive control. Unlike prior approaches that rely on utterance-level or manually specified emotion labels, our method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. The diarized emotion signals are encoded as learned embeddings and used to condition a speech-driven 3D animation model based on a hybrid Transformer-Mamba architecture. This design allows effective disentanglement of linguistic content and emotional style while preserving identity and temporal coherence. We evaluate our approach on a large-scale multi-corpus dataset for speech emotion diarization and on the EmoVOCA dataset for emotional 3D facial animation. Quantitative results demonstrate strong frame-level emotion recognition performance and low geometric and temporal reconstruction errors, while qualitative results show smooth emotion transitions and consistent expression control. These findings highlight the effectiveness of frame-level emotion diarization for expressive and controllable 3D talking head generation.
Figures & tables
Figure 1: SEDTalker is a speech-driven framework for emotion-aware 3D facial animation. By leveraging frame-level speech emotion diarization, the model captures fine-grained emotional cues and directly translates them into 3D facial parameters for precise expressive control.
Dataset
Utterances
Style
Speakers
Contrib.%
MELD [ 3 ]
12,070
Conv.
Multi
20.5
IEMOCAP [ 4 ]
11,032
Script+Improv
10
18.7
JL-Corpus [ 5 ]
10,661
Acted
4
18.1
ESD [ 6 ]
10,500
Professional
10
17.8
CREMA-D [ 7 ]
7,438
Acted
91
12.6
EmoV-DB [ 8 ]
3,353
Varied
4
5.7
Table 1: Dataset composition and characteristics.
Emotion
Utt.
Utt.%
Frames (M)
Frame%
Weight
Happy
16,133
27.4
4.00
32.1
0.31
Angry
15,115
25.7
4.05
32.4
0.34
Sad
10,558
17.9
2.30
18.5
0.48
Neutral
8,041
13.7
0.20
1.6
0.63
Upset
3,573
6.1
1.12
9.0
1.42
Disgust
3,054
5.2
0.53
4.3
1.66
Table 2: Emotion distribution and normalized class weights after frame-level propagation (20 ms).
Emotion
Precision
Recall
F1
Support
Frames %
Happy
0.822
0.773
0.797
595,723
32.2
Angry
0.797
0.824
0.810
594,488
32.1
Sad
0.766
0.736
0.750
351,115
19.0
Neutral
0.766
0.879
0.819
24,052
1.3
Upset
0.694
0.828
0.755
169,368
9.2
Disgust
0.837
0.915
0.874
77,918
4.2
Table 3: Frame-level emotion recognition results on the test set.
Figure 2: Left: Per-emotion precision, recall, and F1-scores on the test set. Disgust achieves the highest F1 (87.4%), while fear remains the most challenging class (62.0%) due to data scarcity. Error bars denote 95% confidence intervals estimated via 1,000 bootstrap resamples. Right: Normalized confusion matrix on the test set. Values represent row-wise percentages (true emotion → predicted emotion). Diagonal entries indicate correct classifications. Major confusions occur between high-arousal emotions ( happy ↔ angry ) and between fear and sad .
Metric
Afraid
Disgust
Irritated1
Sad1
Pleased
Upset
Overall
MVE ↓ ( ×10−3 )
5.9778
5.9473
5.9884
5.9803
5.9919
5.9705
5.9760
LVE ↓ ( ×10−3 )
7.7966
7.8680
7.8839
7.8540
7.7933
7.8876
7.8472
EVE ↓ ( ×10−3 )
7.8083
7.8896
7.9007
7.8738
7.7999
7.9039
7.8627
FFE ↓ ( ×10−3 )
1.8464
1.8410
1.8509
1.8477
1.8607
1.8426
1.8482
FDD ↓ ( ×10−6 )
1.7835
2.1243
2.2790
2.2940
1.3515
2.9052
2.1229
MOD ↓ ( ×10−4 )
1.5946
1.5879
1.6104
1.6169
1.5763
1.5322
1.5864
Table 4: Evaluation metrics with overall performance.
Method
MVE ↓
LVE ↓
EVE ↓
FFE ↓
( ×10−3 )
( ×10−3 )
( ×10−3 )
( ×10−3 )
FaceFormer [18]
6.0063
7.8368
7.8974
1.8618
SEDTalker (Ours)
5.9778
7.7966
7.8083
1.8464
SEDTalker w/o Mamba
6.0358
7.8715
7.8529
1.8780
SEDTalker w/o Transformer
5.9976
7.8229
7.8068
1.8545
Table 5: Quantitative comparison on EmoVOCA test set Afraid emotion (939 train, 120 val, and 117 test). Both methods were trained on identical splits using the same evaluation protocol.
Figure 3: Continuous emotion-controlled facial expression generation. The figure shows temporally ordered facial meshes generated along an emotion trajectory, transitioning across discrete emotion categories ( angry , neutral , happy , and sad ). Despite the discrete labels, facial geometry evolves smoothly with consistent identity preservation, demonstrating continuous interpolation and fine-grained control over expression intensity.
Figure 4: Qualitative comparison of emotional facial animation between our method (Ours) and EmoTalk [ 22 ] . The top example corresponds to an utterance predominantly expressing happy affect with varying intensity, while the bottom example is dominated by anger at different intensity levels. Colored bars indicate the predicted emotion category and intensity over time. Our method more faithfully follows the temporal dynamics of the speech, producing clearer transitions between low, medium, and high affect and more expressive, well-synchronized facial movements compared to EmoTalk.
Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.
Federico Nocentini, Thomas Besnier, Claudio Ferrari +2
Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems +1
Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.
Peng Jia, Li Dai, Zhen Xiao +2
Hefei University of Technology, Hefei 230009, Anhui, China
Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips. While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users' ability to freely control speaking styles. Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio. To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion. Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions. Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.
Xuangeng Chu, Yuan Gan, Ziteng Cui +4
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP