SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

Farzaneh Jafari; Stefano Berretti; Anup Basu

cs.CVApr 14, 2026

SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

Authors: Farzaneh Jafari, Stefano Berretti, Anup Basu

Organizations: University of Alberta, Edmonton, AB, Canada · University of Florence, Florence, Italy

Abstract

We introduce SEDTalker, an emotion-aware framework for speech-driven 3D facial animation that leverages frame-level speech emotion diarization to achieve fine-grained expressive control. Unlike prior approaches that rely on utterance-level or manually specified emotion labels, our method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. The diarized emotion signals are encoded as learned embeddings and used to condition a speech-driven 3D animation model based on a hybrid Transformer-Mamba architecture. This design allows effective disentanglement of linguistic content and emotional style while preserving identity and temporal coherence. We evaluate our approach on a large-scale multi-corpus dataset for speech emotion diarization and on the EmoVOCA dataset for emotional 3D facial animation. Quantitative results demonstrate strong frame-level emotion recognition performance and low geometric and temporal reconstruction errors, while qualitative results show smooth emotion transitions and consistent expression control. These findings highlight the effectiveness of frame-level emotion diarization for expressive and controllable 3D talking head generation.

Figures & tables

Figure 1: SEDTalker is a speech-driven framework for emotion-aware 3D facial animation. By leveraging frame-level speech emotion diarization, the model captures fine-grained emotional cues and directly translates them into 3D facial parameters for precise expressive control.

Dataset	Utterances	Style	Speakers	Contrib.%
MELD [ 3 ]	12,070	Conv.	Multi	20.5
IEMOCAP [ 4 ]	11,032	Script+Improv	10	18.7
JL-Corpus [ 5 ]	10,661	Acted	4	18.1
ESD [ 6 ]	10,500	Professional	10	17.8
CREMA-D [ 7 ]	7,438	Acted	91	12.6
EmoV-DB [ 8 ]	3,353	Varied	4	5.7

Table 1: Dataset composition and characteristics.

Emotion	Utt.	Utt.%	Frames (M)	Frame%	Weight
Happy	16,133	27.4	4.00	32.1	0.31
Angry	15,115	25.7	4.05	32.4	0.34
Sad	10,558	17.9	2.30	18.5	0.48
Neutral	8,041	13.7	0.20	1.6	0.63
Upset	3,573	6.1	1.12	9.0	1.42
Disgust	3,054	5.2	0.53	4.3	1.66

Table 2: Emotion distribution and normalized class weights after frame-level propagation (20 ms).

Emotion	Precision	Recall	F1	Support	Frames %
Happy	0.822	0.773	0.797	595,723	32.2
Angry	0.797	0.824	0.810	594,488	32.1
Sad	0.766	0.736	0.750	351,115	19.0
Neutral	0.766	0.879	0.819	24,052	1.3
Upset	0.694	0.828	0.755	169,368	9.2
Disgust	0.837	0.915	0.874	77,918	4.2

Table 3: Frame-level emotion recognition results on the test set.

Figure 2: Left: Per-emotion precision, recall, and F1-scores on the test set. Disgust achieves the highest F1 (87.4%), while fear remains the most challenging class (62.0%) due to data scarcity. Error bars denote 95% confidence intervals estimated via 1,000 bootstrap resamples. Right: Normalized confusion matrix on the test set. Values represent row-wise percentages (true emotion $\rightarrow$ predicted emotion). Diagonal entries indicate correct classifications. Major confusions occur between high-arousal emotions ( happy $\leftrightarrow$ angry ) and between fear and sad .

Metric	Afraid	Disgust	Irritated1	Sad1	Pleased	Upset	Overall
MVE $\downarrow$ ( $\times 10^{-3}$ )	5.9778	5.9473	5.9884	5.9803	5.9919	5.9705	5.9760
LVE $\downarrow$ ( $\times 10^{-3}$ )	7.7966	7.8680	7.8839	7.8540	7.7933	7.8876	7.8472
EVE $\downarrow$ ( $\times 10^{-3}$ )	7.8083	7.8896	7.9007	7.8738	7.7999	7.9039	7.8627
FFE $\downarrow$ ( $\times 10^{-3}$ )	1.8464	1.8410	1.8509	1.8477	1.8607	1.8426	1.8482
FDD $\downarrow$ ( $\times 10^{-6}$ )	1.7835	2.1243	2.2790	2.2940	1.3515	2.9052	2.1229
MOD $\downarrow$ ( $\times 10^{-4}$ )	1.5946	1.5879	1.6104	1.6169	1.5763	1.5322	1.5864

Table 4: Evaluation metrics with overall performance.

Method	MVE $\downarrow$	LVE $\downarrow$	EVE $\downarrow$	FFE $\downarrow$
	( $\times 10^{-3}$ )	( $\times 10^{-3}$ )	( $\times 10^{-3}$ )	( $\times 10^{-3}$ )
FaceFormer [18]	6.0063	7.8368	7.8974	1.8618
SEDTalker (Ours)	5.9778	7.7966	7.8083	1.8464
SEDTalker w/o Mamba	6.0358	7.8715	7.8529	1.8780
SEDTalker w/o Transformer	5.9976	7.8229	7.8068	1.8545

Table 5: Quantitative comparison on EmoVOCA test set Afraid emotion (939 train, 120 val, and 117 test). Both methods were trained on identical splits using the same evaluation protocol.

Figure 3: Continuous emotion-controlled facial expression generation. The figure shows temporally ordered facial meshes generated along an emotion trajectory, transitioning across discrete emotion categories ( angry , neutral , happy , and sad ). Despite the discrete labels, facial geometry evolves smoothly with consistent identity preservation, demonstrating continuous interpolation and fine-grained control over expression intensity.

Figure 4: Qualitative comparison of emotional facial animation between our method (Ours) and EmoTalk [ 22 ] . The top example corresponds to an utterance predominantly expressing happy affect with varying intensity, while the bottom example is dominated by anger at different intensity levels. Colored bars indicate the predicted emotion category and intensity over time. Our method more faithfully follows the temporal dynamics of the speech, producing clearer transitions between low, medium, and high affect and more expressive, well-synchronized facial movements compared to EmoTalk.

Explore similar work

FreeTalk: Emotional Topology-Free 3D Talking Heads
Mar 16, 2026Federico Nocentini, Thomas Besnier, Claudio Ferrari +2Audio-Driven Facial Animation
Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems +1
CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation
Aug 15, 2026Peng Jia, Li Dai, Zhen Xiao +2Talking Head Generation Audio-Driven Facial Animation
Hefei University of Technology, Hefei 230009, Anhui, China
CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation
May 28, 2026Xuangeng Chu, Yuan Gan, Ziteng Cui +4Audio-Video Generation Talking Head Generation
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP

Mar 16, 2026J/K move · Enter open · S save

FreeTalk: Emotional Topology-Free 3D Talking Heads

Federico Nocentini, Thomas Besnier, Claudio Ferrari, Stefano Berretti+1

Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems+1

Audio-Driven Facial Animation

Open paper AskPDF arXiv

Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.

Abstract

Explore similar work

FreeTalk: Emotional Topology-Free 3D Talking Heads

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation