Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis

Chaolong Yang; Yinuo Guo; Kai Yao; Yuyao Yan; Jie Sun; Guangliang Cheng; Shibin Wu; Bin Dong

cs.CVAug 10, 2026

Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis

Authors: Chaolong Yang, Yinuo Guo, Kai Yao, Yuyao Yan, Jie Sun, Guangliang Cheng, Shibin Wu, Bin Dong,

Organizations: University of Liverpool · Xi’an Jiaotong-Liverpool University · Duke Kunshan University · Carnegie Mellon University ·

Abstract

Precise emotion control in audio-driven talking heads remains a challenge due to the reliance on implicit emotion regulation in existing systems, which often leads to indirect and insufficient control. Additionally, training with explicit emotion-related losses across the entire motion space poses significant difficulties due to the inherent trade-off between accurate lip synchronization and fine-grained emotion control. In this paper, we reveal a key finding: although emotional cues are distributed throughout the motion space, concentrating discriminative supervision on less-principal components achieves a better emotion-lip synchronization balance, as principal components mainly encode high-energy articulation and pose variations. Building on this insight, we propose Xemo-Talker, which first learns a neutral speech-to-motion mapping for stable articulation and lip synchronization, and then introduces a lightweight emotion branch guided by less-principal subspace supervision. To enhance emotion control, we design a Tri-Loss consisting of inter-class separation, intra-class compactness, and less-principal contrastive learning. Given an audio input, a reference image, and an emotion label, Xemo-Talker achieves state-of-the-art emotion classification accuracy while maintaining competitive lip synchronization and high inference efficiency, with performance approaching that measured on real videos.The source code is publicly available at https://github.com/chaolongy/Xemo-Talker.

Figures & tables

Panel of Figure 1 — Figure 1: PCA analysis of the 70-D motion space. Principal components mainly capture high-energy articulation and pose, while the less-principal tail provides a lower-interference region for emotion supervision. Head and Tail denote normalized PCA ranges $0$ – $0.1$ and $0.9$ – $1$ , respectively. PCA is computed on the MEAD training split.

Category	Method	Emotion	Explicit emotion	Factor	Less-principal
Category	Method	guidance	supervision	decoupling	supervision
Foundation	Memo	None	$\times$	$\times$	$\times$
Foundation	Hallo3	None	$\times$	$\times$	$\times$
Implicit	GMTalker	Emotion label	$\times$	$\times$	$\times$
Explicit	EAT	Label/Text	$\checkmark$	$\checkmark$	$\times$
Explicit	EDTalk	Audio/Video	$\checkmark$	$\checkmark$	$\times$

Table 1: Positioning of representative talking-head methods.

Figure 2: Overview of Xemo-Talker. (a) The Geometry Motion Predictor learns emotion-agnostic audio-driven motion through diffusion reconstruction, producing the complete 70-D facial motion sequence. (b) With the predictor frozen, the Geometry Emotion Branch encodes the target emotion and injects multi-scale residual features through zero-initialized adapters to refine the motion prediction. (c) The subspace-aware Tri-Loss organizes global emotion representations using classification and prototype alignment, while applying contrastive discrimination to the less-principal PCA projection. The predicted motion is converted into deformed keypoints and rendered by the frozen LivePortrait renderer ( Guo et al. 2024 ) .

Method	Lip Synchronization		Video Quality				Emotion Accuracy
Method	LSE-C $\uparrow$	LSE-D $\downarrow$	FID $\downarrow$	CPBD $\uparrow$	CSIM $\uparrow$	Avg. Rank $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
Real Video	8.04	7.52	0.00	0.47	1.00	–	85.38
EAT ( Gan et al. 2023 ) [ICCV’23]	7.29	8.35	45.79	0.20	0.68	2.00	75.43
EDTalk ( Tan et al. 2024 ) [ECCV’24]	6.75	8.30	73.00	0.16	0.62	4.33	81.02
DICE-Talk ( Tan et al. 2025 ) [MM’25]	6.61	8.76	74.87	0.31	0.63	3.00	42.03
EmoCAST ( Jiang et al. 2025 ) [arXiv’25]	7.08	8.37	43.35	0.20	0.59	3.00	84.26

Table 2: Quantitative comparison with the state-of-the-art methods on MEAD ( Wang et al. 2020 ) test set. Best in bold and second best in underline . Avg. Rank denotes the average ranking over FID, CPBD, and CSIM.

Figure 3: Qualitative comparison on the MEAD test set for sad and surprised emotions. Xemo-Talker produces clear emotional variations while preserving facial details and source identity.

Model	Emotion	Lip	Overall
EAT ( Gan et al. 2023 )	20.31	15.00	21.56
EDTalk ( Tan et al. 2024 )	11.56	8.75	10.62
DICE-Talk ( Tan et al. 2025 )	13.75	32.81	16.56
EmoCAST ( Jiang et al. 2025 )	25.00	17.50	24.06
Xemo-Talker (Ours)	29.38	25.94	27.19

Table 3: User study results (%) across three evaluation metrics. Best in bold and second best in underline .

Variant	Lip Synchronization		Emotion Accuracy
Variant	LSE-C $\uparrow$	LSE-D $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
w/o $\mathcal{L}_{\mathrm{cls}}$	6.91	8.09	72.89
w/o $\mathcal{L}_{\mathrm{cpa}}$	6.27	8.73	78.88
w/o $\mathcal{L}_{\mathrm{lpc}}$	6.23	8.78	79.09
w/o CSFI	6.36	8.21	82.23
Full model	6.37	8.21	85.28

Table 4: Ablation study of the Tri-Loss terms and CSFI. Each variant removes one component from the full model.

Figure 4: Qualitative results of the ablation study for Tri-loss. The emotion category is Contempt.

Method	Lip Synchronization		Emotion Accuracy
Method	LSE-C $\uparrow$	LSE-D $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
$Full_{(0-1)}$	6.28	8.34	75.84
$Head_{(0-0.5)}$	6.36	8.32	79.49
$Tail_{(0.5-1)}$	6.44	8.20	78.88
$Head_{(0-0.3)}$	6.23	8.36	78.38
$Tail_{(0.7-1)}$	6.39	8.23	79.09

Table 5: Effect of Less-Principal Supervision.

Figure 5: Cross emotion transition from angry to happy with fixed audio.

Figure 6: Xemo-Talker results across diverse emotions. The figure shows the model’s ability to generate clear, expressive, and identity-consistent faces for all eight emotions in challenging out-of-domain scenarios.

Method	FPS $\uparrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
EAT ( Gan et al. 2023 )	22.06	75.43
EDTalk ( Tan et al. 2024 )	5.38	81.02
DICE-Talk ( Tan et al. 2025 )	0.73	42.03
EmoCAST ( Jiang et al. 2025 )	0.41	84.26
Ours (10 steps)	27.02	84.77
Ours (30 steps)	18.27	84.57

Table 6: Inference speed and

\text{Acc}_{\textit{emo}}

on a single RTX 4090.

Figure 7: Emotion intensity scaling and interpolation between surprised and happy.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Disgusted and Angry expressions.

Figure 9: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Neutral and Happy expressions.

Figure 10: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Contempt and Fear expressions.

Panel of Figure 11 — Figure 11: Sensitivity study of the three loss weights and their effects on emotion control and articulation quality.

Figure 12: Visual comparison of Fear and Surprised expressions from MEAD ( Wang et al. 2020 ) .

Audio-Driven Facial Animation

Audio-driven facial animation maps speech to synchronized facial motion for realistic, expressive, and controllable talking faces and avatars.

See all 84 papers

GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
Jul 1, 2026Haijie Yang, Zhenyu Zhang, Yixuan Dong +23D Gaussian Splatting Talking Head Generation
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, 215000, China · Tsientang Institute for Advanced Study, Hangzhou, 310000, China
CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation
Aug 15, 2026Peng Jia, Li Dai, Zhen Xiao +2Talking Head Generation Audio-Driven Facial Animation
Hefei University of Technology, Hefei 230009, Anhui, China
EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization
Mar 22, 2026Haolan Xu, Keli Cheng, Lei Wang +2Talking Head Generation Audio-Driven Facial Animation
Michigan State University, East Lansing, MI 48824 · Qualcomm Technologies Inc., San Diego, CA 92121 · University of North Carolina at Chapel Hill, Chapel Hill, NC 27599

Jul 1, 2026J/K move · Enter open · S save

GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting

Haijie Yang, Zhenyu Zhang, Yixuan Dong, Jianjun Qian+1

PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, 215000, China · Tsientang Institute for Advanced Study, Hangzhou, 310000, China

3D Gaussian Splatting Talking Head Generation Audio-Driven Facial Animation Video Generation Facial Expression Synthesis

Open paper AskPDF arXiv

Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/

Abstract

Audio-Driven Facial Animation

Explore similar work

GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization