Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis

Chaolong Yang; Yinuo Guo; Kai Yao; Yuyao Yan; Jie Sun; Guangliang Cheng; Shibin Wu; Bin Dong

cs.CVAug 10, 2026

Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis

Authors: Chaolong Yang, Yinuo Guo, Kai Yao, Yuyao Yan, Jie Sun, Guangliang Cheng, Shibin Wu, Bin Dong,

Organizations: University of Liverpool · Xi’an Jiaotong-Liverpool University · Duke Kunshan University · Carnegie Mellon University ·

Abstract

Precise emotion control in audio-driven talking heads remains a challenge due to the reliance on implicit emotion regulation in existing systems, which often leads to indirect and insufficient control. Additionally, training with explicit emotion-related losses across the entire motion space poses significant difficulties due to the inherent trade-off between accurate lip synchronization and fine-grained emotion control. In this paper, we reveal a key finding: although emotional cues are distributed throughout the motion space, concentrating discriminative supervision on less-principal components achieves a better emotion-lip synchronization balance, as principal components mainly encode high-energy articulation and pose variations. Building on this insight, we propose Xemo-Talker, which first learns a neutral speech-to-motion mapping for stable articulation and lip synchronization, and then introduces a lightweight emotion branch guided by less-principal subspace supervision. To enhance emotion control, we design a Tri-Loss consisting of inter-class separation, intra-class compactness, and less-principal contrastive learning. Given an audio input, a reference image, and an emotion label, Xemo-Talker achieves state-of-the-art emotion classification accuracy while maintaining competitive lip synchronization and high inference efficiency, with performance approaching that measured on real videos.The source code is publicly available at https://github.com/chaolongy/Xemo-Talker.

Figures & tables

Panel of Figure 1 — Figure 1: PCA analysis of the 70-D motion space. Principal components mainly capture high-energy articulation and pose, while the less-principal tail provides a lower-interference region for emotion supervision. Head and Tail denote normalized PCA ranges $0$ – $0.1$ and $0.9$ – $1$ , respectively. PCA is computed on the MEAD training split.

Category	Method	Emotion	Explicit emotion	Factor	Less-principal
Category	Method	guidance	supervision	decoupling	supervision
Foundation	Memo	None	$\times$	$\times$	$\times$
Foundation	Hallo3	None	$\times$	$\times$	$\times$
Implicit	GMTalker	Emotion label	$\times$	$\times$	$\times$
Explicit	EAT	Label/Text	$\checkmark$	$\checkmark$	$\times$
Explicit	EDTalk	Audio/Video	$\checkmark$	$\checkmark$	$\times$

Table 1: Positioning of representative talking-head methods.

Figure 2: Overview of Xemo-Talker. (a) The Geometry Motion Predictor learns emotion-agnostic audio-driven motion through diffusion reconstruction, producing the complete 70-D facial motion sequence. (b) With the predictor frozen, the Geometry Emotion Branch encodes the target emotion and injects multi-scale residual features through zero-initialized adapters to refine the motion prediction. (c) The subspace-aware Tri-Loss organizes global emotion representations using classification and prototype alignment, while applying contrastive discrimination to the less-principal PCA projection. The predicted motion is converted into deformed keypoints and rendered by the frozen LivePortrait renderer ( Guo et al. 2024 ) .

Method	Lip Synchronization		Video Quality				Emotion Accuracy
Method	LSE-C $\uparrow$	LSE-D $\downarrow$	FID $\downarrow$	CPBD $\uparrow$	CSIM $\uparrow$	Avg. Rank $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
Real Video	8.04	7.52	0.00	0.47	1.00	–	85.38
EAT ( Gan et al. 2023 ) [ICCV’23]	7.29	8.35	45.79	0.20	0.68	2.00	75.43
EDTalk ( Tan et al. 2024 ) [ECCV’24]	6.75	8.30	73.00	0.16	0.62	4.33	81.02
DICE-Talk ( Tan et al. 2025 ) [MM’25]	6.61	8.76	74.87	0.31	0.63	3.00	42.03
EmoCAST ( Jiang et al. 2025 ) [arXiv’25]	7.08	8.37	43.35	0.20	0.59	3.00	84.26

Table 2: Quantitative comparison with the state-of-the-art methods on MEAD ( Wang et al. 2020 ) test set. Best in bold and second best in underline . Avg. Rank denotes the average ranking over FID, CPBD, and CSIM.

Figure 3: Qualitative comparison on the MEAD test set for sad and surprised emotions. Xemo-Talker produces clear emotional variations while preserving facial details and source identity.

Model	Emotion	Lip	Overall
EAT ( Gan et al. 2023 )	20.31	15.00	21.56
EDTalk ( Tan et al. 2024 )	11.56	8.75	10.62
DICE-Talk ( Tan et al. 2025 )	13.75	32.81	16.56
EmoCAST ( Jiang et al. 2025 )	25.00	17.50	24.06
Xemo-Talker (Ours)	29.38	25.94	27.19

Table 3: User study results (%) across three evaluation metrics. Best in bold and second best in underline .

Variant	Lip Synchronization		Emotion Accuracy
Variant	LSE-C $\uparrow$	LSE-D $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
w/o $\mathcal{L}_{\mathrm{cls}}$	6.91	8.09	72.89
w/o $\mathcal{L}_{\mathrm{cpa}}$	6.27	8.73	78.88
w/o $\mathcal{L}_{\mathrm{lpc}}$	6.23	8.78	79.09
w/o CSFI	6.36	8.21	82.23
Full model	6.37	8.21	85.28

Table 4: Ablation study of the Tri-Loss terms and CSFI. Each variant removes one component from the full model.

Figure 4: Qualitative results of the ablation study for Tri-loss. The emotion category is Contempt.

Method	Lip Synchronization		Emotion Accuracy
Method	LSE-C $\uparrow$	LSE-D $\downarrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
$Full_{(0-1)}$	6.28	8.34	75.84
$Head_{(0-0.5)}$	6.36	8.32	79.49
$Tail_{(0.5-1)}$	6.44	8.20	78.88
$Head_{(0-0.3)}$	6.23	8.36	78.38
$Tail_{(0.7-1)}$	6.39	8.23	79.09

Table 5: Effect of Less-Principal Supervision.

Figure 5: Cross emotion transition from angry to happy with fixed audio.

Figure 6: Xemo-Talker results across diverse emotions. The figure shows the model’s ability to generate clear, expressive, and identity-consistent faces for all eight emotions in challenging out-of-domain scenarios.

Method	FPS $\uparrow$	$\text{Acc}_{\textit{emo}}$ $\uparrow$
EAT ( Gan et al. 2023 )	22.06	75.43
EDTalk ( Tan et al. 2024 )	5.38	81.02
DICE-Talk ( Tan et al. 2025 )	0.73	42.03
EmoCAST ( Jiang et al. 2025 )	0.41	84.26
Ours (10 steps)	27.02	84.77
Ours (30 steps)	18.27	84.57

Table 6: Inference speed and

\text{Acc}_{\textit{emo}}

on a single RTX 4090.

Figure 7: Emotion intensity scaling and interpolation between surprised and happy.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Disgusted and Angry expressions.

Figure 9: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Neutral and Happy expressions.

Figure 10: Qualitative comparison on the MEAD ( Wang et al. 2020 ) test set for Contempt and Fear expressions.

Panel of Figure 11 — Figure 11: Sensitivity study of the three loss weights and their effects on emotion control and articulation quality.

Figure 12: Visual comparison of Fear and Surprised expressions from MEAD ( Wang et al. 2020 ) .

Audio-Driven Facial Animation

Audio-driven facial animation maps speech to synchronized facial motion for realistic, expressive, and controllable talking faces and avatars.

See all 84 papers

Jul 1, 2026

GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting

Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/

Haijie Yang, Zhenyu Zhang, Yixuan Dong +2

PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, 215000, China · Tsientang Institute for Advanced Study, Hangzhou, 310000, China

3D Gaussian Splatting Talking Head Generation Audio-Driven Facial Animation Video Generation Facial Expression Synthesis

Ask PDF

Aug 15, 2026

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.

Peng Jia, Li Dai, Zhen Xiao +2

Hefei University of Technology, Hefei 230009, Anhui, China

Talking Head Generation Audio-Driven Facial Animation Facial Expression Synthesis 3D Avatar Generation Continuous Control Video Generation

Ask PDF

Mar 22, 2026

EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization

Audio-driven 3D talking head synthesis has advanced rapidly with Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). By leveraging rich pre-trained priors, few-shot methods enable instant personalization from just a few seconds of video. However, under expressive facial motion, existing few-shot approaches often suffer from geometric instability and audio-emotion mismatch, highlighting the need for more effective emotion-aware motion modeling. In this work, we present EmoTaG, a few-shot emotion-aware 3D talking head synthesis framework built on the Pretrain-and-Adapt paradigm. Our key insight is to reformulate motion prediction in a structured FLAME parameter space rather than directly deforming 3D Gaussians, thereby introducing explicit geometric priors that improve motion stability. Building upon this, we propose a Gated Residual Motion Network (GRMN), which captures emotional prosody from audio while supplementing head pose and upper-face cues absent from audio, enabling expressive and coherent motion generation. Extensive experiments demonstrate that EmoTaG achieves state-of-the-art performance in emotional expressiveness, lip synchronization, visual realism, and motion stability.

Haolan Xu, Keli Cheng, Lei Wang +2

Michigan State University, East Lansing, MI 48824 · Qualcomm Technologies Inc., San Diego, CA 92121 · University of North Carolina at Chapel Hill, Chapel Hill, NC 27599

Talking Head Generation Audio-Driven Facial Animation Facial Expression Synthesis 3D Gaussian Splatting

Ask PDF

Abstract

Audio-Driven Facial Animation

Explore similar work

GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization