3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

Zhongju Wang; Zhenhong Sun; Beier Wang; Yifu Wang; Daoyi Dong; Huadong Mo; Hongdong Li

cs.CVFeb 11, 2026

3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

Authors: Zhongju Wang, Zhenhong Sun, Beier Wang, Yifu Wang, Daoyi Dong, Huadong Mo, Hongdong Li

Organizations: University of New South Wales · Australian National University · Vertex Lab · University of Technology Sydney

Abstract

Audio-driven 3D talking avatar generation is increasingly important in virtual communication, digital humans, and interactive media, where avatars must preserve identity, synchronize lip motion with speech, express emotion, and exhibit lifelike spatial dynamics, collectively defining a broader objective of expressivity. However, achieving this remains challenging due to insufficient training data with limited subject identities, narrow audio representations, and restricted explicit controllability. In this paper, we propose 3DXTalker, an expressive 3D talking avatar through data-curated identity modeling, audio-rich representations, and spatial dynamics controllability. 3DXTalker enables scalable identity modeling via 2D-to-3D data curation pipeline and disentangled representations, alleviating data scarcity and improving identity generalization. Then, we introduce frame-wise amplitude and emotional cues beyond standard speech embeddings, ensuring superior lip synchronization and nuanced expression modulation. These cues are unified by a flow-matching-based transformer for coherent facial dynamics. Moreover, 3DXTalker also enables natural head-pose motion generation while supporting stylized control via prompt-based conditioning. Extensive experiments show that 3DXTalker integrates lip synchronization, emotional expression, and head-pose dynamics within a unified framework, achieves superior performance in 3D talking avatar generation.

Figures & tables

Figure 1: Overview of the proposed expressive 3DXTalker system with plug-in semantic control. Given a static reference image and a driving audio, 3DXTalker generates expressive 3D talking avatars with consistent identity, accurate lip synchronization, rich emotional expression, and controllable head pose dynamics. Demo videos are provided in the supplementary material.

Figure 2: Overview of the proposed 3DXTalker dataset. (a) Comparison of three typical 3D talking-head representation paradigms. (b) Diverse sources of the 3DXTalker dataset from both lab and wild environments, together with a rich word-cloud text corpus. (c) Comparison of existing datasets in subject and sample scale, and attribute coverage, highlighting the suitability of the 3DXTalker dataset for expressive 3D talking avatar. Details of datasets are presented in Appendix B.1 .

Figure 3: Overview of 3DXTalker framework. (a) A multi-branch flow-matching transformer fuses identity and audio cues to model disentangled FLAME parameter space. (b) Frame-wise audio amplitude contributes to coherent mouth aperture and head dynamics. (c) Frame-wise emotion embeddings help modulate emotional expressions.

Figure 4: Qualitative comparisons over selected typical baselines. (a) shows the consistency between generated meshes and the reference image. (b) shows better mouth aperture alignment. (c) shows finer emotional expressiveness. (d) shows predicted natural head pose and camera movements. Full baseline comparisons are provided in Appendix C , and more emotion comparisons are offered in Appendix D.2 . We further present downstream applications of 3DXTalker in Appendix G .

Model	Predition	ID	MVE	CSIM	LVE	LSEC	LSED	UFVE	UFDD	Emo-Fid	BA	MR	Throughput
		Identity			Lip-Sync			Expression			Pose	User Study	Efficiency
	Type	Gen.	( $\times 10^{-3}$ ) $\downarrow$	$\uparrow$	( $\times 10^{-4}$ ) $\downarrow$	$\uparrow$	$\downarrow$	( $\times 10^{-3}$ ) $\downarrow$	( $\times 10^{-4}$ ) $\downarrow$	$\downarrow$	$\uparrow$	$\downarrow$	(FPS) $\uparrow$
FaceFormer	Vert	✗	4.704	0.919	1.801	0.751	13.584	4.094	4.970	0.351	0	4.46	74.640
CodeTalker	Vert	✗	4.795	0.920	1.886	0.777	13.311	4.123	3.783	0.374	0	4.63	64.997
SelfTalk	Vert	✗	4.803	0.921	1.863	0.843	13.385	4.106	3.659	0.336	0	4.45	9.083
FaceDiffuser	Vert	✗	4.752	0.933	1.624	0.636	12.732	4.173	3.567	0.398	0	4.58	16.142

Table 1: Quantitative evaluation with baselines over 200 videos. “ID Gen." denotes whether the model supports identity-referred generalization. Our method achieves better performance on 3D-related metrics, demonstrating improved geometric accuracy and temporal coherence, while it performs well on most 2D perceptual metrics. The BA score assesses the rhythmic alignment of head motions with the driving audio. The MR is averaged over four rating dimensions in a study with 26 participants, each evaluating 20 test cases; lower is better. Throughput was measured on an NVIDIA RTX 5090 GPU. The best results are highlighted in bold and the second-best results are underlined.

Model	LVE	MVE	UFDD	UFVE	LSEC	LSED	CSIM
	( $\times 10^{-4}$ ) $\downarrow$	( $\times 10^{-3}$ ) $\downarrow$	( $\times 10^{-4}$ ) $\downarrow$	( $\times 10^{-3}$ ) $\downarrow$	$\uparrow$	$\downarrow$	$\uparrow$
AbsLatent	1.242	1.558	1.479	1.212	1.794	13.348	0.959
w/o $\mathbf{A}_{amp}$	0.971	1.417	1.328	1.133	0.240	14.116	0.975
w/o $\mathbf{A}_{emo}$	0.926	1.414	1.373	1.142	1.308	13.425	0.966
Ours	0.877	1.371	1.317	1.113	1.674	13.330	0.967

Table 2: Ablation results of 3DXTalker. “AbsLatent” uses absolute latent instead of differential ones in Eq. ( 2 ). “w/o

\mathbf{A}_{\text{emo}}

" and “w/o

\mathbf{A}_{amp}

" remove emotional and amplitude embeddings. The best results in bold and second-best underlined .

Figure 5: Visualizations of ablation results from Table 2 . (a) is conducted on the same audio. (b) extracts each emotion from corresponding videos at the same frame. More cases can be found in Appendix F .

Model	Ang.	Con.	Dis.	Fea.	Hap.	Sad	Sur.
DEEPTalk + $E_{C}$	0.637	0.738	0.602	0.686	0.721	0.659	0.640
Ours + $E_{C}$	0.947	0.935	0.953	0.933	0.921	0.939	0.938

Table 3: Emotion-wise cosine similarity between generated and ground-truth FLAME expression parameters. “

E_{C}

" denotes emotion control. Ang. = Angry, Con. = Contempt, Dis. = Disgust, Fea. = Fear, Hap. = Happy, Sad = Sad, Sur. = Surprise. Mesh visualizations and analyses are offered in Appendix D .

Figure 8: Curves for the ground truth and two predicted sequences, showing correlation with the amplitude-driven mouth aperture.

Figure 9: EMOCA modeling pipeline using the FLAME model. The encoder outputs parametric latent codes: $\boldsymbol{\beta}$ for facial shape, $\boldsymbol{\psi}$ for expression, $\boldsymbol{\theta}$ for head pose and jaw pose dynamics, and $\boldsymbol{\delta}$ for fine-grained appearance details (e.g., texture). We linearly vary $\boldsymbol{\theta}$ to demonstrate controllable changes in head and jaw pose, as illustrated in the bottom row.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Representation	Dimension	Annotation	Source	Audio	Sequence	Duration	FPS	Subject	Identity	Expression	Pose	Detail
Vocaset [ 6 ]	Vertices	5,023 $\times$ 3	4D Scan	Lab	Speech	473	0.56h	60	12	-	-	-	-
BIWI [ 12 ]	Vertices	23,370 $\times$ 3	4D Scan	Lab	Speech	238	0.33	25	6	-	-	-	-
Multiface [ 13 ]	Vertices	6,172 $\times$ 3	4D Scan	Lab	Speech	612	0.67h	30	13	-	-	-	-
MMFace4D [ 14 ]	Vertices	35,709 $\times$ 3	4D Scan	Lab	Speech	35,904	36h	30	431	-	-	-	-
BEAT [ 55 ]	BS	52	ARKit	Lab	Speech	2,508	76h	60	30	-	-	-	-
Song2face [ 56 ]	BS	51	ARKit	Lab	Song	-	1.93h	-	7	-	-	-	-

Appendix

Table 4: Comparison of existing 3D talking-head datasets and our dataset. Our dataset provides the most comprehensive 3D fitting FLAME annotations, including identity, expression, pose, and detail, while also covering both lab and wild scenarios with speech and song audio. BS denotes blendshape coefficients, and ARKit refers to Apple’s facial motion capture parameterization.

Dataset	ID	Environment	Year	Raw Resolution	Size	Subject	Total Duration (s)	Hours (h)	Avg. Duration (s/sample)
GRID	V0	Lab	2006	$720\times 576$	6600	34	99257.81	27.57	15.04
RAVDESS	V1	Lab	2018	$1280\times 1024$	613	24	10071.88	2.80	16.43
MEAD	V2	Lab	2020	$1920\times 1080$	1969	60	42868.77	11.91	21.77
VoxCeleb2	V3	Wild	2018	360P $\sim$ 720P	1326	1k+	21528.20	5.98	16.24
HDTF	V4	Wild	2021	720P $\sim$ 1080P	400	300+	55452.08	15.40	138.63
Celebv-HQ	V5	Wild	2022	$512\times 512$	798	700+	13486.20	3.75	16.90

Appendix

Table 5: Statistics of the constructed 3D talking-head dataset via our data integration pipeline.

Dataset	ID	Total Size	Training Set	Test Set
GRID	V0	6600	6570	30
RAVDESS	V1	613	583	30
MEAD	V2	1969	1939	30
VoxCeleb2	V3	1326	1996	30
HDTF	V4	400	350	50
Celebv-HQ	V5	798	768	30

Appendix

Table 6: Dataset split for training and test.

Figure 10: User study interface. Participants are presented with a ground-truth reference image (top) and eight anonymized video samples generated by all models. The videos are randomly shuffled to ensure a blind assessment.

Figure 11: Visualization comparisons illustrating how mouth-aperture patterns align with phonetic symbols across models.

Figure 12: Neutral face (expression with zero vector) and seven emotion templates across six controllable intensity scales.

Algorithm: Extraction of Emotion Expression Templates from MEAD
Input: MEAD expression-code sessions $\{\Psi_{s}\}$ ; emotion label $e$ .
Output: Mean template $\bar{\psi}^{\,e}$ .
1. Initialize frame set: $\mathcal{X}_{e}\leftarrow\emptyset$ .
2. For each session $s$ containing emotion $e$ :
Load expression sequence $\Psi_{s}\in\mathbb{R}^{T_{s}\times 50}$ .
Reshape to frame-level samples.

Appendix

Table 7: Procedure for extracting emotion templates from the MEAD dataset.

Figure 13: More qualitative comparisons of additional four emotion categories (Disgust, Contempt, Fear, Surprise) across representative baselines. 3DXTalker generates more expressive emotion patterns with clearer facial activations than DEEPTalk and EMOTE.

Figure 14: Head model visualization as the control parameter $\theta$ varies. (a) three parameters for head pose; (b) three parameters for jaw pose. They vary linearly with the control parameter.

Figure 15: Comparison of head pose dynamics between the proposed natural micro-movement modeling and the trajectory-controlled head pose. By incorporating a center motion trajectory, the framework achieves both expressive flexibility and robust dynamic control. Trajectory colors indicate temporal progression (dark $\rightarrow$ light).

Figure 16: Amplitude analysis under different emotions, including (a) happy, (b) sad, and (c) angry.

Panel of Figure 17 — Figure 17: Wan2.2 rendering results comparison. Depth video are extracted from 3D mesh sequences generated by our 3DXTalker. Wan2.2 (Fun Control) synthesizes talking videos by conditioning on both the depth video and the reference image, while Wan2.2 (S2V) generates results directly from audio and the reference image. Both Fun Control and S2V models adopt text prompts to guide the generation process.

Lip Synchronization

See all 19 papers

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation
Aug 15, 2026Peng Jia, Li Dai, Zhen Xiao +2Talking Head Generation Audio-Driven Facial Animation
Hefei University of Technology, Hefei 230009, Anhui, China
MegaAvatar: Controllable Talking Avatar Generation
Sep 30, 2026Junyao Gao, Sibo Liu, Weidong Zhang +2Talking Head Generation 3D Avatar Generation
Tencent · Tongji University
CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation
May 28, 2026Xuangeng Chu, Yuan Gan, Ziteng Cui +4Audio-Video Generation Talking Head Generation
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP

Aug 15, 2026J/K move · Enter open · S save

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

Peng Jia, Li Dai, Zhen Xiao, Xueliang Liu+1

Hefei University of Technology, Hefei 230009, Anhui, China

Talking Head Generation Audio-Driven Facial Animation Facial Expression Synthesis 3D Avatar Generation Continuous Control Video Generation

Open paper AskPDF arXiv

Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.

Abstract

Lip Synchronization

Explore similar work

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

MegaAvatar: Controllable Talking Avatar Generation

CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation