Audio-driven 3D talking avatar generation is increasingly important in virtual communication, digital humans, and interactive media, where avatars must preserve identity, synchronize lip motion with speech, express emotion, and exhibit lifelike spatial dynamics, collectively defining a broader objective of expressivity. However, achieving this remains challenging due to insufficient training data with limited subject identities, narrow audio representations, and restricted explicit controllability. In this paper, we propose 3DXTalker, an expressive 3D talking avatar through data-curated identity modeling, audio-rich representations, and spatial dynamics controllability. 3DXTalker enables scalable identity modeling via 2D-to-3D data curation pipeline and disentangled representations, alleviating data scarcity and improving identity generalization. Then, we introduce frame-wise amplitude and emotional cues beyond standard speech embeddings, ensuring superior lip synchronization and nuanced expression modulation. These cues are unified by a flow-matching-based transformer for coherent facial dynamics. Moreover, 3DXTalker also enables natural head-pose motion generation while supporting stylized control via prompt-based conditioning. Extensive experiments show that 3DXTalker integrates lip synchronization, emotional expression, and head-pose dynamics within a unified framework, achieves superior performance in 3D talking avatar generation.
Figures & tables
Figure 1: Overview of the proposed expressive 3DXTalker system with plug-in semantic control. Given a static reference image and a driving audio, 3DXTalker generates expressive 3D talking avatars with consistent identity, accurate lip synchronization, rich emotional expression, and controllable head pose dynamics. Demo videos are provided in the supplementary material.
Figure 2: Overview of the proposed 3DXTalker dataset. (a) Comparison of three typical 3D talking-head representation paradigms. (b) Diverse sources of the 3DXTalker dataset from both lab and wild environments, together with a rich word-cloud text corpus. (c) Comparison of existing datasets in subject and sample scale, and attribute coverage, highlighting the suitability of the 3DXTalker dataset for expressive 3D talking avatar. Details of datasets are presented in Appendix B.1 .
Figure 3: Overview of 3DXTalker framework. (a) A multi-branch flow-matching transformer fuses identity and audio cues to model disentangled FLAME parameter space. (b) Frame-wise audio amplitude contributes to coherent mouth aperture and head dynamics. (c) Frame-wise emotion embeddings help modulate emotional expressions.
Figure 4: Qualitative comparisons over selected typical baselines. (a) shows the consistency between generated meshes and the reference image. (b) shows better mouth aperture alignment. (c) shows finer emotional expressiveness. (d) shows predicted natural head pose and camera movements. Full baseline comparisons are provided in Appendix C , and more emotion comparisons are offered in Appendix D.2 . We further present downstream applications of 3DXTalker in Appendix G .
Identity
Lip-Sync
Expression
Pose
User Study
Efficiency
Model
Predition
ID
MVE
CSIM
LVE
LSEC
LSED
UFVE
UFDD
Emo-Fid
BA
MR
Throughput
Type
Gen.
( ×10−3 ) ↓
↑
( ×10−4 ) ↓
↑
↓
( ×10−3 ) ↓
( ×10−4 ) ↓
↓
↑
↓
(FPS) ↑
FaceFormer
Vert
✗
4.704
0.919
1.801
0.751
13.584
4.094
4.970
0.351
0
4.46
74.640
CodeTalker
Vert
✗
4.795
0.920
1.886
0.777
13.311
4.123
3.783
0.374
0
4.63
64.997
SelfTalk
Vert
✗
4.803
0.921
1.863
0.843
13.385
4.106
3.659
0.336
0
4.45
9.083
FaceDiffuser
Vert
✗
4.752
0.933
1.624
0.636
12.732
4.173
3.567
0.398
0
4.58
16.142
Table 1: Quantitative evaluation with baselines over 200 videos. “ID Gen." denotes whether the model supports identity-referred generalization. Our method achieves better performance on 3D-related metrics, demonstrating improved geometric accuracy and temporal coherence, while it performs well on most 2D perceptual metrics. The BA score assesses the rhythmic alignment of head motions with the driving audio. The MR is averaged over four rating dimensions in a study with 26 participants, each evaluating 20 test cases; lower is better. Throughput was measured on an NVIDIA RTX 5090 GPU. The best results are highlighted in bold and the second-best results are underlined.
Model
LVE
MVE
UFDD
UFVE
LSEC
LSED
CSIM
( ×10−4 ) ↓
( ×10−3 ) ↓
( ×10−4 ) ↓
( ×10−3 ) ↓
↑
↓
↑
AbsLatent
1.242
1.558
1.479
1.212
1.794
13.348
0.959
w/o Aamp
0.971
1.417
1.328
1.133
0.240
14.116
0.975
w/o Aemo
0.926
1.414
1.373
1.142
1.308
13.425
0.966
Ours
0.877
1.371
1.317
1.113
1.674
13.330
0.967
Table 2: Ablation results of 3DXTalker. “AbsLatent” uses absolute latent instead of differential ones in Eq. ( 2 ). “w/o Aemo " and “w/o Aamp " remove emotional and amplitude embeddings. The best results in bold and second-best underlined .
Figure 5: Visualizations of ablation results from Table 2 . (a) is conducted on the same audio. (b) extracts each emotion from corresponding videos at the same frame. More cases can be found in Appendix F .
Figure 8
Model
Ang.
Con.
Dis.
Fea.
Hap.
Sad
Sur.
DEEPTalk + EC
0.637
0.738
0.602
0.686
0.721
0.659
0.640
Ours + EC
0.947
0.935
0.953
0.933
0.921
0.939
0.938
Table 3: Emotion-wise cosine similarity between generated and ground-truth FLAME expression parameters. “ EC " denotes emotion control. Ang. = Angry, Con. = Contempt, Dis. = Disgust, Fea. = Fear, Hap. = Happy, Sad = Sad, Sur. = Surprise. Mesh visualizations and analyses are offered in Appendix D .
Figure 8: Curves for the ground truth and two predicted sequences, showing correlation with the amplitude-driven mouth aperture.
Figure 9: EMOCA modeling pipeline using the FLAME model. The encoder outputs parametric latent codes: β for facial shape, ψ for expression, θ for head pose and jaw pose dynamics, and δ for fine-grained appearance details (e.g., texture). We linearly vary θ to demonstrate controllable changes in head and jaw pose, as illustrated in the bottom row.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Representation
Dimension
Annotation
Source
Audio
Sequence
Duration
FPS
Subject
Identity
Expression
Pose
Detail
Vocaset [ 6 ]
Vertices
5,023 × 3
4D Scan
Lab
Speech
473
0.56h
60
12
-
-
-
-
BIWI [ 12 ]
Vertices
23,370 × 3
4D Scan
Lab
Speech
238
0.33
25
6
-
-
-
-
Multiface [ 13 ]
Vertices
6,172 × 3
4D Scan
Lab
Speech
612
0.67h
30
13
-
-
-
-
MMFace4D [ 14 ]
Vertices
35,709 × 3
4D Scan
Lab
Speech
35,904
36h
30
431
-
-
-
-
BEAT [ 55 ]
BS
52
ARKit
Lab
Speech
2,508
76h
60
30
-
-
-
-
Song2face [ 56 ]
BS
51
ARKit
Lab
Song
-
1.93h
-
7
-
-
-
-
Appendix
Table 4: Comparison of existing 3D talking-head datasets and our dataset. Our dataset provides the most comprehensive 3D fitting FLAME annotations, including identity, expression, pose, and detail, while also covering both lab and wild scenarios with speech and song audio. BS denotes blendshape coefficients, and ARKit refers to Apple’s facial motion capture parameterization.
Dataset
ID
Environment
Year
Raw Resolution
Size
Subject
Total Duration (s)
Hours (h)
Avg. Duration (s/sample)
GRID
V0
Lab
2006
720×576
6600
34
99257.81
27.57
15.04
RAVDESS
V1
Lab
2018
1280×1024
613
24
10071.88
2.80
16.43
MEAD
V2
Lab
2020
1920×1080
1969
60
42868.77
11.91
21.77
VoxCeleb2
V3
Wild
2018
360P ∼ 720P
1326
1k+
21528.20
5.98
16.24
HDTF
V4
Wild
2021
720P ∼ 1080P
400
300+
55452.08
15.40
138.63
Celebv-HQ
V5
Wild
2022
512×512
798
700+
13486.20
3.75
16.90
Appendix
Table 5: Statistics of the constructed 3D talking-head dataset via our data integration pipeline.
Dataset
ID
Total Size
Training Set
Test Set
GRID
V0
6600
6570
30
RAVDESS
V1
613
583
30
MEAD
V2
1969
1939
30
VoxCeleb2
V3
1326
1996
30
HDTF
V4
400
350
50
Celebv-HQ
V5
798
768
30
Appendix
Table 6: Dataset split for training and test.
Figure 10: User study interface. Participants are presented with a ground-truth reference image (top) and eight anonymized video samples generated by all models. The videos are randomly shuffled to ensure a blind assessment.
Figure 11: Visualization comparisons illustrating how mouth-aperture patterns align with phonetic symbols across models.
Figure 12: Neutral face (expression with zero vector) and seven emotion templates across six controllable intensity scales.
Algorithm: Extraction of Emotion Expression Templates from MEAD
Input: MEAD expression-code sessions {Ψs} ; emotion label e .
Output: Mean template ψˉe .
1. Initialize frame set: Xe←∅ .
2. For each session s containing emotion e :
Load expression sequence Ψs∈RTs×50 .
Reshape to frame-level samples.
Appendix
Table 7: Procedure for extracting emotion templates from the MEAD dataset.
Figure 13: More qualitative comparisons of additional four emotion categories (Disgust, Contempt, Fear, Surprise) across representative baselines. 3DXTalker generates more expressive emotion patterns with clearer facial activations than DEEPTalk and EMOTE.
Figure 14: Head model visualization as the control parameter θ varies. (a) three parameters for head pose; (b) three parameters for jaw pose. They vary linearly with the control parameter.
Figure 15: Comparison of head pose dynamics between the proposed natural micro-movement modeling and the trajectory-controlled head pose. By incorporating a center motion trajectory, the framework achieves both expressive flexibility and robust dynamic control. Trajectory colors indicate temporal progression (dark → light).
Figure 16: Amplitude analysis under different emotions, including (a) happy, (b) sad, and (c) angry.
Figure 17: Wan2.2 rendering results comparison. Depth video are extracted from 3D mesh sequences generated by our 3DXTalker. Wan2.2 (Fun Control) synthesizes talking videos by conditioning on both the depth video and the reference image, while Wan2.2 (S2V) generates results directly from audio and the reference image. Both Fun Control and S2V models adopt text prompts to guide the generation process.
Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.
Peng Jia, Li Dai, Zhen Xiao +2
Hefei University of Technology, Hefei 230009, Anhui, China
This report presents MegaAvatar, a controllable talking avatar generation framework built on top of the Wan2.2-TI2V-5B model. Compared with previous talking-avatar methods that mainly rely on audio or reference-image conditioning, we introduce additional SMPL-X-derived 3D guidance, enabling global control over body pose and head motion. Specifically, we render the driving SMPL-X sequence into dense mesh frames and encode them with a lightweight 3D convolutional encoder, whose outputs are injected into the latent tokens to provide overall motion control. Furthermore, we extend Wan2.2-TI2V-5B with additional audio and face cross-attention modules to enable fine-grained expression control and preserve the input identity, respectively. In addition, we implement an audio-to-SMPL-X model to predict an SMPL-X sequence conditioned on the reference image and input audio, allowing MegaAvatar to support audio-driven inference without user-provided SMPL-X frames. Experiments show that MegaAvatar achieves high-quality talking avatar generation with controllable body and head motion, speech-synchronized facial expressions, and consistent identity preservation. MegaAvatar also supports inference with flexible resolutions and video lengths. Codes, dataset, models will be avaliable in https://github.com/Jeoyal/MegaAvatar
Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips. While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users' ability to freely control speaking styles. Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio. To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion. Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions. Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.
Xuangeng Chu, Yuan Gan, Ziteng Cui +4
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP