Organizations: Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems · Univ. Lille, CNRS, Centrale Lille, Institut Mines-Telecom, UMR 9189 CRIStAL, F-59000 Lille, France
Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.
Figures & tables
Figure 1: FreeTalk takes an unregistered static face mesh (in blue on the left) and an emotional audio speech and predicts a sequence of deformation fields (in cyan ) matching speech-related motions and expression-related deformations.
Emotion
Any mesh
CodeTalker / FaceFormer / SelfTalk
✗
✗
EmoTalk / Emote / Emovoca
✓
✗
ScanTalk
✗
✓
FreeTalk
✓
✓
Table 1: Comparison of talking head models.
Figure 2: Overview of the proposed approach. ATS takes the speech audio signal A and generates a sequence of landmarks displacements (ΔLt)t=1T . Then, STM takes a static neutral mesh M and maps the sparse motion sequence of landmarks (ΔLt)t=1T to dense vertex motions to predict the final animated mesh sequence (M^t)t=1T .
Figure 3: Overview of the Sparse-To-Mesh (STM) module. STM predicts vertex-wise features on a static mesh M , and landmark features from displacement vectors L . The mesh features are enhanced by a learnable mapping that injects a global embedding of the landmark displacements into each vertex via cross-attention.
LVE ↓
MVE ↓
FDD ↓
DTW ↓
DFD ↓
δM↓
δCd↓
Faceformer (2)
12.7
1.87
14.4
3.41
13.3
3.11
0.98
EmoFace (2)
12.8
1.85
14.9
3.46
13.3
3.10
0.99
FreeTalk (1,2,4,5 + 1,2,4,5)
16.4
2.29
3.35
2.65
11.4
4.61
0.83
FreeTalk (1,2,4,5 + 2)
16.6
2.31
3.52
2.65
11.4
4.60
0.83
FreeTalk (2 + 1,2,4,5)
13.7
2.27
4.93
2.61
11.1
3.23
0.82
FreeTalk (2 + 2)
13.8
2.27
5.12
2.60
11.0
3.25
0.81
Table 2: Quantitative comparison on the MEAD-EMOTE test set. We compare FreeTalk with state-of-the-art speech-driven 3D facial animation methods trained on MEAD-EMOTE. Lower values indicate better performance. Numbered settings follow the dataset order defined in Sec. 4.1 : (1) EmoVOCA, (2) MEAD-EMOTE, (3) VOCAset, (4) BIWI 6 , (5) Multiface. For FreeTalk, “ (a,b + c,d) ” denotes that ATS is trained on the datasets before “+” and STM on those after it.
Figure 4: Qualitative comparison on the MEAD-EMOTE test set. Each panel shows a sequence of rendered frames for a single identity. Left : Disgust ; right : Happy . Numbered settings follow the definition in Table 2 .
Figure 5: Results on unseen topologies.
VOCAset
BIWI 6
Multiface
LVE ↓
MVE ↓
FDD ↓
DTW ↓
DFD ↓
δM↓
δCd↓
LVE ↓
MVE ↓
FDD ↓
DTW ↓
DFD ↓
δM↓
δCd↓
LVE ↓
MVE ↓
FDD ↓
DTW ↓
DFD ↓
δM↓
δCd↓
VOCA (sd)
6.99
0.98
2.66
1.77
7.41
1.39
1.02
5.74
2.59
41.5
1.60
7.89
1.47
0.67
4.92
2.76
55.78
1.56
6.51
0.86
1.23
FaceFormer (sd)
6.12
0.93
2.16
1.33
5.39
0.86
0.58
4.08
2.16
37.1
1.56
6.61
0.87
0.55
2.45
1.45
20.2
0.87
4.13
0.22
0.73
FaceDiffuser (sd)
4.35
0.90
2.43
1.73
6.83
0.81
0.60
4.02
2.12
39.6
1.55
6.50
0.85
0.56
3.55
2.39
29.1
1.45
5.24
0.28
0.77
CodeTalker (sd)
3.55
0.89
2.26
1.33
5.66
0.80
0.55
5.19
2.64
20.6
1.50
6.61
0.91
0.59
4.09
2.38
47.9
1.44
5.95
0.55
0.96
SelfTalk (sd)
5.61
0.91
2.32
1.25
5.43
0.72
0.56
3.63
2.06
35.5
1.60
6.93
1.03
0.66
2.28
1.90
37.4
0.96
4.18
0.25
0.75
Table 3: Cross-dataset generalization results on neutral speech datasets. Lower values indicate better performance. (sd) denotes single-dataset training. Numbered settings follow the definition in Table 2 .
Feature extractor
Deformation Decoder
Shared MLP
DiffusionNet
NJF [ 1 ]
Shared MLP
0.0020
0.0023
0.0018
DiffusionNet [ 46 ]
0.0017
0.0021
0.0018
PoissonNet [ 31 ]
0.0016
0.0020
0.0017
Table 4: Component substitution analysis for the STM module. We report the Lip Vertex Error (LVE) of the STM module when trained and tested on VOCAset, BIWI and MultiFace. Each cell correspond to a different model for the mesh feature extractor (left column) and the deformation decoder (first row).
Figure 6: Vertex-wise L2 error on a predicted deformed mesh. We use different models for the deformation decoder. We highlight how a MLP as decoder tends to predict noisy deformations around the lips. On the opposite, Neural Jacobian Fields tends to smooth out "too much" to capture highly localized deformations around the right of the lower lip area.
Feature combination strategy
LVE
Concat.
0.0035
CA
0.0025
CA + Concat.
0.0023
GCN + CA + Concat. (ours)
0.0021
Table 5: Ablation studies for the feature combination component of the STM module.
Figure 7: Heatmap of the attention scores from one landmark to each mesh vertex at one frame. For a selected frame and landmark (in red on the landmark graph on the left of the figure), how strongly each mesh vertex attends to that landmark inside the model’s cross-attention module. Red values mean that the vertex relies more on the selected landmark when building its time-dependent latent feature, while blue values mean that the landmark has less influence there.
We introduce SEDTalker, an emotion-aware framework for speech-driven 3D facial animation that leverages frame-level speech emotion diarization to achieve fine-grained expressive control. Unlike prior approaches that rely on utterance-level or manually specified emotion labels, our method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. The diarized emotion signals are encoded as learned embeddings and used to condition a speech-driven 3D animation model based on a hybrid Transformer-Mamba architecture. This design allows effective disentanglement of linguistic content and emotional style while preserving identity and temporal coherence. We evaluate our approach on a large-scale multi-corpus dataset for speech emotion diarization and on the EmoVOCA dataset for emotional 3D facial animation. Quantitative results demonstrate strong frame-level emotion recognition performance and low geometric and temporal reconstruction errors, while qualitative results show smooth emotion transitions and consistent expression control. These findings highlight the effectiveness of frame-level emotion diarization for expressive and controllable 3D talking head generation.
Farzaneh Jafari, Stefano Berretti, Anup Basu
University of Alberta, Edmonton, AB, Canada · University of Florence, Florence, Italy
Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.
Peng Jia, Li Dai, Zhen Xiao +2
Hefei University of Technology, Hefei 230009, Anhui, China
We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalkRT, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.