We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalkRT, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.
Figures & tables
Figure 2: Overview of baseline audio-driven video diffusion model Dsrc . Because Dsrc adopts wpose , wexp , and wlip to control the degree of motion, AnyTalk is capable of generating dynamic lip movements while maintaining minimal head rotation.
Figure 3: Domain gap between generated and rendered images. From an input image (left), a video is generated using existing talking-head generation models [ 62 , 75 , 30 , 4 ] (middle). A clear mismatch can be observed between their appearances and the rendered images (right).
Figure 4: Finetuning process of DCsF . The rendered images of the target 3D character and the zero condition Czero are used to form a pair for fine-tuning the model. To preserve the motion prior, only the spatial residual network of denoising UNet is trained (flame icon), while the other layers remain frozen.
Figure 5: Inference of DCsF . Non-zero speech passes through the audio encoder to provide motion information, while the latent representations apose , aexp , and alip are scaled by the corresponding factors wpose , wexp , and wlip to achieve dynamic lip motion and a stable head pose.
Figure 6: Optimization process. Blendshape parameters are optimized so that the target 3D character accurately follows the generated video when rendered.
Tab. 1 : Character configurations for experiments. Each character has different number of vertices, number of meshes, number of blendshape parameters, mesh type, and style.
Figure 7: Lip and chin landmarks exhibit the high average displacement, suboptimal for robust homography estimation.
Figure 8: Qualitative results of AnyTalk using two different audio sources. The top two rows shows results of Emily and Malcolm, while bottom two rows shows result of Victor and Morphy.
Figure 9: From input image (left) and audio, 2D video is generated using DCsF (middle). From this video, final 3D speech animation is optimized (right).
Figure 10: Comparison with baselines on Morphy. Each phoneme and its corresponding frame are presented for comparison. Ours best follows the given audio with wider mouth opening and precise lip closure, while ScanTalk, DiffSpeaker + NFR, and CodeTalker + NFR exhibit only subtle movement.
Methods
LSE-D ↓
LSE-C ↑
Ours
11.304
3.155
ScanTalk
12.152
2.395
DiffSpeaker + NFR
13.857
0.665
CodeTalker + NFR
13.840
0.668
Tab. 2 : Quantitative results comparing our method with baselines. Best denoted in bold.
Figure 11: Visual results of generated video. Ours correctly generated the lip shape without artifacts. On the other hand, w/o CsF made double chin, pinched eye, which does not match with the source character. Variants w/o Czero and w/o freezing module produced small and subtle lip motion.
Figure 12: Results of the ablation study on 3D speech animation. Words corresponding to the rendered images of Malcolm and VMan are presented on the left. While our method correctly produced the lip movements corresponing to the given words, the alternatives produced either only subtle lip movements or artifacts.
Methods
LSE-D ↓
LSE-C ↑
Ours
10.695
3.397
w/o CsF
11.207
3.451
w/o Czero
11.203
2.770
w/o freezing module
14.237
1.228
w/o Ltalk
11.044
2.922
w/o landmark filtering
10.777
3.019
Tab. 3 : Quantitative results from the ablation study. The best and the second best results are denoted in bold and underlined .
Methods
LSE-D ↓
LSE-C ↑
Ours
10.695
3.397
w/ audio-attention tuning
10.796
3.307
w/ motion-attention tuning
10.932
3.149
w/o freezing module
14.237
1.228
Tab. 4 : Quantitative results for fine-tuning UNet blocks. The best and the second best results are denoted in bold and underlined , respectively .
Figure 13: Quantitative results for head pose weights. Lip-sync performance degrades as wpose increase.
Methods
Naturalness (%)
Lip-sync(%)
Ours vs. ScanTalk
78.6
76.8
Ours vs. DiffSpeaker + NFR
99.6
99.4
Ours vs. CodeTalker + NFR
99.8
99.8
Tab. 5 : Selection ratio of our method compared to the baselines.
Figure 14: Network architecture for AnyTalkRT .
Figure 15: Distillation process from AnyTalk to AnyTalkRT . Audio2Feat extracts features from the input audio, while Feat2BS predicts blendshape parameters from the extracted features.
Methods
Time (per frame)
LSE-D ↓
LSE-C ↑
AnyTalkRT
9.09 ms
12.19
2.96
AnyTalk
3.12 s
11.74
3.24
Tab. 6 : Quantitative results for AnyTalkRT compared to AnyTalk . Although distillation enabled real-time performance, the lip-sync metric degraded.
Figure 16: Results of AnyTalk and AnyTalkRT , correctly following the phoneme of the given audio and depicting similar speech motion.
Figure 17: Qualitative results of MEMO CsF and AnyTalk MEMO , showing the generalability of AnyTalk Framework.
Figure 1: Mesh components of each character. From the top left to the bottom right: Victor, Emily, VMan, Malcolm, Morphy.
Figure 2: Mesh alignment for baseline methods in Blender.
Figure 3: Comparison with additional baselines on Emily. Each letter and its corresponding frame for comparison is presented. Ours best follows the given audio while Choi et al. + Hallo show only subtle movements.
Methods
LSE-D ↓
LSE-C ↑
Ours
11.304
3.155
Choi et al. + Hallo
16.571
0.285
Appendix
Tab. 1 : Quantitative results comparing our method with baselines. Best denoted in bold.
Characters
LSE-D ↓
LSE-C ↑
Stylized (Morphy, Malcolm)
11.65
3.07
Non-Stylized (Emily, VMan, Victor)
11.05
3.22
Appendix
Tab. 2 : Comparison of lip-sync metrics between stylized and non-stylized characters. The marginal difference suggests that SyncNet-based embeddings remain robust to artistic stylization.
Methods
LSE-D ↓
LSE-C ↑
ScanTalk
10.787
2.955
DiffSpeaker + NFR
13.393
0.559
CodeTalker + NFR
13.392
0.550
Ours w/o CsF
11.207
3.451
Ours
10.695
3.397
Appendix
Tab. 3 : Quantitative comparison with baselines and our non-personalized variant (w/o CsF ) to evaluate the impact of character-specific tuning.
Methods
Jerk ↓
Talk-related jerk ↓
AnyTalk
27.297
14.419
AnyTalkRT
27.603
14.462
Appendix
Tab. 4 : Quantitative comparison of temporal stability of AnyTalk and AnyTalk_RT.
Figure 4: Multi-view generation results for the same audio segment at time t . The top row shows the input character renders for left, frontal, and right viewpoints. The middle and bottom rows display the corresponding generated video frames at identical timestamps, illustrating inconsistent lip motions and facial expressions across views.
Figure 5: Comparison between original meshes (first and third rows) and 3D reconstructed meshes (second and fourth rows) generated by Hunyuan3D-2. While the reconstructed characters resemble the originals, they lack precise pixel-wise alignment, making them unsuitable for direct motion guidance.
Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.
Federico Nocentini, Thomas Besnier, Claudio Ferrari +2
Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems +1
Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips. While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users' ability to freely control speaking styles. Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio. To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion. Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions. Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.
Xuangeng Chu, Yuan Gan, Ziteng Cui +4
The University of Tokyo · Snap Research, Snap Inc. · RIKEN AIP
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.