Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.
Figures & tables
Figure 1: How do we model physically grounded and articulatory-consistent facial motion from speech? (a) Five visible mesh-surface anchors as motion reference points: upper lip (UL), lower lip (LL), left mouth corner (LMC), right mouth corner (RMC), and Jaw. (b) From these anchors, we describe visible articulation with three directional articulatory motions: horizontal rounding/spreading, vertical opening/closing, and depth-wise protrusion/retraction. (c) For an utterance, the trajectories of these motions form temporally localized articulatory events that align with acoustic/phonemic cues.
Figure 2: Overall architecture of our articulation-aware framework. SAM retrieves directional articulatory motion features (Spreading–Rounding, Opening–Closing, Protrusion–Retraction) from speech. TAC composes them into surface-consistent 3D facial motion under the mesh topology.
Dataset
Methods
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
VOCASET
FaceFormer
0.878
0.346
0.143
0.186
CodeTalker
0.913
0.397
0.160
0.204
SelfTalk
0.844
0.305
0.148
0.174
Imitator
0.898
0.384
0.179
0.199
ScanTalk
0.878
0.310
0.152
0.168
UniTalker
0.820
0.282
0.101
0.188
Table 1: Quantitative results on VOCASET and TFHP.
Methods
Distance Error
Velocity Error
ICW ↓
ILD ↓
LP ↓
ICW ↓
ILD ↓
LP ↓
FaceFormer
1.764
4.558
2.429
0.581
1.850
1.230
CodeTalker
1.725
4.777
2.808
0.552
2.009
1.343
SelfTalk
1.680
4.031
2.470
0.544
1.958
1.275
Imitator
1.731
4.851
2.526
0.571
1.873
1.205
ScanTalk
1.737
4.034
2.485
0.558
1.854
1.285
Table 2: Visible articulatory distance and velocity errors on VOCASET.
Figure 3: Radar plot of directional lip-motion tendency strength relative to ground truth.
Figure 4: Sequence-level qualitative results on TFHP. For each method, mesh renderings (top) and per-vertex L1 error against the ground truth (bottom, color-coded from 0 (blue) to 0.015 (red)) are shown over time at phoneme-aligned frames.
Figure 5: Slit-scan visualizations for sequence-level comparison. (a) lip ROI and landmarks (UL, LL, LMC, RMC) defining horizontal (LMC–RMC) and vertical (UL–LL) slit lines. (b) Horizontal and (c) Vertical slit-scans over time, reflecting lip spreading (width) and lip opening (height), respectively.
SAM
TAC
FVE↓
LVE↓
LDTW ↓
✓
✗
0.812
0.255
0.161
✓
✓
0.771
0.235
0.132
Table 3: Ablation of TAC module.
Figure 6: Analysis of Directional articulatory trajectories.
Competitors
Lip Sync
Realism
(% Ours)
(% Ours)
vs. FaceFormer
71.3
65.7
vs. CodeTalker
72.2
75.0
vs. Imitator
77.8
73.1
vs. ScanTalk
73.1
79.6
vs. UniTalker
65.7
67.6
Table 4: User study: pairwise preference rates (%) on VOCASET.
Table S1: Per-direction architecture of the motion autoencoder used to produce zδt . Three encoder–decoder pairs are jointly trained, one for each δ∈{S,O,P} .
Metric
Unit
VOCASET
TFHP
FVE
m
×10−3
×10−2
LVE
m 2
×10−4
×10−3
FDD
m 2
×10−8
×10−8
LDTW
–
×10−3
×10−3
ICW / ILD / LP
m
×10−3
ICW / ILD / LP velocity
m/frame
×10−3
Appendix
Table S2: Scale factors and units for all reported metrics.
Figure S1: FVE and LVE on VOCASET as the number of memory slot N . N=32 ( ⋆ ) achieves the lowest error on both metrics.
Figure S2: Per-phoneme rendered meshes and S / O / P magnitude heatmaps with and without directional decomposition. Each row shows a representative phoneme (/u/, /a/, /o/).
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
w/o directional articulatory decomposition
0.842
0.266
0.122
0.181
w/ directional articulatory decomposition ( S / O / P )
0.771
0.235
0.097
0.132
Appendix
Table S3: Ablation on directional articulatory decomposition (VOCASET).
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
w/o Mkey,Mval,δ
0.823
0.275
0.121
0.188
Full (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S4: Ablation on the explicit motion memory in SAM (VOCASET).
Query
Refinement
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
Content
Content
0.815
0.277
0.127
0.156
Acoustic
Acoustic
0.850
0.372
0.158
0.199
Acoustic
Content (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S5: Ablation on the Query–Refinement design in SAM (VOCASET). “Acoustic” denotes Mel-spectrogram features and “Content” denotes HuBERT features.
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
w/o Lstore
0.811
0.267
0.117
0.144
w/ Lstore (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S6: Ablation on the value memory store loss (VOCASET).
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
SAM + MLP composition
0.799
0.243
0.103
0.146
SAM + TAC (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S7: Topology-independent (MLP) vs. topology-aware (Ours) composition on VOCASET.
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
w/o residual (feedforward)
0.782
0.238
0.111
0.134
w/ residual (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S8: Ablation on the internal residual connection in TAC (VOCASET). The residual path is required for stable optimization across stacked GCN+TCN blocks.
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
w/o Llap
0.788
0.236
0.107
0.139
w/ Llap (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S9: Ablation on the graph Laplacian consistency loss (VOCASET). Llap constrains local surface curvature and improves overall motion fidelity.
Configuration
FVE ↓
LVE ↓
FDD ↓
LDTW ↓
Unified single-stage (from scratch)
0.801
0.245
0.118
0.142
Pretrained + unfrozen fine-tuning
0.798
0.244
0.122
0.144
Pretrained + frozen (Ours)
0.771
0.235
0.097
0.132
Appendix
Table S10: Ablation on the training schedule (VOCASET). Pretraining SAM and freezing it during TAC training outperforms both end-to-end fine-tuning and unified single-stage training.
Figure S3: Effect of half-window size Ω on FVE and LVE (VOCASET). Ω=0 corresponds to frame-independent retrieval without temporal motion context.
Methods
FVE ( ×10−1 ) ↓
LVE ( ×10−2 ) ↓
FDD ( ×10−5 ) ↓
LDTW ( ×10−2 ) ↓
CodeTalker
0.124
0.127
0.461
0.319
SelfTalk
0.120
0.114
0.148
0.285
UniTalker
0.112
0.106
0.353
0.283
Ours (SAM only)
0.115
0.113
0.557
0.296
Ours (SAM+TAC)
0.110
0.106
0.351
0.275
Appendix
Table S11: Topology generalization on BIWI (23,370 vertices).
Methods
Distance Error
Velocity Error
ICW ↓
ILD ↓
LP ↓
ICW ↓
ILD ↓
LP ↓
Mimic
4.320
5.326
5.968
1.460
2.593
2.871
DiffPoseTalk
5.483
5.860
7.006
1.221
2.168
2.438
ARTalk
5.788
6.049
7.378
1.501
2.801
2.644
Ours
4.299
4.829
5.859
1.168
2.083
2.414
Appendix
Table S12: Visible articulatory distance and velocity errors on TFHP.
Figure S4: Stage-wise qualitative comparison on TFHP. Top row: GT. Middle row: Stage 1 (SAM only) result and per-vertex error. Bottom row: Stage 2 (SAM + TAC) result and per-vertex error.
Figure S5: Visualization of the softmax distribution softmax(τα) underlying the TAC modulation gate m on the FLAME template. (a) Per-vertex distribution. (b) Anatomical region segmentation. (c) Region-grouped mean, with the dashed line indicating the uniform baseline 1/V .
Figure S6: Consistency of acoustic-bridge memory addressing vectors. For the same phoneme class, the addressing weights repeatedly peak at similar slots across occurrences, suggesting stable, reusable retrieval.
Figure S7: Visible articulatory lip-motion trajectories of our method and the ground truth along three directions (Spreading–Rounding, Opening–Closing, Protrusion–Retraction). Highlighted segments correspond to the words “Strong”, “support”, and “nation” in the utterance, with GT and Ours mesh sequences shown below.
Figure S8: Additional example of three directional articulatory trajectories (Spreading–Rounding, Opening–Closing, Protrusion–Retraction) for the utterance “I’ve heard directly from our public safety.”
Figure S9: Additional qualitative comparisons on VOCASET and TFHP.
Figure S10: Examples of the conducted user study.
Method
Forehead ↓
Eyes ↓
Scalp ↓
(133 verts)
(574 verts)
(489 verts)
Mimic
0.020
0.099
0.010
DiffPoseTalk
0.015
0.042
0.008
ARTalk
0.044
0.060
0.011
Ours
0.023
0.102
0.010
Appendix
Table S13: Per-region FDD on TFHP ( ×10−7 m 2 , lower is better). Regions follow the official FLAME topology (Fig. S5 (b)).
Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2