Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.
Figures & tables
Figure 1. Similar speech poses hide flatter motion paths. Ground truth and four speech-driven animation methods on /IY-M-EY/, cropped to the lower face. Each row shows five frames from V1→C→V2 , the lip-region trajectory over that interval, and its score Rpw (Section 4 ): 2.03 for ground truth against 1.04–1.21 for the methods. The illustration uses the 60 fps ground-truth timeline with upsampled method outputs, while statistical comparisons use frame-rate-matched ground truth.
Figure 2. The consonant takes the shape of the vowel around it. (A) “too”, with anticipation on the “t” and carryover on the following “m”. (B) “please”, with the lips spreading for the “ee” during the “p” and “l”. Successive frames are 1/30 s apart. (C) Mouth width before and during the consonant, at equal scale: narrower during the “t”, wider during the “l”. See the supplementary video for an animated explanation of coarticulation.
Figure 3. The consonant changes the baseline. Real lip trajectories with illustrative V 1 , C and V 2 mouth poses. Time runs along each colored path. Gray dotted lines join only the endpoints, while red dashed lines go through C. The route-based Rpw distinguishes the bent /m/ path (1.98) from /d/ (1.06). However, endpoint R reverses them (2.22 versus 2.67).
Figure 4. The route through C ranks /hh/ low. Among 21 consonants with n≥20 , /hh/ moves from second under the endpoint chord to twentieth under the route through C, consistent with its weak oral target. The right panel shows per-consonant estimates and speaker-clustered 95% bootstrap intervals. Labials (red) scatter through the ordering rather than leading it.
arm
fps
Rpw
GT − method [95% CI]
β -equiv [95% CI]
ground truth
30
1.143
—
≡1
ground truth
25
1.120
—
≡1
CodeTalker
30
1.107
+0.010 [ −0.001 , +0.040 ]
0.92 [0.68, 1.02]
FaceFormer
30
1.085
+0.020 [ +0.010 , +0.047 ]
0.85 [0.56, 0.93]
DiffPoseTalk
25
1.055
+0.043 [ +0.028 , +0.060 ]
0.41 [0.20, 0.63]
ARTalk
25
1.053
+0.032 [ +0.025 , +0.061 ]
0.56 [0.22, 0.71]
Table 1. All four methods score below ground truth, three of them clearly. Rpw on 183 matched tokens from 38 utterances. Levels are token medians. Gaps are median paired differences against matched-fps ground truth, with utterance-clustered 95% CIs. They are not differences of the displayed levels and are more conservative for every method. The last column gives β -equivalents (Section 5.1 ).
Figure 5. The route-based measure reveals flatter synthesized motion. Deficits relative to frame-rate-matched ground truth for four methods; positive values mean lower method scores. Bars are utterance-clustered 95% bootstrap intervals, and green lines mark zero deficit. Left: all intervals for the endpoint-based ΔR comparison cross zero. Right: three Rpw intervals clear zero; CodeTalker’s crosses it. The β labels express each deficit as an equivalent real-motion setting (Section 5.1 ). The panels use different scales.
β
R
Rpw
resid/amp
amp (mm)
L/(T−1)
0.0
1.17
1.053 [1.042, 1.062]
0.121
0.83
0.10
0.5
1.32
1.093 [1.082, 1.112]
0.166
1.00
0.13
1.0
1.48
1.157 [1.127, 1.179]
0.200
1.21
0.17
1.5
1.64
1.206 [1.172, 1.228]
0.226
1.44
0.20
2.0
1.76
1.240 [1.200, 1.274]
0.250
1.67
0.25
Table 2. Both gain-invariant statistics rise monotonically with β . Controlled articulation ladder on all 421 ground-truth tokens at 30 fps. Entries are medians, with utterance-clustered 95% CIs for Rpw . The β=1 row is unmodified speech in this full set. This set differs from the 183-token matched set behind the per-fps β -equivalents in Table 1 . Here, resid/amp is the companion statistic, amp is motion amplitude, and L/(T−1) is path length per normalized time step in mm.
Figure 6. A trial as participants saw it. Two lower-face renders play side by side under the same audio, and the participant answers on a seven-point scale from “Left much better” to “Right much better”.
Figure 7. Damping is penalized, exaggeration is not. Each group compares real motion ( β=1 ) with one rung of the ladder. Real motion wins clearly against β=0 and β=0.5 , while against β=1.5 and β=2 the altered clip is preferred at least as often. Green favors real motion, gray is equal, and red favors the altered clip. Percentages include ties, and counts are below labels. Model estimates and intervals are reported in text.
Figure 8. Viewers prefer real speech over every method at sentence length, and more weakly on words. (A) Sentences: real motion wins 65–92% per method, 237 of 323 trials pooled. (B) Words: margins narrow and vary by method, from parity for DiffPoseTalk to 68% for ARTalk, with 17.0% ties. Green favors real motion, gray is an equal rating, and red favors the method. Percentages use all trials, and counts appear beneath the method names. FaceFormer was not in the sentence arm.
Figure 9. The articulation dial strengthens both mouth opening and closure. Controlled articulation at ten moments of captured speech. Each triptych shows β=0 (fast component removed), β=1 (unmodified speech), and β=2 (fast component doubled), from left to right. The three renderings share the same source frame, identity and view. Triptychs run in temporal order from left to right and top to bottom; labels give frame indices and times. These examples illustrate the manipulation defined in Section 5.1 and assessed in the β arm of Section 6 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Arm
Comparison
Trials
Real preferred
Equal
Other preferred
Controlled articulation
β=0
259
192 (74.1)
27 (10.4)
40 (15.4)
Controlled articulation
β=0.5
267
121 (45.3)
85 (31.8)
61 (22.8)
Controlled articulation
β=1.5
255
83 (32.5)
75 (29.4)
97 (38.0)
Controlled articulation
β=2
259
93 (35.9)
62 (23.9)
104 (40.2)
Sentences
CodeTalker
104
68 (65.4)
11 (10.6)
25 (24.0)
Sentences
DiffPoseTalk
117
75 (64.1)
20 (17.1)
22 (18.8)
Appendix
Table 3. Perceptual study results after the pre-registered exclusions: 97 usable sessions and 3,523 judgments. Response cells give counts (percentages of all trials, including ties); pooled rows summarize the rows above and add no trials. “Other” means the altered clip in the controlled-articulation arm and the method output in the sentence and word arms. The estimates below repeat the reported analyses: g is the seven-point preference score, with 4 neutral and higher values favoring real motion; the word-level coefficient uses standardized predictors. The sentence interval is binomial. FaceFormer was not included in the sentence arm.
Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.