The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

Danzel Serrano; Przemyslaw Musialski

cs.GROct 2, 2026

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

Authors: Danzel Serrano, Przemyslaw Musialski

Organizations: New Jersey Institute of Technology, USA

Abstract

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

Figures & tables

Figure 1. Similar speech poses hide flatter motion paths. Ground truth and four speech-driven animation methods on /IY-M-EY/, cropped to the lower face. Each row shows five frames from $V_{1}\rightarrow C\rightarrow V_{2}$ , the lip-region trajectory over that interval, and its score $R_{pw}$ (Section 4 ): 2.03 for ground truth against 1.04–1.21 for the methods. The illustration uses the 60 fps ground-truth timeline with upsampled method outputs, while statistical comparisons use frame-rate-matched ground truth.

Figure 2. The consonant takes the shape of the vowel around it. (A) “too”, with anticipation on the “t” and carryover on the following “m”. (B) “please”, with the lips spreading for the “ee” during the “p” and “l”. Successive frames are 1/30 s apart. (C) Mouth width before and during the consonant, at equal scale: narrower during the “t”, wider during the “l”. See the supplementary video for an animated explanation of coarticulation.

Panel of Figure 3 — Figure 3. The consonant changes the baseline. Real lip trajectories with illustrative V 1 , C and V 2 mouth poses. Time runs along each colored path. Gray dotted lines join only the endpoints, while red dashed lines go through C. The route-based $R_{pw}$ distinguishes the bent /m/ path (1.98) from /d/ (1.06). However, endpoint $R$ reverses them (2.22 versus 2.67).

Figure 4. The route through C ranks /hh/ low. Among 21 consonants with

n\geq 20

, /hh/ moves from second under the endpoint chord to twentieth under the route through C, consistent with its weak oral target. The right panel shows per-consonant estimates and speaker-clustered 95% bootstrap intervals. Labials (red) scatter through the ordering rather than leading it.

arm	fps	$R_{pw}$	GT $-$ method [95% CI]	$\beta$ -equiv [95% CI]
ground truth	30	1.143	—	$\equiv 1$
ground truth	25	1.120	—	$\equiv 1$
CodeTalker	30	1.107	$+0.010$ [ $-0.001$ , $+0.040$ ]	0.92 [0.68, 1.02]
FaceFormer	30	1.085	$+0.020$ [ $+0.010$ , $+0.047$ ]	0.85 [0.56, 0.93]
DiffPoseTalk	25	1.055	$+0.043$ [ $+0.028$ , $+0.060$ ]	0.41 [0.20, 0.63]
ARTalk	25	1.053	$+0.032$ [ $+0.025$ , $+0.061$ ]	0.56 [0.22, 0.71]

Table 1. All four methods score below ground truth, three of them clearly.

R_{pw}

on 183 matched tokens from 38 utterances. Levels are token medians. Gaps are median paired differences against matched-fps ground truth, with utterance-clustered 95% CIs. They are not differences of the displayed levels and are more conservative for every method. The last column gives

\beta

-equivalents (Section 5.1 ).

Figure 5. The route-based measure reveals flatter synthesized motion. Deficits relative to frame-rate-matched ground truth for four methods; positive values mean lower method scores. Bars are utterance-clustered 95% bootstrap intervals, and green lines mark zero deficit. Left: all intervals for the endpoint-based

\Delta R

comparison cross zero. Right: three

R_{pw}

intervals clear zero; CodeTalker’s crosses it. The

\beta

labels express each deficit as an equivalent real-motion setting (Section 5.1 ). The panels use different scales.

$\beta$	$R$	$R_{pw}$	resid/amp	amp (mm)	$L/(T{-}1)$
0.0	1.17	1.053 [1.042, 1.062]	0.121	0.83	0.10
0.5	1.32	1.093 [1.082, 1.112]	0.166	1.00	0.13
1.0	1.48	1.157 [1.127, 1.179]	0.200	1.21	0.17
1.5	1.64	1.206 [1.172, 1.228]	0.226	1.44	0.20
2.0	1.76	1.240 [1.200, 1.274]	0.250	1.67	0.25

Table 2. Both gain-invariant statistics rise monotonically with

\beta

. Controlled articulation ladder on all 421 ground-truth tokens at 30 fps. Entries are medians, with utterance-clustered 95% CIs for

R_{pw}

. The

\beta=1

row is unmodified speech in this full set. This set differs from the 183-token matched set behind the per-fps

\beta

-equivalents in Table 1 . Here, resid/amp is the companion statistic, amp is motion amplitude, and

L/(T{-}1)

is path length per normalized time step in mm.

Figure 6. A trial as participants saw it. Two lower-face renders play side by side under the same audio, and the participant answers on a seven-point scale from “Left much better” to “Right much better”.

Figure 7. Damping is penalized, exaggeration is not. Each group compares real motion (

\beta=1

) with one rung of the ladder. Real motion wins clearly against

\beta=0

and

\beta=0.5

, while against

\beta=1.5

and

\beta=2

the altered clip is preferred at least as often. Green favors real motion, gray is equal, and red favors the altered clip. Percentages include ties, and counts are below labels. Model estimates and intervals are reported in text.

Figure 8. Viewers prefer real speech over every method at sentence length, and more weakly on words. (A) Sentences: real motion wins 65–92% per method, 237 of 323 trials pooled. (B) Words: margins narrow and vary by method, from parity for DiffPoseTalk to 68% for ARTalk, with 17.0% ties. Green favors real motion, gray is an equal rating, and red favors the method. Percentages use all trials, and counts appear beneath the method names. FaceFormer was not in the sentence arm.

Figure 9. The articulation dial strengthens both mouth opening and closure. Controlled articulation at ten moments of captured speech. Each triptych shows $\beta=0$ (fast component removed), $\beta=1$ (unmodified speech), and $\beta=2$ (fast component doubled), from left to right. The three renderings share the same source frame, identity and view. Triptychs run in temporal order from left to right and top to bottom; labels give frame indices and times. These examples illustrate the manipulation defined in Section 5.1 and assessed in the $\beta$ arm of Section 6 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Arm	Comparison	Trials	Real preferred	Equal	Other preferred
Controlled articulation	$\beta=0$	259	192 (74.1)	27 (10.4)	40 (15.4)
Controlled articulation	$\beta=0.5$	267	121 (45.3)	85 (31.8)	61 (22.8)
Controlled articulation	$\beta=1.5$	255	83 (32.5)	75 (29.4)	97 (38.0)
Controlled articulation	$\beta=2$	259	93 (35.9)	62 (23.9)	104 (40.2)
Sentences	CodeTalker	104	68 (65.4)	11 (10.6)	25 (24.0)
Sentences	DiffPoseTalk	117	75 (64.1)	20 (17.1)	22 (18.8)

Appendix

Table 3. Perceptual study results after the pre-registered exclusions: 97 usable sessions and 3,523 judgments. Response cells give counts (percentages of all trials, including ties); pooled rows summarize the rows above and add no trials. “Other” means the altered clip in the controlled-articulation arm and the method output in the sentence and word arms. The estimates below repeat the reported analyses:

g

is the seven-point preference score, with

4

neutral and higher values favoring real motion; the word-level coefficient uses standardized predictors. The sentence interval is binomial. FaceFormer was not included in the sentence arm.

Explore similar work

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation
Sep 24, 2026Hyung Kyu Kim, Byungchan Hwang, Hak Gu KimAudio-Driven Facial Animation
Chung-Ang University Seoul, South Korea
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation
Westlake University · Zhejiang University · Tiangong University +2
MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
Apr 3, 2026Bin Liu, Zhixiang Xiong, Zhifen He +1Talking Head Generation Audio-Driven Facial Animation

Sep 24, 2026J/K move · Enter open · S save

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim

Chung-Ang University Seoul, South Korea

Audio-Driven Facial Animation

Open paper AskPDF arXiv

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

Abstract

Explore similar work

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion