Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

Hyung Kyu Kim; Byungchan Hwang; Hak Gu Kim

eess.ASSep 24, 2026

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

Authors: Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim

Organizations: Chung-Ang University Seoul, South Korea

Abstract

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

Figures & tables

Figure 1: How do we model physically grounded and articulatory-consistent facial motion from speech? (a) Five visible mesh-surface anchors as motion reference points: upper lip (UL), lower lip (LL), left mouth corner (LMC), right mouth corner (RMC), and Jaw. (b) From these anchors, we describe visible articulation with three directional articulatory motions: horizontal rounding/spreading, vertical opening/closing, and depth-wise protrusion/retraction. (c) For an utterance, the trajectories of these motions form temporally localized articulatory events that align with acoustic/phonemic cues.

Figure 2: Overall architecture of our articulation-aware framework. SAM retrieves directional articulatory motion features (Spreading–Rounding, Opening–Closing, Protrusion–Retraction) from speech. TAC composes them into surface-consistent 3D facial motion under the mesh topology.

Dataset	Methods	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
VOCASET	FaceFormer	0.878	0.346	0.143	0.186
	CodeTalker	0.913	0.397	0.160	0.204
	SelfTalk	0.844	0.305	0.148	0.174
	Imitator	0.898	0.384	0.179	0.199
	ScanTalk	0.878	0.310	0.152	0.168
	UniTalker	0.820	0.282	0.101	0.188

Table 1: Quantitative results on VOCASET and TFHP.

Methods	Distance Error			Velocity Error
Methods	ICW $\downarrow$	ILD $\downarrow$	LP $\downarrow$	ICW $\downarrow$	ILD $\downarrow$	LP $\downarrow$
FaceFormer	1.764	4.558	2.429	0.581	1.850	1.230
CodeTalker	1.725	4.777	2.808	0.552	2.009	1.343
SelfTalk	1.680	4.031	2.470	0.544	1.958	1.275
Imitator	1.731	4.851	2.526	0.571	1.873	1.205
ScanTalk	1.737	4.034	2.485	0.558	1.854	1.285

Table 2: Visible articulatory distance and velocity errors on VOCASET.

Figure 3: Radar plot of directional lip-motion tendency strength relative to ground truth.

Figure 4: Sequence-level qualitative results on TFHP. For each method, mesh renderings (top) and per-vertex L1 error against the ground truth (bottom, color-coded from 0 (blue) to 0.015 (red)) are shown over time at phoneme-aligned frames.

Figure 5: Slit-scan visualizations for sequence-level comparison. (a) lip ROI and landmarks (UL, LL, LMC, RMC) defining horizontal (LMC–RMC) and vertical (UL–LL) slit lines. (b) Horizontal and (c) Vertical slit-scans over time, reflecting lip spreading (width) and lip opening (height), respectively.

SAM	TAC	$\text{FVE}\!\downarrow$	$\text{LVE}\!\downarrow$	LDTW $\!\downarrow$
✓	✗	0.812	0.255	0.161
✓	✓	0.771	0.235	0.132

Table 3: Ablation of TAC module.

Figure 6: Analysis of Directional articulatory trajectories.

Competitors	Lip Sync	Realism
Competitors	(% Ours)	(% Ours)
vs. FaceFormer	71.3	65.7
vs. CodeTalker	72.2	75.0
vs. Imitator	77.8	73.1
vs. ScanTalk	73.1	79.6
vs. UniTalker	65.7	67.6

Table 4: User study: pairwise preference rates (%) on VOCASET.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Component	Output shape
Input	$\mathbf{v}_{\delta}^{1:T}$ (per-direction motion)	$(T,V)$
Encoder	Input projection ( $V\to D_{\mathrm{SAM}}$ , Linear)	$(T,D_{\mathrm{SAM}})$
	Sinusoidal positional encoding	$(T,D_{\mathrm{SAM}})$
	Transformer encoder (1 layer, 4 heads, FFN $4{\times}D_{\mathrm{SAM}}$ , GELU, dropout 0.1)	$(T,D_{\mathrm{SAM}})$
Latent	$\mathbf{z}_{\delta}^{1:T}$ (motion feature)	$(T,D_{\mathrm{SAM}})$
Decoder	Transformer decoder (1 layer, 4 heads, FFN $4{\times}D_{\mathrm{SAM}}$ , GELU, dropout 0.1)	$(T,D_{\mathrm{SAM}})$

Appendix

Table S1: Per-direction architecture of the motion autoencoder used to produce

\mathbf{z}_{\delta}^{t}

. Three encoder–decoder pairs are jointly trained, one for each

\delta\in\{S,O,P\}

Metric	Unit	VOCASET	TFHP
FVE	m	$\times 10^{-3}$	$\times 10^{-2}$
LVE	m 2	$\times 10^{-4}$	$\times 10^{-3}$
FDD	m 2	$\times 10^{-8}$	$\times 10^{-8}$
LDTW	–	$\times 10^{-3}$	$\times 10^{-3}$
ICW / ILD / LP	m	$\times 10^{-3}$
ICW / ILD / LP velocity	m/frame	$\times 10^{-3}$

Appendix

Table S2: Scale factors and units for all reported metrics.

Figure S1: FVE and LVE on VOCASET as the number of memory slot $N$ . $N=32$ ( $\star$ ) achieves the lowest error on both metrics.

Figure S2: Per-phoneme rendered meshes and $S$ / $O$ / $P$ magnitude heatmaps with and without directional decomposition. Each row shows a representative phoneme (/u/, /a/, /o/).

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
w/o directional articulatory decomposition	0.842	0.266	0.122	0.181
w/ directional articulatory decomposition ( $S$ / $O$ / $P$ )	0.771	0.235	0.097	0.132

Appendix

Table S3: Ablation on directional articulatory decomposition (VOCASET).

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
w/o $\mathbf{M}_{\mathrm{key}},\mathbf{M}_{\mathrm{val},\delta}$	0.823	0.275	0.121	0.188
Full (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S4: Ablation on the explicit motion memory in SAM (VOCASET).

Query	Refinement	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
Content	Content	0.815	0.277	0.127	0.156
Acoustic	Acoustic	0.850	0.372	0.158	0.199
Acoustic	Content (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S5: Ablation on the Query–Refinement design in SAM (VOCASET). “Acoustic” denotes Mel-spectrogram features and “Content” denotes HuBERT features.

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
w/o $\mathcal{L}_{\mathrm{store}}$	0.811	0.267	0.117	0.144
w/ $\mathcal{L}_{\mathrm{store}}$ (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S6: Ablation on the value memory store loss (VOCASET).

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
SAM + MLP composition	0.799	0.243	0.103	0.146
SAM + TAC (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S7: Topology-independent (MLP) vs. topology-aware (Ours) composition on VOCASET.

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
w/o residual (feedforward)	0.782	0.238	0.111	0.134
w/ residual (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S8: Ablation on the internal residual connection in TAC (VOCASET). The residual path is required for stable optimization across stacked GCN+TCN blocks.

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
w/o $\mathcal{L}_{\text{lap}}$	0.788	0.236	0.107	0.139
w/ $\mathcal{L}_{\text{lap}}$ (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S9: Ablation on the graph Laplacian consistency loss (VOCASET).

\mathcal{L}_{\text{lap}}

constrains local surface curvature and improves overall motion fidelity.

Configuration	FVE $\downarrow$	LVE $\downarrow$	FDD $\downarrow$	LDTW $\downarrow$
Unified single-stage (from scratch)	0.801	0.245	0.118	0.142
Pretrained + unfrozen fine-tuning	0.798	0.244	0.122	0.144
Pretrained + frozen (Ours)	0.771	0.235	0.097	0.132

Appendix

Table S10: Ablation on the training schedule (VOCASET). Pretraining SAM and freezing it during TAC training outperforms both end-to-end fine-tuning and unified single-stage training.

Figure S3: Effect of half-window size $\Omega$ on FVE and LVE (VOCASET). $\Omega{=}0$ corresponds to frame-independent retrieval without temporal motion context.

Methods	FVE ( $\times 10^{-1}$ ) $\downarrow$	LVE ( $\times 10^{-2}$ ) $\downarrow$	FDD ( $\times 10^{-5}$ ) $\downarrow$	LDTW ( $\times 10^{-2}$ ) $\downarrow$
CodeTalker	0.124	0.127	0.461	0.319
SelfTalk	0.120	0.114	0.148	0.285
UniTalker	0.112	0.106	0.353	0.283
Ours (SAM only)	0.115	0.113	0.557	0.296
Ours (SAM+TAC)	0.110	0.106	0.351	0.275

Appendix

Table S11: Topology generalization on BIWI (23,370 vertices).

Methods	Distance Error			Velocity Error
Methods	ICW $\downarrow$	ILD $\downarrow$	LP $\downarrow$	ICW $\downarrow$	ILD $\downarrow$	LP $\downarrow$
Mimic	4.320	5.326	5.968	1.460	2.593	2.871
DiffPoseTalk	5.483	5.860	7.006	1.221	2.168	2.438
ARTalk	5.788	6.049	7.378	1.501	2.801	2.644
Ours	4.299	4.829	5.859	1.168	2.083	2.414

Appendix

Table S12: Visible articulatory distance and velocity errors on TFHP.

Figure S4: Stage-wise qualitative comparison on TFHP. Top row: GT. Middle row: Stage 1 (SAM only) result and per-vertex error. Bottom row: Stage 2 (SAM + TAC) result and per-vertex error.

Figure S5: Visualization of the softmax distribution $\mathrm{softmax}(\tau\boldsymbol{\alpha})$ underlying the TAC modulation gate $\mathbf{m}$ on the FLAME template. (a) Per-vertex distribution. (b) Anatomical region segmentation. (c) Region-grouped mean, with the dashed line indicating the uniform baseline $1/V$ .

Figure S6: Consistency of acoustic-bridge memory addressing vectors. For the same phoneme class, the addressing weights repeatedly peak at similar slots across occurrences, suggesting stable, reusable retrieval.

Figure S7: Visible articulatory lip-motion trajectories of our method and the ground truth along three directions (Spreading–Rounding, Opening–Closing, Protrusion–Retraction). Highlighted segments correspond to the words “Strong”, “support”, and “nation” in the utterance, with GT and Ours mesh sequences shown below.

Figure S8: Additional example of three directional articulatory trajectories (Spreading–Rounding, Opening–Closing, Protrusion–Retraction) for the utterance “I’ve heard directly from our public safety.”

Figure S9: Additional qualitative comparisons on VOCASET and TFHP.

Figure S10: Examples of the conducted user study.

Method	Forehead $\downarrow$	Eyes $\downarrow$	Scalp $\downarrow$
Method	(133 verts)	(574 verts)	(489 verts)
Mimic	0.020	0.099	0.010
DiffPoseTalk	0.015	0.042	0.008
ARTalk	0.044	0.060	0.011
Ours	0.023	0.102	0.010

Appendix

Table S13: Per-region FDD on TFHP (

\times 10^{-7}

m 2 , lower is better). Regions follow the official FLAME topology (Fig. S5 (b)).

Explore similar work

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation
Oct 2, 2026Danzel Serrano, Przemyslaw MusialskiAudio-Driven Facial Animation
New Jersey Institute of Technology, USA
MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
Apr 3, 2026Bin Liu, Zhixiang Xiong, Zhifen He +1Talking Head Generation Audio-Driven Facial Animation
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation
Westlake University · Zhejiang University · Tiangong University +2

Oct 2, 2026J/K move · Enter open · S save

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

Danzel Serrano, Przemyslaw Musialski

New Jersey Institute of Technology, USA

Audio-Driven Facial Animation

Open paper AskPDF arXiv

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

Abstract

Explore similar work

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models