Fréchet Oral Distance: A Coordinate-Free Metric and Benchmark for Speech-Driven Oral Animation
Authors: Jaehong Myung, Seungho Eum, Hyunjung Chung, Unsang Park
Organizations: Department of Artificial Intelligence Sogang University Seoul, South Korea · Department of Computer Science Sogang University Seoul, South Korea · Department of Computer Science Department of Artificial Intelligence Sogang University Seoul, South Korea
Recently, speech-driven facial and oral animation technologies have been used in various fields such as education, speech therapy, and healthcare. But methods to objectively evaluate the quality of generated results are still limited. Accordingly, we propose the Fréchet Oral Distance (FOD), which uses embeddings from an Oral Motion Encoder (OME) and a TIMIT-Vis reference distribution. In experiments, FOD reproduced model rankings similar to the coordinate-based metric, Lip Vertex Error (LVE), and showed trends consistent with user preference. Our framework provides objective and reproducible benchmarking without relying on expensive equipment or subjective evaluation, and it can be extended to compare a variety of 2D/3D models.
Figures & tables
Figure 1. (A) Training pipeline: The Oral Motion Encoder is trained using synthetic articulatory videos generated from TIMIT-Vis ( Garofolo et al., 1993 ; Birkholz, 2013 ) (VTL-rendered TIMIT). (B) Evaluation pipeline: Latent vectors extracted from simulation videos are compared to reference distributions using the Fréchet distance ( Heusel et al., 2017 ) . These two stages collectively define the framework for both learning and assessing articulatory dynamics.
Name
TIMIT-Vis
FPS
30
Resolution
256×256
Regions
8
Samples (Train/Test)
6,200/100
F/M
192/438
Table 1. Summary of the TIMIT-Vis benchmark dataset ( Garofolo et al., 1993 ; Birkholz, 2013 ) . 6,200/100 videos (train/test) at 30 FPS, 256×256; 630 speakers, 8 dialect regions (192 female, 438 male).
Figure 2. VocalTractLab (VTL) ( Birkholz, 2013 ) interface. The GUI visualizes articulatory structures and computes acoustic responses; VTL was used to render videos for TIMIT-Vis ( Garofolo et al., 1993 ; Birkholz, 2013 ) .
Metric
FaceFormer ( Fan et al., 2022 )
CodeTalker ( Xing et al., 2023 )
SelfTalk ( Peng et al., 2023 )
UniTalker ( Fan et al., 2024 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
LVE
4.98
1.17
4.79
1.12
4.25
0.96
3.86
0.81
Ours
L1
0.0502
0.0542
0.0501
0.0545
0.0494
0.0532
0.0491
0.0526
L2
13.85
14.92
13.77
14.96
13.61
14.90
13.42
14.23
Frechet-D
190.10
165.14
187.36
167.92
181.51
163.90
176.94
162.14
Table 2. Absolute scores on VOCA/BIWI: LVE (when available) and latent-space L1/L2/FOD. Lower indicates better alignment.
Figure 3. Trend visualization across datasets: LVE, latent L1/L2 distances, and FOD on VOCA and BIWI ( Cudeiro et al., 2019 ; Fanelli et al., 2011 ) . The figure summarizes ranking and separability trends (lower is better), complementing the absolute values in Table 2 .
OS
Ubuntu 22.04
Python
3.10.12
CPU
Xeon Gold 6248R
Torch
2.0.1
GPU
RTX-A6000
CUDA
12.1
Table 3. Experimental system configuration for training and evaluation.
Figure 4. Activation maps of 16 latent channels (reshaped to H×W ). Responses highlight both articulatory motion (e.g., lips, tongue) and static oral structure, indicating joint encoding of dynamics and shape.
Competitors
BIWI-Test-B
VOCA-Test
Lip Sync
Realism
Lip Sync
Realism
CodeTalker vs. VOCA
92.47
89.25
86.02
84.95
CodeTalker vs. MeshTalk
80.65
82.80
95.70
92.47
CodeTalker vs. FaceFormer
53.76
56.99
70.97
69.89
CodeTalker vs. GT
43.01
49.46
43.01
43.01
Table 4. User study (from CodeTalker ( Xing et al., 2023 ) ) on BIWI-Test-B and VOCA-Test using A/B testing. Values are percentages.
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2
Fréchet Audio Distance (FAD) is the de facto standard for evaluating text-to-audio generation, yet its scores depend on the underlying encoder's embedding space. An encoder's training task dictates which acoustic features are preserved or discarded, causing FAD to inherit systematic task-induced biases. We decompose evaluation into Recall, Precision, and Alignment (split into semantic and structural dimensions), using log-scale normalization for fair cross-encoder comparison. Controlled experiments on six encoders across two datasets reveal a four-axis trade-off: reconstruction-based AudioMAE leads precision sensitivity; ASR-trained Whisper dominates structural detection but is blind to signal degradation; classification-trained VGGish maximizes semantic detection but penalizes legitimate intra-class variation. Since no single encoder is a universal evaluator, future metrics must shift toward evaluation-native encoders intrinsically aligned with human perception.
Wonwoo Jeong
Dept. of Computer Science and Engineering, Sogang University, South Korea
We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.