Fréchet Oral Distance: A Coordinate-Free Metric and Benchmark for Speech-Driven Oral Animation
Authors: Jaehong Myung, Seungho Eum, Hyunjung Chung, Unsang Park
Organizations: Department of Artificial Intelligence Sogang University Seoul, South Korea · Department of Computer Science Sogang University Seoul, South Korea · Department of Computer Science Department of Artificial Intelligence Sogang University Seoul, South Korea
Recently, speech-driven facial and oral animation technologies have been used in various fields such as education, speech therapy, and healthcare. But methods to objectively evaluate the quality of generated results are still limited. Accordingly, we propose the Fréchet Oral Distance (FOD), which uses embeddings from an Oral Motion Encoder (OME) and a TIMIT-Vis reference distribution. In experiments, FOD reproduced model rankings similar to the coordinate-based metric, Lip Vertex Error (LVE), and showed trends consistent with user preference. Our framework provides objective and reproducible benchmarking without relying on expensive equipment or subjective evaluation, and it can be extended to compare a variety of 2D/3D models.
Figures & tables
Figure 1. (A) Training pipeline: The Oral Motion Encoder is trained using synthetic articulatory videos generated from TIMIT-Vis ( Garofolo et al., 1993 ; Birkholz, 2013 ) (VTL-rendered TIMIT). (B) Evaluation pipeline: Latent vectors extracted from simulation videos are compared to reference distributions using the Fréchet distance ( Heusel et al., 2017 ) . These two stages collectively define the framework for both learning and assessing articulatory dynamics.
Name
TIMIT-Vis
FPS
30
Resolution
256×256
Regions
8
Samples (Train/Test)
6,200/100
F/M
192/438
Table 1. Summary of the TIMIT-Vis benchmark dataset ( Garofolo et al., 1993 ; Birkholz, 2013 ) . 6,200/100 videos (train/test) at 30 FPS, 256×256; 630 speakers, 8 dialect regions (192 female, 438 male).
Figure 2. VocalTractLab (VTL) ( Birkholz, 2013 ) interface. The GUI visualizes articulatory structures and computes acoustic responses; VTL was used to render videos for TIMIT-Vis ( Garofolo et al., 1993 ; Birkholz, 2013 ) .
Metric
FaceFormer ( Fan et al., 2022 )
CodeTalker ( Xing et al., 2023 )
SelfTalk ( Peng et al., 2023 )
UniTalker ( Fan et al., 2024 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
VOCA ( Cudeiro et al., 2019 )
BIWI ( Fanelli et al., 2011 )
LVE
4.98
1.17
4.79
1.12
4.25
0.96
3.86
0.81
Ours
L1
0.0502
0.0542
0.0501
0.0545
0.0494
0.0532
0.0491
0.0526
L2
13.85
14.92
13.77
14.96
13.61
14.90
13.42
14.23
Frechet-D
190.10
165.14
187.36
167.92
181.51
163.90
176.94
162.14
Table 2. Absolute scores on VOCA/BIWI: LVE (when available) and latent-space L1/L2/FOD. Lower indicates better alignment.
Figure 3. Trend visualization across datasets: LVE, latent L1/L2 distances, and FOD on VOCA and BIWI ( Cudeiro et al., 2019 ; Fanelli et al., 2011 ) . The figure summarizes ranking and separability trends (lower is better), complementing the absolute values in Table 2 .
OS
Ubuntu 22.04
Python
3.10.12
CPU
Xeon Gold 6248R
Torch
2.0.1
GPU
RTX-A6000
CUDA
12.1
Table 3. Experimental system configuration for training and evaluation.
Figure 4. Activation maps of 16 latent channels (reshaped to H×W ). Responses highlight both articulatory motion (e.g., lips, tongue) and static oral structure, indicating joint encoding of dynamics and shape.
Competitors
BIWI-Test-B
VOCA-Test
Lip Sync
Realism
Lip Sync
Realism
CodeTalker vs. VOCA
92.47
89.25
86.02
84.95
CodeTalker vs. MeshTalk
80.65
82.80
95.70
92.47
CodeTalker vs. FaceFormer
53.76
56.99
70.97
69.89
CodeTalker vs. GT
43.01
49.46
43.01
43.01
Table 4. User study (from CodeTalker ( Xing et al., 2023 ) ) on BIWI-Test-B and VOCA-Test using A/B testing. Values are percentages.