Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion

Nhat-Nam Nguyen; Pierre-Andre Vuissoz; Yves Laprie

eess.ASSep 24, 2026

Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion

Authors: Nhat-Nam Nguyen, Pierre-Andre Vuissoz, Yves Laprie

Organizations: Université de Lorraine, CNRS, Inria, F-54000 Nancy, France · Université de Lorraine, Inserm, IADI U1254, F-54000 Nancy, France

Abstract

Cross-speaker acoustic-to-articulatory inversion requires accounting for anatomical differences between speakers. We propose a geometric adaptation framework that uses anatomical landmarks, primarily on vertebrae and dental structures,to transfer predictions from a fixed inversion model to unseen speakers. An affine transformation followed by thin-plate spline (TPS) deformation maps the predicted contours of 10 vocal-tract structures into each target speaker's geometry without retraining. Landmarks are identified in one selected /u/ frame per speaker as a common phonetic reference without assuming identical articulatory configurations across speakers, and the resulting mapping is reused across recordings. We train the model on a single-speaker rt-MRI database and evaluate adaptation on eight speakers from a separate multi-speaker rt-MRI database. We compare affine and TPS configurations using 12 or 14 landmarks. Affine12+TPS14 achieves the lowest mean point-to-closest-point error of 3.19mm. These results support the combined value of anatomical landmark information and nonrigid alignment.

Figures & tables

Figure 1: Anatomical landmark configurations used for geometric adaptation. The 12-landmark set (in blue) is shown together with the two additional lower-incisor landmarks, M1 and L6 (in orange), used in the 14-landmark configuration.

Articulator	Mean $\pm$ SD	Median
Arytenoid cartilage	1.53 $\pm$ 0.76	1.38
Epiglottis	1.71 $\pm$ 1.18	1.39
Lower lip	1.29 $\pm$ 0.64	1.17
Pharyngeal wall	0.96 $\pm$ 0.44	0.85
Soft-palate midline	1.01 $\pm$ 0.55	0.88
Tongue	1.99 $\pm$ 0.74	1.84

Table 1: ASD2 model trained on ten articulators with global normalization. P2CP mean in mm.

Articulator	A12	A14	A12+T12	A12+T14
Arytenoid	5.09 $\pm$ 2.33	4.34 $\pm$ 2.16	4.52 $\pm$ 2.27	3.64 * $\pm$ 1.89
Epiglottis	4.94 $\pm$ 2.74	4.29 $\pm$ 2.40	5.00 $\pm$ 2.76	4.39 $\pm$ 2.43
Lower lip	3.26 $\pm$ 1.50	2.67 $\pm$ 1.06	2.91 $\pm$ 1.20	2.53 $\pm$ 1.11
Pharynx	2.87 $\pm$ 1.79	3.02 $\pm$ 1.91	2.29 $\pm$ 1.41	2.19 $\pm$ 1.27
Soft palate	2.81 $\pm$ 1.26	2.83 $\pm$ 1.42	2.91 $\pm$ 1.19	2.88 $\pm$ 1.15
Tongue	4.58 $\pm$ 1.61	4.06 $\pm$ 1.36	4.41 $\pm$ 1.43	3.62 $\pm$ 1.17

Table 2: Articulator-wise P2CP mean (mm). A14 is the main affine baseline; A12+T14 is the proposed configuration.

Speaker	Raw	A12	A14	A12+T12	A12+T14
P1	9.71 $\pm$ 3.63	5.85 $\pm$ 3.41	5.33 $\pm$ 2.94	5.34 $\pm$ 3.51	3.95 * $\pm$ 2.31
P3	9.69 $\pm$ 4.26	5.06 $\pm$ 3.38	4.42 $\pm$ 2.49	4.31 $\pm$ 2.89	3.46 * $\pm$ 1.76
P4	8.14 $\pm$ 4.26	3.28 $\pm$ 1.64	2.95 $\pm$ 1.36	3.06 $\pm$ 1.93	3.17 * $\pm$ 2.10
P5	4.42 $\pm$ 2.12	3.13 $\pm$ 1.36	2.93 $\pm$ 1.21	2.64 $\pm$ 1.03	2.46 * $\pm$ 1.00
P6	7.79 $\pm$ 3.37	3.41 $\pm$ 1.97	2.94 $\pm$ 1.58	3.32 $\pm$ 1.69	2.82 * $\pm$ 1.36
P7	11.23 $\pm$ 4.55	3.41 $\pm$ 2.27	3.34 $\pm$ 2.26	3.69 $\pm$ 2.15	3.17 * $\pm$ 2.14

Table 3: Speaker-wise Full10 P2CP mean (mm). A14 is the main affine baseline; A12+T14 is the proposed configuration.

Figure 2: Adaptation example for P8 containing /k/ in avec . Orange solid: predictions; cyan dashed: ground truth. Frame-level P2CP errors are shown below each panel.

Explore similar work

Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
Sep 9, 2026Hong Nguyen, Sean Foley, Christina Hagedorn +4Vocal Tract Shape Speaker
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA, USA · Department of English, College of Staten Island, City University of New York, New York, NY, USA · Department of Linguistics, University of Potsdam, Germany +1
ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion
Jun 15, 2026Hyung Kyu Kim, Byungchan Hwang, Hak Gu KimArticulatory Augmentation
Department of Imaging Science and Arts, Chung-Ang University, South Korea · Department of Metaverse Convergence, Chung-Ang University, South Korea
Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
Sep 24, 2026Yoomee Cho, Jisun LeeCross-Lingual Voice Cloning Speaker Similarity
ESTsoft, Seoul, Republic of Korea

Sep 9, 2026, cs.SDJ/K move · Enter open · S save

Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis

Hong Nguyen, Sean Foley, Christina Hagedorn, Yijing Lu+3

Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA, USA · Department of English, College of Staten Island, City University of New York, New York, NY, USA · Department of Linguistics, University of Potsdam, Germany+1

Vocal Tract Shape Speaker Grapheme-To-Phoneme S-Jepa Magnetic Resonance Imaging Video World Models

Open paper AskPDF arXiv

Real-time MRI (rtMRI) captures the dynamics of the entire vocal tract during speech, but labeled data are scarce and the modality - single-slice, grayscale, low-resolution - differs substantially from the natural videos that video foundation models are trained on. We introduce Arti-JEPA, a joint embedding predictive architecture to model vocal tract rtMRI by continuing its self-supervised objective on about 62h of unlabelled vocal-tract videos, and evaluate the frozen representation on three tasks: cross-domain phoneme prediction (on typical speakers), fluent-vs-disfluent classification (a corpus containing stuttered speech), and characterizing pre/post-operative transfer (after partial glossectomy). Three key findings emerge. (1) A temporal video prior decisively outperforms per-frame image encoders, and latent prediction (V-JEPA) is at least as strong as pixel reconstruction (VideoMAE), with the edge on fine-grained phonemes. (2) Domain adaptation is \emph{task-dependent}: it roughly doubles cross-domain phoneme prediction

κ

(to 0.352) but does not help binary stuttering classification. (3) Arti-JEPA was able to recover phoneme signal from pre/post glossectomy speech --- an in-domain probe decodes patients at least as well as a typical speaker, indicating that the residual transfer gap is cross-speaker/domain misalignment, not surgical signal loss, and post-operative decoding does not fall below performance on pre-operative speech. Together, these position a frozen, domain-adapted rtMRI encoder as a reusable measurement tool for articulatory and clinical speech science.

Abstract

Explore similar work

Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis

ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion

Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning