Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

Eric Ming Chen; Jin Woo Lee; Vincent Sitzmann

cs.SDSep 29, 2026

Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

Authors: Eric Ming Chen, Jin Woo Lee, Vincent Sitzmann

Organizations: MIT CSAIL

Abstract

The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.

Figures & tables

Figure 1 : Method Overview. We introduce a method to reconstruct a MRI video of a person’s moving vocal tract from their speech alone. (a) Given an MRI image of a person’s vocal tract, we extract cross-sectional areas along the length of the tract. (b) These areas define an acoustic tube . Our differentiable simulator $\Phi$ propagates sound through the acoustic tube to synthesize speech. Gradients from the loss flow back through $\Phi$ and the generative model into a set of latent vectors, reconstructing the MRI video from the target speech signal.

Figure 2 : FDS Leads to Smoother Loss Landscapes. (a) Starting from the vowel /u/, we interpolate its area in the direction of the /a/ geometry and /e/ geometry. (b) We plot the log Mel spectrogram loss between the sound produced by the interpolated geometries and the sound produced by /u/. The loss from time domain synthesis is noiser than the loss from frequency domain synthesis.

Figure 4 : Neural Fields Smoothly Reconstruct Acoustic Tubes. Directly optimizing a discrete acoustic tube fails to fit basic vowels. Optimization is trapped around the initialization, and the recovered geometries sometimes suffer from high-frequency oscillation. In comparison, neural fields couple the geometry of each section with shared weights leading to smooth reconstructions.

	/A/	/e/	/i/	/1/	/o/	/u/
Discrete	14.66 $\pm$ 2.25	26.22 $\pm$ 1.36	27.66 $\pm$ 2.04	29.65 $\pm$ 2.34	27.42 $\pm$ 3.60	28.19 $\pm$ 2.54
RFF	1.96 $\pm$ 4.17	1.29 $\pm$ 0.90	2.46 $\pm$ 4.01	0.47 $\pm$ 0.32	1.63 $\pm$ 0.40	27.26 $\pm$ 22.30
MFN	5.06 $\pm$ 3.64	15.00 $\pm$ 7.06	5.96 $\pm$ 1.37	5.79 $\pm$ 10.52	8.83 $\pm$ 3.22	19.98 $\pm$ 5.04

Table 2 : Area Reconstruction. We report the mean squared error of each area parameterization in

\text{cm}^{4}

. Both neural field methods, RFF and MFN, significantly outperform the discrete baseline.

	Ground Truth		TensorTract2 [ 31 ]			Ours
	WER	CER	WER	CER	Speaker ID	WER	CER	Speaker ID
English	3.91 $\pm$ 0.84	2.05 $\pm$ 0.70	13.00 $\pm$ 1.82	7.37 $\pm$ 1.11	16.86 $\pm$ 1.01	5.30 $\pm$ 0.94	2.82 $\pm$ 0.75	52.0 $\pm$ 1.39
German	5.53 $\pm$ 0.78	2.40 $\pm$ 0.47	17.59 $\pm$ 1.15	10.85 $\pm$ 0.73	14.07 $\pm$ 1.42	12.49 $\pm$ 1.54	7.89 $\pm$ 0.74	54.9 $\pm$ 1.45
Dutch	9.24 $\pm$ 0.87	2.59 $\pm$ 0.30	27.08 $\pm$ 1.63	12.89 $\pm$ 0.91	26.97 $\pm$ 1.15	17.19 $\pm$ 1.27	7.77 $\pm$ 0.65	59.3 $\pm$ 1.17
French	4.72 $\pm$ 0.79	2.27 $\pm$ 0.59	19.65 $\pm$ 1.71	11.37 $\pm$ 1.17	15.11 $\pm$ 1.34	12.91 $\pm$ 1.38	7.69 $\pm$ 0.85	47.4 $\pm$ 1.46
Spanish	3.35 $\pm$ 0.69	1.37 $\pm$ 0.44	9.71 $\pm$ 0.93	5.13 $\pm$ 0.44	14.74 $\pm$ 1.49	9.27 $\pm$ 1.25	4.43 $\pm$ 0.59	52.9 $\pm$ 1.81

Table 3: Speech Autoencoding Metrics. We report WER, CER and ID metrics, with their 95

\%

confidence intervals, across 11 languages. Our self-supervised autoencoder better encodes intelligible utterances than TensorTract2, an autoencoder trained on synthetic consonant-vowel clusters.

Figure 5 : Encoded areas form a spectrum.

Figure 6 : Phonetic Structure Emerges without Supervision. We randomly generate 1,000 MRIs from our GAN, then compute their formants with our simulator. The convex hull of formants resemble a trapezoid corresponding almost exactly to the IPA vowel chart. The anatomy of each generated MRI corresponds closely to textbook illustrations of vowels from Fant [17] .

	Ours	$\begin{subarray}{c}\text{\large Speech}\\ \text{\large 2rtMRI}\end{subarray}$
Audio
SI-SDR $\uparrow$	13.08	N/A
STOI $\uparrow$	0.92	N/A
PESQ $\uparrow$	2.09	N/A
Visual
FVD $\downarrow$	623	2949

Table 4 : MRI Metrics

Panel of Figure 7 — Figure 7 : Acoustic Grounding Prevents Video Collapse. Because Speech2rtMRI [ 43 ] does not leverage a physical representation, it tends to degenerate over time. Our method remains stable.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8 : A schematic diagram illustrating three application cases of the differentiable $\Phi$ , as described in Section 6 . The problem of estimating $A(x,t)$ from speech can thus be solved through various gradient-based optimization techniques, and none of these methods presuppose a ground truth $A$ , but rather resolve this inverse problem solely by utilizing the real-world speech signal $p_{\mathrm{tgt}}$ .

	Discrete	RFF [ 59 ]	MFN [ 19 ]
SI-SDR	10.78 $\pm$ 2.22	14.51 $\pm$ 3.78	16.39 $\pm$ 2.42
STOI	0.81 $\pm$ 0.02	0.84 $\pm$ 0.05	0.93 $\pm$ 0.02
PESQ	1.57 $\pm$ 0.12	1.76 $\pm$ 0.26	1.98 $\pm$ 0.26

Appendix

Table 5 : Neural Field Reconstruction

Figure 9 : A Larger UMAP Projection of Encoded Vocal Tract Area Functions. We provide a UMAP projection of an even larger set of phonemes. The embedded area functions form a continuous space spanning the set of IPA symbols.

Explore similar work

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI
May 18, 2026Daiqi Liu, Lukas Mulzer, Md Hasan +11Vocal Tract Shape Magnetic Resonance Imaging
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Harvard Medical School / Massachusetts General Hospital, Boston, USA +3
SIREM: Speech-Informed MRI Reconstruction with Learned Sampling
May 18, 2026Md Hasan, Nyvenn Castro, Daiqi Liu +6Magnetic Resonance Imaging Reconstruction Speaker
1Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · 3Institut für Informationsverarbeitung, Leibniz Universität Hannover, Hannover, Germany +1
Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion
Sep 24, 2026Nhat-Nam Nguyen, Pierre-Andre Vuissoz, Yves LaprieArticulatory Speaker
Université de Lorraine, CNRS, Inria, F-54000 Nancy, France · Université de Lorraine, Inserm, IADI U1254, F-54000 Nancy, France

May 18, 2026, cs.CVJ/K move · Enter open · S save

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI

Daiqi Liu, Lukas Mulzer, Md Hasan, Nyvenn de Castro+10

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Harvard Medical School / Massachusetts General Hospital, Boston, USA+3

Vocal Tract Shape Magnetic Resonance Imaging Modality-Specific Encoders Multimodal Contrastive Learning Grapheme-To-Phoneme

Open paper AskPDF arXiv

Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisitions may provide synchronized acoustic signals, existing methods discard this information, and the few multimodal approaches that incorporate audio cannot be deployed when audio is unavailable. We propose a three-stage framework that leverages acoustic and phonological supervision during training while requiring only the rtMRI image at inference: phonological representations are converted into spatial bounding-box priors for articulator localization, visual and acoustic encoders are aligned via dual-level cross-modal contrastive pretraining, and the learned representations are fused through a cross-attention decoder, effectively transferring multimodal knowledge into a single-modality inference pipeline. Evaluated on 75-Speaker~Annot-16 and USC-TIMIT datasets, our method outperforms existing unimodal and multimodal methods, demonstrating that multimodal supervision provides transferable benefits for precise and clinically deployable vocal tract segmentation.

Abstract

Explore similar work

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI

SIREM: Speech-Informed MRI Reconstruction with Learned Sampling

Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion