This paper presents the acoustic and aerodynamic extension of DYNARTmo, a dynamic articulatory model for visualizing and synthesizing speech production. Gestural control trajectories determine vocal-tract geometry, which is converted into an acoustic tube with nasal branching and effective lateral passages. A lumped aerodynamic network relates prescribed glottal motion and pulmonary pressure to airflow and supraglottal pressure storage. Phonatory excitation, oral friction, and glottal aspiration are synthesized as separate contributions. Frame-specific acoustic responses are combined by overlap-and-add, preserving response tails beyond individual control frames. Three German words, Tasse'', Muse'', and ``Lade'', illustrate the coordination of articulation, pressure, flow, and acoustic sources across vowels, nasals, laterals, fricatives, and plosives. The implementation has been optimized by the author primarily for sound recognizability; independent perceptual evaluation remains outstanding, and fricative-noise naturalness requires further refinement. The model provides an inspectable, articulatorily controlled production plant for future neural simulations of babbling, imitation, and early word acquisition. Further work will improve acoustic realism, reduce empirical source corrections, and extend the sensory information available to developmental learning models.
Figures & tables
Figure 1: The model’s sampling grid for the vowel targets /i/ and /u/. P1 and P2 are the fixed posterior and anterior fan origins; pale lines show construction rays and darker blue segments the sampled cavity distances. G is the midpoint of the two inferior glottal reference points. The labial reference points delimit the adaptive lip region. The same regional line counts are used for both configurations; their intersections with the articulator contours change. The diagrams are regenerated from the current model geometry, rather than traced from screenshots.
Figure 2: Distance-to-area conversion and acoustic consequences for the same /i/ (left) and /u/ (right) model targets as in Fig. 1 . (a,b) Sampled midsagittal distances, expressed in the original contour units. (c,d) Mapped areas (blue) and fixed-length acoustic tube sections (orange). (e,f) Tract-inlet flow-impulse to internal lip-pressure response magnitudes, computed with the model’s fixed IR glottal-boundary convention and displayed relative to one common maximum over both curves in the plotted frequency band. They are not phonatory spectra or far-field radiated transfer functions. The velopharyngeal port is closed in both configurations; no nasal branch, friction arrow, or application annotation is displayed.
Figure 3: Area functions for the bilabial nasal /m/ with the velum lowered to open the nasal port. The oral path is closed at the lips while the nasal port remains open. The blue curve is the mapped oral area function; the orange steps are the effective acoustic tube sections. The green steps show the equivalent nasal branch, including its coupling section in purple. The branch origin is aligned horizontally with the actual junction in the oral–pharyngeal tube, indicated by the arrow. Both profiles use the same length and area scales; the nasal baseline is shifted upward for display, with its own area labels. Horizontal distance along the nasal profile is measured from the junction and does not denote an anatomical continuation of the oral axis. The arrow marks connectivity, not a one-way flow constraint.
Figure 4: Tasse /tas@/. The initial apical gesture reaches its target 25 ms earlier with the release time retained, placing closure before pulmonary pressure onset. The five upper tiers show vocalic, consonantal, velar, glottal, and pulmonary/CT control. Pale green envelopes with dotted outlines show the actual gesture activations, including onset and offset ramps; their heights are scaled for display. In the vocalic and velum tiers, activation zero coincides with trajectory zero. Positive velum values indicate raising/closing and negative values lowering/opening. The vertical lines mark segment-timeline reference boundaries, not gesture boundaries. The shared amplitude panel shows voicing, oral friction, and aspiration, as the absolute RMS amplitudes of their separately synthesized radiated components in non-overlapping 5-ms windows. For display only, voicing is multiplied by 1, oral friction by 6, and aspiration by 5. These fixed factors, amplitude limits, and synthesis gains are identical across all three words; no per-source or per-word peak normalization is applied. These are output amplitudes in arbitrary units, not calibrated sound pressures. Area and flow are reconstructed from responses associated with glottal pulse times; pressures use successive control frames. The final panel shows the radiated floating-point synthetic output before PCM quantization.
Figure 5: Lade /la:d@/. The five upper tiers show vocalic, consonantal, velar, glottal, and pulmonary/CT control. Pale green envelopes with dotted outlines show the actual gesture activations, including onset and offset ramps; their heights are scaled for display. In the vocalic and velum tiers, activation zero coincides with trajectory zero. Positive velum values indicate raising/closing and negative values lowering/opening. The vertical lines mark segment-timeline reference boundaries, not gesture boundaries. The shared amplitude panel shows voicing, oral friction, and aspiration, as the absolute RMS amplitudes of their separately synthesized radiated components in non-overlapping 5-ms windows. For display only, voicing is multiplied by 1, oral friction by 6, and aspiration by 5. These fixed factors, amplitude limits, and synthesis gains are identical across all three words; no per-source or per-word peak normalization is applied. These are output amplitudes in arbitrary units, not calibrated sound pressures. Area and flow are reconstructed from responses associated with glottal pulse times; pressures use successive control frames. The final panel shows the radiated floating-point synthetic output before PCM quantization.
Figure 6: Muse /mu:z@/. The five upper tiers show vocalic, consonantal, velar, glottal, and pulmonary/CT control. Pale green envelopes with dotted outlines show the actual gesture activations, including onset and offset ramps; their heights are scaled for display. In the vocalic and velum tiers, activation zero coincides with trajectory zero. Positive velum values indicate raising/closing and negative values lowering/opening. The vertical lines mark segment-timeline reference boundaries, not gesture boundaries. The shared amplitude panel shows voicing, oral friction, and aspiration, as the absolute RMS amplitudes of their separately synthesized radiated components in non-overlapping 5-ms windows. For display only, voicing is multiplied by 1, oral friction by 6, and aspiration by 5. These fixed factors, amplitude limits, and synthesis gains are identical across all three words; no per-source or per-word peak normalization is applied. These are output amplitudes in arbitrary units, not calibrated sound pressures. Area and flow are reconstructed from responses associated with glottal pulse times; pressures use successive control frames. The final panel shows the radiated floating-point synthetic output before PCM quantization.
The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.
Recent advances in machine learning and the availability of articulatory datasets allow vocal tract synthesis to be conditioned on phonetic sequences, a primary task of articulatory speech synthesis. However, quality assessment needs a better definition. Generally, ranking generative models is tricky due to subjectivity. However, articulatory synthesis has the additional difficulty of requiring specialized knowledge in vocal tract anatomy and acoustics. To address this problem, this paper proposes to evaluate speech articulation synthesis using phoneme recognition as a proxy. Our hypothesis is that phoneme recognition using articulatory features better captures nuances in phoneme production, such as correct places of articulation, which traditional metrics (e.g., point-wise distance metrics) do not. We train a neural network with acoustic and articulatory features extracted from a single-speaker RT-MRI dataset. Then, we compare the recognition performance when testing the model with different synthetic articulatory features. Our results show that our articulatory feature set is phonetically rich and helps exploring additional dimensions on speech articulation synthesis.
Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.