VocalTrax recovers interpretable articulatory controls from a recording by gradient descent through a differentiable Pink Trombone synthesizer. Its vocal tract is a Kelly-Lochbaum waveguide updated twice per audio sample, so every gradient step backpropagates through a long recurrence, which is slow on GPUs. We propose Convolutional VocalTrax, which computes the exact frequency response of the same waveguide and applies it as a causal convolution. We assume that the tract is stationary within 5.8 ms blocks and that its lossy impulse response can be truncated after 93 ms. Gradients then pass through FFTs and a short recursion along the tract instead of through time. On 600 AudioMNIST recordings, the tract forward and backward pass runs 47 to 63 times faster on an A100 GPU, and a classifier trained only on real speech recognizes our reconstructions more often than those of VocalTrax (14.5% vs. 10.0%), although the log-mel distance to the target is 11% higher. Parameters fitted with our model keep this gain when rendered by the original VocalTrax (13.8%).
Figures & tables
Block H
Fixed
5% motion
20% motion
Time (s)
64
8.1×10−6
0.79%
2.95%
0.36
128
8.1×10−6
0.86%
3.33%
0.19
256
8.1×10−6
1.04%
4.00%
0.17
512
8.1×10−6
1.46%
5.55%
0.11
Table 1: Validation of the two assumptions. Top: relative waveform error of our model against VocalTrax for fixed geometry and for diameter motion of 5% and 20%, and CPU time of one forward pass over 8192 samples. Bottom: sensitivity of one output sample of VocalTrax to past source samples, as the lag containing 99% and 99.9% of the energy and the energy fraction beyond 100 ms.
Figure 1: Spectrograms of the digit “zero” (speaker 55) and its 1000-step reconstructions. (d) renders the parameters fitted by our model with the original recurrent VocalTrax, without further optimization. All panels share one dB scale, and no signal is normalized.
Audio
Resampled to 44.1 kHz, right-padded to a multiple of 1024 samples, one extra control frame at the end
F0
CREPE full model, 23 ms hop; confidence ≤0.5 set to 1 Hz; interpolated to the control frames; fixed during fitting
Controls
5 per 1024-sample frame, normalized to [0,1] : tongue diameter [2.05,3.5] , tongue position [12,29] , throat and lip constriction [−0.99,0.99] , tenseness [0.1,1]
Projection to [0,1] after every update and every smoothing
Smoothing
Savitzky-Golay, window 10, order 7, every 200 steps, averaged 1:1 with the current values; skipped below 10 frames
Table 2: Fitting protocol shared by both models.
Acc. (%) ↑
Log-mel ↓
F0 (ct) ↓
V-F1 ↑
Real
86.7 [81.6, 91.1]
VocalTrax
10.0 [7.7, 12.3]
0.250
46.0
0.843
Ours
14.5 [11.8, 17.0]
0.278
40.1
0.846
Transfer
13.8 [10.9, 16.5]
0.310
41.1
0.849
Table 3: Recognition by a classifier trained on real speech, and distances to the reference, on 600 AudioMNIST recordings. Brackets give 95% intervals. Transfer renders our parameters with VocalTrax. F0 : RMS error in cents; V-F1: voicing F1 score.
Tract, fwd + bwd
Full optimizer step
Audio
VocalTrax
Ours
Ratio
VocalTrax
Ours
Ratio
1 s
1.111
0.023
47.5
1.45
0.50
2.9
8 s
9.30
0.148
62.6
11.77
3.62
3.3
16 s
17.63
0.297
59.3
22.98
7.13
3.2
30 s
34.46
0.546
63.1
42.46
13.35
3.2
60 s
68.06
1.092
62.3
85.19
28.98
2.9
Table 4: Time per call in seconds on one A100 at batch size one. The tract columns add the forward and backward passes of the tract. The full step includes the shared LF source, the loss and the optimizer.
Descriptor
VocalTrax
Ours
Transfer
Log-STFT L1 ↓
1.566
1.579
1.714
MFCC L2 ↓
7.19
7.80
7.74
MFCC DTW cost ↓
6.35
6.66
6.46
Centroid error (Hz) ↓
2587
2727
2813
Contrast error (dB) ↓
5.27
5.83
5.35
Δ MFCC L1 ↓
0.589
0.572
0.576
Table 5: Further descriptors averaged over the 600 recordings. Pitch measures use the 581 recordings voiced in all conditions.
Figure 2: Time per call of both tracts and the shared LF source, at batch size one on one A100.
Figure 3: Per digit (60 recordings each). Top: log-mel MSE. Bottom: frame-to-frame change of the four tract controls, ours divided by VocalTrax.
The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.
Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Scalable Matrix Extension (SME). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.
Compact acoustic models face a challenging quality-capacity trade-off. We investigate two factors in this regime: encoder context and Mel-spectrogram supervision. A receptive-field-scaling study shows that expanding self-attention beyond 15 phonemes provides no consistent gains in pitch, energy, or duration prediction. Guided by this finding, we introduce a fixed-receptive-field convolutional encoder that reduces the respective prediction errors by 36.0%, 17.3%, and 3.4%. We further show that directly transferring image-domain gradient-variance supervision restores fine-scale variation but degrades predicted quality, motivating a Mel-specific formulation with axis-specific gradients, overlapping local statistics, and log-domain variance matching. GrainSpeech contains only 264.8K parameters and achieves 17.9x real-time Mel generation on a microcontroller (MCU), while attaining UTMOS scores comparable to substantially larger models with less than 1.5% of their parameters. Source code and demos are available at https://github.com/lab-emi/GrainSpeech.
Zitao Liang, Chang Gao
Department of Microelectronics, Delft University of Technology, The Netherlands