cs.SDOct 11, 2026

Synthesizing Ultrasonic Echo Profiles from Text for Open-Vocabulary Silent Speech Recognition

Authors: Jiacheng Liu, Ruidong Zhang, François Guimbretière, Cheng Zhang

Organizations: Cornell University, Ithaca, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.

Figures & tables

Figure 1: Overview of SoniGAN. (a) The recognizer encodes echo profiles with a ResNet encoder and decodes text with a Transformer; crossCon contrasts its output against soft speech units uu , with a DTW path aligning the silent profile to its voiced counterpart. (b) Stage 1 encodes uu and the mode vector eme_{m} into cc and expands it into an envelope, under two discriminators and Lperc\mathcal{L}_{\text{perc}} from a frozen encoder. (c) Stage 2 concatenates uu with an envelope and predicts the residual that carries the fringe structure, under Lperc\mathcal{L}_{\text{perc}} , Lenv\mathcal{L}_{\text{env}} on the Hilbert envelope, and a second pair of discriminators. The circled plus denotes element-wise addition and the circled C concatenation along the channel axis.
MethodReal (sess.)Syn. (h)WER (%)
SoniSpeech [ 23 ]140–26.3
SoniGAN140–25.9
+ synthetic data1402624.6
+ synthetic data1405123.8
+ synthetic data14010423.6
+ 20 noisy-channel sessions16010422.2
Table 1: Silent-speech WER on the SoniSpeech test set. Rows are cumulative; each addition is applied on top of the row above.
 WER (%)
Training sessionsw/o dup.w/ dup.
140 (intact only)23.623.5
160 (incl. 20 faulty)22.822.2
Table 2: Contribution of the 20 sessions with a faulty microphone and of the channel-duplication augmentation. All rows start from the same 104 h pretrained checkpoint.
 Synthetic WER (%)Downstream
Pretraining dataSoniSpeech [ 23 ]Corpus [ 5 , 12 ]WER (%)
None––25.9
Noise––27.6
Real data30.4–26.2
SoniGAN29.052.023.6
w/o Lenv\mathcal{L}_{\text{env}}28.350.824.3
Table 3: Ablations of the pretraining data and of the generator’s supervision.
More in this topic

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Explore similar work

CardsList

  1. SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

    Aug 1, 2026Ruidong Zhang, Jiacheng Liu, François Guimbretière +1Automatic Speech Recognition

    Cornell University, Ithaca, NY, USA

  2. Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading

    Jun 8, 2026Eder del Blanco, David Gimeno-Gómez, Eva Navas +2Missing-Modality LearningMultimodal Fusion

    Aholab research group within the HiTZ Center at University of the Basque Country (UPV/EHU), Barrio Sarriena s/n, 48940 Leioa, Bizkaia, Spain · PRHLT research center, Universitat Polit`ecnica de Val`encia (UPV), Camino de Vera s/n, 46022, Val`encia, Spain

  3. Textual Echo Cancellation

    Aug 13, 2020Shaojin Ding, Ye Jia, Ke Hu +1Sequence-to-Sequence ModelsSpeech Enhancement

    Google LLC, USA