NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction

Jun Rekimoto; Yu Nishimura; Bojian Yang

cs.HCMar 11, 2026

NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction

Authors: Jun Rekimoto, Yu Nishimura, Bojian Yang

Organizations: Sony CSL - Kyoto · Sony CSL

Abstract

Silent and whispered speech offer promise for always-available voice interaction with AI, yet existing methods struggle to balance vocabulary size, wearability, silence, and noise robustness. We present NasoVoce, a nose-bridge-mounted interface that integrates a microphone and a vibration sensor. Positioned at the nasal pads of smart glasses, it unobtrusively captures both acoustic and vibration signals. The nasal bridge, close to the mouth, allows access to bone- and skin-conducted speech and enables reliable capture of low-volume utterances such as whispered speech. While the microphone captures high-quality audio, it is highly sensitive to environmental noise. Conversely, the vibration sensor is robust to noise but yields lower signal quality. By fusing these complementary inputs, NasoVoce generates high-quality speech robust against interference. Evaluation with Whisper Large-v2, PESQ, STOI, and MUSHRA ratings confirms improved recognition and quality. NasoVoce demonstrates the feasibility of a practical interface for always-available, continuous, and discreet AI voice conversations.

Figures & tables

Figure 1. NasoVoce integrates a microphone and a vibration sensor at the nasal bridge for capturing normal and whispered speech. This location enables effective capture of both air-conducted and bone/skin-conducted signals even in very noisy environment, while remaining unobtrusive for daily wear.

name	method	silence	noise tolerance	command/speech
AlterEgo ( Kapur et al., 2018 )	neuromuscular	silent	good	command
SottoVoce ( Kimura et al., 2019 )	ultrasound	silent	good	limited vocab.
SilentVoice ( Fukumoto, 2018 )	ingressive speech	low	unknown	speech
WESPER ( Rekimoto, 2023 )	mic	normal/whisper	unknown	speech
LipLerner ( Su et al., 2023 )	lip-reading	silent	good	command
V-Speech ( Maruri et al., 2018 )	BC on nosepad	normal voice	good	speech

Table 1. Comparison of silent and whispered speech / wearable speech technologies

Figure 2. NasoVoce sensor configuration: A MEMS microphone and MEMS vibration sensor can be acquired as time-synchronized digital data as the left and right channels of a TDM audio interface. An example usage is shown with the sensor mounted on the nose pad of a smart glasses frame.

Figure 3. By covering the mouth and nose with one’s hand, the spoken content cannot be inferred through lip reading. This posture further serves as a social signal, indicating that the person is engaged in a conversation with the device.

Figure 4. D-DCCRN (Dual-DCCRN) accepts composite inputs from a microphone (Mic) and a vibration sensor (Vib). D-DCCRN generalizes the design of DCCRN audio enhancement model to jointly process the real and imaginary components of both Mic and Vib signals.

Figure 5. The NasoVoce training method: it combines audio enhancement loss ( $L_{ae}$ ) and Knowledge Distillation loss ( $L_{kd}$ )

Figure 6. Audio quality improvement examples (normal and whispered speeches) through the combined use of microphone (Mic) input and vibration sensor (Vib) input: While the Mic input contains external noise, the audio enhancement results obtained from both Mic and Vib inputs (Enhancement) closely approximate the Ground Truth. This outcome is demonstrated through a simulation in which noise is added to the Ground Truth to form the Mic input, and an audio enhancement model is subsequently applied.

Figure 7. Speech recognition accuracy (WER, CER) for normal and whispered speech: MEMS microphone (Mic), MEMS vibration sensor (Vib), audio enhancement by D-DCCRN

Figure 8. PESQ and STOI scores of MEMS microphone (Mic), MEMS vibration sensor (Vib), and enhanced voice using the D-DCCRN model by combining Mic and Vib (D-DCCRN) under various noise conditions.

Figure 9. MUSHRA-based audio quality evaluation: We assessed the Clean speech (Hidden Reference), the simultaneously recorded MEMS vibration sensor (Vib), a microphone signal with added noise (Mic, SNR

10\text{\,}\mathrm{d}\mathrm{B}

0\text{\,}\mathrm{d}\mathrm{B}

-10\text{\,}\mathrm{d}\mathrm{B}

), and an Enhanced signal obtained by fusing Mic and Vib and processing with D-DCCRN.

Figure 10. In-the-wild trial of NasoVoce: We recorded normal and whispered speech in four everyday environments—inside a café, on a busy roadside, while walking, and inside a train car—and compared signals obtained with a conventional microphone and with NasoVoce.

Figure 11. Alternative use cases of the proposed Vib-Mic sensor: (Left) Placed against the skin between the upper lip and the nose, it can reliably capture whispered speech via skin conduction. (Right) Applied as a throat microphone, it acquires both skin-conducted and vocal signals through the neck.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces
Aug 1, 2026Ruidong Zhang, Jiacheng Liu, François Guimbretière +1Automatic Speech Recognition
Cornell University, Ithaca, NY, USA
Soft Active Electromyography Interface for Machine Learning-Enabled Silent Speech Recognition
Aug 27, 2026Yuta Kurotaki, Shusuke Yamakoshi, Reitaro Yoshida +6Automatic Speech Recognition
Department of Mechanical Engineering, Yokohama National University, 79-5 Tokiwadai, Hodogaya-Ku, Yokohama, Kanagawa 240-8501, Japan · Pepabo Research and Development Institute, GMO Pepabo, Inc., 26-1 Sakuragaoka, Shibuya Ward, Tokyo 150-8512, Japan
Human-robot conversation with multiple participants in noisy public spaces
Sep 1, 2026Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes +10Spoken Dialogue Systems Speech Enhancement
1Kyoto University Graduate School of Informatics · 2Osaka University, Graduate School of Engineering Science · 3National University of Singapore +1

Aug 1, 2026J/K move · Enter open · S save

SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

Ruidong Zhang, Jiacheng Liu, François Guimbretière, Cheng Zhang

Cornell University, Ithaca, NY, USA

Automatic Speech Recognition

Open paper AskPDF arXiv

Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies. Approaches achieving larger vocabularies require obtrusive hardware such as facial electrodes. We present SoniSpeech, the first large-scale, open-vocabulary, trimodal dataset for wearable SSI using acoustic-sensing eyewear. It contains 34 hours across 18,000 utterances with three synchronized modalities: ultrasound echo profiles, voiced audio, and frontal video, in both voiced and silent modes. The corpus draws from the SODA dialogue dataset, providing contemporary conversational English with 5,356 unique words and full phoneme coverage. A CTC-based ResNet-34 baseline achieves 26.3% word error rate (WER) on open-vocabulary silent speech recognition, the first benchmark for this task. Dataset is available at https://doi.org/10.7298/xjjr-9m85

Abstract

Automatic Speech Recognition

Explore similar work

SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

Soft Active Electromyography Interface for Machine Learning-Enabled Silent Speech Recognition

Human-robot conversation with multiple participants in noisy public spaces