The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication

Hanlong Li; Karishma Kamalahasan; Jiahui Li; Kazuhiro Nakadai; Shreyas Kousik

cs.ROMar 7, 2026

The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication

Authors: Hanlong Li, Karishma Kamalahasan, Jiahui Li, Kazuhiro Nakadai, Shreyas Kousik

Organizations: Institute of Science Tokyo · Georgia Institute of Technology

Abstract

We present Artoo, a learned acoustic communication system for robots that replaces hand-designed signal processing with end-to-end co-trained neural networks. Our system pairs a lightweight text-to-speech (TTS) transmitter (1.18M parameters) with a conformer-based automatic speech recognition (ASR) receiver (938K parameters), jointly optimized through a differentiable channel. Unlike human speech, robot-to-robot communication is paralinguistics-free: the system need not preserve timbre, prosody, or naturalness, only maximize decoding accuracy under channel distortion. Through a three-phase co-training curriculum, the TTS transmitter learns to produce distortion-robust acoustic encodings that surpass the baseline under noise, achieving 8.3% CER at 0 dB SNR. The entire system requires only 2.1M parameters (8.4 MB) and runs in under 13 ms end-to-end on a CPU, making it suitable for deployment on resource-constrained robotic platforms.

Figures & tables

Fig. 1: We propose Artoo : a text-to-speech (TTS) and automatic speech recognition (ASR) pipeline for robot-robot communication. The goal of our method is to enable robots to communicate in a way that is robust to environmental noise; our key insight is to relax the need for TTS and ASR models to replicate human speech.

Parameter	Value
Model dimension $d$	128
Feed-forward dimension $d_{\text{ff}}$	256
Attention heads	4
Encoder layers $L_{\text{enc}}$	4
Decoder layers $L_{\text{dec}}$	4
Mel bins $N_{\text{mel}}$	40

TABLE I : TTS hyperparameters.

Component	Params	Key Details
TTS Transmitter	1.18 M	4+4 Transformer layers, $d$ =128, 4 heads
Griffin-Lim	0	32 iterations
ASR Receiver	938 K	4 Conformer blocks, Conv $k$ =15, CTC
Procedural Synth.	0	3-harmonic, 60 ms/token
Total	2.1 M	8.4 MB (fp32)

TABLE II: Artoo Model architecture summary.

Hyperparameter	Value
Optimizer	AdamW (separate for TTS/ASR)
Learning rate (TTS / ASR)	$5{\times}10^{-5}$ / $1{\times}10^{-4}$
LR schedule	Warmup (500 steps) + cosine decay
Weight decay	$10^{-2}$
Batch size	128
Training steps	$\sim$ 30,000 (300 epochs)

TABLE III: Training hyperparameters.

System	Type	Params	Custom Tokens
Ours (co-trained)	Learned codec	2.1 M	✓
Artoo Untrained (ours)	Ablation	2.1 M	✓
Whisper-tiny + gTTS	Off-the-shelf	39 M	—
GGWave	FSK modem	—	✓

TABLE IV: Systems under evaluation.

Fig. 2: Illustration of the real-world over-the-air deployment setup used for hardware evaluation.

Config	Transmitter	Receiver
Laptop	Laptop speaker	Laptop mic
RPi	RPi Zero 2W + USB speaker	RPi 5 + USB mic

TABLE V: Physical testbed hardware configurations.

∗ Dominated by network round-trip to gTTS API.
System	Params	Size	Latency
Artoo Co-trained (ours)	2.1 M	8.4 MB	12 ms
Artoo Untrained	2.1 M	8.4 MB	11.4 ms
Whisper + gTTS	39 M	150 MB	342 ms ∗
GGWave	—	$<$ 1 MB	106 ms

TABLE VI: System footprint and latency.

† Silent drop reported as 100% CER.
SNR (dB)	$-10$	$-5$	$0$	$5$	$10$	clean
Artoo Co-trained (ours)	38.4	21.7	8.3	3.1	2.2	0.4
Artoo Untrained	100	100	100	99.7	98.2	99.1
Whisper + gTTS	89.2	64.5	35.8	7.6	5.4	0.1
GGWave	100 †	100 †	28.3	12.1	0.0	0.0

TABLE VII: Noise robustness: CER (%) at varying SNR.

System	Gaussian	Pink	Brown
Artoo Co-trained (ours)	2.1	4.8	2.6
Artoo Untrained (ours)	100	99.8	99.6
Whisper + gTTS	7.9	9.2	8.6
GGWave	9.2	14.7	8.5

TABLE VIII: Multi-noise robustness: CER (%) at

5\text{\,}\mathrm{d}\mathrm{B}

SNR.

† Includes silent drops as 100% per-message CER.
System	CER ( $\downarrow$ )	WER ( $\downarrow)$	EM ( $\uparrow$ )	Latency ( $\downarrow)$
Artoo Co-trained (ours)	3.1%	8.4%	72%	13 ms
Artoo Untrained (ours)	100%	100%	0%	12.7 ms
Whisper + gTTS	7.6%	9.3%	68%	394.72 ms
GGWave	12.1% †	—	91%	147.97 ms

TABLE IX: E2E simulation results (

5\text{\,}\mathrm{d}\mathrm{B}

SNR, 200 messages).

Length (chars)	5	10	20	40	60	100	200	500	1000
Artoo Co-trained (ours)	1.8	2.4	3.1	4.3	4.7	3.8	3.5	3.3	3.4
Artoo Untrained (ours)	99.8	97.4	99.1	100.0	100.0	99.7	98.8	99.9	97.2
Whisper + gTTS	98.5	30.1	25.9	7.4	6.5	5.2	6.7	6.8	7.2
GGWave	0.0	4.2	12.1	14.0	21.6	13.7	— †	— †	— †
† GGWave only allows message length less than 140 characters.

TABLE X: CER (%) vs. message length at

5\text{\,}\mathrm{d}\mathrm{B}

SNR.

Config	System	0.5 m	1.0 m	3.0 m
Laptop	Artoo (ours)	1.2	2.8	7.4
RPi	Artoo (ours)	2.4	4.1	11.3

TABLE XI: Over-the-air results: CER (%) by distance and hardware configuration (quiet lab, no injected noise).

Config	System	CER	EM	Latency
Laptop	Ours	6.3%	58%	28 ms
RPi	Ours	8.7%	51%	47 ms

TABLE XII: Over-the-air with injected noise (

\sim

5\text{\,}\mathrm{d}\mathrm{B}

SNR,

1.0\text{\,}\mathrm{m}

distance).

Configuration	CER	EM	Note
PS + Co-trained ASR	1.8%	86%	ASR generalizes to PS
PS + Detached ASR	3.4%	71%	PS-only baseline
TTS_MEL + Co-trained	0.4%	94%	Encoding quality
TTS_WAV + Co-trained	0.6%	91%	Deploy path
TTS_MEL + Detached	42.1%	3%	No co-training
TTS_WAV + Detached	56.8%	1%	No co-training

TABLE XIII: Ablation study: effect of co-training (clean).

Channel Effect	TTS_WAV	PS_WAV
Noise ( $5\text{\,}\mathrm{d}\mathrm{B}$ )	3.1	8.7
Noise ( $0\text{\,}\mathrm{d}\mathrm{B}$ )	8.3	22.4
Reverb	1.4	18.6
Clipping	2.1	31.2
Resample drift	1.8	14.9
Combined	12.6	54.3

TABLE XIV: Channel effects: CER (%) under individual distortions. TTS_WAV = co-trained TTS, PS_WAV = procedural synthesizer.

Robust ASR

See all 23 papers

Jul 13, 2026

Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems

Automatic speech recognition (ASR) has become a critical component of modern robotic systems because it is one of the most natural and intuitive ways for humans to interact with robots. A commonly used method is to directly use API services online. But is that all we can do? This article provides an overview of how ASR technologies are integrated into various intelligent robots and machines. We discuss the evolution of speech recognition from established approaches to state-of-the-art deep learning models, such as OpenAI's Whisper. We also list large-scale datasets and open source toolkits that have been widely used in both industry and academia. We structure the survey around ASR model families, deployment strategies in robotics (especially ROS-based, cloud-based, and hybrid solutions), and several real-world robotic platforms. Finally, we outline the challenges of deploying robust speech recognition in robots and discuss future directions, including multimodal interaction in diverse and dynamic environments. This paper can help social robotics researchers better navigate the emerging domain of language-based natural human-robot interaction.

Sheng Li, Jing Li, Felix Schijve +2

Institute of Science Tokyo, Yokohama, Japan · Department of Industrial Design, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands · Department of Biomedical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands

Robotics Automatic Speech Recognition Human-Robot Interaction

Ask PDF

Sep 1, 2026

Human-robot conversation with multiple participants in noisy public spaces

For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.

Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes +10

1Kyoto University Graduate School of Informatics · 2Osaka University, Graduate School of Engineering Science · 3National University of Singapore +1

Spoken Dialogue Systems Speech Enhancement Spatial Audio Human-Robot Interaction Microphone Array Processing

Ask PDF

Sep 30, 2026

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-aligned acoustic features with token-level linguistic context, preserving their distinct granularities. Trained with rectified flow matching, it models one-to-many utterance-motion relationships directly in robot space. To support training and evaluation, we introduce a BEAT2-derived audio-text-robot dataset and a benchmark covering co-speech characteristics, robot-motion quality, and runtime efficiency. Comparative evaluation supports direct robot-space generation over the evaluated human-motion generation and retargeting pipelines, while modality ablations highlight the benefits of joint audio-text conditioning. We further demonstrate deployment on a physical humanoid robot. A complementary video-rating study also favors joint conditioning over the alternatives. The dataset and training, inference, and evaluation code are available through our project page.

Yizhao Li, Pusen Gao, Ming Wang +3

Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1

Text-to-Motion Generation Human Motion Generation Robot Motion Generation Co-Speech Gesture Generation Diffusion Models

Ask PDF

Abstract

Robust ASR

Explore similar work

Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems

Human-robot conversation with multiple participants in noisy public spaces

ECHO-G: Embodied Co-speech Humanoid mOtion Generation