The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication

Hanlong Li; Karishma Kamalahasan; Jiahui Li; Kazuhiro Nakadai; Shreyas Kousik

cs.ROMar 7, 2026

The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication

Authors: Hanlong Li, Karishma Kamalahasan, Jiahui Li, Kazuhiro Nakadai, Shreyas Kousik

Organizations: Institute of Science Tokyo · Georgia Institute of Technology

Abstract

We present Artoo, a learned acoustic communication system for robots that replaces hand-designed signal processing with end-to-end co-trained neural networks. Our system pairs a lightweight text-to-speech (TTS) transmitter (1.18M parameters) with a conformer-based automatic speech recognition (ASR) receiver (938K parameters), jointly optimized through a differentiable channel. Unlike human speech, robot-to-robot communication is paralinguistics-free: the system need not preserve timbre, prosody, or naturalness, only maximize decoding accuracy under channel distortion. Through a three-phase co-training curriculum, the TTS transmitter learns to produce distortion-robust acoustic encodings that surpass the baseline under noise, achieving 8.3% CER at 0 dB SNR. The entire system requires only 2.1M parameters (8.4 MB) and runs in under 13 ms end-to-end on a CPU, making it suitable for deployment on resource-constrained robotic platforms.

Figures & tables

Fig. 1: We propose Artoo : a text-to-speech (TTS) and automatic speech recognition (ASR) pipeline for robot-robot communication. The goal of our method is to enable robots to communicate in a way that is robust to environmental noise; our key insight is to relax the need for TTS and ASR models to replicate human speech.

Parameter	Value
Model dimension $d$	128
Feed-forward dimension $d_{\text{ff}}$	256
Attention heads	4
Encoder layers $L_{\text{enc}}$	4
Decoder layers $L_{\text{dec}}$	4
Mel bins $N_{\text{mel}}$	40

TABLE I : TTS hyperparameters.

Component	Params	Key Details
TTS Transmitter	1.18 M	4+4 Transformer layers, $d$ =128, 4 heads
Griffin-Lim	0	32 iterations
ASR Receiver	938 K	4 Conformer blocks, Conv $k$ =15, CTC
Procedural Synth.	0	3-harmonic, 60 ms/token
Total	2.1 M	8.4 MB (fp32)

TABLE II: Artoo Model architecture summary.

Hyperparameter	Value
Optimizer	AdamW (separate for TTS/ASR)
Learning rate (TTS / ASR)	$5{\times}10^{-5}$ / $1{\times}10^{-4}$
LR schedule	Warmup (500 steps) + cosine decay
Weight decay	$10^{-2}$
Batch size	128
Training steps	$\sim$ 30,000 (300 epochs)

TABLE III: Training hyperparameters.

System	Type	Params	Custom Tokens
Ours (co-trained)	Learned codec	2.1 M	✓
Artoo Untrained (ours)	Ablation	2.1 M	✓
Whisper-tiny + gTTS	Off-the-shelf	39 M	—
GGWave	FSK modem	—	✓

TABLE IV: Systems under evaluation.

Fig. 2: Illustration of the real-world over-the-air deployment setup used for hardware evaluation.

Config	Transmitter	Receiver
Laptop	Laptop speaker	Laptop mic
RPi	RPi Zero 2W + USB speaker	RPi 5 + USB mic

TABLE V: Physical testbed hardware configurations.

∗ Dominated by network round-trip to gTTS API.
System	Params	Size	Latency
Artoo Co-trained (ours)	2.1 M	8.4 MB	12 ms
Artoo Untrained	2.1 M	8.4 MB	11.4 ms
Whisper + gTTS	39 M	150 MB	342 ms ∗
GGWave	—	$<$ 1 MB	106 ms

TABLE VI: System footprint and latency.

† Silent drop reported as 100% CER.
SNR (dB)	$-10$	$-5$	$0$	$5$	$10$	clean
Artoo Co-trained (ours)	38.4	21.7	8.3	3.1	2.2	0.4
Artoo Untrained	100	100	100	99.7	98.2	99.1
Whisper + gTTS	89.2	64.5	35.8	7.6	5.4	0.1
GGWave	100 †	100 †	28.3	12.1	0.0	0.0

TABLE VII: Noise robustness: CER (%) at varying SNR.

System	Gaussian	Pink	Brown
Artoo Co-trained (ours)	2.1	4.8	2.6
Artoo Untrained (ours)	100	99.8	99.6
Whisper + gTTS	7.9	9.2	8.6
GGWave	9.2	14.7	8.5

TABLE VIII: Multi-noise robustness: CER (%) at

5\text{\,}\mathrm{d}\mathrm{B}

SNR.

† Includes silent drops as 100% per-message CER.
System	CER ( $\downarrow$ )	WER ( $\downarrow)$	EM ( $\uparrow$ )	Latency ( $\downarrow)$
Artoo Co-trained (ours)	3.1%	8.4%	72%	13 ms
Artoo Untrained (ours)	100%	100%	0%	12.7 ms
Whisper + gTTS	7.6%	9.3%	68%	394.72 ms
GGWave	12.1% †	—	91%	147.97 ms

TABLE IX: E2E simulation results (

5\text{\,}\mathrm{d}\mathrm{B}

SNR, 200 messages).

Length (chars)	5	10	20	40	60	100	200	500	1000
Artoo Co-trained (ours)	1.8	2.4	3.1	4.3	4.7	3.8	3.5	3.3	3.4
Artoo Untrained (ours)	99.8	97.4	99.1	100.0	100.0	99.7	98.8	99.9	97.2
Whisper + gTTS	98.5	30.1	25.9	7.4	6.5	5.2	6.7	6.8	7.2
GGWave	0.0	4.2	12.1	14.0	21.6	13.7	— †	— †	— †
† GGWave only allows message length less than 140 characters.

TABLE X: CER (%) vs. message length at

5\text{\,}\mathrm{d}\mathrm{B}

SNR.

Config	System	0.5 m	1.0 m	3.0 m
Laptop	Artoo (ours)	1.2	2.8	7.4
RPi	Artoo (ours)	2.4	4.1	11.3

TABLE XI: Over-the-air results: CER (%) by distance and hardware configuration (quiet lab, no injected noise).

Config	System	CER	EM	Latency
Laptop	Ours	6.3%	58%	28 ms
RPi	Ours	8.7%	51%	47 ms

TABLE XII: Over-the-air with injected noise (

\sim

5\text{\,}\mathrm{d}\mathrm{B}

SNR,

1.0\text{\,}\mathrm{m}

distance).

Configuration	CER	EM	Note
PS + Co-trained ASR	1.8%	86%	ASR generalizes to PS
PS + Detached ASR	3.4%	71%	PS-only baseline
TTS_MEL + Co-trained	0.4%	94%	Encoding quality
TTS_WAV + Co-trained	0.6%	91%	Deploy path
TTS_MEL + Detached	42.1%	3%	No co-training
TTS_WAV + Detached	56.8%	1%	No co-training

TABLE XIII: Ablation study: effect of co-training (clean).

Channel Effect	TTS_WAV	PS_WAV
Noise ( $5\text{\,}\mathrm{d}\mathrm{B}$ )	3.1	8.7
Noise ( $0\text{\,}\mathrm{d}\mathrm{B}$ )	8.3	22.4
Reverb	1.4	18.6
Clipping	2.1	31.2
Resample drift	1.8	14.9
Combined	12.6	54.3

TABLE XIV: Channel effects: CER (%) under individual distortions. TTS_WAV = co-trained TTS, PS_WAV = procedural synthesizer.

Robust ASR

See all 23 papers

Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems
Jul 13, 2026Sheng Li, Jing Li, Felix Schijve +2Robotics Automatic Speech Recognition
Institute of Science Tokyo, Yokohama, Japan · Department of Industrial Design, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands · Department of Biomedical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
Human-robot conversation with multiple participants in noisy public spaces
Sep 1, 2026Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes +10Spoken Dialogue Systems Speech Enhancement
1Kyoto University Graduate School of Informatics · 2Osaka University, Graduate School of Engineering Science · 3National University of Singapore +1
ECHO-G: Embodied Co-speech Humanoid mOtion Generation
Sep 30, 2026Yizhao Li, Pusen Gao, Ming Wang +3Text-to-Motion Generation Human Motion Generation
Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1

Jul 13, 2026J/K move · Enter open · S save

Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems

Sheng Li, Jing Li, Felix Schijve, Jun Hu+1

Institute of Science Tokyo, Yokohama, Japan · Department of Industrial Design, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands · Department of Biomedical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands

Robotics Automatic Speech Recognition Human-Robot Interaction

Open paper AskPDF arXiv

Automatic speech recognition (ASR) has become a critical component of modern robotic systems because it is one of the most natural and intuitive ways for humans to interact with robots. A commonly used method is to directly use API services online. But is that all we can do? This article provides an overview of how ASR technologies are integrated into various intelligent robots and machines. We discuss the evolution of speech recognition from established approaches to state-of-the-art deep learning models, such as OpenAI's Whisper. We also list large-scale datasets and open source toolkits that have been widely used in both industry and academia. We structure the survey around ASR model families, deployment strategies in robotics (especially ROS-based, cloud-based, and hybrid solutions), and several real-world robotic platforms. Finally, we outline the challenges of deploying robust speech recognition in robots and discuss future directions, including multimodal interaction in diverse and dynamic environments. This paper can help social robotics researchers better navigate the emerging domain of language-based natural human-robot interaction.

Abstract

Robust ASR

Explore similar work

Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems

Human-robot conversation with multiple participants in noisy public spaces

ECHO-G: Embodied Co-speech Humanoid mOtion Generation