Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator

Daniel Varela Sánchez; Guillermo Martín-Coello Juárez

cs.SDOct 9, 2026

Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator

Authors: Daniel Varela Sánchez, Guillermo Martín-Coello Juárez

Organizations: Lokutor (Hashing Works, S.L.L.), Madrid, Spain

Abstract

Speech recognition on low-cost microcontrollers is, in practice, limited to keyword spotting or a fixed list of commands. We present Oído, an inference engine that runs open-vocabulary English speech recognition entirely on an ESP32-S3 (240 MHz dual-core Xtensa LX7, 512 KB of SRAM, 8 MB of PSRAM, 16 MB of flash, no neural accelerator). Oído executes a 13M-parameter Conformer-CTC model with int8 weights, per-frame dynamic int8 activations and int8 attention on the chip's SIMD instructions, with a schedule designed around flash and PSRAM bandwidth. On LibriSpeech test-clean/test-other the on-device arithmetic reaches 3.70%/8.23% word error rate (WER), within 0.12 points of full precision, and 3.33%/7.15% with an on-chip 1.3M-parameter language model; a transducer head gives 2.99%/6.67% and an int4 variant fits in 8.3 MB. This is 2.3-2.8x fewer errors than the vendor's recognizer on the same chip, and fewer than Whisper tiny.en and Moonshine tiny in full precision on a laptop, including in real noise and reverberation. Measured on a physical board, the firmware runs at 1.97x the duration of the audio on one core, with transcripts identical to an instruction-exact emulator: accurate, but not yet real time (a 4 s command is transcribed about 8.7 s after it ends). The two-core schedule gives wrong results on silicon and is disabled. A streaming variant shows partial text while the speaker talks and delivers the final text 4.1-11.1 s after clips of 3.6-10.6 s (7.2-22.4 s in utterance mode) at 4.86%/11.03% WER; a Spanish model makes 34-60% fewer errors than multilingual Whisper tiny. Code and models are released.

Figures & tables

Panel of Figure 1 — Figure 1: Oído on the ESP32-S3: pipeline and memory placement. Weights are streamed from flash or PSRAM once per 64-frame block. The activations of the current block and the int32 accumulation tiles stay in on-chip SRAM.

Features	80 log-mel, 25 ms window, 10 ms hop
Subsampling	2 $\times$ conv $3{\times}3$ , stride 2, 176 ch $\to$ 25 Hz
Encoder	16 Conformer blocks, $d{=}176$ , FF 704
Attention	4 heads, relative position [ 9 ]
Convolution	depthwise, kernel 31, BatchNorm
Output	CTC [ 19 ] , 1024 SentencePiece [ 27 ] + blank
Parameters	13 M (int8: 14.0 MB; int4: 8.3 MB)

Table 1: Model architecture (Conformer-CTC Small).

System	Size	clean	other
Conformer-CTC S, fp32 (reference)	52 MB	3.68	8.11
Oído int8, greedy	14.0 MB	3.70	8.23
Oído int8 + GRU LM	15.3 MB	3.33	7.15
Oído int4 (QAT), greedy	8.3 MB	4.61	9.98
Oído int4 + GRU LM	9.6 MB	3.78	8.43
Oído Transducer int8, greedy	15.5 MB	2.99	6.67

Table 2: LibriSpeech wer (%). “Chip” rows use the on-device engine, compiled for the host (same int8 arithmetic; on the board a few hard clips differ in a word, Section 7.5 ). Laptop rows are full precision on 500-utterance subsets. † wer published by the vendor; size of the model files in ESP-SR.

Oído, host build of the on-chip engine
System	CV	Vox	AMI	PS	E22
int8, greedy	11.4	9.3	20.2	11.6	22.4
int8 + LM (books)	10.4	9.2	20.4	12.2	21.6
int8 + LM (books + transcripts)	10.1	9.0	19.0	11.0	20.8
int4 + LM (books)	11.8	11.1	20.2	10.7	21.1
Transducer int8	9.6	8.2	19.2	11.6	21.0

Table 3: wer (%) on held-out sets. CV: Common Voice; Vox: VoxPopuli; PS: People’s Speech (noisy references); E22: Earnings-22. “Books”: LM trained on public-domain books only (released).

Condition	T	C	C+LM	M	W	V
Clean	2.5	3.5	3.3	4.3	5.6	9.1
Car, 10 dB	2.6	3.6	3.3	4.3	5.8	9.1
Car, 5 dB	2.7	3.6	3.2	4.4	5.9	9.3
Kitchen, 10 dB	3.0	4.0	3.6	5.1	6.6	10.7
Kitchen, 5 dB	3.3	4.7	4.2	6.1	7.5	12.4
Living room, 10 dB	3.2	4.4	4.0	5.8	6.7	11.5

Table 4: wer (%) on 300 test-clean utterances under real noise (DEMAND), babble and simulated reverberation. T: Oído Transducer int8; C: Oído CTC int8 greedy; C+LM: with GRU LM; M: Moonshine tiny; W: Whisper tiny.en; V: Vosk small.

Mode	clean	other	robust.	final text ‡
Utterance mode (original)	3.33	7.15	7.5	6.9–21.3 s
Streaming, $C{=}32$ , greedy	6.27	13.06	–	–
Streaming, $C{=}32$	4.86	11.03	9.0	4.1–11.1 s
Streaming, $C{=}16$	6.14	13.23	–	–
Same model, full context	3.43	7.78	6.2	7.2–22.4 s

Table 5: One streaming-trained model on the ESP32-S3 engine (int8, with the language model unless stated), against the original utterance-mode model. Robust.: mean wer over the 14 conditions of Table 4 . ‡ Measured on the board, one core, six clips of 3.6–10.6 s: time from the last sample of the clip to the final text, with the streaming rows fed at the rate of speech and the utterance rows compute only. The 0.8 s end-of-speech wait comes on top.

System	CV	MLS	Vox	FLEURS
Oído + LM (host build)	13.3	10.6	15.5	11.3
Oído greedy	20.3	14.2	20.0	16.9
Oído streaming $C{=}32$ + LM	15.2	12.1	16.3	12.8
Whisper tiny (laptop)	33.0	21.5	28.7	17.1

Table 6: Spanish wer (%) on 400-utterance subsets (Common Voice 17, MLS, VoxPopuli, FLEURS es_419).

Model	clips	RTF	range	= host
CTC int8 (released)	27	1.97	1.91–2.02	22/27
CTC int4	10	1.76	1.73–1.83	8/10
Transducer int8, no LM	10	1.89	1.88–1.91	10/10
Stream. model, utterance	12	2.04	1.98–2.11	12/12
Stream. model, $C{=}32$	12	1.96	–	10/12
Spanish, utterance	12	2.06	1.98–2.13	9/12

Table 7: Measured on the board, one core, 240 MHz, with the language model unless noted. RTF: compute time divided by audio duration (mean, and range over clips). “= host”: clips whose transcript equals the host build’s. The firmware’s transcripts equal the emulator’s on all 27 clips of the first row. Clips: 214.8 s of audio in the first row, 59–103 s in the others.

Stage	Minstr/s	Mcyc/s	cyc/instr
log-mel	10	14	1.48
conv0	14	20	1.45
im2col	6	20	3.66
matrix products	103	216	2.09
layernorm	7	15	2.07
activation quant	15	32	2.16

Table 8: Cycles on the board against instructions in the emulator, per second of audio (int8 CTC model, 27 clips on the board, 39.7 s of speech in the emulator).

Panel of Figure 3 — Figure 3: Executed instructions per second of audio, by stage, for the int8 CTC model (both cores, exact counts from instruction-counting emulation).

Step	G instr.	$\Delta$
Baseline int8 engine	9.82	–
+ merged attention score pass	9.63	$-$ 1.9%
+ x-stationary GEMM for $K{\leq}256$	9.39	$-$ 2.5%
+ K-split, batched front end †	9.66	+2.9%
+ int8 positional table, buffer aliasing	9.58	$-$ 0.8%
+ fused im2col quantization	9.29	$-$ 3.0%

Table 9: Engine optimizations: executed instructions for 39.7 s of audio (int8 CTC, greedy), cumulative. † Adds instructions but cuts weight traffic from flash from 18 to 7.5 MB/s.

Profile	RTF	text after clip
CTC int8 + LM, utterance mode	1.97	6.9–7.4 s
CTC int4 + LM, utterance mode	1.76	6.2–6.7 s
Transducer int8, utterance mode	1.89	7.0 s †
Streaming model, utterance mode	2.04	7.2–7.6 s
Streaming model, streaming $C{=}32$	1.96	4.1–4.2 s

Table 10: Time from the last sample of a 3.6–3.8 s command to its text, measured on the board (one core). † Computed from the measured RTF (this model was not run on a clip of that length).

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Sep 24, 2026

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement

Deploying real-time speech enhancement on resource-constrained devices requires meeting strict latency, memory, and energy constraints. Microcontroller NPUs can accelerate neural inference under these constraints, but only through a restricted set of operators in static, integer-quantized graphs. Recent speech-enhancement networks have reduced parameter counts and MACs to levels nominally suitable for microcontrollers, but their operators and execution patterns often remain incompatible with restricted NPUs. We address this gap by redesigning LiSenNet, a 37k parameter sub-band dual-path model, for the STM32N6570-DK Neural-ART accelerator. We replace its recurrent bottleneck with convolutional frequency and temporal mixers, reformulate unsupported operations as static int8-compatible primitives, and use bounded decoder activations to preserve quality after quantization. On VoiceBank-DEMAND, the final NPU-compatible model matches or exceeds the recurrent LiSenNet baseline, reaching PESQ 3.08 versus 3.01 in FP32 and 3.01 versus 2.93 in int8. Deployed on a microcontroller, it processes each 16 ms input hop in 4.83 ms, corresponding to a real-time factor of 0.30. Stateless receptive-field recomputation is an order of magnitude slower at the same frame rate despite higher accelerator utilization. These results show that parameter count and operator compatibility, quantization range, and persistent streaming state must be co-designed to achieve efficient real-time speech enhancement on restricted NPUs.

Clément Laroche, Rasmus Kongsgaard Olsson

GN A/S, Audio Research, Lauptrupbjerg 7, 2750, Denmark

Efficient Neural Network Inference Speech Enhancement Neural Network Quantization

Ask PDF

Oct 17, 2026

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose PARDON, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of 70%, PARDON achieves the highest recall by identifying up to 52% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24% under a budget that accounts for both interruption frequency and the lengths of re-uttered spans.

Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi

IIT Bombay

Automatic Speech Recognition Streaming ASR

Ask PDF

Jun 9, 2026

Massive Open-Vocabulary Keyword Spotting

Automatic speech recognition systems have been shown to under-perform when it comes to transcribing words rarely seen in the training data, namely specialized terminology. Open-vocabulary keyword spotting, combined with contextual biasing, has been shown to mitigate this issue. However, existing systems can only handle glossaries of a few hundred terms without becoming an infeasible bottleneck. We propose a system that stores features with a memory footprint up to 128 times smaller than a comparable baseline and allows users to process massive databases while remaining open-vocabulary. Without fine-tuning the speech recognition model, our system achieves a comparable entity recall as uncompressed solutions, even in languages not seen during training.

Leonor Barreiros, Raul Monteiro, Afonso Mendes +1

Priberam Labs, Lisboa, Portugal · Instituto Superior Técnico, Lisboa, Portugal · Instituto de Telecomunicações, Lisboa, Portugal

Keyword Spotting Automatic Speech Recognition

Ask PDF

Abstract

Automatic Speech Recognition

Explore similar work

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

Massive Open-Vocabulary Keyword Spotting