Whisper-RIR-Mega: A Paired Clean-Reverberant Speech Benchmark for ASR Robustness to Room Acoustics
Organizations: Acoustics Researcher, WA, USA
Abstract
We introduce Whisper-RIR-Mega, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics. Each sample pairs a clean LibriSpeech utterance with the same utterance convolved with a real room impulse response from the RIR-Mega corpus, with stratified splits by reverberation time (RT60) and direct-to-reverberant ratio (DRR). We evaluate five Whisper models (tiny through large-v3) on 1600 test samples and report word error rate (WER) and character error rate (CER) under clean and reverberant conditions. Reverberation consistently degrades performance across all model sizes; the reverb penalty in WER ranges from 2.31 to 15.50 percentage points depending on the model. Whisper-large-v3 shows the smallest penalty; Whisper-tiny shows the largest. We release the dataset, evaluation code, and baseline results to support reproducible research on robust ASR.
Figures & tables
| Model | WER (clean) | WER (reverb) | WER | CER (clean) | CER (reverb) | CER |
|---|---|---|---|---|---|---|
| Whisper-tiny | 54.88 | 70.38 | 15.50 | 3.72 | 7.52 | 3.80 |
| Whisper-base | 46.50 | 57.94 | 11.44 | 3.08 | 4.88 | 1.80 |
| Whisper-small | 35.88 | 43.31 | 7.44 | 2.05 | 3.04 | 0.99 |
| Whisper-medium | 30.06 | 36.00 | 5.94 | 1.90 | 2.38 | 0.48 |
| Whisper-large-v3 | 29.00 | 31.31 | 2.31 | 3.07 | 4.51 | 1.44 |