"OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detection
Organizations: Telefónica Innovación Digital, Madrid, Spain · Universidad Autónoma de Madrid, Madrid, Spain
Abstract
Voice-based interfaces are widely used; however, achieving fair Wake-up Word detection across diverse speaker populations remains a critical challenge due to persistent demographic biases. This study evaluates the effectiveness of demographics-agnostic training techniques in mitigating performance disparities among speakers of varying sex, age, and accent. We utilize the OK Aura database for our experiments, employing a training methodology that excludes demographic labels, which are reserved for evaluation purposes. We explore (i) data augmentation techniques to enhance model generalization and (ii) knowledge distillation of pre-trained foundational speech models. The experimental results indicate that these demographics-agnostic training techniques markedly reduce demographic bias, leading to a more equitable performance profile across different speaker groups. Specifically, one of the evaluated techniques achieves a Predictive Disparity reduction of 39.94% for sex, 83.65% for age, and 40.48% for accent when compared to the baseline. This study highlights the effectiveness of label-agnostic methodologies in fostering fairness in Wake-up Word detection.
Figures & tables
| Sex | # Samples | Percentage |
|---|---|---|
| Female | 2131 | 41.74% |
| Male | 2974 | 58.26% |
| Sex | # Samples | # Speakers |
|---|---|---|
| Female | 254 (44.88%) | 20 |
| Male | 321 (55.12%) | 27 |
| Age Group | # Samples | # Speakers |
|---|---|---|
| 0-20 | 0 | 0 |
| 21-30 | 135 | 11 |
| 31-40 | 138 | 11 |
| 41-50 | 295 | 24 |
| 51+ | 7 | 1 |
| Accent | # Samples | # Speakers |
|---|---|---|
| central southern | 313 | 26 |
| eastern & balearic | 15 | 1 |
| non native | 49 | 4 |
| northern | 90 | 7 |
| southern | 84 | 7 |
| unknown | 12 | 2 |
| Attribute | Advantaged Group | Disadvantaged Group | Disparate Impact |
|---|---|---|---|
| Sex | Male | Female | 0.7170 |
| Age | 41-50 | 21-30 | 0.6804 |
| Accent | central_southern | northern | 0.1692 |
| Group | F1-score | Support |
| Male | 0.9863 | 296 |
| Female | 0.9825 | 204 |
| 21–30 | 0.9956 | 115 |
| 31–40 | 0.9828 | 118 |
| 41–50 | 0.9827 | 265 |
| southern_spain | 0.9818 | 84 |
| Classifier | Sex RRPD (%) | Age RRPD (%) | Accent RRPD (%) |
|---|---|---|---|
| w2v-BERT2-kws | 79.64 | 85.35 | 41.05 |
| Experimental Setting | Sex RRPD (%) | Age RRPD (%) | Accent RRPD (%) |
|---|---|---|---|
| DIR | 67.35 | 0.00 | -20.13 |
| FreqMixStyle | -21.42 | 34.12 | 40.48 |
| FilterAugment | 88.26 | 30.14 | -40.19 |
| FreqMasking | 39.94 | 83.65 | 40.48 |
| KD | 67.35 | 15.10 | -20.13 |
| KD + FreqMasking | 21.24 | 15.10 | -40.19 |
| Group | F1-score | Support |
| Male | 0.9828 | 296 |
| Female | 0.9851 | 204 |
| 21–30 | 0.9880 | 115 |
| 31–40 | 0.9828 | 118 |
| 41–50 | 0.9847 | 265 |
| southern_spain | 0.9818 | 84 |
Demographic Bias in ASR
Demographic bias in automatic speech recognition (ASR) concerns unequal transcription accuracy and user burdens across speaker groups and language varieties.
See all 30 papers