Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation

Srishti Ginjala; Eric Fosler-Lussier; Srinivasan Parthasarathy

cs.CLSep 30, 2026

Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation

Authors: Srishti Ginjala, Eric Fosler-Lussier, Srinivasan Parthasarathy

Organizations: Department of Computer Science and Engineering The Ohio State University, Columbus, OH, USA

Abstract

Demographic fairness gaps in automatic speech recognition are almost always reported from a single training run. We fine-tune the Q-former projector and LoRA adapters of a speech LLM at five audio compression factors and six random seeds, holding the encoder, base decoder, data and decoding fixed, and evaluate every run on Common Voice and Fair-Speech. At 460 h of clean LibriSpeech, the seed moves fairness metrics more than compression does on most demographic axes. A balanced 3x3 decomposition attributes 85.3% of the variation in Fair-Speech ethnicity normalized gap to the seed against 8.3% to compression (p = 0.009), though compression explains more on age and gender. Held-out LibriSpeech word error rate spreads by 0.04 points across those seeds while Common Voice spreads by 8.57, so these are not failed runs, and the effect survives controlling for accuracy and dropout. Scaling and diversifying the adaptation set to 960 h damps the effect but does not remove it. On Fair-Speech ethnicity, two single-run systems must differ by more than 0.30 in normalized gap to exceed seed variability.

Figures & tables

Figure 1: Training variance against evaluation sampling variance. Per-group word error rate on Fair-Speech ethnicity at factor 5, 460 h. Points are the six training seeds (training variance); bars are the 95% bootstrap interval over utterances for one run (evaluation sampling variance). On the worst-served group the seeds span 4.68 points against a 1.76 point interval, so training variance is

2.7\times

wider than the uncertainty normally reported.

Figure 2: Absolute change in MMR from the widest compression pair (factor 3 versus 8) and from the widest of six seeds, with 95% bootstrap intervals over utterances, Benjamini-Hochberg corrected within each family of seven axes. The seed row is an effect-size display and not a test, for the reason given in Section 3 . Compression survives on two secondary axes, in opposite directions.

axis	metric	% fac.	% seed	$p$	12-run
FS ethnicity	NormGap	8.3	85.3	0.009	83.6
CV accent (exp.)	NormGap	1.1	91.0	0.064	91.6
FS ethnicity	MMR	40.6	53.2	0.073	68.9
FS L1 group	MMR	15.4	59.1	0.142	63.0
FS socioecon.	MMR	28.0	46.3	0.095	61.3
FS gender	NormGap	87.5	9.4	0.048	52.5

Table 1: Balanced

3\times 3

factorial at 460 h: factors

\{3,5,8\}

crossed with seeds

\{1234,2222,3333\}

, one run per cell, additive two-way decomposition. One observation per cell makes the residual the factor

\times

seed interaction rather than a separate error term, so no model carrying both is identifiable. The

p

value comes from permuting seed labels independently within each factor column 10,000 times, breaking the shared seed identity across columns and holding the factor margin fixed. The last column is the descriptive seed share from the unbalanced 12-run grid.

axis	metric	ratio	$\|r\|$	ret.	quadrant
FS ethnicity	NormGap	2.82 $\times$	0.31	0.81	carries
FS socioecon.	NormGap	1.43 $\times$	0.33	1.45	carries
CV accent	MMR	7.26 $\times$	0.94	0.04	confound.

Table 2: Effect size against accuracy dependence at 460 h. “ret.” is seed variance after residualizing on overall word error rate over seed variance before; the regression is fitted over all runs while the variance is taken over the six factor-5 seeds, so the ratio can exceed 1. Thresholds (

1.0\times

|r|<0.5

) were fixed before inspection; “carries” marks cells clearing both.

axis	metric	$\sigma$	$d{=}0.5$	$d{=}1.0$	min. diff.
FS ethnicity	NormGap	0.108	1	1	0.30
FS ethnicity	MMR	0.278	5	2	0.77
CV accent	MMR	1.219	94	24	3.38
FS gender	MMR	0.193	3	1	0.53

Table 3: Seeds needed per configuration for 80% power at

\alpha=0.05

, and the smallest single-run difference exceeding the observed seed spread (

2.77\sigma

), from the six seeds at factor 5.

Demographic Bias in ASR

Demographic bias in automatic speech recognition (ASR) concerns unequal transcription accuracy and user burdens across speaker groups and language varieties.

See all 30 papers

Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs
Sep 29, 2026Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi ShekharLLM Pruning Algorithmic Fairness
School of Computer Science and Electronic Engineering, University of Essex, UK
Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition
Apr 23, 2026Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers +1ASR Evaluation Demographic Bias in ASR
The Ohio State University, Columbus, OH, USA · Air Force Research Laboratory, USA
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Apr 7, 2026Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9Domain Adaptation for ASR Cross-Modal Alignment
Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland +2

Sep 29, 2026J/K move · Enter open · S save

Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs

Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi Shekhar

School of Computer Science and Electronic Engineering, University of Essex, UK

LLM Pruning Algorithmic Fairness Demographic Bias in ASR Speech Language Models Demographic Bias in Language Models

Open paper AskPDF arXiv

Speech-LLMs are expensive to run, making compression important for real-world deployment. However, compressed models are usually selected using aggregate word error rate (WER), which can hide how pruning affects different demographic groups. In this work, we systematically study the effect of audio encoder pruning on SLAM-ASR for different demographic groups. Using the Fair-Speech and Common Voice datasets, we found that the pruning does not affect all demographic groups equally; the gap between best- and worst-performing groups increases in fold. These disparities appear across all three encoder scales, but only the largest model initially hides them behind aggregate WER. LoRA adaptation improves WER for every group, but benefits groups already performing well more strongly and widens for certain groups. On Common Voice English, Danish, and Dutch, accent gaps persist but do not clearly widen, showing that the fairness effects of pruning vary across datasets and must be measured directly. Our findings suggest that for pruned models, deployment decisions should include per-group WER, with the worst-performing group's error rate as an explicit criterion.

Abstract

Demographic Bias in ASR

Explore similar work

Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs

Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR