The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

Shree Harsha Bokkahalli Satish; Christoph Minixhofer; Maria Teleki; James Caverlee; Ondřej Klejch; Peter Bell; Gustav Eje Henter; Éva Székely

eess.ASMar 15, 2026

The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

Authors: Shree Harsha Bokkahalli Satish, Christoph Minixhofer, Maria Teleki, James Caverlee, Ondřej Klejch, Peter Bell, Gustav Eje Henter, Éva Székely

Organizations: Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden · Centre for Speech Technology Research, University of Edinburgh, UK · Texas A&M University, USA

Abstract

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect recurring directional disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. Responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, showing stronger accent-level contrasts.

Figures & tables

Model	CN	EE	IN	LA	US	GB	All
LFM2-Audio	32.2	57.4	116.2	43.1	33.2	15.1	50.1
OmniVinci	8.3	5.8	8.4	7.3	8.0	9.1	7.8
Qwen3-Omni	7.1	5.8	8.0	7.3	6.2	8.4	7.1
Whisper (small)	11.5	10.8	11.0	10.8	10.8	10.8	10.9

Table 1: SpeechLLM and Whisper ASR transcription WER (%) by accent. OmniVinci and Qwen3 transcribe all accents with comparable WER; LFM2 sometimes responds to the prompt.

Figure 1: Per-model accent and gender identification accuracy; dashed lines = chance level.

Figure 2: Mean helpfulness scores (1–5 Likert scale). Left: by accent and model. Right: by accent and gender (averaged across models $\pm$ Standard Error). Eastern European $\times$ Female is the most disadvantaged subgroup.

Figure 3: Left: Proportion of judge LLM BWS selections by accent across four evaluation dimensions. Right: Pairwise helpfulness win rates (%) aggregated across all models. No win rate is over 50% because of ties.

Intersectional Bias in Language Models

See all 9 papers

From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction
Mar 19, 2026Shree Harsha Bokkahalli Satish, Maria Teleki, Christoph Minixhofer +3Intersectional Bias in Language Models Language Model Bias Evaluation
KTH Royal Institute of Technology Stockholm, Sweden · Texas A&M University College Station, USA · University of Edinburgh Edinburgh, UK
HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models
Sep 28, 2026Shen Yan, Duc Le, Irina-Elena VelicheAudio-Language Model Evaluation Audio-Language Models
Meta Superintelligence Labs
Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition
Apr 23, 2026Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers +1ASR Evaluation Demographic Bias in ASR
The Ohio State University, Columbus, OH, USA · Air Force Research Laboratory, USA

Mar 19, 2026J/K move · Enter open · S save

From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction

Shree Harsha Bokkahalli Satish, Maria Teleki, Christoph Minixhofer, Ondrej Klejch+2

KTH Royal Institute of Technology Stockholm, Sweden · Texas A&M University College Station, USA · University of Edinburgh Edinburgh, UK

Intersectional Bias in Language Models Language Model Bias Evaluation Speech Language Models Spoken Dialogue Systems

Open paper AskPDF arXiv

SpeechLLMs process spoken language directly from audio, but accent and vocal identity cues can lead to biased behaviour. Current bias evaluations often miss how such bias manifests in end-to-end speech interactions and how users experience it. We distinguish quality-of-service disparities (e.g., off-topic or low-effort responses) from content-level bias in coherent outputs, and examine intersectional effects of accent and perceived gender. In this work, we explore a two-part evaluation approach: (1) a controlled test cohort spanning six accents and two gender presentations, analysed with judge-free prompt-response metrics, and (2) an interactive study design using voice conversion to let users experience identical content through different vocal identities. Across two studies (Interactive, N=24; Observational, N=19), we find that voice conversion increases trust and acceptability for benign responses and encourages perspective-taking, while automated analysis in search of quality-of-service disparities, reveals {accent x gender} disparities in alignment and verbosity across SpeechLLMs. These results highlight voice conversion for probing and experiencing intersectional voice bias while our evaluation suite provides richer bias evaluations for spoken conversational AI.

Abstract

Intersectional Bias in Language Models

Explore similar work

From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction

HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models

Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition