Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics

Yangyang Qu; Massimiliano Todisco; Chiara Galdi; Nicholas Evans

cs.SDMar 11, 2026

Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics

Authors: Yangyang Qu, Massimiliano Todisco, Chiara Galdi, Nicholas Evans

Organizations: EURECOM Sophia Antipolis, France

Abstract

Voice biometric systems can exhibit sex-related performance gaps even when overall verification accuracy is strong. We attribute these gaps to two practical mechanisms: (i) demographic shortcut learning, where speaker classification training exploits spurious correlations between sex and speaker identity, and (ii) feature entanglement, where sex-linked acoustic variation overlaps with identity cues and cannot be removed without degrading speaker discrimination. We propose Fair-Gate, a fairness-aware and interpretable risk-gating framework that addresses both mechanisms in a single pipeline. Fair-Gate applies risk extrapolation to reduce variation in speaker-classification risk across proxy sex groups, and introduces a local complementary gate that routes intermediate features into an identity branch and a sex branch. The gate provides interpretability by producing an explicit routing mask that can be inspected to understand which features are allocated to identity versus sex-related pathways. Experiments on VoxCeleb1 show that Fair-Gate improves the utility--fairness trade-off, yielding more sex-fair ASV performance under challenging evaluation conditions.

Figures & tables

Fig. 1: Desired decision vs. demographic shortcut in speaker verification under a shared threshold. A verifier should base its decision on identity evidence by comparing enrollment and test utterances (solid arrow). However, because sex affects acoustics (e.g., $F_{0}$ and formant structure) and can be spuriously correlated with speaker identity in the training data, the model may also exploit sex-linked cues as a shortcut (dashed arrow). Such shortcut reliance can shift score distributions differently for male and female speakers, leading to subgroup error-rate gaps when deploying a single global decision threshold.

Fig. 2: Overview of Fair-Gate. The encoder produces frame-level features $\mathbf{U}$ , which are complementarily soft-routed by a local mask $A$ (gate) into an identity branch and a sex branch. The identity branch produces the embedding $z_{\mathrm{id}}$ , which is the only representation used for automatic speaker verification (ASV) at inference. During training, the sex branch learns a sex embedding $z_{\mathrm{sex}}$ and predicts proxy sex labels $\hat{s}$ via a sex classifier ( $\mathcal{L}_{\mathrm{sex}}$ ). The identity branch is optimized for speaker classification ( $\mathcal{L}_{\mathrm{spk}}$ ), regularized by Risk Extrapolation (REx) across proxy sex groups ( $\mathcal{L}_{\mathrm{rex}}$ ), and constrained by an adversarial sex classifier implemented through a Gradient Reversal Layer (GRL) ( $\mathcal{L}_{\mathrm{adv}}$ ). A decorrelation loss ( $\mathcal{L}_{\mathrm{decor}}$ ) encourages separation between $z_{\mathrm{id}}$ and $z_{\mathrm{sex}}$ , while gate regularizers ( $\mathcal{L}_{\mathrm{cap}}$ , $\mathcal{L}_{\mathrm{sat}}$ ) prevent degenerate routing.

Protocol	Speakers	# Trials	Description
Vox1-O	40	37,720	Original test list
Vox1-E	1,251	581,480	Expanded trial list
Vox1-H	1,251	552,536	Hard (same nationality & sex impostors)

TABLE I: VoxCeleb1 verification protocols used for evaluation.

Model	Vox1-O			Vox1-E			Vox1-H
Model	EER $\downarrow$	minDCF $\downarrow$	GARBE $\downarrow$	EER $\downarrow$	minDCF $\downarrow$	GARBE $\downarrow$	EER $\downarrow$	minDCF $\downarrow$	GARBE $\downarrow$
ECAPA-TDNN [ 23 ]	1.12	0.14	0.16	1.34	0.17	0.11	2.62	0.25	0.10
ECAPA-TDNN + GRL	0.98	0.13	0.22	1.25	0.14	0.12	2.57	0.28	0.10
VoxDisentangler [ 24 ]	0.82	0.12	0.17	1.15	0.14	0.11	2.40	0.26	0.10
Fair-Gate (ours)	0.92	0.11	0.26	1.11	0.14	0.05	2.25	0.26	0.07

TABLE II: Results on VoxCeleb1 (O/E/H). We report EER [%], minDCF, and sex fairness measured by GARBE at the threshold

\tau_{1\%}

Setting	Cap	Sat	Gs	Adv	REx	EER $\downarrow$	GARBE $\downarrow$	FNMR M	FNMR F	FMR M	FMR F
Main w/o Cap	–	✓	✓	✓	✓	2.66	0.09	1.03	0.95	4.76	6.30
Main w/o Sat	✓	–	✓	✓	✓	2.30	0.07	0.95	1.07	3.98	4.68
Main w/o Gs	✓	✓	–	✓	✓	2.66	0.09	1.03	0.95	4.76	6.30
Main w/o Adv	✓	✓	✓	–	✓	2.27	0.07	0.96	1.06	3.81	4.48
Main w/o REx	✓	✓	✓	✓	–	2.55	0.08	0.96	1.06	4.54	5.60
Main, Full	✓	✓	✓	✓	✓	2.25	0.07	0.96	1.07	3.80	4.49

TABLE III: Ablation results on Vox1-H. We report EER [%],

\mathrm{GARBE}(\tau_{1\%})

, and subgroup-specific FNMR/FMR [%] at the same shared threshold

\tau_{1\%}

. Bold indicates the best value in each column.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models
Sep 23, 2026Kian Shamsaie, Iman ModarressiDisentangled Representation Learning Audio Representation Learning
People Make Things
Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias
May 9, 2026Aishwarya Fursule, Shruti Kshirsagar, Anderson R. AvilaAudio Deepfake Detection Algorithmic Fairness
School of Computing, Wichita State University, Wichita, KS, USA · Institut national de la recherche scientifique (INRS-EMT), Montreal, QC, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada
Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis
Mar 9, 2026Aishwarya Fursule, Shruti Kshirsagar, Anderson R. AvilaAudio Deepfake Detection Algorithmic Fairness
School of Computing, Wichita State University, Wichita, KS, USA · Institut national de la recherche scientifique (INRS–EMT), Montreal, QC, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada

Sep 23, 2026J/K move · Enter open · S save

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models

Kian Shamsaie, Iman Modarressi

People Make Things

Disentangled Representation Learning Audio Representation Learning Algorithmic Fairness Audio Understanding Demographic Bias in ASR

Open paper AskPDF arXiv

Speech technology penalizes some voices: recognition errs nearly twice as often for Black speakers, and accuracy declines for second-language accents and older speakers. We introduce TRIAD, an audit grid crossing 120 texts, 24 rendered demographic voice profiles (gender, age band, accent), and ten expressive styles via controllable text-to-speech, isolating perceived demographic attributes from content and affect. For ten open-weights encoders we define axis-fidelity functionals, principal-angle leakage between axis subspaces, and group-conditional gaps; a proposition proves that average probe disparity grows with the same aggregate voice-semantic leakage

Λ

we measure, and a corollary shows that peak leakage forces worst-case disparity inside an active region. The measured mean-square probe disparity tracks

Λ

(Pearson r = 0.93), and a black-box protocol exposes the same signature in two closed-source models. ORCA, an adapter combining axis-specific contrastive heads, an orthogonality penalty, and group-balanced sampling, cuts leakage 72% and roughly halves the gaps.

Abstract

Speaker Verification

Explore similar work

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models

Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias

Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis