Bias and Fairness in Self-Supervised Acoustic Representations for Cognitive Impairment Detection

Kashaf Gulzar; Korbinian Riedhammer; Elmar Nöth; Andreas K. Maier; Paula Andrea Pérez-Toro

eess.ASMar 3, 2026

Bias and Fairness in Self-Supervised Acoustic Representations for Cognitive Impairment Detection

Authors: Kashaf Gulzar, Korbinian Riedhammer, Elmar Nöth, Andreas K. Maier, Paula Andrea Pérez-Toro

Organizations: Zentrum für Künstliche Intelligenz (KIZ), Technische Hochschule Nürnberg, Nürnberg, Germany · Pattern Recognition Lab, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany

Abstract

Speech-based detection of cognitive impairment (CI) offers a promising non-invasive approach for early diagnosis, yet performance disparities across demographic and clinical subgroups remain underexplored, raising concerns around fairness and generalizability. This study presents a systematic bias analysis of acoustic-based CI and depression classification using the DementiaBank Pitt Corpus. We compare traditional acoustic features (MFCCs, eGeMAPS) with contextualized speech embeddings from Wav2Vec 2.0 (W2V2), and evaluate classification performance across gender, age, and depression-status subgroups. For CI detection, higher-layer W2V2 embeddings outperform baseline features (UAR up to 80.6%), but exhibit performance disparities; specifically, females and younger participants demonstrate lower discriminative power ( $AUC$ : 0.769 and 0.746, respectively) and substantial specificity disparities ( $Δ_{spec}$ up to 18% and 15%, respectively), leading to a higher risk of misclassifications than their counterparts. These disparities reflect representational biases, defined as systematic differences in model performance across demographic or clinical subgroups. Depression detection within CI subjects yields lower overall performance, with mild improvements from low and mid-level W2V2 layers. Cross-task generalization between CI and depression classification is limited, indicating that each task depends on distinct representations. These findings emphasize the need for fairness-aware model evaluation and subgroup-specific analysis in clinical speech applications, particularly in light of demographic and clinical heterogeneity in real-world applications.

Figures & tables

	Imbalanced		CI-balanced		CI-gender balanced
	CI [F/M]	NCI [F/M]	CI [F/M]	NCI [F/M]	CI [F/M]	NCI [F/M]
Gender	97/42	51/39	63/27	51/39	39/39	39/39
Age grp. 1	22/13	29/25	19/10	29/25	14/12	21/25
Age grp. 2	75/29	22/14	44/17	22/14	25/27	18/14
CI-low	0/0	51/39	0/0	51/39	0/0	39/39
CI-mild	75/28	0/0	56/18	0/0	34/25	0/0

TABLE I: Demographic and clinical information of subjects for Imbalanced, CI-balanced, and CI-gender balanced datasets. CI: CI patients, NCI: NCI subjects, Age grp: Age group, F: Female, M: Male. MMSE and HAM-D scores are expressed as mean (standard deviation).

Fig. 1: Subject distribution based on CI and gender.

Fig. 2: Workflow diagram illustrating the key steps in the data analysis.

Dataset	Feature	Clf	Accuracy	UAR	Sensitivity	Specificity
IMB	MFCCs	MLP	66.38 (2.81)	63.39 (3.36)	77.14 (4.42)	49.63 (7.98)
CI-B		MLP	63.33 (4.74)	63.33 (4.74)	65.18 (5.54)	61.48 (6.02)
CIG-B		MLP	66.81 (3.95)	66.75 (4.02)	64.35 (10.79)	69.17 (7.73)
IMB	eGeMAPS	SVM	59.42 (2.25)	49.73 (0.96)	94.29 (7.16)	5.19 (5.54)
CI-B		SVM	50.74 (5.31)	50.74 (5.31)	57.04 (6.87)	44.45 (13.25)
CIG-B		SVM	46.81 (6.31)	47.25 (6.31)	68.69 (19.90)	25.83 (19.61)

TABLE II: Top classification results for CI vs. NCI across acoustic feature types, dataset configurations, and classifiers. Results are reported as accuracy, UAR, sensitivity, and specificity, averaged over 5 fixed random data splits, and expressed as mean (standard deviation)%. MFCCs and eGeMAPS are used as baseline features. The highest UAR for each dataset configuration is highlighted. Asterisks (*) indicate comparisons where paired t-tests yielded p

<

0.05 . Clf: classifier, IMB: imbalanced, CI-B: cognitive impairment balanced, CIG-B: cognitive impairment and gender balanced, Train-Bal/Test-Rem: train on balanced and test on remaining data, W2V-HL: Wav2Vec 2.0 hidden layer.

Fig. 3: Comparison of classifier performance across W2V2 embedding layers in CI vs. NCI and D-CI vs. ND-CI classifications for imbalanced and CI-balanced datasets. Best results are highlighted for each classifier.

Dataset	Feature	Clf	Accuracy	UAR	Sensitivity	Specificity
IMB	MFCCs	RF	58.57 (5.13)	51.78 (6.51)	28.00 (12.22)	75.56 (3.78)
CI-B		SVM	62.22 (1.48)	50.65 (1.29)	6.00 (12.00)	95.29 (9.41)
CIG-B		MLP	58.33 (6.97)	52.89 (6.76)	31.11 (8.31)	74.67 (8.84)
IMB	eGeMAPS	MLP	55.24 (4.10)	51.26 (3.74)	37.33 (11.62)	65.19 (9.83)
CI-B		SVM	61.48 (3.78)	53.76 (5.15)	24.00 (20.59)	83.53 (13.62)
CIG-B		MLP	61.67 (9.65)	59.11 (11.36)	48.89 (19.37)	69.33 (6.79)

TABLE III: Top classification results for D-CI vs. ND-CI across acoustic feature types, dataset configurations, and classifiers. Results are reported in terms of accuracy, UAR, sensitivity, and specificity, averaged over 5 fixed random data splits and expressed as mean (standard deviation)%. MFCCs and eGeMAPS are used as baseline features. The highest UAR for each dataset configuration is highlighted. Clf: classifier, IMB: imbalanced, CI-B: cognitive impairment balanced, CIG-B: cognitive impairment and gender balanced, Train-Bal/Test-Rem: train on balanced and test on remaining data, W2V-HL: Wav2Vec 2.0 hidden layer.

Dataset	Age Groups								Gender								Clinical Status
	Group 1			Group 2			Age $\Delta$		Male			Female			Gender $\Delta$		N-Depr.			Depr.			Clin. $\Delta$
	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$
IMB	80	76	4	71	83	-12	9	-7	86	76	10 ∗	68	83	-15 ∗	18 ∗	-7	73	81	-8	95	81	14 ∗	-22 ∗	0
CIB	84	78	6	69	75	-6	15 ∗	3	85	69	16 ∗	72	80	-8 ∗	13 ∗	-11 ∗	74	77	-3	100	74	26 ∗	-26 ∗	3
CIG-B	79	74	5	80	74	6	-1	0	86	74	12 ∗	73	74	-1	13 ∗	0	76	77	-1	100	70	30 ∗	-24 ∗	7

TABLE IV: Subgroup performance and disparity metrics for CI vs. NCI classification using Wav2Vec 2.0 hidden layer 9 with an SVM classifier.

\delta

denotes Intra-group performance imbalance (Sp - Se); positive values reflect bias toward NCI (higher specificity), while negative values indicate bias toward CI (higher sensitivity).

\Delta

quantifies Inter-group performance disparity.Asterisks (*) indicate comparisons where paired t-tests yielded p

<

0.05 . All values are reported as percentages, with biased results highlighted. Se: Subgroup-specific sensitivity, Sp:Subgroup-specific specificity, IMB: imbalanced, CI-B: cognitive impairment balanced, CIG-B: cognitive impairment and gender balanced, N-Depr.: non-depressed, Depr.: depressed.

Fig. 4: SVM decision score distributions across demographic and clinical subgroups for Wav2Vec 2.0 hidden layer 9 using CI-gender balanced dataset. Histograms and Kernel Density Estimate (KDE) curves illustrate the separation between NCI and CI classes. The dashed line at 0 represents the SVM decision threshold. AUC values are provided for each subgroup to indicate classification performance.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Feature	Clf	Accuracy	UAR	Sensitivity	Specificity
Train-CI Test-D	W2V HL 7	SVM	48.70 (7.48)	52.37 (6.13)	69.56 (6.81)	35.18 (9.46)
	W2V HL 9	RF	45.22 (6.51)	49.31 (6.03)	67.69 (14.10)	30.93 (16.01)
	W2V HL 10	RF	46.09 (6.51)	49.64 (4.75)	67.46 (9.79)	31.81 (13.93)
Train-D Test-CI	W2V HL 2	MLP	52.34 (5.65)	51.76 (5.61)	38.00 (9.54)	65.53 (10.84)
	W2V HL 6	MLP	57.45 (2.33)	55.86 (3.09)	29.41 (5.32)	82.30 (3.75)

Appendix

TABLE V: Classification performance for cross-condition evaluations on the CI and gender-balanced dataset. Models trained on CI status were tested on depression status and vice versa. Results are reported in terms of accuracy, UAR, sensitivity, and specificity, averaged over 5 fixed random data splits and expressed as mean (standard deviation)%. Clf: classifier, Train-CI Test-D: train on CI and test for depression status, Train-D Test-CI: train on depression and test for CI status, W2V-HL: Wav2Vec 2.0 hidden layer.

Dataset	Age Groups								Gender								Clinical Status
	Group 1			Group 2			Age $\Delta$		Male			Female			Gender $\Delta$		N-Depr.			Depr.			Clin. $\Delta$
	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$	Sp	Se	$\delta$	Sp	Se	$\delta$	$\Delta_{Sp}$	$\Delta_{Se}$
IMB	79	78	1	73	79	-6	6	-1	75	76	-1	78	80	-2	-3	-4	73	77	-4	95	81	14 ∗	-22 ∗	-4
CIB	80	78	2	75	70	5	5	8	78	69	9 ∗	78	75	3	0	-6	74	75	-1	100	67	33 ∗	-26 ∗	8
CIG-B	72	74	-2	82	78	4	-10	-4	82	74	8 ∗	70	79	-9 ∗	12 ∗	-5	74	77	-3	88	76	12 ∗	-14 ∗	1

Appendix

TABLE VI: Subgroup performance and disparity metrics for CI vs. NCI classification using Wav2Vec 2.0 hidden layer 10 with an MLP classifier.

\delta

denotes Intra-group performance imbalance (Sp - Se); positive values reflect bias toward NCI (higher specificity), while negative values indicate bias toward CI (higher sensitivity).

\Delta

quantifies Inter-group performance disparity.Asterisks (*) indicate comparisons where paired t-tests yielded p

<

Explore similar work

Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy
May 26, 2026Serli Kopar, Roshan Prakash Rane, Christian Mychajliw +6Speech Processing Audio Understanding
Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Tübingen AI Center, University of Tübingen, Tübingen, Germany · Department of Psychology, Humboldt-Universität zu Berlin +8
Segment-Level Mandarin Chinese Speech-Based Cognitive Impairment Detection via an Autoencoder with Contrastive Learning
Jun 18, 2026Yongqi Shao, Hong Huo, Flavio Bertini +2Audio Representation Learning Representation Learning
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; Key Laboratory of System Control and Information Processing, Ministry of Education of China; Shanghai Key Laboratory of Perception and Control in Industrial Network Systems, Shanghai, 200240, China · Department of Mathematical, Physical and Computer Sciences, University of Parma, Parco Area delle Sciences 53/A, Parma, 43124, Italy · Department of Computer Science and Engineering, University of Bologna, Mura Anteo Zamboni 7, Bologna, 40126, Italy
Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models
Jul 23, 2026Yingchao Huang, Xin Wang, Yuhan Su +1Multimodal Large Language Models Speech-Based Dementia Detection
Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada

May 26, 2026J/K move · Enter open · S save

Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy

Serli Kopar, Roshan Prakash Rane, Christian Mychajliw, Lydia Federmann+5

Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Tübingen AI Center, University of Tübingen, Tübingen, Germany · Department of Psychology, Humboldt-Universität zu Berlin+8

Speech Processing Audio Understanding Self-Supervised Speech Representation Learning Clinical Speech Processing

Open paper AskPDF arXiv

This study examines the relationship between speech representations and the hierarchical structure of cognitive assessment in mild cognitive impairment. Utilizing 5,754 German neuropsychological assessment recordings, we evaluate six cognitive tasks across three score levels: task, domain, and global levels. We compare hand-crafted acoustic features with self-supervised learning (SSL) embeddings. Results show that although SSL representations generally outperform hand-crafted features at lower levels, this trend reverses for MCI classification. Furthermore, task-specific constraints influence performance: tasks with greater response freedom exhibit performance dilution as hierarchical levels increase, suggesting

specialist'' representations, whereas the performance of highly structured tasks increases toward higher levels, suggesting

generalist'' representations. These findings show links between task constraints and assessment hierarchy in automated clinical speech analysis.

Abstract

Explore similar work

Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy

Segment-Level Mandarin Chinese Speech-Based Cognitive Impairment Detection via an Autoencoder with Contrastive Learning

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models