Improved automatic speech-based cognitive assessment using neuropsychologically calibrated diagnostic labels

Madhurananda Pahar; Caitlin Illingworth; Adam Hampshire; Peter J Hellyer; Agata J Hampshire; Daniel Blackburn; Annalena Venneri; Heidi Christensen

cs.SDOct 20, 2026

Improved automatic speech-based cognitive assessment using neuropsychologically calibrated diagnostic labels

Authors: Madhurananda Pahar, Caitlin Illingworth, Adam Hampshire, Peter J Hellyer, Agata J Hampshire, Daniel Blackburn, Annalena Venneri, Heidi Christensen

Organizations: School of Computer Science, The University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), The University of Sheffield, Sheffield, S10 2HQ, UK · Centre for Neuroimaging Sciences, IoPPN, King’s College London, London, UK · Wolfson Sensory, Pain and Regeneration Centre (SPaRC ), King’s College London, UK

Abstract

Although automatic detection of cognitive impairment from conversational speech is a promising way, it becomes less accurate, as large real-world datasets often contain unreliable diagnostic labels due to self-referral and limited clinical verification, and here we investigated whether neuropsychologically calibrated diagnostic relabeling improves it. We analysed 138 hours of conversational speech from 720 participants, recalibrating unreliable diagnostic labels using attention, executive function, and memory performance via external clinical thresholds, Gaussian, and normative z-score calibration. Transformer-based classifiers were trained using both original and recalibrated labels and evaluated internally on 150 clinically verified participants and externally on Pitt, ADReSS and PROCESS-2 datasets. Memory- and MoCA-based z-score calibration produced the greatest improvements. Using our internal dataset, $F_1$ increased from $0.70$ to $0.83$ ( $+13\%$ ) and from $0.54$ to $0.66$ ( $+12\%$ ) for binary and multiclass classification, respectively. For external validation, the increases are $+4\%$ , $+5\%$ and $+9\%$ on Pitt, ADReSS and PROCESS-2 corpus. Neuropsychologically calibrated relabeling substantially improves speech-based cognitive assessment, highlighting label quality, rather than model architecture, as a critical bottleneck in clinical AI for dementia screening.

Figures & tables

Figure 1: Overview of the proposed improved neuropsychological performance-based diagnostic relabelling and speech-based classification framework. Participants with memory concerns were recruited across the United Kingdom (UK) from both clinically confirmed and non-clinical community sources and completed remote cognitive assessments using the CognoMemory platform. (A) Neuropsychological label reassignment. Instead of directly using inherited diagnostic labels, neuropsychological performances across attention, executive function and memory domains were analysed to generate refined diagnostic labels. Three complementary threshold-based sorting strategies were applied: external FLAME [ 14 ] thresholds, data-driven Gaussian calibration (Cogno_G), and normative z-score deviation thresholds (Cogno_Z). Two selection strategies were evaluated: ‘with MoCA’ integrates clinically validated diagnoses and MoCA-derived priors when available, whereas ‘No MoCA’ derives labels solely from neuropsychological performance, enabling large-scale weakly supervised dataset curation. Threshold values are noted as $T_{1}$ and $T_{2}$ , which are used to relabel participants as dementia, mild cognitive impairment (MCI), and healthy control (HC) groups. (B) Speech-based model training and evaluation. Texts from speech recordings collected during cognitive testing were generated using three ASRs, such as Whisper, Wav2Vec 2.0 and NeMo. Classification was performed using those texts by three transformer-based models, like BART, RoBERTa and DistilBERT. Models were trained using neuropsychologically calibrated diagnostic labels and evaluated against an independent test set with ground-truth clinical diagnoses, demonstrating improved performance compared to the original label usage.

Evaluation	Group	Diagnosis	N	Src(C)	Src(NC)	Age	M	F	Audio (hr)	Whisper	Wav2Vec2	NeMo	MoCA
Train	Case	Dementia	25	6	19	71.76 $\pm$ 8.93	18	7	5.76	31.53	34.19	32.99	22
	Case	MCI	90	21	69	69.16 $\pm$ 7.61	45	44	19.52	121.33	126.53	120.46	69
	Control	HC	465	9	456	65.95 $\pm$ 8.28	172	286	85.6	564.89	593.68	571.92	8
	OTHER	Unconfirmed	140	0	140	66.15 $\pm$ 8.43	51	86	27.13	182.46	192.17	182.68	10
SUBTOTAL	—	—	720	36	684	66.58 $\pm$ 8.38	286	423	137.99	900.21	946.56	908.05	109
Test	Case	Dementia	23	23	0	78 $\pm$ 7.4	11	12	4.15	20.89	21.86	19.50	21

Table 1: Dataset description. A total of 720 participants who completed both CognoMemory and the neuropsychological assessments were used in training. Another 150 participants who participated in only CognoMemory with ground truth diagnostic labels were used for testing. The gender distribution excludes 11 undisclosed participants. The number of words generated by Whisper, Wav2Vec 2.0, and NeMo is listed as 1,000. Clinical and non-clinical are noted as Src(C) and Src(NC) .

Panel of Figure 2 — Figure 2: Domain-level neuropsychological performance analysis and dataset comparison. (a) Mean neuropsychological performance comparison between CognoMemory and the FLAME dataset. Bars indicate mean $\pm$ standard deviation. Red asterisks mark cases where between-source mean differences exceed the combined within-group variability, indicating pronounced distributional separation. (b) Gaussian-fitted histograms illustrating score distributions for dementia, MCI, and healthy control (HC) participants across attention, executive function, and memory domains. Solid curves represent fitted normal distributions, while dashed lines denote 95% confidence interval (CI) boundaries estimated as $\mu\pm 1.96\sigma$ . (c) Raincloud plots comparing neuropsychologically derived scores originating from clinically verified and non-clinical data sources. Statistical significance between sources within each diagnostic group was assessed using independent two-sample t-tests, with ns (not significant), $*$ ( $p<0.1$ ), $**$ ( $p<0.05$ ), and $***$ ( $p<0.01$ ).

Figure 3: Diagnostic thresholds by three sorting strategies: Clinically anchored diagnostic thresholds derived using FLAME clinical references, data-calibrated Gaussian midpoints, and healthy-control z-score deviations across memory, attention, and executive function domains. The figure illustrates how small amounts of trusted clinical information may inform decision boundaries.

Figure 4: Distribution of diagnostic labels generated using neuropsychologically calibrated reassignment strategies. The figure compares class distributions obtained from the original dataset annotations, the internal CognoMemory test cohort, and automatically reassigned diagnostic labels produced using three thresholding strategies (FLAME clinical reference thresholds, Cogno Gaussian midpoint thresholds, and Cogno Z-score normative deviation thresholds) across memory (MEM), attention (ATT), and executive function (EXE) domains. The top row shows label distributions when Montreal Cognitive Assessment (MoCA) scores were incorporated as an auxiliary clinical override, whereas the bottom row shows label assignments derived purely from cognition, without MoCA information. Bars represent counts of participants assigned to Dementia, Mild Cognitive Impairment (MCI), Healthy Control (HC), and Unknown categories. Across strategies, neuropsychologically anchored reassignment reduces reliance on uncertain original annotations while preserving clinically plausible class proportions. The comparison highlights how different thresholding assumptions influence class balance and demonstrates the stabilising effect of incorporating clinical priors during large-scale automatic label construction.

Panel of Figure 5 — Figure 5: Summary of binary cognitive decline classification performance. (a) Heatmap of $F_{1}$ -scores across individual speech tasks comparing original clinical labels with proposed neuropsychologically informed labelling strategies. Columns represent FLAME clinical thresholds, data-driven Gaussian calibration (Cogno-G), and normative z-score calibration (Cogno-Z). Associated automatic speech recognition (ASR) and transformer-based model configurations are shown for reference. (b) Average classification performance obtained with and without incorporation of MoCA-derived clinical priors. MoCA integration produces consistent performance gains across domains, supporting the role of global cognitive screening as a stabilising supervisory signal. (c) Distribution of $F_{1}$ -scores across labelling strategies, demonstrating a systematic upward shift in performance relative to the original labels. (d) Confusion matrix obtained using original diagnostic labels, showing increased misclassification between impaired and healthy participants. (e) Confusion matrix obtained using Cogno-Z memory-based labels, illustrating improved class separation and reduced diagnostic ambiguity.

Panel of Figure 6 — Figure 6: Summary of multiclass cognitive decline classification performance. (a) Heatmap showing multiclass $F_{1}$ -scores across speech tasks and labelling strategies. Neuropsychologically calibrated labels consistently outperform original annotations, particularly for memory-intensive tasks. (b) Comparison of performance with and without MoCA-derived priors, demonstrating improved stability when separating dementia, MCI, and healthy control groups. (c) Distributional comparison of $F_{1}$ -scores illustrating global performance improvements achieved through clinically anchored thresholding. (d) Confusion matrix obtained using original labels, highlighting difficulty distinguishing intermediate cognitive states. (e) Confusion matrix obtained using Cogno-Z memory-derived labels, showing improved discrimination between dementia, MCI, and healthy controls.

Validation Type	Dataset	Size ( $N$ )	Task	Method	Mean_diff ( $\Delta F_{1}$ )	Cohens_d	Result
Internal	CognoMemory	Case: 73 Control: 77	Binary	FLAME_ATTEN	-0.032	-0.697	0.008 (**)
	(Table 1 )			FLAME_EXEC	-0.027	-0.482	0.057 (ns)
				FLAME_MEM	0.070	2.635	$<$ 0.001 (****)
				Cogno_G_ATTEN	0.035	1.029	$<$ 0.001 (***)
				Cogno_G_EXEC	0.041	1.379	$<$ 0.001 (****)
				Cogno_G_MEM	0.075	2.707	$<$ 0.001 (****)

Table 2: Performance Comparison across Internal and External Validation Cohorts (Binary vs. Multiclass Classification). The strategies and the neuropsychological domains performing significantly better are shown in bold.

Figure 7: TF-IDF analysis of semantic fluency responses using Cogno_Z_MEM diagnostic labels. The figure shows the ten most distinctive animal words produced by participants in each diagnostic group (Dementia, MCI, and Healthy Control). TF-IDF scores were normalised within each group and expressed as percentages relative to the most distinctive word (100%) in that group. Higher percentages indicate words that are disproportionately characteristic of a specific cognitive group rather than frequently used across all participants. Distinct lexical patterns emerge across diagnostic categories, suggesting systematic differences in semantic retrieval strategies associated with cognitive impairment.

Speech-Based Dementia Detection

Speech-based dementia detection uses spoken language and vocal patterns to identify cognitive impairment and support accessible screening.

See all 44 papers

PROCESS-2: A Benchmark Speech Corpus for Early Cognitive Impairment Detection
May 14, 2026Madhurananda Pahar, Caitlin H. Illingworth, Bahman Mirheidari +7Speech-Based Dementia Detection Speech Processing
School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), University of Sheffield, Sheffield, S10 2HQ, UK
Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment
Jun 17, 2026Franziska Braun, Christopher Witzl, Andreas Erzigkeit +4Speech-Based Dementia Detection Automated Evaluation
Technische Hochschule Nürnberg · Geromed GmbH · PMU Klinikum Nürnberg
Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech
Sep 28, 2026Ziyun Cui, Wen Wu, Chuan Shi +10Explainable Artificial Intelligence Speech-Based Dementia Detection
Tsinghua University, Beijing, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, China. · Peking University Sixth Hospital, Beijing, China. +3

May 14, 2026J/K move · Enter open · S save

PROCESS-2: A Benchmark Speech Corpus for Early Cognitive Impairment Detection

Madhurananda Pahar, Caitlin H. Illingworth, Bahman Mirheidari, Hend Elghazaly+6

School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), University of Sheffield, Sheffield, S10 2HQ, UK

Speech-Based Dementia Detection Speech Processing

Open paper AskPDF arXiv

Speech-based analysis offers a scalable and non-invasive approach for detecting cognitive decline, yet progress has been constrained by the limited availability of clinically validated datasets collected under realistic conditions. We introduce PROCESS-2, a large-scale speech dataset designed to support research on automatic assessment of cognitive impairment from spontaneous and task-oriented speech. The dataset comprises recordings from 200 healthy controls, 150 mild cognitive impairment, and 50 dementia diagnoses collected using the CognoMemory digital assessment platform. Each participant completed a single assessment session, including picture description and verbal fluency tasks, accompanied by manually verified transcripts and participant-level metadata. PROCESS-2 contains approximately 21 hours of speech audio with predefined train/test partitions. Comprehensive technical validation evaluated demographic balance, clinical consistency, recording stability, embedding-space structure, and reproducible baseline modelling performance, demonstrating clinically meaningful group separation and stable performance across modelling approaches while preserving real-world conversational variability. PROCESS-2 is released under controlled access via Hugging Face to enable responsible reuse while protecting participant privacy, providing a reproducible benchmark resource for speech-based cognitive assessment research.

Abstract

Speech-Based Dementia Detection

Explore similar work

PROCESS-2: A Benchmark Speech Corpus for Early Cognitive Impairment Detection

Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment

Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech