Improved automatic speech-based cognitive assessment using neuropsychologically calibrated diagnostic labels
Authors: Madhurananda Pahar, Caitlin Illingworth, Adam Hampshire, Peter J Hellyer, Agata J Hampshire, Daniel Blackburn, Annalena Venneri, Heidi Christensen
Organizations: School of Computer Science, The University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), The University of Sheffield, Sheffield, S10 2HQ, UK · Centre for Neuroimaging Sciences, IoPPN, King’s College London, London, UK · Wolfson Sensory, Pain and Regeneration Centre (SPaRC ), King’s College London, UK
Although automatic detection of cognitive impairment from conversational speech is a promising way, it becomes less accurate, as large real-world datasets often contain unreliable diagnostic labels due to self-referral and limited clinical verification, and here we investigated whether neuropsychologically calibrated diagnostic relabeling improves it. We analysed 138 hours of conversational speech from 720 participants, recalibrating unreliable diagnostic labels using attention, executive function, and memory performance via external clinical thresholds, Gaussian, and normative z-score calibration. Transformer-based classifiers were trained using both original and recalibrated labels and evaluated internally on 150 clinically verified participants and externally on Pitt, ADReSS and PROCESS-2 datasets. Memory- and MoCA-based z-score calibration produced the greatest improvements. Using our internal dataset, F1 increased from 0.70 to 0.83 (+13%) and from 0.54 to 0.66 (+12%) for binary and multiclass classification, respectively. For external validation, the increases are +4%, +5% and +9% on Pitt, ADReSS and PROCESS-2 corpus. Neuropsychologically calibrated relabeling substantially improves speech-based cognitive assessment, highlighting label quality, rather than model architecture, as a critical bottleneck in clinical AI for dementia screening.
Figures & tables
Figure 1: Overview of the proposed improved neuropsychological performance-based diagnostic relabelling and speech-based classification framework. Participants with memory concerns were recruited across the United Kingdom (UK) from both clinically confirmed and non-clinical community sources and completed remote cognitive assessments using the CognoMemory platform. (A) Neuropsychological label reassignment. Instead of directly using inherited diagnostic labels, neuropsychological performances across attention, executive function and memory domains were analysed to generate refined diagnostic labels. Three complementary threshold-based sorting strategies were applied: external FLAME [ 14 ] thresholds, data-driven Gaussian calibration (Cogno_G), and normative z-score deviation thresholds (Cogno_Z). Two selection strategies were evaluated: ‘with MoCA’ integrates clinically validated diagnoses and MoCA-derived priors when available, whereas ‘No MoCA’ derives labels solely from neuropsychological performance, enabling large-scale weakly supervised dataset curation. Threshold values are noted as T1 and T2 , which are used to relabel participants as dementia, mild cognitive impairment (MCI), and healthy control (HC) groups. (B) Speech-based model training and evaluation. Texts from speech recordings collected during cognitive testing were generated using three ASRs, such as Whisper, Wav2Vec 2.0 and NeMo. Classification was performed using those texts by three transformer-based models, like BART, RoBERTa and DistilBERT. Models were trained using neuropsychologically calibrated diagnostic labels and evaluated against an independent test set with ground-truth clinical diagnoses, demonstrating improved performance compared to the original label usage.
Evaluation
Group
Diagnosis
N
Src(C)
Src(NC)
Age
M
F
Audio (hr)
Whisper
Wav2Vec2
NeMo
MoCA
Train
Case
Dementia
25
6
19
71.76 ± 8.93
18
7
5.76
31.53
34.19
32.99
22
MCI
90
21
69
69.16 ± 7.61
45
44
19.52
121.33
126.53
120.46
69
Control
HC
465
9
456
65.95 ± 8.28
172
286
85.6
564.89
593.68
571.92
8
OTHER
Unconfirmed
140
0
140
66.15 ± 8.43
51
86
27.13
182.46
192.17
182.68
10
SUBTOTAL
—
—
720
36
684
66.58 ± 8.38
286
423
137.99
900.21
946.56
908.05
109
Test
Case
Dementia
23
23
0
78 ± 7.4
11
12
4.15
20.89
21.86
19.50
21
Table 1: Dataset description. A total of 720 participants who completed both CognoMemory and the neuropsychological assessments were used in training. Another 150 participants who participated in only CognoMemory with ground truth diagnostic labels were used for testing. The gender distribution excludes 11 undisclosed participants. The number of words generated by Whisper, Wav2Vec 2.0, and NeMo is listed as 1,000. Clinical and non-clinical are noted as Src(C) and Src(NC) .
Figure 2: Domain-level neuropsychological performance analysis and dataset comparison. (a) Mean neuropsychological performance comparison between CognoMemory and the FLAME dataset. Bars indicate mean ± standard deviation. Red asterisks mark cases where between-source mean differences exceed the combined within-group variability, indicating pronounced distributional separation. (b) Gaussian-fitted histograms illustrating score distributions for dementia, MCI, and healthy control (HC) participants across attention, executive function, and memory domains. Solid curves represent fitted normal distributions, while dashed lines denote 95% confidence interval (CI) boundaries estimated as μ±1.96σ . (c) Raincloud plots comparing neuropsychologically derived scores originating from clinically verified and non-clinical data sources. Statistical significance between sources within each diagnostic group was assessed using independent two-sample t-tests, with ns (not significant), ∗ ( p<0.1 ), ∗∗ ( p<0.05 ), and ∗∗∗ ( p<0.01 ).
Figure 3: Diagnostic thresholds by three sorting strategies: Clinically anchored diagnostic thresholds derived using FLAME clinical references, data-calibrated Gaussian midpoints, and healthy-control z-score deviations across memory, attention, and executive function domains. The figure illustrates how small amounts of trusted clinical information may inform decision boundaries.
Figure 4: Distribution of diagnostic labels generated using neuropsychologically calibrated reassignment strategies. The figure compares class distributions obtained from the original dataset annotations, the internal CognoMemory test cohort, and automatically reassigned diagnostic labels produced using three thresholding strategies (FLAME clinical reference thresholds, Cogno Gaussian midpoint thresholds, and Cogno Z-score normative deviation thresholds) across memory (MEM), attention (ATT), and executive function (EXE) domains. The top row shows label distributions when Montreal Cognitive Assessment (MoCA) scores were incorporated as an auxiliary clinical override, whereas the bottom row shows label assignments derived purely from cognition, without MoCA information. Bars represent counts of participants assigned to Dementia, Mild Cognitive Impairment (MCI), Healthy Control (HC), and Unknown categories. Across strategies, neuropsychologically anchored reassignment reduces reliance on uncertain original annotations while preserving clinically plausible class proportions. The comparison highlights how different thresholding assumptions influence class balance and demonstrates the stabilising effect of incorporating clinical priors during large-scale automatic label construction.
Figure 5: Summary of binary cognitive decline classification performance. (a) Heatmap of F1 -scores across individual speech tasks comparing original clinical labels with proposed neuropsychologically informed labelling strategies. Columns represent FLAME clinical thresholds, data-driven Gaussian calibration (Cogno-G), and normative z-score calibration (Cogno-Z). Associated automatic speech recognition (ASR) and transformer-based model configurations are shown for reference. (b) Average classification performance obtained with and without incorporation of MoCA-derived clinical priors. MoCA integration produces consistent performance gains across domains, supporting the role of global cognitive screening as a stabilising supervisory signal. (c) Distribution of F1 -scores across labelling strategies, demonstrating a systematic upward shift in performance relative to the original labels. (d) Confusion matrix obtained using original diagnostic labels, showing increased misclassification between impaired and healthy participants. (e) Confusion matrix obtained using Cogno-Z memory-based labels, illustrating improved class separation and reduced diagnostic ambiguity.
Figure 6: Summary of multiclass cognitive decline classification performance. (a) Heatmap showing multiclass F1 -scores across speech tasks and labelling strategies. Neuropsychologically calibrated labels consistently outperform original annotations, particularly for memory-intensive tasks. (b) Comparison of performance with and without MoCA-derived priors, demonstrating improved stability when separating dementia, MCI, and healthy control groups. (c) Distributional comparison of F1 -scores illustrating global performance improvements achieved through clinically anchored thresholding. (d) Confusion matrix obtained using original labels, highlighting difficulty distinguishing intermediate cognitive states. (e) Confusion matrix obtained using Cogno-Z memory-derived labels, showing improved discrimination between dementia, MCI, and healthy controls.
Validation Type
Dataset
Size ( N )
Task
Method
Mean_diff ( ΔF1 )
Cohens_d
Result
Internal
CognoMemory
Case: 73 Control: 77
Binary
FLAME_ATTEN
-0.032
-0.697
0.008 (**)
(Table 1 )
FLAME_EXEC
-0.027
-0.482
0.057 (ns)
FLAME_MEM
0.070
2.635
< 0.001 (****)
Cogno_G_ATTEN
0.035
1.029
< 0.001 (***)
Cogno_G_EXEC
0.041
1.379
< 0.001 (****)
Cogno_G_MEM
0.075
2.707
< 0.001 (****)
Table 2: Performance Comparison across Internal and External Validation Cohorts (Binary vs. Multiclass Classification). The strategies and the neuropsychological domains performing significantly better are shown in bold.
Figure 7: TF-IDF analysis of semantic fluency responses using Cogno_Z_MEM diagnostic labels. The figure shows the ten most distinctive animal words produced by participants in each diagnostic group (Dementia, MCI, and Healthy Control). TF-IDF scores were normalised within each group and expressed as percentages relative to the most distinctive word (100%) in that group. Higher percentages indicate words that are disproportionately characteristic of a specific cognitive group rather than frequently used across all participants. Distinct lexical patterns emerge across diagnostic categories, suggesting systematic differences in semantic retrieval strategies associated with cognitive impairment.
Speech-based analysis offers a scalable and non-invasive approach for detecting cognitive decline, yet progress has been constrained by the limited availability of clinically validated datasets collected under realistic conditions. We introduce PROCESS-2, a large-scale speech dataset designed to support research on automatic assessment of cognitive impairment from spontaneous and task-oriented speech. The dataset comprises recordings from 200 healthy controls, 150 mild cognitive impairment, and 50 dementia diagnoses collected using the CognoMemory digital assessment platform. Each participant completed a single assessment session, including picture description and verbal fluency tasks, accompanied by manually verified transcripts and participant-level metadata. PROCESS-2 contains approximately 21 hours of speech audio with predefined train/test partitions. Comprehensive technical validation evaluated demographic balance, clinical consistency, recording stability, embedding-space structure, and reproducible baseline modelling performance, demonstrating clinically meaningful group separation and stable performance across modelling approaches while preserving real-world conversational variability. PROCESS-2 is released under controlled access via Hugging Face to enable responsible reuse while protecting participant privacy, providing a reproducible benchmark resource for speech-based cognitive assessment research.
Madhurananda Pahar, Caitlin H. Illingworth, Bahman Mirheidari +7
School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), University of Sheffield, Sheffield, S10 2HQ, UK
Early detection of cognitive impairment relies on neuropsychological tests to minimize subjectivity by assessing multiple cognitive domains. Speech-based evaluation can support diagnostics and improve accessibility, but transcription errors and the omission of nonverbal subtests (e.g., motor skills) limit accuracy. Beyond conventional test scores, speech-derived features can provide additional insights into cognitive status. This study investigates the speech-based evaluation of the German "Syndrom-Kurz-Test," a standardized dementia screening test comprising verbal and motor subtests. We train models that integrate transcript-derived scores and Whisper embeddings per verbal subtest to reduce scoring errors. To compensate for missing motor subtests, we then leverage these fused representations to approximate expert overall ratings. Despite omitting subtests, our models strongly correlate with expert ratings and efficiently and accurately discriminate between cognitive status groups.
Franziska Braun, Christopher Witzl, Andreas Erzigkeit +4
Alzheimer's disease (AD) and mild cognitive impairment (MCI), which may precede AD, manifest early through subtle linguistic and acoustic alterations. Traditional diagnostics, however, are often resource-intensive and lack scalability for mass screening. To address these challenges, we introduce a novel bilingual speech large language model framework for automated, explainable cognitive screening. Unlike conventional pipelines that rely on error-prone automatic speech recognition, our system directly processes raw speech to learn joint acoustic-semantic representations, preserving critical prosodic cues often lost in transcription. Utilising our newly collected PUTH-AD dataset alongside multiple open-source corpora, we implemented a multi-task learning objective that simultaneously performs cognitive status classification and generates clinician-understandable natural language explanations. Our system achieved the highest average accuracy and AUROC across six dataset/task conditions, comparing three representative baselines. The system demonstrated cross-task transfer to held-out PUTH-AD task subsets, maintaining classification accuracy on an entirely unseen cognitive task without task-specific fine-tuning. Furthermore, clinician evaluation confirms that the generated explanations are both clinically relevant and largely consistent with the underlying speech evidence, supporting their potential utility in clinical interpretation. This study provides a scalable, objective, and explainable framework for speech-based cognitive screening, combining cognitive status classification with natural language explanations that clinicians can assess and verify, bridging the gap between advanced AI and clinical utility.