Automatic estimation of verbal fluency index in people with Motor Neuron Disease using ASR alignment and pause modelling

Bahman Mirheidari; Leslie Ing; Daniel Blackburn; Sharon Abrahams; Christopher McDermott; Heidi Christensen

cs.CLSep 22, 2026

Automatic estimation of verbal fluency index in people with Motor Neuron Disease using ASR alignment and pause modelling

Authors: Bahman Mirheidari, Leslie Ing, Daniel Blackburn, Sharon Abrahams, Christopher McDermott, Heidi Christensen

Organizations: School of Computer Science, University of Sheffield, UK · Sheffield Institute for Translational Neuroscience (SITraN) · Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, UK · Euan MacDonald Centre for Motor Neuron Disease Research, University of Edinburgh, UK

Abstract

Monitoring cognitive impairment (CI) in motor neuron disease (MND) is essential for timely treatment and care, yet challenging due to co-occurring speech difficulties. The Edinburgh Cognitive and Behavioural ALS Screen (ECAS) provides a robust metric for CI assessment, with the Verbal Fluency Index (VFI) a central element. Building on recent advances in automated speech analysis, this study proposes a system for estimating VFI. It leverages a unique MND dataset and combines ASR (WhisperX) and VAD (Silero) with refined timestamping to predict the VFI and extract several clinically interpretable measures. Our approach outperformed systems based on traditional acoustic features and self-supervised embeddings, evaluated using multiple regression algorithms. Clinically inspired features consistently outperformed the other sets, with the best models achieving strong results (P-words: R2 0.9, NRMSE 0.05; S-words: R2 0.8, NRMSE 0.08), demonstrating the feasibility of automated VFI estimation.

Figures & tables

Figure 1: Automatic VFI scoring pipeline.

Panel of Figure 2 — Figure 2: Manual alignment vs ASR, VAD and adjustments (top: typical speech, bottom: dysarthric).

Characteristic	Dys (n=20)	Typ (n=22)	All (n=42)
Age	66.6 $\pm$ 13.1	63.0 $\pm$ 13.0	64.7 $\pm$ 13.0
Female %	25.0	27.3	26.2
Education (years)	14.0 $\pm$ 3.2	12.0 $\pm$ 2.4	13.0 $\pm$ 2.9
ECAS Total	112.7 $\pm$ 12.0	110.7 $\pm$ 14.8	110.8 $\pm$ 14.8
ALSFRS-R	2.4 $\pm$ 0.8	3.6 $\pm$ 0.5	3.2 $\pm$ 1.0
VFI (P)	4.7 $\pm$ 2.3	3.7 $\pm$ 1.7	4.2 $\pm$ 2.1

Table 1: Participants’ demographic information, grouped by dysarthric (Dys) speech vs. typical (Typ).

Name	Type (No.)	E.g.
eGeMaps	Acoustic (88)	Pitch, energy, intensity
eGeMaps (2)	Acoustic (175)	Stats of eGeMaps (Low-Level Descriptors)
Spectral/Temporal	Acoustic (43)	MFCCs, Vowel duration
Formants	Acoustic (8)	F1, F2
Voice Quality	Acoustic (19)	Jitter, Shimmer
Articulation	Acoustic (9)	Formant trajectories, speech rates

Table 2: Summary of the extracted features.

Task	Removed fillers	Dys $\downarrow$	Typ $\downarrow$	All $\downarrow$
P	No	28.8	17.1	21.5
P	Yes	22.2	8.1	14.0
S	No	37.1	12.8	23.0
S	Yes	24.0	15.0	18.7

Table 3: WERs [%] dysarthric (Dys) vs. Typical (Typ) speech.

VAD	Adjust.	P-words% $\downarrow$	S-words % $\downarrow$
Silero [ 25 ]	Longest	6.9	11.8
CNN-BiLSTM [ 27 ]	Last	11.0	13.5
FSMN [ 26 ]	Longest	15.6	20.7

Table 4: Total segmentation (weighted speech/pause) errors.

Model	Feat.	R² $\uparrow$	NRMSE $\downarrow$
LASSO	Clin-Man (P)	0.797/0.131	0.064/0.024
SVR	Clin-Man (P)	0.793/0.124	0.067/0.033
LASSO	Clin-Auto (P)	0.856/0.111	0.053/0.015
SVR	Clin-Auto (P)	0.808/0.135	0.065/0.028
SVR	Base-HuBERT-8 (P)	0.526/0.190	0.102/0.008
SVR	Base-HuBERT-9 (P)	0.497/0.210	0.105/0.009

Table 5: VFI estimation results for the top two models trained on manual clinical features vs baseline features. (Average/STD).

Panel of Figure 3 — Figure 3: Scatter plots for the best 2 models (top: P-words, bottom: S-words)

Clinical Speech Processing

Clinical speech processing uses speech recognition and acoustic analysis to support health assessment, monitoring, and communication.

See all 57 papers

CCMAN: Cognitive Instability-Aware Cross-Modal Attention Network for Interpretable Temporal Biomarkers of Verbal Fluency Speech
Sep 13, 2026Madhurananda Pahar, Caitlin Illingworth, Dorota Braun +2Speech-Based Dementia Detection
School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), University of Sheffield, Sheffield, S10 2HQ, UK
Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment
Jun 17, 2026Franziska Braun, Christopher Witzl, Andreas Erzigkeit +4Speech-Based Dementia Detection Automated Evaluation
Technische Hochschule Nürnberg · Geromed GmbH · PMU Klinikum Nürnberg
SAND: The Challenge on Speech Analysis for Neurodegenerative Disease Assessment
Apr 7, 2026Giovanna Sannino, Ivanoe De Falco, Nadia Brancati +10Clinical Speech Processing Clinical Outcome Prediction
National Research Council of Italy (CNR), Institute for High-Performance Computing and Networking (ICAR), Naples, 80131, Italy · Department of Mathematics and Physics, University of Campania “Luigi Vanvitelli”, Caserta, 81100, Italy · Department of Electrical Engineering and Information Technologies, University of Naples “Federico II”, Naples, 80125, Italy +3

Sep 13, 2026J/K move · Enter open · S save

CCMAN: Cognitive Instability-Aware Cross-Modal Attention Network for Interpretable Temporal Biomarkers of Verbal Fluency Speech

Madhurananda Pahar, Caitlin Illingworth, Dorota Braun, Daniel Blackburn+1

School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Sheffield Institute for Translational Neuroscience (SITraN), University of Sheffield, Sheffield, S10 2HQ, UK

Speech-Based Dementia Detection

Open paper AskPDF arXiv

Early detection of cognitive decline from speech offers a scalable and non-invasive alternative to conventional clinical assessment. Verbal fluency tasks are particularly informative, but most automated approaches aggregate features across an entire recording, overlooking temporal speech dynamics. We propose the Cognitive Instability-Aware Cross-Modal Attention Network (CCMAN), a transfer learning framework that learns task-agnostic cognitive speech representations from multiple memory-probing tasks before fine-tuning on a minute-long semantic and phonemic verbal fluency task. CCMAN integrates semantic, acoustic, and linguistic information through bidirectional cross-attention, gated multimodal fusion, and transformer-based temporal modelling to derive interpretable biomarkers of cognitive decline. Experiments were conducted on 165.44 hours of speech from 843 participants (498 healthy controls, 245 with mild cognitive impairment, and 100 with dementia). CCMAN achieved Macro-F1 scores of 0.81 and 0.59 for binary and multiclass semantic fluency classification, and 0.77 and 0.53 for phonemic fluency, consistently outperforming strong static and temporal baselines. Statistical analyses showed that semantic drift variance and pause variance, but not mean semantic drift, were significantly elevated in both MCI and dementia relative to healthy controls, while pause duration increased progressively over the task with the steepest slope in dementia, supporting global and progressive temporal speech instability as interpretable biomarkers. Evaluation on the independent PROCESS-2 benchmark further demonstrated the generalisability of the proposed framework, improving the baseline Macro-F1 by up to 9%. These findings support temporal speech instability as a dynamic speech biomarker for robust, interpretable, and generalisable early detection of cognitive decline.

Abstract

Clinical Speech Processing

Explore similar work

CCMAN: Cognitive Instability-Aware Cross-Modal Attention Network for Interpretable Temporal Biomarkers of Verbal Fluency Speech

Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment

SAND: The Challenge on Speech Analysis for Neurodegenerative Disease Assessment