Cross-Linguistic Acoustic Biomarkers of Cognitive Impairment Across Interactive and Single-Speaker Speech Tasks
Authors: Kenyu Ikeuchi, Tomoaki Ohtsuki, Michelle Cohn, Alyssa Weakley, Yui Ishihara, Joseph Hoang, Taishiro Kishimoto, Sarah Tomaszewski Farias, Jet M. J. Vonk, Chen-Nee Chuah
Organizations: Keio University, Yokohama, Japan · University of California - Davis, Davis, CA, USA · University of California - Davis Health, Sacramento, CA, USA · Keio University School of Medicine, Tokyo, Japan · University of California - San Francisco, San Francisco, CA, USA
Multilingual dementia detection is essential for scalable and equitable screening. In recent years, automated speech analysis has emerged as a promising approach for supporting dementia detection by enabling non-invasive and scalable assessment of cognitive function. However, existing studies that use automated speech analysis often rely on one type of task, such as picture description, and use black-box models. These studies often neglect the role of task and language. To address these gaps, we performed a cross-linguistic and cross-task acoustic analysis that integrates a conversational dataset with structured speech tasks across four languages (Spanish, Mandarin, English, Japanese). We applied statistical, linguist-/clinician-guided, and machine learning model-based feature selection to derive compact acoustic feature sets while maintaining classification performance. This enabled systematic comparison of cognitive impairment-relevant features across datasets and tasks. Our results revealed that acoustic features did not generalize uniformly across tasks and languages. Temporal features remained robust across all datasets, consistently capturing timing disruption regardless of language or task. In contrast, speech rate showed strong task dependence -- emerging as informative mainly in single-speaker tasks with externally anchored content and pacing, but becoming less reliable when speech content was highly individualized. Pitch-related features further exhibited language-specific patterns, favoring chroma-related features in tonal languages (e.g., Mandarin) and F0-related features in non-tonal languages. These findings caution against indiscriminate pooling of heterogeneous speech data and underscore the need for interpretable, context-aware approaches to multilingual dementia detection using automated speech analysis.
Figures & tables
Dataset (Lang, Task)
Normal
Impaired
ADReSSo (English, PD)
114 (HC)
116 (AD)
Ivanova (Spanish, Reading)
150 (HC)
127 (AD, MCI)
Chou (Mandarin, PD)
43 (HC)
44 (MCI)
PROMPT (Japanese, Conv)
74 (HC)
52 (CI)
ADRC (English, AM)
98 (HC)
110 (MCI, Dementia)
TABLE I: Dataset composition based on clinical diagnosis (after excluding MMSE < 10).
Fig. 1: Overview of the proposed method
Feature group
Single
Conversation
Temporal features (Utterance, Speech Rate)
20
27
pyAudioAnalysis
136
136
openSMILE eGeMAPS
88
88
Praat (Pitch, Intensity, Rhythm)
11
11
Total
255
262
TABLE II: Feature counts for single-speaker and conversational tasks without any feature filtering. Response pause features are not applicable to single-speaker tasks.
Dataset
(A) All
(B) KS-only
(C) KS + Expert
(D) KS + RFECV
(E) KS + Expert + RFECV
ADReSSo
255 / 0.74 (0.70–0.78)
186 / 0.76 (0.72–0.80)
171 / 0.76 (0.73–0.79)
156 / 0.75 (0.72–0.78)
142 / 0.76 (0.73–0.79)
Ivanova
255 / 0.77 (0.74–0.80)
199 / 0.77 (0.75–0.79)
178 / 0.77 (0.75–0.79)
121 / 0.77 (0.74–0.80)
114 / 0.77 (0.75–0.79)
Chou
255 / 0.66 (0.62–0.70)
253 / 0.66 (0.62–0.70)
227 / 0.67 (0.63–0.71)
131 / 0.66 (0.62–0.70)
106 / 0.67 (0.63–0.71)
PROMPT
262 / 0.79 (0.76–0.82)
251 / 0.79 (0.76–0.82)
227 / 0.79 (0.76–0.82)
156 / 0.79 (0.76–0.82)
101 / 0.79 (0.76–0.82)
ADRC
262 / 0.65 (0.63–0.67)
202 / 0.66 (0.64–0.68)
180 / 0.67 (0.65–0.69)
102 / 0.65 (0.63–0.67)
86 / 0.67 (0.65–0.69)
TABLE III: Effect of feature filtering and RFECV on feature dimensionality and classification performance. Each cell reports the number of retained features and the corresponding ROC-AUC (shown as #Features / AUC mean (95% confidence interval)).
Dataset
Ivanova
Chou
ADReSSo
ADRC
PROMPT
Language
Spanish
Mandarin
English
English
Japanese
Task
Reading
Picture Description
Picture Description
Autobiographical Memory
Conversation
The number of stable important features
53
23
127
12
29
Temporal features (Duration, Speech Rate)
High
High
Moderate
High
Moderate
Pitch (F0 variability and slope)
Moderate
Low
Moderate
High
Moderate
Pitch (chroma-based mean and std)
Moderate
High
Moderate
Low
High
TABLE IV: Cross-dataset comparison of dominant acoustic features. Top rows summarize dataset attributes, while bottom rows indicate the relative dominance ( High , Moderate , Low ) of each acoustic feature family within stable important features.
Feature
# Datasets
Family
timing_medial_pause_mean
4
Temporal
pause_ratio
3
Temporal
speech_rate
3
Temporal
timing_medial_pause_std
3
Temporal
timing_utterance_median
3
Temporal
utterance_ratio
3
Temporal
TABLE V: Most consistently selected task- and language-agnostic individual temporal features across datasets.
Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment. Yet most speech-based dementia detection systems depend on transcription, discard within-recording temporal structure, and are validated on a single English corpus with known recording artifacts. We propose an ASR-agnostic framework operating directly on Mel spectrograms. Our key contribution is extracting spectrotemporal displacement fields from consecutive spectrogram frames, capturing shifting spectral energy patterns as digital biomarkers of cognitive decline. These features are fused with CNN-ConvGRU acoustic embeddings via a learned cross-attention mechanism and aggregated using a Transformer encoder with learnable query pooling. A composite temporal loss enforces smoothness and contrastive coherence across segments. We train independent models on English DementiaBank, Slovak EWA-DB, and Spanish Ivanova corpora, using clinical elicitation protocols taxing IADL-relevant cognitive domains. The Slovak model achieves 83.9% accuracy, and Spanish achieves, while the English baseline yields 53.2%, confirming known artifacts. Cross-lingual ablation studies reveal distinct fusion regimes: removing cross-attention collapses Spanish performance to 53.7%, below unimodal models, while the Slovak audio encoder alone outperforms the full model, 93.7% vs. 83.9%, and all English configurations remain near chance. Thus, multimodal fusion's value is corpus-dependent: essential when signal is distributed across modalities, counterproductive when one dominates, and irrelevant when no signal exists. Auxiliary temporal losses converge to language-invariant values, indicating cross-lingual architectural stability.
Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging. We propose a multimodal framework leveraging Whisper for dual-purpose extraction: acoustic representations from encoder outputs and transcripts via automatic speech recognition (ASR). For the acoustic pathway, temporal networks with attention pooling aggregate variable-length sequences into fixed-dimensional embeddings. For the linguistic pathway, we prompt a large language model (LLM) to extract interpretable features spanning lexical diversity, syntactic complexity, semantic coherence, and discourse patterns. A gated fusion network integrates both modalities. On ADReSS and ADReSSo, our method achieves F1-scores of 89.47% and 90.14%, demonstrating effective integration of acoustic and LLM-augmented linguistic features. Ablation shows that multimodal fusion consistently outperforms either modality alone.
Olivier Jiyoun Jung, Jonghyeon Park, Myungwoo Oh
Division of Communication and Media, Ewha Womans University, South Korea · NAVER Cloud, South Korea
The development of multilingual Alzheimer's Disease Dementia (AD) detection models presents significant challenges due to the resource-intensive and time-consuming nature of language-specific model training. We propose a novel solution using cross-language training to detect AD in languages beyond those used for model training. This study investigates multilingual deep learning models for detecting AD across different languages and cognitive impairment levels. Using datasets in English, Chinese, Arabic, and Hindi, we developed transformer-based models for binary AD classification. Our approach achieved F1 scores of 82% across all languages, demonstrating strong cross-linguistic generalization. The rapid inference time (0.5 seconds) supports potential real-time screening applications, while consistent performance across languages indicates feasibility for global deployment.