Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech

Ziyun Cui; Wen Wu; Chuan Shi; Shuguang Yang; Xueying Gui; Yan Zheng; Qiong Yang; Haiyan Zhao

eess.ASSep 28, 2026

Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech

Authors: Ziyun Cui, Wen Wu, Chuan Shi, Shuguang Yang, Xueying Gui, Yan Zheng, Qiong Yang, Haiyan Zhao, +5 more

Organizations: Tsinghua University, Beijing, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, China. · Peking University Sixth Hospital, Beijing, China. · Peking University, Beijing, China. · OPPO Research Institute, Shenzhen, China. · Peking University Third Hospital, Beijing, China.

Abstract

Alzheimer's disease (AD) and mild cognitive impairment (MCI), which may precede AD, manifest early through subtle linguistic and acoustic alterations. Traditional diagnostics, however, are often resource-intensive and lack scalability for mass screening. To address these challenges, we introduce a novel bilingual speech large language model framework for automated, explainable cognitive screening. Unlike conventional pipelines that rely on error-prone automatic speech recognition, our system directly processes raw speech to learn joint acoustic-semantic representations, preserving critical prosodic cues often lost in transcription. Utilising our newly collected PUTH-AD dataset alongside multiple open-source corpora, we implemented a multi-task learning objective that simultaneously performs cognitive status classification and generates clinician-understandable natural language explanations. Our system achieved the highest average accuracy and AUROC across six dataset/task conditions, comparing three representative baselines. The system demonstrated cross-task transfer to held-out PUTH-AD task subsets, maintaining classification accuracy on an entirely unseen cognitive task without task-specific fine-tuning. Furthermore, clinician evaluation confirms that the generated explanations are both clinically relevant and largely consistent with the underlying speech evidence, supporting their potential utility in clinical interpretation. This study provides a scalable, objective, and explainable framework for speech-based cognitive screening, combining cognitive status classification with natural language explanations that clinicians can assess and verify, bridging the gap between advanced AI and clinical utility.

Figures & tables

Figure 1 : Overview of the proposed bilingual speech-based cognitive screening framework. (a) Dataset composition includes a newly collected PUTH-AD dataset, completing four distinct cognitive tasks, alongside established open-source datasets covering English and Chinese languages with varying task types. (b) The system pipeline consists of two components: Evidence text generation uses Gemini to generate explanations from speech recordings and cognitive status labels, serving as training targets; the proposed SpeechLLM-based system processes raw speech with task prompts to simultaneously output cognitive status classification and natural language explanations.

Figure 2 : Demographic and cognitive status characteristics of datasets. (a) Distribution of cognitive status categories (HC/MCI/AD) across ADReSS, NCMMSC-AD, and PUTH-AD datasets. (b) Age distribution by cognitive status categories for ADReSS and PUTH-AD datasets (NCMMSC-AD lacks age information). (c) Gender distribution by cognitive status categories across all three datasets. (d) Mini-Mental State Examination (MMSE) score distribution by cognitive status categories for the ADReSS dataset. (e-f) Montreal Cognitive Assessment (MoCA) and Hong Kong Brief Cognitive Test (HKBC) score distributions by cognitive status categories for the PUTH-AD dataset.

Figure 3 : Classification accuracy comparison across datasets and speech tasks. Bar charts show the performance of the SSL-Model baseline, the Gemini-few-shot baseline, the Text-only-LLM baseline, and our system on six separate datasets and speech tasks. Bars show mean accuracy across five runs with different random seeds, with error bar showing standard deviation. Asterisks indicate statistical comparisons between each baseline and Ours-Kimi-Audio. Bars marked “NS” denote non-significant differences.

*p<0.05;**p<0.01;***p<0.001

indicate statistical significance.

Dataset	Speech Task	Accuracy	AUROC
ADReSS	-	0.893 [0.825, 0.952]	0.921 [0.849, 0.980]
NCMMSC-AD	-	0.857 [0.807, 0.904]	0.943 [0.906, 0.973]
PUTH-AD	AFT	0.553 [0.427, 0.677]	0.675 [0.587, 0.763]
	FPD	0.507 [0.404, 0.612]	0.673 [0.584, 0.763]
	CTD	0.553 [0.442, 0.662]	0.681 [0.572, 0.792]
	PR	0.638 [0.523, 0.750]	0.770 [0.674, 0.875]

Table 1 : Classification performance of the Kimi-Audio model on each dataset/task. Accuracy and AUROC across five random seeds are reported in formant of mean [95% CIs].

Figure 4 : Ablation study comparing classification-only, generation-only, and multi-task training for SALMONN and Kimi-Audio. Panels show accuracy, AUROC, BERT-Score, and hit rate (HR), with markers indicating mean performance across five random seeds. Squares, triangles, and circles denote classification-only, generation-only, and multi-task training, respectively; lines connect training variants within each backbone. Generation-only accuracy is derived from the generated explanations. AUROC is reported for classification-only and multi-task, which have continuous class scores, and generation metrics for generation-only and multi-task which produce explanations. ADR and NCM denote ADReSS and NCMMSC-AD; AFT, FPD, CTD, and PR denote the four PUTH-AD tasks.

Figure 5 : Comparison of separate and joint training with the Kimi-Audio backbone. Blue and red markers show mean performance across five random seeds under separate and joint training, respectively. Joint training includes ADReSS (ADR), NCMMSC-AD (NCM), and the PUTH-AD Personal Recall (PR) subset. The dashed line separates these data from the PUTH-AD AFT, FPD, and CTD subsets, which are excluded from joint training and evaluated on held-out test subjects. Lines connect the two training settings within each condition. Voting denotes participant-level hard voting across the four PUTH-AD tasks, evaluated with accuracy.

Figure 6 : Clinical evaluation for classification and explanation outputs by the joint-training Kimi-Audio model using the predefined random seed of 42. (a) Class-wise F1, sensitivity, and specificity computed in a one-vs-rest manner, after pooling test predictions from ADReSS, NCMMSC-AD, and PUTH-AD voting result. (b) Agreement between original Gemini-generated explanations and clinician-corrected Gemini explanations, measured by BERT-Score and hit rate. (c) Clinician scores for Gemini-generated explanation in two dimensions, clinical relevance and evidence faithfulness. (d) Clinician scores for Kimi-Audio explanations, shown overall and separately for correctly and incorrectly classified samples. Significance was assessed by Welch’s two-sample t-test between the correct and wrong groups.

***p<0.001

; n.s., not significant.

Figure 7 : Model Architecture. Dual output heads simultaneously do classification and generation. The model is optimised through both losses.

Panel of Figure 9 — Supplementary Figure 1 : Two representative successful examples of generated explanations on the AFT task and the FPD task. Each example includes the ASR transcription of the participant’s speech, the system’s generated explanation output, and the ground truth category. The original transcription and output are in Chinese, and translated to English here.

Panel of Figure 10 — Supplementary Figure 2 : Two representative failure cases of generated explanations on the CTD task and PR task. Each example includes the ASR transcription of the participant’s speech, the system’s generated explanation output, and the ground truth category. The original transcription and output are in Chinese, and have been translated to English here.

Panel of Figure 11 — Supplementary Figure 3 : Prompt templated for generating explanation texts using Gemini-2.5-Flash. The generation process employs a two-stage pipeline to ensure thorough analysis and concise output: Stage 1 guides detailed step-by-step analysis of speech characteristics, while Stage 2 extracts the most salient points from the detailed explanation. The prompt is original in Chinese and translated into English here.

Supplementary Figure 4 : Prompt templated for Gemini-few-shot baseline, with NCMMSC-AD shown as an example. The prompt contains the same general clinical and task descriptions as the explanation generation prompt, but is adapted for direct cognitive status classification. Six in-context examples are provided, including two examples from each category (HC, MCI, and AD). The prompt is original in Chinese and translated into English here.

Supplementary Figure 5 : Web interface for clinician evaluation of generated explanations. The left panel presents the audio player, audio path, transcription, task instruction, and task image when applicable. The right panel presents the original Gemini-generated explanation, scoring fields for the Gemini explanation, an editable text box for clinician correction, the Kimi-Audio-generated explanation, and scoring fields for the Kimi-Audio explanation. The interface was implemented in Chinese and deployed on a local server. A default value of 0 indicates that a rating has not yet been assigned; valid clinician ratings range from 1 to 5.

Dataset	Speech Task		Separate Training
Dataset	Speech Task	Unseen	Acc	AUROC	BERT	HR
ADReSS	-	-	0.893 ± 0.016	0.921 ± 0.013	0.772 ± 0.001	0.697 ± 0.037
NCMMSC-AD	-	-	0.857 ± 0.023	0.943 ± 0.004	0.727 ± 0.007	0.580 ± 0.032
PUTH-AD	AFT	-	0.553 ± 0.016	0.675 ± 0.031	0.731 ± 0.012	0.598 ± 0.053
	FPD	-	0.507 ± 0.029	0.673 ± 0.011	0.743 ± 0.009	0.570 ± 0.043
	CTD	-	0.553 ± 0.041	0.681 ± 0.032	0.752 ± 0.004	0.569 ± 0.022

Supplementary Table 3 : Comparison of separate training versus cross-domain joint training with Kimi-Audio backbone. Joint training data includes ADReSS, NCMMSC-AD, and PUTH-AD Personal Recall data. “Unseen” denotes PUTH-AD task subsets excluded from joint training, which are evaluated on the held-out PUTH-AD test subjects for generalisation assessment. Voting represents the result of hard voting across all PUTH-AD tasks. Results reported as mean

\pm

standard deviation across the five different seeds.

Explore similar work

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models
Jul 23, 2026Yingchao Huang, Xin Wang, Yuhan Su +1Mild Cognitive Impairment Alzheimer
Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada
From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection
Jun 26, 2026Yasaman Haghbin, Sina Rashidi, Ali Zolnour +6Explainability Mild Cognitive Impairment
Independent Researcher · Columbia University, United States · Chalmers University of Technology, Sweden
LSEAD: A Privacy-Preserving LLM-Based Speech Analysis Framework for Early Alzheimer's Disease Screening
Aug 7, 2026Xin Wang, Yingchao Huang, Yuhan Su +2Alzheimer Speaker
Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada +1

Jul 23, 2026, eess.SPJ/K move · Enter open · S save

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada

Mild Cognitive Impairment Alzheimer Multimodal Large Language Models Multimodal Embeddings

Open paper AskPDF arXiv

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.

Abstract

Explore similar work

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection

LSEAD: A Privacy-Preserving LLM-Based Speech Analysis Framework for Early Alzheimer's Disease Screening