Multilingual Cognitive Impairment Detection in the Era of Foundation Models

Damar Hoogland; Boshko Koloski; Jaya Caporusso; Tine Kolenik; Ana Zwitter Vitez; Senja Pollak; Christina Manouilidou; Matthew Purver

cs.CLApr 8, 2026

Multilingual Cognitive Impairment Detection in the Era of Foundation Models

Authors: Damar Hoogland, Boshko Koloski, Jaya Caporusso, Tine Kolenik, Ana Zwitter Vitez, Senja Pollak, Christina Manouilidou, Matthew Purver

Organizations: Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Institute of Synergetics and Psychotherapy Research, Paracelsus Medical University, Salzburg, Austria · University of Ljubljana, Ljubljana, Slovenia ·

Abstract

We evaluate cognitive impairment (CI) classification from transcripts of speech in English, Slovene, and Korean. We compare zero-shot large language models (LLMs) used as direct classifiers under three input settings -- transcript-only, linguistic-features-only, and combined -- with supervised tabular approaches trained under a leave-one-out protocol. The tabular models operate on engineered linguistic features, transcript embeddings, and early or late fusion of both modalities. Across languages, zero-shot LLMs provide competitive no-training baselines, but supervised tabular models generally perform better, particularly when engineered linguistic features are included and combined with embeddings. Few-shot experiments focusing on embeddings indicate that the value of limited supervision is language-dependent, with some languages benefiting substantially from additional labelled examples while others remain constrained without richer feature representations. Overall, the results suggest that, in small-data CI detection, structured linguistic signals and simple fusion-based classifiers remain strong and reliable signals.

Figures & tables

Dataset	Condition	Patients	Controls	Total	Patient/Control (%)
English	AD	78	78	156	50.0 / 50.0
Slovene	AD	12	15	27	44.4 / 55.6
Korean	MCI	40	37	77	51.9 / 48.1
Total	–	130	130	260	50.0 / 50.0

Table 1: Participant statistics per dataset with within-dataset class proportions. AD: Alzheimer’s Disease; MCI: Mild Cognitive Impairment.

Non-learning baseline
Method	Input	English	Slovene	Korean
Majority	—	0.333	0.357	0.342
LLMs (zero-shot † )
MedGemma-27B †	Full	0.361	0.518	0.595
	Ling.	0.333	0.357	0.342
	Trans.	0.413	0.492	0.579

Table 2: LOO Macro-F1 across all models and input configurations. Classical ML, tabular foundation models (TFMs), gradient boosting, and LLM zero-shot baselines. Best per language in bold .

\dagger

= zero-shot (no training data).

LLM zero-shot reference (best across models & modalities)
Method	$k$	English	Slovene	Korean
Best LLM †	0	0.556	0.617	0.621
Tabular foundation models
TabPFN	1	0.439	0.846	0.538
	2	0.442	0.815	0.504
	3	0.436	0.852	0.560

Table 3: Few-shot Macro-F1 (embeddings-only,

k

shots/class, 3 seeds) vs. LLM zero-shot reference. Best overall per language in bold .

Language	CKA	Spearman $\rho$	Procrustes	Overlap@5	Purity feat @5	Purity emb @5
English	0.024	0.001	1.758	0.032	0.658	0.501
Slovene	0.181	0.116	1.432	0.200	0.511	0.563
Korean	0.016	0.005	1.835	0.049	0.610	0.610

Table 4: Feature–embedding space alignment. Low CKA and Spearman

\rho

indicate weak alignment between representations; Procrustes disparity Williams et al. (2021) near 2.0 indicates geometric dissimilarity. Overlap@5 = shared

k

NN neighbours; Purity@5 = same-class neighbours.

Speech-Based Dementia Detection

Speech-based dementia detection uses spoken language and vocal patterns to identify cognitive impairment and support accessible screening.

See all 44 papers

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models
Jul 23, 2026Yingchao Huang, Xin Wang, Yuhan Su +1Multimodal Large Language Models Speech-Based Dementia Detection
Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada
Fair Cognitive Impairment Detection Through Unlearning
Jun 17, 2026William Nguyen, Jiali Cheng, Hadi AmiriAdversarial Representation Learning Algorithmic Fairness
University of Massachusetts Lowell, USA
Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models
Oct 23, 2026Xiutian Zhao, Luqi Sun, Björn Schuller +1Audio-Language Models Circuits in Language Models
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Jul 23, 2026J/K move · Enter open · S save

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

Faculty of Digital Innovation, Arts & Sciences, Saskatchewan Polytechnic, Regina SK S4S 5X1, Canada · School of Basic Medical Sciences, Hebei University, Baoding 071000, China · Department of Civil & Environmental Engineering and School of Mining & Petroleum Engineering, University of Alberta, Edmonton AB T6G 2H5, Canada

Multimodal Large Language Models Speech-Based Dementia Detection Multimodal Classification

Open paper AskPDF arXiv

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.

Abstract

Speech-Based Dementia Detection

Explore similar work

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

Fair Cognitive Impairment Detection Through Unlearning

Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models