Exploiting Speech LLM Representations for Multilingual and Cross-Lingual Parkinson's Disease Detection

Sarthak Giri; Zi Haur Pang; Tatsuya Kawahara

cs.SDSep 13, 2026

Exploiting Speech LLM Representations for Multilingual and Cross-Lingual Parkinson's Disease Detection

Authors: Sarthak Giri, Zi Haur Pang, Tatsuya Kawahara

Organizations: Graduate School of Informatics, Kyoto University, Japan

Abstract

Speech Large Language Models (Speech LLMs) have shown strong performance across diverse tasks, yet their utility for pathological speech analysis remains underexplored. In this work, we investigate the effectiveness of internal representations from encoder and decoder components of Speech LLMs for Parkinson's Disease (PD) detection across multilingual and cross-lingual settings. Our findings reveal that encoder representations consistently outperform their decoder counterparts in most models and settings and that pathological cues may be progressively attenuated as audio representations are projected into the language model space. We further show that generative outputs are less reliable for clinical tasks compared to internal representations. To leverage information spread across multiple layers, we propose a Squeeze-and-Excitation (SE)-based dynamic layer aggregation framework, which surpasses best-layer selection in multiple experiments, suggesting that PD-relevant acoustic cues are distributed across transformer layers rather than concentrated in one.

Figures & tables

Fig. 1: Overview of Speech LLM pipeline showing the encoder/decoder representation probing, zero-shot generative output, and the proposed SE-based layer aggregation.

Language	Sustained Vowel	Read	DDK
Spanish	450/447	90/90	90/90
German	176/175	88/88	150/88
Czech	100/100	100/100	100/100

TABLE I: Utterance-level sample distribution by language, task, and class. Counts are reported as PD/HC.

Model	Setting	Audio Only	Biomarker Guided
Qwen2-Audio	Multi	0.550	0.513
	ES	0.558	0.551
	CS	0.497	0.498
	DE	0.485	0.473
Phi-4	Multi	0.540	0.523
	ES	0.512	0.581

TABLE II: UAR of zero-shot generative classification using audio-only and biomarker-guided prompts alongside audio.

		Best Layer						Mean Performance Across Layers
		AUC			UAR			AUC			UAR
Model	Setting	E	D	$\Delta$	E	D	$\Delta$	E	D	$\Delta$	E	D	$\Delta$
Qwen2-Audio	Multi	0.911	0.858	+0.053*	0.851	0.793	+0.058*	0.892	0.848	+0.044*	0.805	0.774	+0.031*
	DE+CS $\rightarrow$ ES	0.828	0.736	+0.092*	0.719	0.680	+0.038*	0.700	0.689	+0.011	0.638	0.628	+0.009
	DE+ES $\rightarrow$ CS	0.755	0.595	+0.160*	0.643	0.565	+0.078*	0.667	0.570	+0.097*	0.580	0.540	+0.040*
	ES+CS $\rightarrow$ DE	0.638	0.598	+0.040*	0.595	0.574	+0.021*	0.597	0.582	+0.015	0.568	0.555	+0.013

TABLE III: Encoder(E) vs. decoder(D) representation performance across Speech LLMs.

\Delta

= Encoder

-

Decoder.

Setting	AUC			UAR
Setting	XLS-R 2B	Whisper	HuBERT	XLS-R 2B	Whisper	HuBERT
Multi	0.806	0.789	0.798	0.710	0.728	0.725
DE+CS $\rightarrow$ ES	0.723	0.709	0.664	0.662	0.651	0.604
DE+ES $\rightarrow$ CS	0.783	0.748	0.763	0.707	0.640	0.693
ES+CS $\rightarrow$ DE	0.725	0.671	0.689	0.662	0.636	0.630

TABLE IV: AUC and UAR performance of best-performing layers of speech foundation models across multilingual and cross-lingual experiments.

		AUC			UAR
Model	Setting	Mean	Best	SE-Based (Ours)	Mean	Best	SE-Based (Ours)
Qwen2-Audio	Multi	0.899	0.903	0.905	0.822	0.830	0.807
	DE+CS $\rightarrow$ ES	0.763	0.702	0.850 ∗†	0.695	0.659	0.671
	ES+CS $\rightarrow$ DE	0.646	0.598	0.619	0.603	0.566	0.598
	DE+ES $\rightarrow$ CS	0.716	0.695	0.742 ∗†	0.548	0.557	0.573
Phi-4	Multi	0.860	0.852	0.906 ∗†	0.769	0.760	0.803

TABLE V: Comparison of mean layer representations, validation-selected best-layer, and proposed SE-based layer aggregation for audio encoder representations of Speech LLMs and Speech Foundation models.

Explore similar work

Motor, Cognitive, or Corpus? What Survives Cross-Lingual Transfer in Speech-Based Parkinsons Disease Detection
Aug 13, 2026Serli Kopar, Sam Gijsen, Abner Hernandez +2Speech Processing Parkinson's Disease
Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Tübingen AI Center, University of Tübingen, Tübingen, Germany · Charité–Universitätsmedizin, Department of Psychiatry and Psychotherapy, Berlin, Germany +1
Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease
Mar 23, 2026Abner Hernandez, Eunjung Yeo, Kwanghee Choi +12Domain Adaptation Medical Diagnosis
1FAU Erlangen-Nürnberg, Germany · 2UT Austin, USA · 3CMU, USA +5
Cross-Lingual Parkinson's Disease Severity Assessment Using Pre-trained Speech Embeddings: A Multi-Class Evaluation
Sep 16, 2026Simon Pals, Cristian Tejedor-GarciaParkinson's Disease Clinical Speech Processing
Centre for Language Studies Radboud University Nijmegen, the Netherlands

Aug 13, 2026J/K move · Enter open · S save

Motor, Cognitive, or Corpus? What Survives Cross-Lingual Transfer in Speech-Based Parkinsons Disease Detection

Serli Kopar, Sam Gijsen, Abner Hernandez, Paula Andrea Perez-Toro+1

Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Tübingen AI Center, University of Tübingen, Tübingen, Germany · Charité–Universitätsmedizin, Department of Psychiatry and Psychotherapy, Berlin, Germany+1

Speech Processing Parkinson's Disease Self-Supervised Speech Representation Learning Clinical Speech Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

Self-supervised learning (SSL) speech representations achieve strong performance for Parkinson's disease (PD) detection within individual corpora. However, it remains unclear whether these models capture disease-related characteristics or exploit dataset-specific confounds, particularly since most SSL backbones are pretrained exclusively on healthy speech. To investigate this question, we perform a layer-wise analysis of nine SSL speech backbones using a low-capacity logistic regression probe across three languages. We structure the evaluation as multiple scenarios that progressively introduce distribution shifts in participant identity, recording conditions, language, and pathology. Our results reveal two key findings. First, layer selection is highly corpus-dependent: the optimal representation layer is determined primarily by the source dataset rather than by the SSL architecture itself. Second, the transferred discriminative signal lacks pathological specificity: classifiers trained to detect PD assign similarly high probabilities to both PD and dementia speech in the target corpus. These results highlight critical limitations that must be addressed before speech-based pathology recognition models can be reliably deployed in clinical settings.

Abstract

Explore similar work

Motor, Cognitive, or Corpus? What Survives Cross-Lingual Transfer in Speech-Based Parkinsons Disease Detection

Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease

Cross-Lingual Parkinson's Disease Severity Assessment Using Pre-trained Speech Embeddings: A Multi-Class Evaluation