Behind the [MASK]: Disentangling Representation and Faithfulness in DAPF-Based Dementia Detection

Pardis Ranjbar-Noiey; Natalie Parde

cs.CLAug 25, 2026

Behind the [MASK]: Disentangling Representation and Faithfulness in DAPF-Based Dementia Detection

Authors: Pardis Ranjbar-Noiey, Natalie Parde

Organizations: University of Illinois Chicago Chicago, IL, USA

Abstract

Spoken-language analysis via prompt-based domain-adaptive models is a promising direction for low-resource, non-invasive dementia screening, but such models remain internally opaque. We study the interpretability of the Domain-Adapted models via Prompt-based Fine-tuning (DAPF) framework, which casts dementia detection as diagnosis-related masked-token prediction. We interpret DAPF and strong baselines using a variety of probing and analysis techniques, finding that DAPF achieved the best overall performance (accuracy=0.83 and macro-F1=0.83) with diagnosis most recoverable from its [MASK] representation. However, this representational advantage did not extend to token-level explanation faithfulness. DAPF attributions primarily reflected language task vocabulary, discourse markers, and transcription artifacts, with perturbation tests showing weak or negative effects. This suggests that its masked-token interface determines diagnosis information without producing faithful token-level explanations.

Figures & tables

Dataset Split	Total	AD	HC
Source Train (CCC)	105	57	48
Target Train (ADReSS)	108	54	54
Target Test (ADReSS)	48	24	24

Table 1: Dataset splits. AD =Alzheimer’s disease or related dementia; HC =Healthy control.

Component	DAPF-BERT	BERT-CLS	BERT-CLS+Prompt
Input format	Prompt-wrapped transcript	Transcript only	Prompt-wrapped transcript
Prediction interface	MLM/verbalizer at [MASK]	Classifier over [CLS]	Classifier over [CLS]
Template	Manual template ID 7	-	Manual template ID 7
Verbalizer	Manual	-	-

Table 2: Model-specific differences between DAPF-BERT, BERT-CLS, and BERT-CLS+Prompt. All models share the same backbone, data split, training configuration, class weighting, and random seeds (Appendix Table 15 ).

Model	Acc.	Macro-F1	AUROC	ECE
DAPF	0.8292	0.8287	0.8757	0.1683
BERT-CLS	0.8083	0.8063	0.8906	0.1695
BERT-CLS+Prompt	0.8083	0.8065	0.8819	0.1680

Table 3: Mean test performance over five random seeds.

Correct Model Combination	Count
All three correct	170
All three incorrect	19
DAPF only	20
BERT-CLS only	3
BERT-CLS+Prompt only	2
BERT-CLS + BERT-CLS+Prompt	17

Table 4: Paired error analysis over 48 ADReSS test samples

\times

five random seeds (

n

=240 decisions). Counts show which model combinations were correct.

Model	Representation	Probe Macro-F1
DAPF	[CLS]	0.7550
DAPF	[MASK]	0.7989
BERT-CLS	[CLS]	0.7704
BERT-CLS+Prompt	[CLS]	0.7683
BERT-CLS+Prompt	auxiliary [MASK]	0.7662

Table 5: Final-layer representation probing. BERT-CLS has no diagnosis [MASK] . For BERT-CLS+Prompt, [MASK] is auxiliary and not used for prediction.

Representation	Diagnosis Probe	Random Probe
DAPF [CLS]	0.7550 $\pm$ 0.0113	0.474 $\pm$ 0.007
DAPF [MASK]	0.7989 $\pm$ 0.0189	0.470 $\pm$ 0.004

Table 6: Random-label control, with macro-F1 averaged across five random seeds. Random probes use the same DAPF representations and probing procedure but randomly permute diagnosis labels.

Panel of Figure 1 — Figure 1: Layer-wise probing across models. Macro-F1 is averaged across five random seeds.

Token	Total	T/S	Attr.	SD	\|Attr.\|
reach	55	8	0.2540	0.1900	0.3374
out	240	32	0.1832	0.1569	0.2337
something	25	5	0.1827	0.0946	0.2568
anything	40	6	0.1262	0.0808	0.1584
cookies	150	21	0.1130	0.1906	0.2982
quiet	25	5	0.0921	0.1086	0.1211

Table 7: Top positive DAPF cross-seed mean attribution tokens. Positive values indicate average contribution toward the Dementia class. Total =total occurrences; T/S =mean transcripts per seed; Attr. =mean signed attribution; SD =standard deviation across seeds; |Attr.| =mean absolute attribution.

Token	Total	T/S	Attr.	SD	\|Attr.\|
okay	60	12	-0.5181	0.3719	0.5775
lid	40	7	-0.1401	0.0474	0.1456
guess	45	8	-0.1141	0.0574	0.1407
cabinets	35	5	-0.1017	0.0865	0.1366
curtains	55	11	-0.0977	0.0196	0.1726
laughing	35	7	-0.0813	0.0523	0.1502

Table 8: Top negative DAPF cross-seed mean attribution tokens. Negative values indicate average contribution away from the Dementia class. Total =total occurrences; T/S =mean transcripts per seed; Attr. =mean signed attribution; SD =standard deviation across seeds; |Attr.| =mean absolute attribution.

Model	Test	5%	20%	50%
DAPF	Deletion	0.0080	0.0055	-0.0318
DAPF	Insertion	-0.0187	-0.0352	-0.0916
BERT-CLS	Deletion	0.0259	0.0142	0.0067
BERT-CLS	Insertion	0.0369	0.0813	0.1223
BERT-CLS +Prompt	Deletion	0.0302	0.0426	0.0259
BERT-CLS +Prompt	Insertion	0.0279	0.0787	0.1225

Table 9: Positive-ranking faithfulness comparison using tokens ranked by positive attribution toward Dementia . Positive values indicate behavior consistent with dementia-supporting evidence. Results are shown for 5%, 20%, and 50% perturbation fractions.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Configuration
Representation probing	Logistic regression probe
Probe evaluation	Stratified 5-fold CV
Probe metric	Macro-F1
Primary attribution method	Integrated Gradients
Attribution sensitivity check	Partition SHAP (DAPF seed 0)
Attribution target	$P(\mathrm{\textsc{Dementia}})$

Appendix

Table 10: Interpretability configuration.

Model	Acc.	Macro-F1	AUROC	ECE
DAPF	0.8292 $\pm$ 0.0174	0.8287 $\pm$ 0.0173	0.8757 $\pm$ 0.0183	0.1683 $\pm$ 0.0284
BERT-CLS	0.8083 $\pm$ 0.0243	0.8063 $\pm$ 0.0247	0.8906 $\pm$ 0.0102	0.1695 $\pm$ 0.0349
BERT-CLS+Prompt	0.8083 $\pm$ 0.0083	0.8065 $\pm$ 0.0082	0.8819 $\pm$ 0.0156	0.1680 $\pm$ 0.0476

Appendix

Table 11: Mean test performance over five random seeds.

Model	Representation	Probe Macro-F1
DAPF	[CLS]	0.7550 $\quad\pm$ 0.0113
DAPF	[MASK]	0.7989 $\quad\pm$ 0.0189
BERT-CLS	[CLS]	0.7704 $\quad\pm$ 0.0260
BERT-CLS+Prompt	[CLS]	0.7683 $\quad\pm$ 0.0152
BERT-CLS+Prompt	auxiliary [MASK]	0.7662 $\quad\pm$ 0.0374

Appendix

Table 12: Final-layer representation probing. BERT-CLS has no diagnosis [MASK] . For BERT-CLS+Prompt, [MASK] is auxiliary and not used for prediction.

Layer	DAPF [CLS]	DAPF [MASK]	BERT-CLS [CLS]	BERT-CLS+Prompt [CLS]	BERT-CLS+Prompt aux. [MASK]
1	0.7101 $\pm$ 0.0102	0.5654 $\pm$ 0.0107	0.7138 $\pm$ 0.0148	0.7802 $\pm$ 0.0283	0.5437 $\pm$ 0.0093
2	0.6976 $\pm$ 0.0216	0.6009 $\pm$ 0.0183	0.7860 $\pm$ 0.0130	0.6911 $\pm$ 0.0091	0.5396 $\pm$ 0.0143
3	0.7644 $\pm$ 0.0116	0.6306 $\pm$ 0.0266	0.7419 $\pm$ 0.0142	0.7379 $\pm$ 0.0405	0.6439 $\pm$ 0.0380
4	0.6746 $\pm$ 0.0132	0.7439 $\pm$ 0.0106	0.7979 $\pm$ 0.0150	0.7277 $\pm$ 0.0274	0.6773 $\pm$ 0.0233
5	0.7488 $\pm$ 0.0204	0.7916 $\pm$ 0.0315	0.8040 $\pm$ 0.0214	0.7498 $\pm$ 0.0304	0.6922 $\pm$ 0.0481
6	0.7490 $\pm$ 0.0375	0.7361 $\pm$ 0.0175	0.7900 $\pm$ 0.0193	0.7924 $\pm$ 0.0289	0.7466 $\pm$ 0.0511

Appendix

Table 13: Layer-wise diagnosis probing across DAPF, transcript-only BERT-CLS, and prompt-as-input BERT-CLS+Prompt. Values are macro-F1 scores averaged across five random seeds. For DAPF, [MASK] is the supervised masked-token diagnosis prediction position. For BERT-CLS+Prompt, the auxiliary [MASK] token is present in the input but is not the supervised prediction site; prediction is made through a standard [CLS] classification head.

Statistic	Mean Signed Attribution	Mean Abs. Attribution
Mean	-0.0015	0.0724
Standard Dev.	0.0608	0.0637
Minimum	-0.5181	0.0177
25th Percentile	-0.0174	0.0427
Median	0.0000	0.0540
75th Percentile	0.0147	0.0753

Appendix

Table 14: Summary statistics for aggregated DAPF token attributions (

n=163

Component	Shared setting
Backbone	BERT-base-uncased
Training data	CCC + ADReSS train
Test data	ADReSS test
Test size	48 transcripts
Epochs	10
Batch size	4

Appendix

Table 15: Experimental settings shared by DAPF-BERT, BERT-CLS, and BERT-CLS+Prompt.

Metric	Value
IG top-20 tokens with matching SHAP direction	11 / 20
Spearman correlation	0.362

Appendix

Table 16: Agreement between Integrated Gradients and Partition SHAP token attributions.

Positive Mean SHAP Attribution					Negative Mean SHAP Attribution
Token	Freq.	Trans.	Mean	Mean Abs.	Token	Freq.	Trans.	Mean	Mean Abs.
dry	12	9	0.0448	0.0464	flow	23	18	-0.0307	0.0349
ing	21	18	0.0428	0.0477	cookies	30	21	-0.0225	0.0267
fall	22	19	0.0323	0.0335	cookie	76	36	-0.0222	0.0252
reach	11	8	0.0278	0.0278	open	15	12	-0.0167	0.0172
get	23	13	0.0243	0.0284	guess	10	9	-0.0164	0.0218

Appendix

Table 17: Tokens with the highest positive and negative mean Partition SHAP attributions across the ADReSS test set after applying the same token-reporting and normalization procedure used for Integrated Gradients. Only tokens occurring at least 10 times and in at least five transcripts are included. Freq. is the total number of occurrences, Trans. is the number of transcripts containing the token, Mean is the mean signed attribution to the Dementia output, and Mean Abs. is the mean absolute attribution.

Model	Test	5%	20%	50%
DAPF	Deletion Drop	0.0080 $\pm$ 0.0155	0.0055 $\pm$ 0.0132	-0.0318 $\pm$ 0.0144
DAPF	Insertion Recover	-0.0187 $\pm$ 0.0192	-0.0352 $\pm$ 0.0395	-0.0916 $\pm$ 0.0612
BERT-CLS	Deletion Drop	0.0259 $\pm$ 0.0287	0.0142 $\pm$ 0.0331	0.0067 $\pm$ 0.0280
BERT-CLS	Insertion Recover	0.0369 $\pm$ 0.0226	0.0813 $\pm$ 0.0392	0.1223 $\pm$ 0.0558
BERT-CLS +Prompt	Deletion Drop	0.0302 $\pm$ 0.0277	0.0426 $\pm$ 0.0409	0.0259 $\pm$ 0.0444
BERT-CLS +Prompt	Insertion Recover	0.0279 $\pm$ 0.0185	0.0787 $\pm$ 0.0291	0.1225 $\pm$ 0.0417

Appendix

Table 18: Positive-ranking faithfulness comparison using tokens ranked by positive attribution toward Dementia . Positive values indicate behavior consistent with dementia-supporting evidence. Results are shown for 5%, 20%, and 50% perturbation fractions.

Model	Test	5%	20%	50%
DAPF	Deletion Drop	-0.0511 $\pm$ 0.0471	-0.0939 $\pm$ 0.0499	-0.1799 $\pm$ 0.0483
DAPF	Insertion Recover	-0.0709 $\pm$ 0.0607	-0.1902 $\pm$ 0.1175	-0.3453 $\pm$ 0.1019
BERT-CLS	Deletion Drop	-0.0166 $\pm$ 0.0267	-0.1318 $\pm$ 0.0524	-0.3318 $\pm$ 0.1184
BERT-CLS	Insertion Recover	0.0407 $\pm$ 0.0247	0.0082 $\pm$ 0.0353	-0.1940 $\pm$ 0.1054
BERT-CLS+Prompt	Deletion Drop	-0.0997 $\pm$ 0.0775	-0.2207 $\pm$ 0.0907	-0.3883 $\pm$ 0.1016
BERT-CLS+Prompt	Insertion Recover	0.0008 $\pm$ 0.0102	-0.0189 $\pm$ 0.0395	-0.2402 $\pm$ 0.0823

Appendix

Table 19: Absolute-ranking faithfulness. Results should not be interpreted as target-specific rationales.

Faithfulness of Language Model Explanations

Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.

See all 106 papers

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features
Jun 26, 2026Jonghyeon Park, Olivier Jiyoun Jung, Myungwoo OhMulti-View Learning Speech-Based Dementia Detection
NAVER Cloud, South Korea · Division of Communication and Media, Ewha Womans University, South Korea
Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech
May 25, 2026Rez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales +3Multilingual Language Model Evaluation Speech-Based Dementia Detection
Ateneo de Manila Senior High School · Analog Devices, Inc.
From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection
Jun 26, 2026Yasaman Haghbin, Sina Rashidi, Ali Zolnour +6Transformer Interpretability Explainable Artificial Intelligence
Independent Researcher · Columbia University, United States · Chalmers University of Technology, Sweden

Jun 26, 2026J/K move · Enter open · S save

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

Jonghyeon Park, Olivier Jiyoun Jung, Myungwoo Oh

NAVER Cloud, South Korea · Division of Communication and Media, Ewha Womans University, South Korea

Multi-View Learning Speech-Based Dementia Detection

Open paper AskPDF arXiv

Early detection of dementia enables timely intervention, and reflecting cognitive impairment, spontaneous speech offers a non-invasive screening modality. Conventional approaches often focus on a single representational dimension -- such as acoustic descriptors, pause modeling, automatic speech recognition (ASR) transcripts, or multimodal fusion -- limiting integrative reasoning across heterogeneous cognitive symptoms. We propose a low-rank adaptation (LoRA)-tuned large language model (LLM) that performs structured multi-view reasoning over four complementary speech-derived signals: ASR transcripts with pause markers, discourse-level topic cues, temporal fluency statistics, and phonological sequences. These cues are encoded within a unified prompt, enabling a single LLM to learn a coherent decision function without modality-specific encoders or late-stage fusion. On ADReSSo, our best model achieves an F1-score of 90.14%, and ablation confirms the complementary contribution of each view.

Abstract

Faithfulness of Language Model Explanations

Explore similar work

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech

From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection