NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification

Qingming Li; Ruiyan Sun; Satoshi Nakamura

cs.CLOct 20, 2026

NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification

Authors: Qingming Li, Ruiyan Sun, Satoshi Nakamura

Organizations: The Chinese University of Hong Kong Shenzhen

Abstract

Picture-description dementia classification uses a shared visual scene, but a transcript-level prediction alone does not identify the scene evidence underlying its inputs. NarraSpace links words and speech times to named picture elements and retrieves utterances for predefined scene events. Narrative-spatial descriptors summarize coverage and temporal organization. A frozen language model contrasts each event's text under examples from Alzheimer's disease and healthy-control training participants, alongside transcript surprisal and coverage measurements. Retained source records link event scores to their source utterances, speech intervals, and lexical matches. On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, a fixed support vector classifier achieves $80.88\%\pm2.56\%$ mean accuracy over fifty seeds; averaged probabilities yield $81.25\%$ accuracy and $91.32\%$ area under the receiver operating characteristic curve. NarraSpace makes the construction of scene-linked classification inputs explicit and inspectable by connecting event-indexed language-model measurements to their source utterances, speech timing, and extraction records. Such source-linked measurements could support clinician review of picture descriptions by presenting quantitative summaries alongside the original speech evidence.

Figures & tables

Figure 1: NarraSpace constructs features while retaining links to their source text and speech times. The highlighted fragment illustrates a match; the language model scores the complete utterances retrieved for an event, concatenated in narrative order. The classifier path below the diagram expands its feature-selection step. Event-record summaries support source inspection and are excluded from classifier inputs (Appendix A.5 ). AD denotes Alzheimer’s disease and HC healthy control.

Figure 2: Two inputs, one scoring model: AD training examples followed by the target event text, and HC training examples followed by that same text. Only target tokens are scored. Their mean log-likelihood difference gives an event contrast; the cloze contrast restricts the average to lexicon-matched subwords of that text. The illustrated excerpt is abridged, and the hand-written cloze template is not a model input.

	Accuracy (%)			AUC (%)
Method	Mean $\pm$ std.	Best	Aggregated	Mean $\pm$ std.	Best	Aggregated
A. Primary NarraSpace system
Support vector machine (radial basis function)	$80.88\pm 2.56$	85.42	81.25	$90.65\pm 1.32$	93.23	91.32
B. Reference configurations under their original evaluation procedures
Xiao–Mistral reproductions with harmonized transcripts †
$\tilde{D}_{\log}$	$80.92\pm 4.40$	87.50	79.17	$88.62\pm 1.86$	93.23	89.41

Table 1: Classification performance on the 48 official-test participants. Block A presents the primary NarraSpace system; block B presents Xiao–Mistral reproductions and NarraSpace classifier backends under their original evaluation procedures. Values are percentages. Mean

\pm

standard deviation summarizes fifty seeds; Best denotes the maximum for each metric. Bold marks column maxima within block B, including ties. Aggregation and evaluation procedures are specified below and in Appendix E .

Removed evidence family	Accuracy	AUC	Accuracy	AUC
	Fifty-seed mean		Aggregated probability
None (full)	$80.88\pm 2.56$	$90.65\pm 1.32$	81.25	91.32
Speech timing	$83.25\pm 2.24$	$91.14\pm 1.42$	85.42	92.19
Acoustic	$81.08\pm 2.49$	$90.89\pm 1.39$	79.17	91.15
Linguistic	$80.88\pm 2.63$	$89.76\pm 1.71$	81.25	90.97
Image–text relevance	$81.25\pm 2.57$	$91.23\pm 1.20$	79.17	91.84

Table 2: Corrected official-test results with the same support vector machine and sixty selected inputs throughout. Values are percentages;

\pm

is the population standard deviation over fifty seeds. The full representation is the primary system fixed before these family-removal comparisons. Each removal refits preprocessing, feature selection, classifier, and training-derived thresholds. AUC denotes area under the receiver operating characteristic curve.

Figure 3: Inspecting scene-linked measurements through their source speech. Reports for two test participants connect retrieved utterances with scene-anchor matches, event-field coverage, and likelihood contrasts. E1–E7 denote cookie reaching, stool instability, girl observing, dishwashing, overflow, unawareness, and kitchen context. Filled fields indicate lexical matches; excerpts are abridged, while scores use the full retrieved text. Header means summarize event-field coverage across the 48 official-test participants. For each participant, the displayed probability is the model-estimated probability of Alzheimer’s disease, averaged across fifty runs. E7’s “no aligned span” indicates an unavailable event-text contrast.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Evidence family	Representation	Added indicators
Speech timing	21	0
Acoustic	88	0
Linguistic	27	0
Image–text relevance	10	0
Narrative-spatial	70	0
Scene-likelihood and coverage	20	20

Appendix

Table 3: Full-system feature accounting: 236 constructed features, 256 selection candidates, and sixty selected inputs. The 31 event-record summaries are separate from all three stages and are used for source inspection only.

Identifier	Canonical scene event
E1	Boy reaching for cookies
E2	Stool instability
E3	Girl observing
E4	Woman dishwashing
E5	Sink overflow
E6	Woman unaware of the hazard

Appendix

Table 4: The seven canonical event records. Each has actor, action, object, location, and consequence fields. Utterances are assigned through lexical or anchor matches to this schema.

Subfamily	Count	Represented property
Narrative profile	40	Coverage, duration, path, graph structure, and mapping status
Reference deviation	3	Coverage, first-mention order, and dwell allocation relative to a healthy-control reference
Timing–space interaction	10	Relationships between pauses, dwell, and anchor transitions
Temporal segmentation	14	Changes between the first and second halves of the visit interval
Transition sequences	3	Repetition and entropy of anchor pairs and triples
Total	70

Appendix

Table 5: The seventy narrative-spatial candidate features comprise

40+3+10+14+3

descriptors. The forty-feature narrative-profile group is included within this total. The family-removal analysis evaluates all seventy descriptors together; Appendix I examines specified smaller alternatives.

Descriptors	Count	Control group	Evaluation handling
Cloze mean, minimum, positive count	3	Event scores	Reaggregated
Cloze defined-event coverage	1	Coverage	Reaggregated
Seven paired contrasts, mean, positive count	9	Event scores	Reaggregated
Scene mean, maximum, variance; vague, action, object surprisal	6	Surprisal	Reused
Unmatched-concept count	1	Coverage	Reused

Appendix

Table 6: One inventory of the twenty likelihood-stream columns. “Reaggregated” means recomputed from existing class-context scoring records; “reused” means copied from saved features without new model inference. Positive-event counts depend on score signs.

Stage / target row	Exemplars and healthy-control reference	Preprocessing and classifier
Outer fitting rows	Other four folds; also exclude the target itself	Fit medians, ranks, scaler, and classifier on these rows
Outer validation rows	Same other four folds	Apply that outer fit; save held-out probabilities
Final training rows	Each target’s own-fold-excluded version	Fit final preprocessing and classifier on all 108 rows
Official test rows	All 108 training participants	Apply the final fit
Internal probability folds	Keep the feature values already supplied to the classifier	Fit the sigmoid internally; do not rebuild references or upstream selection

Appendix

Table 7: Data available at each fitting stage. An outer validation fold is one fifth of the 108 official training participants; the official test participants never enter these training pools. “Own fold” is the outer fold containing the target participant.

System	Mean accuracy	Aggregated accuracy	Aggregated AUC
Full	80.88 $[70.75,89.88]$	81.25 $[68.75,91.67]$	91.32 $[82.47,97.57]$
Without speech timing	83.25 $[73.63,91.75]$	85.42 $[75.00,93.75]$	92.19 $[83.68,98.26]$
Without acoustic	81.08 $[70.83,90.17]$	79.17 $[66.67,89.58]$	91.15 $[82.12,97.40]$
Without linguistic	80.88 $[71.29,89.63]$	81.25 $[68.75,91.67]$	90.97 $[81.77,97.57]$
Without image–text	81.25 $[71.29,90.21]$	79.17 $[66.67,89.58]$	91.84 $[83.33,97.74]$
Without narrative-spatial	78.21 $[67.58,87.79]$	81.25 $[68.75,91.67]$	87.67 $[76.56,96.18]$

Appendix

Table 8: Corrected point estimates with participant-bootstrap 95% confidence intervals underneath. Accuracy and area under the receiver operating characteristic curve (AUC) are shown on a 0–100 scale. Mean accuracy averages fifty seed-specific accuracies; aggregated metrics use each participant’s mean probability.

System	Threshold	Sensitivity	Specificity
Full	0.56	75.00	87.50
Without speech timing	0.59	79.17	91.67
Without acoustic	0.62	70.83	87.50
Without linguistic	0.41	79.17	83.33
Without image–text	0.51	75.00	83.33
Without narrative-spatial	0.47	83.33	79.17

Appendix

Table 9: Operating points for aggregated predictions. Thresholds are on the 0–1 probability scale; sensitivity and specificity are percentages. Each system uses its own training-derived threshold.

Removed family	Mean accuracy	Aggregated accuracy	Aggregated AUC
speech timing	-2.38 $[-6.63,0.46]$	-4.17 $[-10.42,0.00]$	-0.87 $[-4.86,2.78]$
acoustic	-0.21 $[-1.54,1.25]$	2.08 $[0.00,6.25]$	0.17 $[-1.56,2.08]$
linguistic	0.00 $[-2.71,3.17]$	0.00 $[-6.25,6.25]$	0.35 $[-2.43,3.47]$
image–text	-0.38 $[-2.08,1.25]$	2.08 $[0.00,6.25]$	-0.52 $[-3.13,1.56]$
narrative-spatial	2.67 $[-3.67,9.00]$	0.00 $[-10.42,10.42]$	3.65 $[-2.78,10.59]$
likelihood stream	-0.75 $[-9.33,7.71]$	0.00 $[-12.50,12.50]$	2.43 $[-4.86,9.90]$

Appendix

Table 10: Full-minus-removal differences and paired participant-bootstrap 95% confidence intervals underneath. All values are differences on the 0–100 metric scale: percentage points for accuracy and AUC points for AUC. Positive values favor the full system.

Removed family	Improved	Regressed	$p$
speech timing	0	2	0.500
acoustic	1	0	1.000
linguistic	1	1	1.000
image–text	1	0	1.000
narrative-spatial	3	3	1.000
likelihood stream	5	5	1.000

Appendix

Table 11: Discordant aggregated decisions comparing the full system with each family removal. Counts are participants;

p

is the exact two-sided McNemar test

p

-value.

Sample	True	$p_{\mathrm{AD}}$	Decision	$p_{\mathrm{AD}}$	Decision	Change
		Without likelihood		Full system
S162	AD	0.6159	AD	0.2339	HC	Regressed
S164	AD	0.3121	HC	0.7523	AD	Improved
S168	AD	0.5966	AD	0.5184	HC	Regressed
S169	AD	0.4828	AD	0.3436	HC	Regressed
S170	HC	0.5071	AD	0.3320	HC	Improved

Appendix

Table 12: All ten changed aggregated decisions. Probabilities are the mean AD probability over fifty seeds, rounded to four decimals for display; decisions use the unrounded values and thresholds

0.45

(without likelihood) and

0.56

(full). Change is defined when moving from without likelihood to the full system.

Method	Protocol / reported summary	Accuracy	AUC
Published point estimates
Official linguistic baseline ( Luz et al., 2020 )	Official 108/48 split	75.00	—
SpeechHGT ( Abid et al., 2025 )	New 70/30 split of all 156	86.32	—
Bidirectional long short-term memory transfer ( Pahar et al., 2025 )	CognoMemory training; official 48 external test	85.00	84.00
TITAN ( Li et al., 2025 )	Official 108/48; last-five-epoch average	89.58	94.10
Pause-augmented ERNIE3p ( Yuan et al., 2021 )	Official 108/48; 35-model majority vote	89.60	—

Appendix

Table 13: Published results under their original evaluation protocols. This table separates reported point estimates from explicitly reported fifty-seed statistics. These literature values provide context; they are not a common-protocol ranking with Table 1 . A dash denotes an unreported metric. All values are percentages.

	Fifty-seed mean		Aggregated probability
Historical configuration	Accuracy	AUC	Accuracy	AUC
NarraSpace full representation	80.25	90.70	81.25	91.32
Whole-transcript replacement	81.38	93.25	87.50	94.79
Consistent event-column permutation	80.71	90.94	81.25	91.32
Xiao–Mistral, $\tilde{D}_{\log}$	80.92	88.62	79.17	89.41
Xiao–Mistral, $\bar{D}$	80.67	90.03	83.33	91.84

Appendix

Table 14: Historical descriptive results under the previous evaluation pipelines, before the partition-isolation correction. These numbers are retained for transparency and are not corrected head-to-head comparisons. Values are percentages; the first two columns are means over fifty seeds.

Scene anchor	Correct / reviewed	Precision (%)
Boy	33 / 33	100.00
Girl	43 / 44	97.73
Woman	14 / 15	93.33
Stool	25 / 26	96.15
Cookie jar	68 / 68	100.00
Cookies	0 / 1	0.00

Appendix

Table 15: Per-anchor precision for the single-annotator audit of accepted phrase-rule and prototype assignments. The cookies and curtain rows have only one and four reviewed assignments; their percentages are not stable anchor-specific estimates.

Pool	Candidates	Accuracy	AUC	Accuracy	AUC
		Fifty-seed mean		Aggregated
Full reference	256	$80.88\pm 2.56$	$90.65\pm 1.32$	81.25	91.32
Coverage + flags	238	$79.33\pm 0.57$	$89.93\pm 0.00$	79.17	89.93
Coverage + surprisal + flags	244	$80.17\pm 1.73$	$92.01\pm 0.00$	79.17	92.01
Coverage + event scores + flags	250	$79.88\pm 2.91$	$89.73\pm 1.81$	81.25	90.63
Compact spatial	192	$77.96\pm 3.78$	$87.92\pm 1.47$	83.33	89.06

Appendix

Table 16: All targeted representation controls. Each of the three coverage-based rows retains the 216 base and narrative-spatial features in addition to the named groups. Candidate dimensions include twenty missingness flags; every row selects sixty inputs. Performance values are percentages, with population standard deviations over fifty seeds. Thresholds are selected from training validation probabilities. These post-exposure controls do not replace the fixed primary system.

Left minus right	Mean accuracy	Aggregated accuracy	Aggregated AUC
Full $-$ coverage	$1.54$	$2.08$	$1.39$
	$[-8.62,11.67]$	$[-10.42,14.58]$	$[-4.86,7.64]$
Full $-$ (coverage + surprisal)	$0.71$	$2.08$	$-0.69$
	$[-6.42,7.92]$	$[-8.33,12.50]$	$[-6.94,4.69]$
Full $-$ (coverage + scores)	$1.00$	$0.00$	$0.69$
	$[-4.75,6.75]$	$[-8.33,8.33]$	$[-4.17,5.38]$

Appendix

Table 17: All eight paired representation-control contrasts, in percentage points. Each cell gives left-minus-right difference and its unadjusted participant bootstrap 95% interval. “Coverage,” “surprisal,” and “scores” refer to the numeric groups in Table 6 . Coverage-based pools also retain all 216 base and narrative-spatial features; every pool retains the twenty flags.

Left minus right	Improved	Regressed	$p$
Full $-$ coverage	6	5	1.00
Full $-$ (coverage + surprisal)	4	3	1.00
Full $-$ (coverage + scores)	2	2	1.00
Full $-$ compact spatial	4	5	1.00
Full $-$ two-coordinate removal	3	3	1.00
Full $-$ quality removed	1	0	1.00

Appendix

Table 18: Hard-decision discordance for the same eight contrasts. Improved means left correct and right incorrect; regressed means the reverse. Exact two-sided McNemar tests concern aggregated decisions, not fractional mean-over-seed correctness. No comparison has zero discordance.

Feature	Final fits / 50	Outer-fold fits / 250
Cloze mean	49	—
Cloze minimum	47	—
Cloze coverage	0	—
Cloze positive-event count	2	—
E1: cookie reaching	35	—
E2: stool instability	8	—

Appendix

Table 19: Full-reference feature-selection counts. Counts for grouped rows apply to each named feature separately. A dash denotes a count not tabulated here, not zero. Event names follow Table 4 .

Group	$n$	Filled fields	Nonempty events	Unique utterances	Shared (%)
Training HC	54	19.26	6.81	9.57	80.12
Training AD	54	14.65	6.20	8.44	79.08
Test HC	24	20.75	6.88	9.63	81.59
Test AD	24	15.42	6.71	7.54	81.11

Appendix

Table 20: Mechanical source descriptions, separated by official split and diagnosis. Participant-level entries are group means: filled fields out of 35, nonempty events out of seven, and distinct retrieved utterances. Shared utterances are those retrieved for multiple events, divided by unique retrieved utterances within each participant, then averaged. Record counts describe field availability; they are not semantic or timing accuracy estimates.

Speech-Based Dementia Detection

Speech-based dementia detection uses spoken language and vocal patterns to identify cognitive impairment and support accessible screening.

See all 44 papers

Jun 26, 2026

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

Early detection of dementia enables timely intervention, and reflecting cognitive impairment, spontaneous speech offers a non-invasive screening modality. Conventional approaches often focus on a single representational dimension -- such as acoustic descriptors, pause modeling, automatic speech recognition (ASR) transcripts, or multimodal fusion -- limiting integrative reasoning across heterogeneous cognitive symptoms. We propose a low-rank adaptation (LoRA)-tuned large language model (LLM) that performs structured multi-view reasoning over four complementary speech-derived signals: ASR transcripts with pause markers, discourse-level topic cues, temporal fluency statistics, and phonological sequences. These cues are encoded within a unified prompt, enabling a single LLM to learn a coherent decision function without modality-specific encoders or late-stage fusion. On ADReSSo, our best model achieves an F1-score of 90.14%, and ablation confirms the complementary contribution of each view.

Jonghyeon Park, Olivier Jiyoun Jung, Myungwoo Oh

NAVER Cloud, South Korea · Division of Communication and Media, Ewha Womans University, South Korea

Multi-View Learning Speech-Based Dementia Detection

Ask PDF

May 25, 2026

A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning

Alzheimer's disease (AD) is a progressive neurodegenerative disorder and the leading cause of dementia, affecting memory, reasoning, communication, and daily functioning. Early diagnosis is particularly important, as timely intervention may help slow cognitive decline and improve patient care. Recent studies have demonstrated that spontaneous speech contains valuable linguistic and acoustic biomarkers associated with dementia. However, existing approaches often rely on independently trained modality-specific models, feature concatenation strategies, ensemble methods, or attention-based fusion mechanisms that do not explicitly maximize the dependency between speech and transcript representations. In this work, we propose a multimodal deep learning framework for automatic dementia detection that jointly exploits speech and transcript information in an end-to-end trainable manner. Specifically, speech recordings are divided into 10-second segments and passed through a pre-trained HuBERT model to extract contextualized acoustic representations. To better capture informative temporal speech characteristics, attentive statistics pooling is employed to aggregate frame-level acoustic embeddings. For the textual modality, transcripts are encoded using a pre-trained BERT model, where the [CLS] token representation is used as the linguistic embedding. The acoustic and textual representations are subsequently combined using an attention-based Audio-Text Fusion (AT-Fusion) mechanism. In addition, we introduce a MINE objective to maximize the mutual information between modalities and improve multimodal representation alignment. The fused multimodal representation is finally used for dementia classification. Experiments conducted on the publicly available ADReSS Challenge and PROCESS-2 dataset demonstrate the effectiveness and robustness of the proposed approach for speech-based dementia assessment.

Loukas Ilias, Dimitris Askounis

Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece

Speech-Based Dementia Detection Speech Processing Alzheimer's Disease Classification Mutual Information Maximization Multimodal Fusion

Ask PDF

Sep 11, 2026

DementiaCare-Bench: A Modality-Validated Video Benchmark

Dementia affects an estimated 57 million people worldwide, and for most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia (BPSD): agitation, wandering, resistance to care, sundowning. Understanding these symptoms requires more than recognizing the behavior itself; it also requires knowing what happened beforehand. The same behavior may call for a different response depending on its trigger. Video-language models (VLMs) could potentially support caregivers, yet no existing benchmark evaluates this capability. To fill this gap, we present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories, with 2023 questions generated by a multi-agent pipeline that grounds every clinical claim in a verbatim transcript span. Each question is then probed under four visual conditions and labelled by the least it requires, so its visual demand is measured rather than assumed. Measurement contradicts intent: we wrote 77.7% of the questions to require ordered frames, and 34.8% do. Across 12 current VLMs the pattern is uniform. The best reach 85% overall, but that average is carried by questions a language model can answer from clinical knowledge alone; accuracy falls by 17 points on average on questions that require the ordered clip, and a leading open model scores at chance on judging whether a caregiver's response was appropriate. A lightweight LoRA fine-tune, DemCare-VLM, moves video dependence from -3.3 to +4.5 points, so what the benchmark exposes can be repaired and not only measured.

Afrouz Sheikholeslami, Yuankai Qi, Xuyun Zhang +4

School of Computing, Macquarie University, Sydney, NSW, Australia · School of Electrical and Computer Engineering, University of Sydney, Sydney, NSW, Australia · University of California, Merced, CA, USA

Alzheimer's Disease Healthcare Video-Language Model Evaluation Dementia Care Video QA

Ask PDF

Abstract

Speech-Based Dementia Detection

Explore similar work

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning

DementiaCare-Bench: A Modality-Validated Video Benchmark