cs.AIOct 26, 2026

Faithful Explanations under Missing Modalities in Multimodal Dementia Prognosis

Authors: Firuz Kamalov

Organizations: Canadian University Dubai, City Walk, P.O. Box 117781, Dubai, United Arab Emirates

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models are increasingly used to turn the feature attributions of multimodal prognostic models into narratives for clinicians, yet costly modalities such as amyloid PET and cerebrospinal fluid biomarkers are often missing, and whether these explanations stay faithful under missing evidence has not been tested. We separate two failure points: phantom attribution, in which the model credits modalities that were never observed, and unfaithful narration, in which the report omits, cites or misjudges missing modalities. Using 307 participants of the Alzheimer's Disease Neuroimaging Initiative with complete data, we masked every subset of four costly modalities, predicted 36-month conversion from mild cognitive impairment to Alzheimer's disease, and restored the masked data to obtain interventional ground truth. All five missing-data strategies assigned 12% to 23% of attribution to unobserved modalities, and up to 45% with four absent; missingness-aware strategies minimised it at no cost in accuracy. Across 51,200 reports from four language models and four prompting regimes, naming absent modalities in the prompt raised acknowledgement from 48% to 88% and cut phantom evidence twenty-fold, but advice on whether acquiring a missing test would change the prediction was correct in only 27% to 41% of cases until checked against executed restorations, which raised accuracy to 69% for open-weight narrators and 99.9% for a proprietary model. Higher phantom attribution was associated with more phantom evidence and fewer faithful direction claims, and direction fidelity, measured by a calibrated language-model judge, depended on narrator scale rather than prompting. Narrators told to name absent modalities sometimes declared observed ones absent, an error that a structured template eliminated. We release the masking protocol and evaluation code.

Figures & tables

Figure 1: Participant flow and analytic cohorts. The complete-data cohort supplies the interventional ground truth through designed masking and restoration; the naturally incomplete cohort supplies a test under the missingness that ADNI exhibits.
Figure 2: Phantom attribution mass against the number of absent modalities, by missing-data strategy, under the background-sample reference. Bands are 95% bootstrap intervals over the fifteen runs.
Figure 3: Phantom attribution mass with one modality absent, by absent modality and missing-data strategy, under the background-sample reference. Error bars are 95% bootstrap intervals.
Figure 4: Acknowledgement recall, phantom-evidence rate, attribution fidelity, and value-of-information accuracy by narration regime for the three open-weight narrators pooled, 9,600 reports per regime. Attribution fidelity is measured by the language-model judge on 308 to 320 sampled reports per regime. Error bars are 95% bootstrap intervals clustered by participant.
ModalityNarratorsStated: would changeStated: would not changeExecuted ≥δ\geq\deltaExecuted <δ<\deltaOverstatement
MRIOpen-weight1,3808035321,6530.102
Amyloid PETOpen-weight2,4364931,0001,9290.028
CSFOpen-weight1,4395065141,4310.023
APOE genotypeOpen-weight571599591,1110.034
MRISonnet 5.53101,0342171,1270.184
Amyloid PETSonnet 5.54617693508800.091
Table 5: Acquisition claims against executed restoration at δ\delta = 0.10, pooled over regimes. The overstatement rate is the fraction of cases with an executed change of at least δ\delta for which the narrator stated that acquiring the modality would not change the assessment.
QuantityPipeline vs Qwen2.5-14B judge, n = 180Pipeline vs Sonnet 5.5 judge, n = 240Judge vs judge, n = 180
Acknowledgement of each absent modalityagreement 0.84, precision 1.00, recall 0.83, κ\kappa 0.23agreement 0.86, precision 0.94, recall 0.89, κ\kappa 0.45agreement 0.86, κ\kappa 0.26
Phantom evidence, report levelagreement 0.96, precision 0.14, recall 0.33, κ\kappa 0.18agreement 0.95, precision 0.88, recall 0.41, κ\kappa 0.54agreement 0.91, κ\kappa 0.08
Direction error, report levelagreement 0.65, precision 0.14, recall 0.02, κ\kappa −- 0.04agreement 0.60, precision 0.75, recall 0.06, κ\kappa 0.05agreement 0.61, κ\kappa 0.23
Stated direction of acquisition claimagreement 0.84, κ\kappa 0.63, n = 49agreement 0.89, κ\kappa 0.77, n = 64agreement 0.86, κ\kappa 0.71, n = 152
Confidence levelweighted κ\kappa 0.77, n = 22weighted κ\kappa 0.78, n = 29weighted κ\kappa 0.74, n = 158
Table 6: Agreement between the automatic pipeline and two independent language-model judges, and between the judges. Precision and recall treat the judge as reference.
NarratorUnconstrainedExplicit missingnessStructuredVerified
Qwen2.5-3B0.97% [0.66%, 1.31%], n = 311.78% [1.34%, 2.25%], n = 570, n = 00, n = 0
Qwen2.5-7B0.09% [0.00%, 0.22%], n = 31.09% [0.75%, 1.47%], n = 350, n = 00, n = 0
Qwen2.5-14B0.31% [0.16%, 0.50%], n = 100.16% [0.03%, 0.31%], n = 50, n = 00, n = 0
Table 7: False-absence rate by narrator and regime over 3,200 reports per cell, with 95% bootstrap intervals and the number of flagged reports. A false absence is a statement that an observed modality was unavailable.
RegimeAcknowledgement, naturalAcknowledgement, designedPhantom evidence, naturalPhantom evidence, designed
Unconstrained0.443 [0.395, 0.500]0.496 [0.475, 0.515]0.0097 [0.0056, 0.0136]0.0131 [0.0103, 0.0163]
Explicit missingness0.860 [0.821, 0.894]0.883 [0.865, 0.900]0.0004 [0.0000, 0.0010]0.0008 [0.0003, 0.0015]
Structured0.866 [0.829, 0.899]0.866 [0.854, 0.878]0.0000 [0.0000, 0.0000]0.0013 [0.0005, 0.0023]
Verified0.864 [0.832, 0.894]0.856 [0.841, 0.870]0.0000 [0.0000, 0.0000]0.0010 [0.0004, 0.0018]
Table 8: Natural against designed missingness for the open-weight narrators, restricted to the seven patterns present in the natural sub-sample: PET with CSF in 26 participants, PET in 23, CSF in 6, and four rarer patterns in 5. Attribution fidelity is not reported for this comparison because the judged sample contains complete-data reports only.
More in this topic

Faithfulness of Language Model Explanations

Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.

See all 107 papers

Explore similar work

CardsList
May 13, 2026

PRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities

Missing modalities are prevalent in real-world Alzheimer's disease (AD) assessment and pose a significant challenge to multimodal learning, particularly when the distribution of observed modality subsets differs between training and deployment. Such missingness pattern mismatch induces a conditional representation shift across modality subsets. Existing approaches that rely on implicit imputation or modality synthesis often fail to explicitly model modality availability and uncertainty, leading to overconfident dependence on synthesized features, reduced robustness, and miscalibrated uncertainty estimates. To address these limitations, we propose PRA-PoE, an incomplete multimodal learning framework that is equipped with Prototype-anchored Representation Alignment (PRA) and an Uncertainty-aware Product of Experts (UA-PoE) fusion mechanism. First, PRA uses learnable global prototypes and availability-conditioned tokens to encode modality availability, distinguish observed from missing modalities, re-synthesize features for missing modalities, and adaptively refine observed representations to align latent spaces across modality subsets, with the goal of reducing representation shift under varying missingness patterns. Second, UA-PoE models each modality as a Gaussian expert and performs closed-form Product of Experts fusion, where experts with higher uncertainty are automatically down-weighted via lower precision, improving uncertainty reliability. We evaluate PRA-PoE under a clinically realistic protocol by training with naturally missing data and testing on all non-empty modality combinations. PRA-PoE consistently outperforms the state-of-the-art across datasets, achieving a 5.4% relative improvement in average accuracy on ADNI and a 10.9% relative gain in average F1 on OASIS-3 over the strongest baseline across all non-empty modality subsets.

Guangqian Yang, Ye Du, Wenlong Hou +2

Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo,2026 Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China

Missing-Modality LearningAlzheimer's DiseaseMultimodal RobustnessMedical DiagnosisCross-Modal AlignmentLearning with Missing Data+2
AskPDF
Sep 22, 2026

MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction

Clinical decision making heavily relies on predicting the disease progression trajectory by seeking to understand patient's health status which is characterised by multimodal medical data. AI holds great potential for learning useful representations from multimodal medical data to predict disease progression and aid clinical decision making. However, development of predictive AI models is constrained by missing modalities and incomplete tabular data frequently occurring in medical datasets. In addition, disease labels alone may only provide limited supervisory signals for learning representations from high-dimensional multimodal data. Here, we present MMAP, a novel Multimodal Missing-aware Alignment Pretraining method for learning image-tabular representations from incomplete data. An image encoder is pretrained with efficient sigmoid contrastive learning combined with generative reconstruction. A tabular encoder is built upon a tabular foundation model. A missing token generator enables the two encoders to take incomplete data as input, enabling the model to be robust against missing modalities, either with missing images or missing tabular data. We evaluate the clinical usefulness of the learnt multimodal representations on two challenging longitudinal clinical tasks for Alzheimer's disease: predicting disease stage conversion and predicting amyloid status. The proposed method outperforms strong multimodal and unimodal baselines.

Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2

Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1

Missing-Modality LearningCross-Modal AlignmentMultimodal PretrainingMultimodal Contrastive LearningAlzheimer's Disease Progression Prediction
AskPDF
Feb 19, 2026

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.

Divyam Madaan, Sumit Chopra, Kyunghyun Cho

New York University. · New York University Grossman School of Medicine · Genentech +1

Missing-Modality LearningMultimodal LearningMultimodal ClassificationLatent Variable ModelsMultimodal Model Interpretability
AskPDF