cs.AIMar 30, 2026

Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

Authors: Doan Nam Long Vu, Simone Balloccu

Organizations: Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Trustworthy clinical AI must use real evidence and avoid relying on surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on two clinical neuroimaging cohorts for binary classification of affective disorders and cognitive decline. Both cohorts include structural magnetic resonance imaging (MRI) acquired under their original research protocols. Prior work does not establish the included neuroimaging inputs as reliable stand-alone diagnostic evidence for the present tasks. Nevertheless, when neuroimaging context is introduced, smaller VLMs gain up to 0.66 F1 under the evaluated augmented conditions, becoming competitive with models an order of magnitude larger. Confidence estimation shows that most of the calibration improvement for the analyzed smaller models occurs after the MRI reference is added to the prompt, before any image is supplied. Our preliminary expert case study finds that faithfulness remains low in every condition examined, with the reviewed model introducing unverified clinical details. Finally, in our single-model intervention, preference alignment suppresses MRI-referencing behavior but reduces the augmented-condition advantage, leaving the underlying issue unresolved. These results caution against reading surface metric gains as evidence of true multimodal integration, with direct implications for clinical VLM deployment.

Figures & tables

Figure 1: Overview of the VLM evaluation pipeline.
Cond.InputCSVPrompt(MRI)Parcel.ImageImage Type
C1text(csv)✓    
C2text(csv) ++ prompt(mri)✓✓   
C3text(csv) ++ prompt(mri) ++ plot(MRI)✓✓ ✓nilearn brain plot
C4text(csv, parcel) ++ prompt(mri) ++ plot(MRI)✓✓✓✓nilearn brain plot
C5text(csv, parcel) ++ prompt(mri) ++ plot(swap)✓✓✓✓OOD image (ablation)
Table 1: Experimental conditions and their input components. Conditions are designed to vary the supplied input components: C1 uses tabular text only, C2 adds the MRI preamble without imaging data, C3–C4 progressively add MRI content, and C5 replaces the subject-specific MRI plot with an out-of-distribution control image to test whether performance depends on image content or merely image presence.
DatasetGroup / Status# Samples
FOR2107Active MDD701
Control1,071
OASIS-3Cognitive Decline487
Cognitive Normal849
Table 2: Class distribution of the clinical subsets used in this work.
Figure 2: F1 scores on both datasets for the 12 tested VLMs. Top row : text-only baseline (C1, teal-green) versus the best augmented setup per model (best of C2–C5). Bottom row : full breakdown across all five input conditions. Models are sorted from smallest to largest (parameter count).
Figure 3: FOR2107 per-group mean P^(MDD)\hat{P}(\text{MDD}) (top) and P^(Control)\hat{P}(\text{Control}) (bottom) across the three input conditions. Both diagnostic groups exhibit an upward shift in P^(MDD)\hat{P}(\text{MDD}) ; the corresponding decrease in P^(Control)\hat{P}(\text{Control}) is implied by two-class normalization. Shaded bands show one sample standard deviation around each group mean and are clipped to the [0,1][0,1] probability range.
Cond.Qwen2.5-VL-3BMinistral-3-3B
ECE ↓\downarrowBrier ↓\downarrowECE ↓\downarrowBrier ↓\downarrow
C10.3760.3760.3700.3700.3610.3610.3540.354
C20.1640.1640.178\mathbf{0.178}0.1990.1990.2190.219
C40.155\mathbf{0.155}0.1840.1840.183\mathbf{0.183}0.207\mathbf{0.207}
Table 3: Expected calibration error (ECE; 15 equal-width bins) and Brier score by condition. Lower ECE and Brier indicate better calibration. Bold indicates the best score per model.
Grp.Cond., ClassFaith.  ↑\,\uparrowClin. Acc.  ↑\,\uparrowReasoning  ↑\,\uparrow
0C1, Control2.172.172.252.252.462.46
1C1, MDD2.08‾\underline{2.08}2.17‾\underline{2.17}2.17‾\underline{2.17}
2C4, Control2.792.792.63\mathbf{2.63}2.96\mathbf{2.96}
3C4, MDD2.83\mathbf{2.83}2.462.462.832.83
Table 4: Mean ratings by a clinical psychologist of reasoning traces generated by Qwen2.5-VL-3B , using a 1–4 scale with 0.5-point increments ( n=12n=12 traces per group, FOR2107 ). Bold indicates the best score and underlining indicates the worst score in each metric column. See Table 1 for a description of the notation.
Cond.InputFOR2107OASIS-3
Qwen2.5-VL-3BMinistral-3-3BQwen2.5-VL-3BMinistral-3-3B
C1text(csv)0.1530.0640.2620.504
C2text(csv) ++ prompt(mri)0.7280.4800.5890.541
C2 †text(csv) ++ prompt(fmri)0.7020.3610.3790.569
C2 ‡text(csv) ++ prompt(weather)0.0560.0310.1480.425
Table 5: F1 under two preamble ablations—a factually false fMRI mention and a semantically irrelevant weather preamble—evaluated with Qwen2.5-VL-3B and Ministral-3-3B across both cohorts. Bold indicates the best score for each model and cohort.
Figure 4: Shift in mean P^(MDD)\hat{P}(\text{MDD}) relative to the text(csv) baseline (vertical) versus cosine similarity to the MRI-preamble direction (horizontal), for candidate phrases across semantic categories, evaluated with Qwen2.5-VL-3B on 15 participants from the FOR2107 MDD group. Error bars show one standard deviation across participants. Phrases in the top-right quadrant are associated with both a preamble-aligned hidden-state shift and increased mean P^(MDD)\hat{P}(\text{MDD}) without providing imaging data.
Cond.Before MPO  ↑\,\uparrowAfter MPO  ↑\,\uparrow
C10.1530.459
C20.7280.496
C40.7230.477
Table 6: F1 of Qwen2.5-VL-3B on FOR2107 before and after MPO. Random baseline F1 = 0.40. Bold indicates the best score. See Table 1 for a description of the notation.
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

ModelPrefix token IDs
GLM-4.1V-9B7302222361985152203305471788330 
GLM-4.6V-Flash7302222361985152203305471788330 
LLaVA-OV-1.5-4B7359422361985152203305471788330 
Ministral-3-3B1975835310102030103214291727828111429 
Ministral-3-14B1975835310102030103214291727828111429 
Qwen2.5-VL-3B7359422361985152203305471788330 
Appendix
Table 7: Token IDs of the JSON prefix for all evaluated models in the paper.
AbbreviationCheckpoint (Link to HF)
GLM-4.1V-9Bzai-org/GLM-4.1V-9B-Thinking
GLM-4.6V-Flashzai-org/GLM-4.6V-Flash
LLaVA-OV-1.5-4Blmms-lab/LLaVA-OneVision-1.5-4B-Instruct
Ministral-3-3Bmistralai/Ministral-3-3B-Instruct-2512-BF16
Ministral-3-14Bmistralai/Ministral-3-14B-Instruct-2512-BF16
Qwen2.5-VL-3BQwen/Qwen2.5-VL-3B-Instruct
Appendix
Table 8: Model abbreviations and hyperlinked checkpoint paths.
Figure 9: Distribution of input-token counts in the FOR2107 dataset.
Figure 10: Distribution of input-token counts in the OASIS-3 dataset.
Figure 11: Out-of-distribution images used in condition C5. Both replace the subject-specific MRI plot while the rest of the input remains unchanged.
ModelΔ\Delta F1 (C2 −- C1)Δ\Delta F1 (C4 −- C2)Δ\Delta F1 (C5 −- C4)
Qwen2.5-VL-3B+0.575 [+0.531, +0.620]−- 0.005 [ −- 0.039, +0.030]+0.005 [ −- 0.024, +0.035]
Ministral-3-3B+0.416 [+0.370, +0.461]+0.197 [+0.148, +0.247]−- 0.002 [ −- 0.045, +0.040]
InternVL3.5-4B+0.255 [+0.215, +0.295]−- 0.003 [ −- 0.059, +0.053]+0.147 [+0.090, +0.205]
GLM-4.1V-9B+0.119 [+0.068, +0.170]+0.080 [+0.035, +0.125]+0.012 [ −- 0.030, +0.052]
Appendix
Table 9: Condition-wise F1 differences with approximate 95% confidence intervals for FOR2107 , calculated from Table 11 . C2 −- C1 compares the MRI preamble with the text-only baseline, C4 −- C2 adds parcellation text and the subject-specific MRI, and C5 −- C4 replaces that MRI with an out-of-distribution image.
ModelΔ\Delta F1 (C2 −- C1)Δ\Delta F1 (C4 −- C2)Δ\Delta F1 (C5 −- C4)
Qwen2.5-VL-3B+0.327 [+0.262, +0.391]+0.006 [ −- 0.053, +0.066]+0.024 [ −- 0.033, +0.081]
Ministral-3-3B+0.037 [ −- 0.010, +0.087]+0.022 [ −- 0.019, +0.061]+0.005 [ −- 0.041, +0.052]
InternVL3.5-4B+0.020 [ −- 0.008, +0.047]+0.019 [ −- 0.014, +0.053]+0.020 [ −- 0.017, +0.055]
GLM-4.1V-9B+0.118 [+0.051, +0.184]+0.059 [ −- 0.008, +0.125]+0.067 [+0.002, +0.132]
Appendix
Table 10: Condition-wise F1 differences with approximate 95% confidence intervals for OASIS-3 , calculated from Table 12 . The three comparisons follow the definitions in Table 9 .
ModelF1PRACC
C1: text(csv)
InternVL3.5-4B0.0000.0000.0000.604
InternVL3.5-14B0.5590.9460.3970.752
GLM-4.1V-9B0.5210.9580.3580.740
GLM-4.6V-Flash0.7950.8960.7150.854
LLaVA-OV-1.5-4B0.6630.9540.5080.796
Appendix
Table 11: Full results on FOR2107 across all five input conditions (C1–C5) and all twelve models. Metrics are F1, Precision (P), Recall (R), and Accuracy (ACC). Best F1 per condition is shown in bold . Left panel: text-only and preamble/image conditions (C1–C3). Right panel: parcellation-augmented and out-of-distribution swap conditions (C4–C5).
ModelF1PRACC
C1: text(csv)
InternVL3.5-4B0.6160.4700.8930.594
InternVL3.5-14B0.7730.8650.6980.850
GLM-4.1V-9B0.2780.9090.1640.689
GLM-4.6V-Flash0.3590.9010.2240.708
LLaVA-OV-1.5-4B0.6790.5910.7990.725
Appendix
Table 12: Full results on OASIS-3 across all five input conditions (C1–C5) and all twelve models. Metrics are F1, Precision (P), Recall (R), and Accuracy (ACC). Best F1 per condition is shown in bold . Left panel: text-only and preamble/image conditions (C1–C3). Right panel: parcellation-augmented and out-of-distribution swap conditions (C4–C5). Note that Qwen3-VL-2B produces near-degenerate outputs across all conditions on this dataset, and Qwen2.5-VL-32B under C1–C2 exhibits near-constant positive predictions (ACC ≈\approx 0.40, R ≈\approx 1.00), consistent with a near-degenerate prediction strategy that has little discriminative ability.
DatasetCond.medgemma-4b-itmedgemma-27b-it
  F1PRACCF1PRACC
FOR2107C10.1300.8910.0700.6290.4670.9820.3070.724
C20.1000.9020.0530.6230.5820.9140.4270.757
C30.0031.0000.0010.6050.6250.9250.4720.776
C40.0141.0000.0070.6070.3860.9660.2410.696
C50.0031.0000.0010.6050.3220.9510.1940.677
Appendix
Table 13: Ablation of medgemma models on FOR2107 and OASIS-3 : F1, Precision (P), Recall (R), and Accuracy (ACC) across the five input conditions. Bold marks the higher F1 between the two model sizes per row. See Table 1 for a description of the notation.
Figure 12: FOR2107 per-group mean P^(MDD)\hat{P}(\text{MDD}) (top) and P^(Control)\hat{P}(\text{Control}) (bottom) across the three input conditions. Both diagnostic groups exhibit an upward shift in P^(MDD)\hat{P}(\text{MDD}) , with the complementary decrease in P^(Control)\hat{P}(\text{Control}) implied by two-class normalization. Shaded bands show one sample standard deviation around each group mean and are clipped to the [0,1][0,1] probability range.
ModelCond.ECE 15Brier
InternVL3.5-4BC10.3820.371
C20.2890.283
C40.2650.264
GLM-4.1V-9BC10.2650.265
C20.1960.196
C40.2040.204
Appendix
Table 14: Confidence estimation for InternVL3.5-4B and GLM-4.1V-9B on FOR2107 , extending the analysis in Section 5 (Table 3 ). Results are computed over all 1,772 participants (MDD and Control combined). Bold denotes the lower score for each model.
Figure 13: Logit-lens trajectory across all 36 layers of Qwen2.5-VL-3B on FOR2107 , under conditions C1, C2, and C4. The label carries negligible weight through the early and middle layers and emerges only in the final layers (onset ≈\approx layer 32). Past this onset, the preamble conditions (C2, C4) sit above the C1 baseline, showing an association between the preamble conditions and higher projected weight toward the target label. We extract activations from the layer immediately preceding the onset.
#CategoryPhrasecos⁡ ⁣↑\cos\!\uparrowδ ⁣↑\delta\!\uparrow
1• MRI / neuroimagingBrain MRI findings are available.+0.43+0.02
2• General clinicalDiagnostic results are available.+0.24+0.02
3• MRI / neuroimagingBrain scans have been performed.+0.23-0.01
4• Structural / formatClassification result:+0.00-0.02
5• General clinicalA clinical diagnosis has been established.-0.07-0.01
6• Structural / formatThe answer is:-0.25-0.09
Appendix
Table 15: Numbered phrases from Figure 4 with their scores. cos⁡\cos : cosine similarity between the hidden-state shift associated with the phrase and the MRI-preamble direction. δ\delta : change in mean P^(MDD)\hat{P}(\text{MDD}) for the FOR2107 MDD group relative to the mean P^(MDD)\hat{P}(\mathrm{MDD}) under text(csv) when the phrase replaces the MRI preamble. Higher values on both axes indicate a stronger preamble-like pattern.
Cond.InputFOR2107OASIS-3
InternVL3.5-4BGLM-4.1V-9BInternVL3.5-4BGLM-4.1V-9B
C1text(csv)0.0000.5210.6160.278
C2text(csv) ++ prompt(mri)0.2550.6400.6360.396
C2 †text(csv) ++ prompt(fmri)0.1860.6610.6520.527
C2 ‡text(csv) ++ prompt(weather)0.0000.5120.5940.339
Appendix
Table 16: Extended preamble ablation on InternVL3.5-4B and GLM-4.1V-9B , following the design of Table 5 . Bold indicates the best score for each model and cohort.
HyperparameterValue
Rank ( rr )64
Alpha ( α\alpha )128
Dropout0.05
Biasnone
Target modulesq, k, v, o, gate, up, down_proj
Appendix
Table 17: LoRA adapter configuration.
HyperparameterValue
MPO losses
Loss typessigmoid, bco_pair, sft
Loss weights0.8, 0.2, 1.0
KL penalty ( β\beta )0.1
Optimization
Learning rate5×10−55\times 10^{-5}
Appendix
Table 18: MPO training hyperparameters.
NameDescription
ProbandTest subject
Datum_InterviewDate interview as stated on the interview form
Datum_FragebogenDate of the questionnaire
GeburtsjahrYear of birth
AlterAge
GeschlechtGender
Appendix
Table 19: FOR2107 - Demographics and Clinical Information. The "Group" (“Patient grouping or diagnosis”) field is the true label for reference. We do not pass it to the model input for classification.
NameDescription
RS-25: Resilience Scale
RS251If I have plans, I follow them through.
RS252I usually manage everything somehow.
RS253I can rely on myself rather than on others.
RS254It is important for me to stay interested in many things.
RS255If I have to, I can be alone.
Appendix
Table 20: FOR2107 - Items for the Resilience Scale (RS-25) and Perceived Stress Scale (PSS).
NameDescription
FSozU1I have people who can look after my home (flowers, pets) when I’m not there.
FSozU2There are people who accept me for who I am.
FSozU3It is important for my friends/relatives to know my opinion on certain things.
FSozU4I would like more understanding and care from others.
FSozU5I have a very trusted person whose help I can always count on.
FSozU6I can borrow tools and food if necessary.
Appendix
Table 21: FOR2107 - Questionnaire on Social Support (FSozU) Items.
NameDescription
LEQ_pn1 / LEQ1Health: Serious illness of one’s own (Type of influence / Influence on life)
LEQ_pn2 / LEQ2Health: Major change in eating habits
LEQ_pn3 / LEQ3Health: Major change in sleeping habits
LEQ_pn4 / LEQ4Health: Significant change in the type or amount of leisure activities
LEQ_pn5 / LEQ5Health: Major dental procedure
LEQ_pn6 / LEQ6Health: Pregnancy
Appendix
Table 22: FOR2107 - Life Experiences Questionnaire (LEQ) Part 1. Note: ’pn’ designates Type of Influence, while the number alone designates Influence on Life.
NameDescription
LEQ_pn29 / LEQ29Love/Partnership: Entering into an engagement
LEQ_pn30 / LEQ30Love/Partnership: Problems with boyfriend or girlfriend
LEQ_pn31 / LEQ31Love/Partnership: Separation from boyfriend/girlfriend or breaking engagement
LEQ_pn32 / LEQ32Love/Partnership: Pregnancy of wife or girlfriend
LEQ_pn33 / LEQ33Love/Partnership: Miscarriage or abortion of wife or girlfriend
LEQ_pn34 / LEQ34Love/Partnership: Marriage or domestic partnership
Appendix
Table 23: FOR2107 - Life Experiences Questionnaire (LEQ) Part 2.
NameDescription
LEQ_pn56 / LEQ56Personal/Social: Greater personal achievement
LEQ_pn57 / LEQ57Personal/Social: Important decision regarding your immediate future
LEQ_pn58 / LEQ58Personal/Social: Changes in your personal habits (clothing, lifestyle, hobbies)
LEQ_pn59 / LEQ59Personal/Social: Changes in your religious beliefs
LEQ_pn60 / LEQ60Personal/Social: Changes in your political views
LEQ_pn61 / LEQ61Personal/Social: Loss or damage to your personal property
Appendix
Table 24: FOR2107 - Life Experiences Questionnaire (LEQ) Part 3.
NameDescription
SozDemo1 – SozDemo5Current living, work situation, occupation, social contacts
Haushaltu14 / Haushaltab14People in the household under/over 14 years
HaushaltsnettoHousehold net income
SchulabschlussHighest educational qualification achieved by subject
Schule_Vater / Schule_MutterHighest educational qualification achieved by father / mother
GebJahr_Mutter / GebJahr_VaterMother’s / Father’s year of birth
Appendix
Table 25: FOR2107 - Socio-Demographics, Family Composition, and Pregnancy Variables.
NameDescription
FzT: Drinking Habits
FzT1 – FzT5Questions on frequency and quantity of alcohol consumption (now/past).
FzT6 – FzT9Questions on binge drinking (6+ drinks) and inability to stop (now/past).
FzT10 – FzT13Failing expectations due to alcohol; morning drinking (now/past).
FzT14 – FzT17Guilt, remorse, and memory loss due to alcohol (now/past).
FzT18 – FzT20Injuries, advice to reduce, and professional help sought for alcohol.
Appendix
Table 26: FOR2107 - Alcohol Use (FzT).
NameDescription
Subject Demographics & Family
OASISIDOASIS subject ID
GENDERSubject’s gender
RACESubject’s race
HANDSubject’s Handedness
TWINIs this a new informant? (Note: Label suggests ’Twin’ but description asks about new informant status)
Appendix
Table 27: OASIS-3 - Subject Demographics, Family, and Living Situation.
NameDescription
INSEXInformant’s sex
INHISPDoes informant report being of Hispanic/Latino ethnicity?
INHISPORIf yes, what are the informant’s reported origins?
INRACEWhat does informant report as his/her race?
INRASECWhat additional race does informant report?
INRATERWhat additional race, beyond what was indicated above, does informant report?
Appendix
Table 28: OASIS-3 - Informant Demographics and Contact Frequency.
NameDescription
General Medical History
HYPERTENHypertension
HYPERCHOHypercholesterolemia
DIABETESDiabetes
B12DEFB12 deficiency
THYROIDThyroid Disease
Appendix
Table 29: OASIS-3 - Subject Medical History, Cardiovascular Conditions, and Vitals.
NameDescription
Lifestyle, Sleep & Trauma
TOBAC100Smoked more than 100 cigarettes in life?
SMOKYRSTotal years smoked
ALCOHOLSubstance abuse - alcohol (clinically significant over a 12 month period)
TRAUMEXTTraumatic brain injury with extended loss of consciousness ( ≥\geq 5 minutes)
TRAUMCHRTraumatic brain injury with chronic deficit or dysfunction
Appendix
Table 30: OASIS-3 - Lifestyle Factors, Sensory Capabilities.

Explore similar work

CardsList

  1. Medical Context Distorts Decisions in Clinical Vision Language Models

    May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2VLM EvaluationVLM Robustness

    1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1

  2. Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI

    Aug 3, 2026Amir Sabbaghziarani, Mohammadsajad Abavisani, Sergey PlisVLM EvaluationVLM Hallucination

    TReNDS Center Atlanta, GA, USA

  3. NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

    Mar 25, 2026Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska +5Medical VLMsMultimodal Model Evaluation

    Faculty of computer science and engineering, Ss. Cyril and Methodius University, Skopje, North Macedonia.