Medical Vision Language Models (VLMs) can change their answers when clinicians rephrase the same question, a failure mode that threatens deployment safety. We introduce PSF-Med, a benchmark of 26,850 chest X-ray questions paired with 92,856 meaning-preserving paraphrases across MIMIC-CXR, PadChest, and VinDr-CXR, spanning clinical populations in the US, Spain, and Vietnam. Every paraphrase is validated by an LLM judge using a bidirectional clinical entailment rubric, with 91.6% cross-family agreement. Across nine VLMs, including general-purpose models, we find flip rates from 3% to 37%. However, low flip rate does not imply visual grounding: text-only baselines show that some models stay consistent even when the image is removed, suggesting they rely on language priors. To study mechanisms in one model, we apply GemmaScope 2 Sparse Autoencoders (SAEs) to MedGemma 4B and analyze FlipBank, a curated set of 158 flip cases. We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts. In causal patching, removing this feature's contribution recovers 45% of the yes-minus-no logit margin on average and fully reverses 15% of flips. Acting on this finding, we show that clamping the identified feature at inference reduces flip rates by 31% relative with only a 1.3 percentage-point accuracy cost, while also decreasing text-prior reliance. These results suggest that flip rate alone is not enough; robustness evaluations should test both paraphrase stability and image reliance.
Figures & tables
Figure 1: The robustness-grounding trade-off. Models with low flip rates (left) often show high text-only agreement, meaning they ignore the image. Models that attend to visual evidence (right) can be more sensitive to phrasing. Evaluations should measure both.
MIMIC
PadChest
VinDr
Total
Images
999
4,523
4,263
9,785
Questions
3,266
14,935
8,649
26,850
Paraphrases
8,938
59,573
24,345
92,856
Para/question
2.7
4.0
2.8
3.5
Table 1: PSF-Med v2 dataset statistics after LLM-judge filtering. The benchmark spans three clinical populations with validated paraphrases.
MIMIC
PadChest
VinDr
Model
Acc
Flip
Acc
Flip
Acc
Flip
Targeted LoRA
89.9
4.2
82.1
7.5
74.7
8.5
Full LoRA
84.3
3.3
40.0
10.3
53.5
5.8
MedGemma-4B
88.3
6.7
59.0
13.1
57.2
12.7
MedGemma-27B
76.7
10.6
55.3
17.8
57.6
12.4
MedGemma-1.5-4B
–
11.5
73.2
12.0
63.8
12.0
Table 2: Accuracy (%) and per-paraphrase flip rates (%) on judge-filtered paraphrases. Lower flip rates indicate greater consistency. Results marked ∗ use 500 hardest questions; † denotes degenerate yes-bias.
Figure 2: Flip rates by paraphrase transformation type. Negation-adjacent paraphrases (presence/absence framing changes that preserve meaning) cause the highest flip rates across all models, while simple lexical substitutions are most robust. Error bars show 95% bootstrap confidence intervals.
Figure 3: Question embedding distance predicts flips. Left: Cosine similarity distributions for flip (orange) vs no-flip (blue) cases. Right: Euclidean distance. Flipped pairs show lower similarity and higher distance, though effect size is small.
Model
Flip †
Text-Only
Swap Sens.
MedGemma-4B
18.2%
66.4%
30.8%
MedGemma-27B
9.4%
85.0%
19.6%
CheXagent
28.1%
71.2%
26.4%
† Flip rate with real images.
Table 3: Visual grounding analysis on MIMIC-CXR presence questions (N=2,499). Text-Only Agree measures answer stability when image is removed. Swap Sens. measures answer change when a different image is shown.
Metric
Flip Cases
No-Flip Cases
GT Coverage
8.4%
14.2%
Precision
10.6%
29.0%
Table 4: Attention-bounding box analysis on PadChest (N=200 samples with ground truth boxes). Flip cases show substantially lower attention to pathology regions.
Figure 4: Robustness vs grounding trade-off. MedGemma-27B achieves low flip rates but high text-only agreement; MedGemma-4B shows stronger visual dependence but higher flip rates.
Question Phrasing
Feat. 3818
Response
“Is there evidence of X?”
302–386
Conservative
“Is X present?”
280–340
Conservative
“Can you see X?”
0–50
Permissive
“Does this show X?”
0–50
Permissive
Table 5: Feature 3818 activation by question phrasing. The feature activates for formal clinical language and deactivates for casual phrasing.
Question
F3818
“Evidence” in both:
“Is there radiographic evidence of X?”
342
“Any evidence of X here?”
68
Same content, different register:
“Is cardiomegaly present?”
312
“Big heart?”
23
Table 6: Token-controlled test for Feature 3818. The feature responds to formality register, not specific tokens.
Table 14: Compute requirements for main experiments.
Model
HuggingFace Identifier
MedGemma-4B
google/medgemma-4b-it
MedGemma-27B
google/medgemma-27b-it
MedGemma-1.5-4B
google/medgemma-1.5-4b-it
LLaVA-Rad
microsoft/llava-rad-v1
RadFM
chaoyi-wu/RadFM
CheXagent
stanford-crfm/chexagent-2-3b
Appendix
Table 15: Model versions used in evaluation.
Statistic
Value
Total flip cases
158
Mean embedding similarity
0.97
Yes → No flips
94 (59%)
No → Yes flips
64 (41%)
Unique images
142
Unique findings queried
23
Appendix
Table 16: FlipBank statistics.
Original Question
Paraphrase
Orig.
Para.
“Does this X-ray show a pneumothorax?”
“Is there radiographic evidence of a collapsed lung?”
Yes
No
“Can you see cardiomegaly?”
“Is there evidence of an enlarged cardiac silhouette?”
Yes
No
“Is there pleural effusion?”
“Does the image demonstrate fluid in the pleural space?”
Yes
No
“Does this show atelectasis?”
“Is there evidence of lung collapse?”
No
Yes
“Is the lung hyperinflated?”
“Does this chest X-ray show signs of hyperinflation?”
No
Yes
Appendix
Table 17: Representative flip cases from MedGemma 4B on MIMIC-CXR. All cases show semantically equivalent questions receiving opposite binary answers.
Figure 6: Cross-model flip correlation matrix. Higher values indicate that the same questions cause flips in both models. Models within the same family (MedGemma variants) show higher correlation, suggesting shared failure modes. RadFM and LLaVA-Rad show lowest correlation with other models.
Table 19: Feature 3818 activation strength by layer. The feature is most active at layers 15–19, with peak activation at layer 17.
Model / Finding
Text-Only Agree
Image Swap Sens.
Blank-Image Flip Rate
MedGemma-4B
Presence questions
68.2%
28.4%
44.3%
Location questions
61.5%
35.2%
38.7%
Severity questions
72.1%
22.8%
41.2%
MedGemma-27B
Presence questions
84.3%
20.1%
15.2%
Appendix
Table 20: Text-only baseline detailed results by model and question type. Text-Only Agree and Image Swap Sensitivity use the same definitions as Table 3 . Blank-Image Flip Rate is the paraphrase flip rate measured when real images are replaced with blank input, isolating text-driven inconsistency; this is a different metric from the standard flip rate in Tables 2 – 3 .
Phenomenon
Mean Distance
Std Distance
Flip Rate
Lexical substitution
0.237
0.046
7.7%
Scope/quantification
0.229
0.057
10.4%
Specificity modulation
0.250
0.048
7.2%
Syntactic restructuring
0.253
0.043
8.3%
Negation pattern
0.274
0.035
39.4%
Appendix
Table 21: Embedding distance by paraphrase phenomenon. Negation patterns show both the largest embedding distances and highest flip rates, suggesting that semantic distance in embedding space partially explains flip likelihood.
Model
MIMIC-CXR
PadChest
MedGemma-4B
3.2%
4.1%
MedGemma-27B
2.8%
3.4%
MedGemma-1.5-4B
2.1%
2.9%
CheXagent
5.4%
6.2%
LLaVA-Rad
8.7%
11.3%
RadFM
12.1%
14.6%
Appendix
Table 22: Exclusion rates by model and dataset. Higher rates indicate more ambiguous model outputs.
Paraphrases
N Questions
Flip Rate
3
1,247
12.4%
4
2,018
15.1%
5
1,733
18.2%
Appendix
Table 23: Flip rate by number of paraphrases per question (MedGemma-4B on MIMIC-CXR).
Model
Accuracy
Text-Only Agree
MedGemma-4B
71.2%
66.4%
MedGemma-27B
74.8%
85.0%
CheXagent
68.4%
71.2%
Appendix
Table 24: Ground-truth agreement on MIMIC-CXR presence questions with verified labels (N=1,842).
Original Question
Normalized Form
“Does this X-ray show a collapsed lung?”
“Is pneumothorax present in this chest radiograph?”
“Can you see any signs of fluid buildup?”
“Is pleural effusion present in this chest radiograph?”
“Is there radiographic evidence of cardiomegaly?”
“Is cardiomegaly present in this chest radiograph?”
“Big heart?”
“Is cardiomegaly present in this chest radiograph?”
Appendix
Table 25: Prompt normalization examples. Each original question is mapped to a canonical clinical phrasing.
Paraphrase Type
Baseline Flip Rate
Clamped Flip Rate
Relative Δ
Negation-adjacent
28.4%
17.6%
−38%
Syntactic restructuring
12.1%
9.4%
−22%
Lexical substitution
7.2%
5.8%
−19%
Register change
18.7%
12.1%
−35%
Appendix
Table 26: Effect of feature clamping on flip rate by paraphrase transformation type. Clamping is most effective on negation-adjacent paraphrases, consistent with Feature 3818’s role in modulating conservative versus permissive responding.
Component
Time (ms)
% of Total
Baseline inference (no intervention)
624
(base)
+ SAE encode (layer 17)
+8.2
+1.3%
+ Feature read & subtract
+0.4
+0.1%
+ Continue forward pass
+3.1
+0.5%
Total with clamping
636
+1.9%
Appendix
Table 27: Latency overhead of feature clamping intervention. Measurements on NVIDIA A100-80GB with batch size 1.
Method
Flip Rate
Accuracy
Text-Only Agree
Swap Sens.
Baseline
42.4%
69.1%
58.7%
38.2%
+ Random feature clamp (control)
42.2%
69.0%
58.6%
38.3%
+ Feature 3818 clamp
33.8%
67.6%
55.4%
40.1%
+ Prompt normalization
35.2%
68.4%
58.1%
37.8%
+ Feature 3818 clamp + prompt norm.
30.5%
66.8%
54.8%
40.9%
Appendix
Table 28: Mitigation results on PadChest (N=3,812 questions, ∼ 18k paraphrase pairs). Feature clamping generalizes across datasets with a smaller but significant effect.
Medical Vision-Language Models can give different yes or no answers to rephrasings of the same clinical question. We study this in MedGemma-4B using PSF-Med Sadanandan and Behzadan (2025), which provides paraphrase pairs for systematic consistency evaluation on medical VQA. On MIMIC-CXR binary questions (n = 158), the baseline flip rate is 14.6% and mean margin difference is 1.63 logits. We validate that Gemma Scope 2 Sparse Autoencoders (SAEs) transfer to MedGemma activations, achieving R2 ~= 0.997 on both medical and general text (n = 100 prompts each, p < 0.001 for exceeding a 0.95 threshold). We then fine-tune Low-Rank Adaptation (LoRA) adapters with a combined loss that balances paraphrase consistency with answer accuracy. This combined approach prevents mode collapse that occurs with pure consistency training while reducing flip rate from 14.6% to 4.4% (p = 0.002, two-proportion z-test) and margin difference from 1.63 to 0.33 (79.5% reduction). Accuracy remains stable at 84.2% baseline versus 82.3% after training (-1.9pp, not significant). On PadChest Balanced (n = 250), flip rate drops from 13.6% to 7.8%, mean margin difference drops from 1.08 to 0.35 (67.9% reduction), and accuracy increases from 66.4% to 69.4%. A layer-range ablation shows that early layers reduce margin differences more than mechanistically selected middle layers.
Medical Vision-Language Models (VLMs) answering binary presence questions on chest radiographs can fail in two linked ways: they are confidently wrong, and they change answers when a clinically equivalent question is rephrased. In a binary answer head both failures track the same logit margin, so we test how well one score screens for both. On MedGemma-4B-IT across MIMIC-CXR (in-distribution, n=196) and PadChest (out-of-distribution, n=861), single-pass predictive entropy predicts which yes/no answers flip under rephrasing with an area under the receiver operating characteristic curve (AUROC) of 0.823 on PadChest and 0.821 on MIMIC, and the signal is stable across five random seeds (0.828±0.021). The same signal replicates on LLaVA-RAD-7B (AUROC 0.83) and is invariant to the flip definition (all, operator-preserving, or negation-excluded rewrites), so the screen needs no semantic filter at inference. One low-entropy threshold drives error and flip rate down together (3.3% and 5.7% at 80% coverage), and threshold selection transfers to a held-out split. Under cross-site shift the more expensive methods we test add nothing practical: at the 5% risk target a single forward pass answers 88.5% of cases, matching Monte Carlo (MC) Dropout, and beats the uniformly averaged five-seed adapter ensemble on error detection by 0.111 AUROC. Even a model that is 91% accurate still contradicts itself on 13% of items under rephrasing, and one forward pass flags them.
Binesh Sadanandan, Vahid Behzadan
SAIL Lab, University of New Haven, West Haven, CT, USA
Consistency under paraphrase, the property that semantically equivalent prompts yield identical predictions, is increasingly used as a proxy for reliability when deploying medical vision-language models (VLMs). We show this proxy is fundamentally flawed: a model can achieve perfect consistency by relying on text patterns rather than the input image. We introduce a four-quadrant per-sample safety taxonomy that jointly evaluates consistency (stable predictions across paraphrased prompts) and image reliance (predictions that change when the image is removed). Samples are classified as Ideal (consistent and image-reliant), Fragile (inconsistent but image-reliant), Dangerous (consistent but not image-reliant), or Worst (inconsistent and not image-reliant). Evaluating five medical VLM configurations across two chest X-ray datasets (MIMIC-CXR, PadChest), we find that LoRA fine-tuning dramatically reduces flip rates but shifts a majority of samples into the Dangerous quadrant: LLaVA-Rad Base achieves a 1.5% flip rate on PadChest while 98.5% of its samples are Dangerous. Critically, Dangerous samples exhibit high accuracy (up to 99.6%) and low entropy, making them invisible to standard confidence-based screening. We observe a negative correlation between flip rate and Dangerous fraction (r = -0.89, n=10) and recommend that deployment evaluations always pair consistency checks with a text-only baseline: a single additional forward pass that exposes the false reliability trap.
Binesh Sadanandan, Vahid Behzadan
SAIL Lab, University of New Haven, West Haven, CT, USA