Evaluating Cross-Signal Coherence in Mammography Vision-Language Models
Authors: Rebeca Nam, Chase M. Fensore, Christian A. Fauria-Robinson, Judy Gichoya, Joyce C. Ho
Organizations: Yonsei University, 50 Yonsei-ro, Seodaemun-gu, 03722, Seoul, Republic of Korea. · Department of Computer Science, Emory University, 400 Dowman Drive, Atlanta, 30322, GA, USA. · Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, 30322, GA, USA.
As vision-language models (VLMs) move toward interpreting medical images directly, their outputs increasingly include not just a label but a stated confidence and a written justification. Yet the field still predominantly evaluates VLMs using just accuracy. We evaluate the coherence between answer, confidence, and explanation across four VLMs on breast density and BI-RADS categorization, using two mammography datasets. We perform controlled perturbations of the question, multiple choice option orders, and the images and assess changes in the three outputs. Our results suggest confidence does not decrease even as the mammography image is changed. Shuffling the options also changes most VLMs' answers. Notably, there are cases where a model's own stated reasoning contradicts its final answer, a distinction accuracy alone cannot capture but one that governs whether an AI-generated answer can be trusted.
Figures & tables
Figure 1: Our proposed evaluation methodology with six conditions: the original image and five perturbations.
Figure 2: Prompt used to jointly elicit an explicit answer, per-option confidence scores, and reasoning in a single response.
Model
Output acc. %
NOTA sel. %
RSNA – BI-RADS (6 true classes; majority 59.4%)
MedGemma-4B
42.4
0.0
Lingshu-7B
12.0
0.0
Qwen2-VL-7B
2.9
62.8
InternVL3-8B
7.7
13.7
RSNA – Density (ACR) (4 true classes; majority 42.6%)
Table 1: Per-task output accuracy and “None of the above” selection rate, with the majority-class prior. The answer is the model’s explicitly stated single-best choice.
Figure 3: Answer × reasoning coherence on RSNA. Each item is classified by whether the explicit answer y^=y and the reasoning answer y^e=y : A (both correct), B (answer correct, reasoning wrong), C (answer wrong, reasoning correct; the disconnect ), D (both wrong).
Figure 4: Two models on the same RSNA mammogram (ground truth BI-RADS 0). Each model’s stated answer agrees with the category implied by its explanation, yet the correct answer (InternVL, BI-RADS 0) rests on a self-contradictory rationale—it calls the exam “normal” (a BI-RADS 1 finding) while assigning BI-RADS 0—whereas the incorrect answer (MedGemma, BI-RADS 1) follows validly from the same normal reading.
Figure 5: Mean confidence on the condition-correct answer, cy (solid) the original category on real-image conditions, NOTA on the visual perturbations vs. on the explicitly chosen answer, cy^ (dashed), across the six RSNA conditions.
Model
Real %
Visual %
Δ (pp)
BI-RADS
MedGemma-4B
0.0
0.0
+0.0
Lingshu-7B
0.0
20.6
+20.6
Qwen2-VL-7B
62.8
91.7
+28.9
InternVL3-8B
13.7
91.0
+77.3
Density (ACR)
Table 2: RSNA “None of the above” selection rate on the real image vs. its average over the three visual perturbations ( unrelated / blank / no image ), where NOTA is the correct response, with Δ= Visual − Real.
Figure 6: Perturbed-Options condition effect on RSNA. Bars = share of items whose stated answer changes when the options are reordered, split by effect on accuracy.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Hugging Face checkpoint
Domain
Params
MedGemma-4B
google/medgemma-4b-it
Medical
4B
Lingshu-7B
lingshu-medical-mllm/Lingshu-7B
Medical
7B
Qwen2-VL-7B
Qwen/Qwen2-VL-7B-Instruct
General
7B
InternVL3-8B
OpenGVLab/InternVL3-8B
General
8B
Appendix
Table 3: Model configurations. All models use the default (latest) Hugging Face checkpoint revision, bfloat16 precision (no quantization), greedy decoding (temperature 0 ), and a 384-token generation budget.
Cohort
Task
N
Majority-class prior
RSNA
BI-RADS
5,106
59.4%
RSNA
Density
5,821
42.6%
EMBED
BI-RADS
14,303
62.1%
Appendix
Table 4: Cohort and task composition. Majority-class prior is the accuracy of always predicting the most frequent label.
Model
Task
Parsed %
Answer acc. %
Reasoning acc. %
Δ (pp)
MedGemma-4B
BI-RADS
99.8
42.4
35.8
−6.7
Density
94.1
33.8
33.5
−0.3
Lingshu-7B
BI-RADS
97.6
12.3
16.6
+4.3
Density
92.2
48.4
48.3
+0.0
Qwen2-VL-7B
BI-RADS
66.3
3.7
7.1
+3.4
Density
75.9
32.5
32.9
+0.5
Appendix
Table 5: Explicit answer vs. reasoning accuracy on the parsable fraction of RSNA results. Both accuracies are computed only on items whose explanation yields a readable category, so the comparison within each row is paired. Parse rate varies across model × task cells, from 66% (Qwen2-VL-7B, BI-RADS) to ∼100% (MedGemma-4B, BI-RADS); low-parse cells rest on a correspondingly smaller subset. The gap Δ=(reasoning−answer) is small and mixed in sign ( −6.7 to +4.3 ): the free-text rationale is no more accurate than the explicitly stated answer.
Figure 7: Example explanations for the four reasoning-parser outcomes: BI-RADS label, ACR-density label, “None of the above,” and unparsable. Highlighted text is the phrase the parser matches.
Comparison
Agreement %
κ
Matcher vs. human
80.8
0.777
Judge vs. human
82.5
0.798
Matcher vs. judge
78.3
0.753
Appendix
Table 6: Reasoning-extraction agreement on 120 hand-labeled RSNA explanations.
Model
Task
Agreement %
MedGemma-4B
BI-RADS
93.5
Density
74.5
Lingshu-7B
BI-RADS
58.0
Density
66.7
Qwen2-VL-7B
BI-RADS
84.9
Density
59.1
Appendix
Table 7: Matcher vs. LLM-judge agreement at scale ( 1,000 explanations per task, judge blinded). Overall ≈70.6% .
Figure 8: NOTA-selection rate (left) and mean confidence when NOTA is chosen (right), for BI-RADS (top) and Density/ACR (bottom), across all six conditions. Shaded band marks the three conditions where NOTA is the correct response.
Figure 9: Effect of rewording the question, on items whose wording actually changed. Each bar is the share of items whose answer changed, split by effect on accuracy. The cross-hatched portions are the share that changed to a NOTA. Numbers above the bars give the net accuracy change (percentage points).
Model
Task
Pert.
Orig.%
Pert.%
C → W
W → C
Net Δ
MedGemma-4B
BI-RADS
Question
46.0
45.6
88
75
−0.3
BI-RADS
Options
42.5
17.7
1685
431
−24.8 *
Density
Question
31.5
26.0
544
303
−5.4 *
Density
Options
33.9
21.7
1406
697
−12.2 *
Lingshu-7B
BI-RADS
Question
10.5
10.0
52
36
−0.4
BI-RADS
Options
14.3
5.5
433
68
−8.9 *
Appendix
Table 8: McNemar test results for the two elicitation perturbations (reworded question, shuffled options), per model and task, on the explicitly stated answer. Density refers to ACR breast density. C → W and W → C are item counts; Net Δ is the percentage-point change in accuracy. *: p<0.05 .
Model
Parsed %
Answer acc. %
Reasoning acc. %
Δ (pp)
MedGemma-4B
96.3
36.8
20.9
−15.9
Lingshu-7B
99.8
10.3
57.6
+47.3
Qwen2-VL-7B
91.4
15.4
2.3
−13.1
InternVL3-8B
82.4
4.8
4.7
−0.1
Appendix
Table 9: Explicit answer vs. reasoning accuracy on the parsable fraction of EMBED BI-RADS, both computed on items with parseable reasoning (paired). Parsed % is relative to the ∼14.3 k item set; Δ= reasoning − answer.
Figure 11: Mean confidence on the condition-correct answer cy (solid; original category on real images, NOTA on the visual perturbations) vs. the explicitly chosen answer cy^ (dashed), across the six EMBED (BI-RADS) conditions.
Figure 12: NOTA-selection rate on the real image vs. junk images (EMBED, BI-RADS). Evidence-driven NOTA should rise under junk; most models do not.
Figure 13: Perturbation sensitivity (EMBED, BI-RADS): question wording (left) vs. option shuffling + fillers (right). Option shuffling flips most answers without raising accuracy.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Joint Georgia Institute of Technology / Georgia State University / Emory University Center for Translational Research in Data Science and Neuroimaging · Georgia State University