MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models

Junhyeok Lee; Songsoo Kim; Kyu Sung Choi

cs.CLAug 25, 2026

MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models

Authors: Junhyeok Lee, Songsoo Kim, Kyu Sung Choi

Organizations: Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine · Department of Radiology, Seoul National University Hospital · Department of Radiology, Seoul National University College of Medicine · Healthcare AI Research Institute, Seoul National University Hospital

Abstract

Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging. Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image. We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation. Each case fixes the current image and target finding while presenting matched reliable and misleading context across text and prior CXR, with visual overlays where available. MC-CXR defines three task families and two paired metrics, the switch-to-wrong rate and the context-aligned error rate. We evaluate ten VLMs spanning open-source general, medical-domain, and closed-source systems. Image-only accuracy is necessary but insufficient. Mean switch rates range from 45.6-78.1% across misleading textual sources and 35.7-61.7% across misleading visual sources. Among switched predictions, 74.6% align with the misleading label for text versus 17.6% for visual context, a 57.0-point gap (95% CI 50.9-62.8). This text-visual asymmetry is observed under the standardized direct-answer protocol. The dataset is available on PhysioNet.

Figures & tables

Figure 1: Overview of context-robust CXR evaluation and MC-CXR . Clinical image interpretation often combines current-image evidence with auxiliary context. We formalize context-robust visual reasoning into three task families, Image-only Recognition (IOR), Reliable Context Stability (RCS), and Misleading Context Resistance (MCR), and evaluate them across textual, prior-image, and visual-overlay perturbations.

Benchmark	Primary task	Disrupt.	Ind.	Img.	Rep.	Note	Overlay
ReXVQA ( Pal et al., 2025 )	Multi-task CXR VQA	–	–	–	–	–	–
MS-CXR-T ( Bannur et al., 2023 )	Temporal progression	–	–	✓	✓	–	–
MI-CXR ( Cho et al., 2026 )	Longitudinal reasoning	–	–	✓	✓	–	–
MC-CXR (ours)	Context-robust reasoning	✓	✓	✓	✓	✓	✓

Table 1: Comparison of MC-CXR with existing medical VQA and longitudinal CXR benchmarks. Columns indicate whether each benchmark evaluates paired context-induced disruption and which auxiliary inputs it supports across indication, prior image, prior report, preliminary note, and overlay. Among the benchmarks compared here, MC-CXR combines all five context sources with explicit disruption metrics.

Figure 2: Overview of MC-CXR construction. We derive eligible cases from MIMIC-CXR and MIMIC-CXR-JPG, select target and misleading labels through radiologist review, and instantiate matched reliable and misleading context conditions. The resulting cohort supports paired evaluation of reliable-context stability and misleading-context resistance.

	Image-only		Reliable context					Misleading context
			Textual			Visual		Textual			Visual
Model	IOR	$\kappa$	ind	rep	note	img	overlay	ind	rep	note	img	overlay
InternVL3-8B	26.7	0.165	26.2	27.9	70.8	23.8	22.7	12.9	11.2	6.2	22.9	10.6
Qwen3.5-9B	30.0	0.198	37.1	37.9	68.8	32.9	28.8	25.0	17.5	9.2	30.8	23.6
Gemma-3-12B-IT	9.2	0.001	20.8	25.8	68.8	8.8	15.3	5.8	3.8	4.6	10.0	6.0
Llama-3.2-11B-V	8.3	0.008	22.9	28.3	58.3	7.5	15.3	5.4	4.6	3.3	8.8	11.1

Table 2: Absolute accuracy across context conditions for the ten evaluated VLMs, together with image-only recognition (IOR) accuracy and Cohen’s

\kappa

as model-level summaries. These columns are evaluated without IOR-correct conditioning and therefore reflect conventional target-label classification accuracy. The high current-report-derived preliminary-note column reflects by-construction target evidence, whereas the low misleading-context columns reflect context-induced disruption under misleading conditions. Bold and underline mark the best and second-best result per column, respectively (higher is better).

	Textual			Visual
Model	ind	rep	note	img	overlay
InternVL3-8B	63.9	67.4	83.9	20.0	22.5
Qwen3.5-9B	73.7	64.5	93.0	33.3	20.0
Gemma-3-12B-IT	57.9	65.0	86.4	9.1	5.6
Llama-3.2-11B-V	85.7	68.8	80.0	20.0	22.2
Phi-4-multimodal	70.0	58.6	79.4	4.2	16.0

Table 3: Directional

Y

-alignment among switches induced by misleading context across the ten evaluated VLMs. Results report the fraction of switched errors that land on the context-implied label

Y

for each misleading source. Textual conditions show systematic directional pull, whereas visual conditions produce lower and more scattered alignment. Bold and underline mark the best and second-best result per column, respectively, and lower values are better.

Figure 3: Switch-to-wrong rate across context conditions on the IOR-correct subset for the ten evaluated VLMs (lower is better). Coloured dots are per-model rates and black stars mark the mean across models. The rewritten preliminary-note condition produces the strongest misleading-text disruption, the current-report-derived preliminary-note condition is the most stability-preserving reliable context, and the off-finding overlay condition shows strong visual perturbation without corresponding directional alignment.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Alias	SDK	Pinning
Claude Opus 4.7	claude-opus-4-7	anthropic	alias
GPT-5.5	gpt-5.5	openai	alias
Gemini 3.5 Flash	gemini-3.5-flash	google-genai	alias

Appendix

Table 4: Closed-source API provenance. All three providers were queried in May 2026 through service aliases; no immutable backend snapshot identifier was exposed in the response envelope. Decoding used temperature=0, max_tokens=1024, minimal reasoning effort.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2VLM Evaluation VLM Robustness
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays
Jun 19, 2026Geon Choi, Hangyul Yoon, Nalee Kim +5VLM Evaluation Medical Imaging
1KAIST · 2Samsung Medical Center · 3Konkuk University Medical Center
CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs
May 7, 2026Zhengru Fang, Yanan Ma, Yu Guo +5VLM Evaluation Chest X-Ray Classification
City University of Hong Kong · Tsinghua Shenzhen International Graduate School

May 17, 2026J/K move · Enter open · S save

Medical Context Distorts Decisions in Clinical Vision Language Models

David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis+1

1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK+1

VLM Evaluation VLM Robustness Prompt Sensitivity Radiology VLMs Vision-Language Models Healthcare+1

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios that require integrating both visual and textual context from medical records remains poorly characterized. This paper identifies three failure modes: (1) modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs. We evaluate a diverse set of general-domain and medically-tuned open and closed VLMs on chest x-ray tasks using MIMIC-CXR. By systematically manipulating image-text alignment, clinical history, and prompt formulations, we found that VLM decisions are dominated by the text modality, even when visual evidence is available. Moreover, we observed that VLMs are heavily influenced by irrelevant reports, while minor prompt changes can reverse correct image-based predictions. Our findings underscore the need for explicit safeguards and stress-testing before considering the use of these models in clinical practice.

Abstract

VLM Evaluation

Explore similar work

Medical Context Distorts Decisions in Clinical Vision Language Models

CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs