Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models

Xinye Yang; Zhusi Zhong; Scott Collins; Grayson Baird; Xuyu Wang; Zhicheng Jiao

eess.IVOct 1, 2026

Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models

Authors: Xinye Yang, Zhusi Zhong, Scott Collins, Grayson Baird, Xuyu Wang, Zhicheng Jiao

Organizations: Department of Computer Science Brown University Providence, RI, USA · Brown University Brown University Health Providence, RI, USA · School of Computing and Information Sciences Florida International University Miami, FL, USA

Abstract

Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies treat these factors in isolation. We introduce a reliability stress test for chest X-ray interpretation built on two balanced datasets (a private report-backed set and a curated MIMIC subset). Three medical VLMs (CheXagent, MedGemma-4B, and MedGemma-27B) are evaluated across three prompt styles and two workflows (single-VLM and multi-agent), producing 36 configurations. We show that exact-match accuracy alone can overstate the effectiveness of conservative models that default to "Normal" predictions. Diagnostic reliability also depends heavily on model family and scale: multi-agent reasoning helps some configurations but hurts others. Building on these observations, we propose a decision-time routing framework that selectively escalates to multi-agent inference only when beneficial, improving the cost-quality trade-off over fixed workflows. Our results highlight the need for evaluation protocols that jointly consider prompt sensitivity, failure-mode diversity, and workflow choice before clinical deployment.

Figures & tables

Dataset	Cases	Normal	Abnormal	Reports
Private formal	50	25	25	50
MIMIC formal	50	25	25	50

TABLE I: Dataset Summary.

Fig. 1: Decision-time routing framework for chest X-ray interpretation. A single-VLM pass produces initial predictions and routing features. The decision router then either returns the low-cost output directly or escalates the case to a multi-agent pipeline with extraction, formulation, and synthesis stages.

Model	Prompt	Workflow	Exact	Partial	Macro F1	Hall FP	Normal Acc	Abn Partial	Latency (s)	Tokens
CheXagent	structured	single	6.0%	30.0%	0.118	0.351	12.0%	48.0%	1.32	22.2
CheXagent	structured	multi	24.0%	32.0%	0.160	0.089	40.0%	24.0%	6.30	144.5
CheXagent	concise	single	6.0%	28.0%	0.093	0.380	12.0%	44.0%	1.44	23.9
CheXagent	concise	multi	38.0%	42.0%	0.127	0.073	72.0%	12.0%	3.96	70.9
CheXagent	labels-only	single	6.0%	28.0%	0.094	0.367	12.0%	44.0%	1.37	23.3
CheXagent	labels-only	multi	32.0%	46.0%	0.127	0.125	64.0%	28.0%	2.92	49.7

TABLE II: Comprehensive Formal Results on the Curated MIMIC Subset.

Model	Prompt	Workflow	Exact	Partial	Macro F1	Hall FP	Normal Acc	Abn Partial	Latency (s)	Tokens
CheXagent	structured	single	36.0%	78.0%	0.436	0.147	64.0%	92.0%	0.95	13.7
CheXagent	structured	multi	52.0%	62.0%	0.317	0.038	96.0%	28.0%	4.58	54.2
CheXagent	concise	single	32.0%	74.0%	0.429	0.182	56.0%	92.0%	1.11	15.8
CheXagent	concise	multi	54.0%	62.0%	0.217	0.044	100.0%	24.0%	2.95	36.9
CheXagent	labels-only	single	46.0%	82.0%	0.457	0.129	84.0%	80.0%	1.00	11.9
CheXagent	labels-only	multi	50.0%	56.0%	0.118	0.049	100.0%	12.0%	2.61	21.9

TABLE III: Comprehensive Formal Results on the Private Dataset.

Fig. 2: Routing trade-off surface: macro F1 versus average latency for different routing strategies.

Fig. 3: Comparison of exact-match and macro F1 scores. High exact-match scores alone can mask poor clinical coverage, because conservative models achieve them by defaulting to “Normal” while producing low macro F1.

Fig. 4: Performance difference (

\Delta

Macro F1) between multi-agent and single-VLM workflows. The direction and magnitude of the change depend strongly on the base model and dataset.

Dataset	Comparison	Setting Pair	$\Delta$ Macro F1	$\Delta$ Abn Partial	Primary Failure Signal
MIMIC	Multi vs. Single	MedGemma-27B, structured	+0.174	+0.362	Mitigation of residual normal overcalls
Private	Single vs. Multi	CheXagent, labels-only	+0.295	+0.678	Abnormal-to-normal pathological collapse
MIMIC	Model contrast (multi)	MedGemma-27B vs. CheXagent, structured	+0.160	+0.325	Reduced false-positive propensity in MedGemma-27B

TABLE IV: Key comparisons showing configuration-dependent performance shifts and failure types.

Configuration	Routing Strategy	Macro F1	Latency (s)	Tokens	Multi-Agent Esc. (%)
MIMIC / MedGemma-27B / structured	always-single	0.138	4.86	104.3	0.0
MIMIC / MedGemma-27B / structured	always-multi	0.335	6.27	135.1	100.0
MIMIC / MedGemma-27B / structured	oracle	0.338	5.08	109.0	20.0
MIMIC / MedGemma-27B / structured	heuristic A	0.138	4.86	104.3	0.0
MIMIC / MedGemma-27B / structured	heuristic B	0.331	5.26	112.5	40.0
MIMIC / MedGemma-27B / structured	learned (5-fold LR)	0.162	4.78	102.2	10.0

TABLE V: Evaluation of Decision-Time Routing Topologies Across Core Representative Settings.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2VLM Evaluation VLM Robustness
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical VQA VLM Interpretability
University of Virginia · National Institutes of Health
CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays
Jun 19, 2026Geon Choi, Hangyul Yoon, Nalee Kim +5VLM Evaluation Medical Imaging
1KAIST · 2Samsung Medical Center · 3Konkuk University Medical Center

May 17, 2026J/K move · Enter open · S save

Medical Context Distorts Decisions in Clinical Vision Language Models

David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis+1

1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK+1

VLM Evaluation VLM Robustness Prompt Sensitivity Radiology VLMs Vision-Language Models Healthcare+1

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios that require integrating both visual and textual context from medical records remains poorly characterized. This paper identifies three failure modes: (1) modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs. We evaluate a diverse set of general-domain and medically-tuned open and closed VLMs on chest x-ray tasks using MIMIC-CXR. By systematically manipulating image-text alignment, clinical history, and prompt formulations, we found that VLM decisions are dominated by the text modality, even when visual evidence is available. Moreover, we observed that VLMs are heavily influenced by irrelevant reports, while minor prompt changes can reverse correct image-based predictions. Our findings underscore the need for explicit safeguards and stress-testing before considering the use of these models in clinical practice.

Abstract

Medical VLMs

Explore similar work

Medical Context Distorts Decisions in Clinical Vision Language Models

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays