NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

Katarina Trojachanec Dineva; Stefan Andonov; Ilinka Ivanoska; Ivan Kitanovski; Sasho Gramatikov; Tamara Kostova; Monika Simjanoska Misheva; Kostadin Mishev

cs.CVMar 25, 2026

NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

Authors: Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska, Ivan Kitanovski, Sasho Gramatikov, Tamara Kostova, Monika Simjanoska Misheva, Kostadin Mishev

Organizations: Faculty of computer science and engineering, Ss. Cyril and Methodius University, Skopje, North Macedonia.

Abstract

Recent advances in multimodal large language models enable new possibilities for image-based decision support. However, their reliability and operational trade-offs in neuroimaging remain insufficiently understood. We present a comprehensive benchmarking study of vision-enabled large language models for 2D neuroimaging using curated MRI and CT datasets covering multiple sclerosis, stroke, brain tumors, other abnormalities, and normal controls. Models are required to generate multiple outputs simultaneously, including diagnosis, diagnosis subtype, imaging modality, specialized sequence, and anatomical plane. Performance is evaluated across four directions: discriminative classification with abstention, calibration, structured-output validity, and computational efficiency. A multi-phase framework ensures fair comparison while controlling for selection bias. Across twenty frontier multimodal models, the results show that technical imaging attributes such as modality and plane are nearly solved, whereas diagnostic reasoning, especially subtype prediction, remains challenging. Tumor classification emerges as the most reliable task, stroke is moderately solvable, while multiple sclerosis and rare abnormalities remain difficult. Few-shot prompting improves performance for several models but increases token usage, latency, and cost. Gemini-2.5-Pro and GPT-5-Chat achieve the strongest overall diagnostic performance, while Gemini-2.5-Flash offers the best efficiency-performance trade-off. Among open-weight architectures, MedGemma-1.5-4B demonstrates the most promising results, as under few-shot prompting, it approaches the zero-shot performance of several proprietary models, while maintaining perfect structured output. These findings provide practical insights into performance, reliability, and efficiency trade-offs, supporting standardized evaluation of multimodal LLMs in neuroimaging.

Figures & tables

Figure 1: Overview of the evaluation setup

Figure 2: Illustrative overview of selected output fields and evaluation dimensions assessed in NeuroVLM-Bench. The stacked bars summarize representative metrics capturing diagnostic discrimination (Diagnosis F1 and DiagnosisDetailed F1), imaging attribute recognition (Modality F1, SpecializedSequence F1, Plane F1), output reliability (Schema Validity), uncertainty behavior (Abstention Rate), and confidence calibration (ECE). This visualization provides a high-level summary of relative model behavior and is not intended as a complete performance ranking.

Identity			Count	Modality		Sequence (MRI)				Plane
Dataset	Class	Subclass	Count	CT	MRI	FLAIR	T1C	T1	T2	Axial	Sagittal
Brain Tumor Dataset Cheng (2017)	Tumor	Glioma	1426	0	1426	0	1426	0	0	0	0
		Meningioma	708	0	708	0	708	0	0	0	0
		Pituitary Tumor	930	0	930	0	930	0	0	0	0
	Dataset Total	-	3064	0	3064	0	3064	0	0	0	0
Brain Tumor MRI Images Dataset with 44 Classes Fernando (2022b)	Tumor	Carcinoma	251	0	251	0	112	66	73	0	0

Table 1: Benchmark dataset counts by source dataset and diagnostic category.

Figure 3: Overview of the data origin and class composition

Identity		Count	Modality		Sequence (MRI)				Plane
Main Class	Subclass	Count	CT	MRI	FLAIR	T1C+	T1	T2	Axial	Sagittal
Tumor	Glioma	3962	0	3962	0	2398	812	718	1317	0
	Meningioma	2881	0	2881	0	1702	617	562	1299	0
	Pituitary Tumor	930	0	930	0	930	0	0	0	0
	Neurocitoma	999	0	999	0	484	299	216	542	0
	Schwannoma	935	0	935	0	388	301	246	470	0

Table 2: Class distribution in the benchmark dataset after merging all source datasets, grouped hierarchically by primary diagnosis and detailed diagnostic category, and further stratified by imaging modality, specialized MRI sequence (for MRI studies only), and imaging plane (when available).

Figure 4: Sunburst visualization of the benchmark’s output fields, illustrating the distribution of samples across diagnostic class, diagnostic subclass, and imaging modality.

Panel of Figure 5 — Figure 5: Distribution of imaging-specific output fields in the benchmark.

Model	Context	Cost per $1M$ tokens
Model	Context	Input	Output
GPT-5 Mini	400k	0.25	2.00
GPT-5 Chat	128k	1.25	10.00
GPT-4o Mini	128k	0.15	0.60
GPT-4o	128k	2.50	10.00
GPT-4.1 (April 2025)	128k	5.00	15.00

Table 3: Pricing of models per million tokens.

Panel of Figure 6 — Figure 6: Overview of the staged benchmarking pipeline. Phase 0 calibrates and freezes the experimental setup. Phase 1 performs screening on a stratified 30% pool and filters candidate models. Phase 2 confirms performance at development scale. Phase 3 reports final results on a strictly held-out test split under zero-shot and few-shot prompting.

Figure 7: Sample image input, the corresponding ground truth and the responses from some the models. We are trying to get structured response from the models, where each field (target) describes a specific part of the diagnostic report for the provided input.

Model	Macro-F1	Macro-F1-weighted	Micro-F1	Accuracy	AUC	ECE	Brier	Valid JSON (%)	Abstention Rate (%)
Amazon Nova Lite 1.0	$0.326\,410$ ( $0.316\,474$ – $0.338\,939$ )	$0.515\,218$ ( $0.493\,558$ – $0.536\,154$ )	$0.584\,861$ ( $0.570\,944$ – $0.599\,652$ )	$0.371\,288$ ( $0.364\,739$ – $0.381\,738$ )	$0.571\,193$	$0.336\,982$	$0.353\,870$	$99.86$	$0.0000$
Amazon Nova Pro 1.0	$0.323\,336$ ( $0.308\,361$ – $0.338\,806$ )	$0.510\,315$ ( $0.497\,380$ – $0.528\,184$ )	$0.512\,220$ ( $0.494\,535$ – $0.525\,297$ )	$0.329\,633$ ( $0.307\,718$ – $0.357\,518$ )	$0.564\,061$	$0.419\,851$	$0.421\,972$	$99.86$	$0.0000$
Gemini 2.0 Flash	$0.536\,523$ ( $0.517\,023$ – $0.553\,664$ )	$0.714\,269$ ( $0.698\,675$ – $0.728\,868$ )	$0.711\,428$ ( $0.696\,668$ – $0.728\,577$ )	$0.602\,889$ ( $0.556\,100$ – $0.639\,214$ )	$0.764\,689$	$0.142\,947$	$0.198\,768$	$99.97$	$0.0000$
Gemini 2.5 Flash	$0.569\,297$ ( $0.546\,298$ – $0.589\,599$ )	$0.739\,263$ ( $0.722\,197$ – $0.751\,537$ )	$0.728\,630$ ( $0.713\,365$ – $0.743\,284$ )	$0.617\,461$ ( $0.575\,666$ – $0.673\,048$ )	$0.615\,188$	$0.222\,948$	$0.238\,120$	$99.93$	$0.0000$
Gemini 2.5 Pro	$0.573\,162$ ( $0.553\,469$ – $0.592\,033$ )	$0.749\,430$ ( $0.733\,475$ – $0.764\,675$ )	$0.748\,050$ ( $0.732\,274$ – $0.761\,131$ )	$0.580\,074$ ( $0.557\,211$ – $0.614\,725$ )	$0.591\,086$	$0.219\,047$	$0.234\,076$	$99.97$	$0.0000$
Gemma 3 27B IT	$0.325\,991$ ( $0.308\,369$ – $0.340\,862$ )	$0.477\,328$ ( $0.456\,238$ – $0.495\,702$ )	$0.484\,407$ ( $0.466\,873$ – $0.499\,839$ )	$0.348\,651$ ( $0.315\,191$ – $0.376\,756$ )	$0.761\,128$	$0.392\,359$	$0.374\,202$	100	$0.0000$

Table 4: Phase 1 - Model performance on the primary diagnostic task evaluated for abstentions penalized discriminative performance metrics (Macro-F1, weighted Macro-F1, Micro-F1, Accuracy, AUC), calibration (ECE and Brier score), and structured output reliability (valid JSON rate, abstention rate). Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best observed point estimate per column.

Model	Diagnosis	Detailed diagnosis	Modality	Specialized sequence	Plane
Amazon Nova Lite	$0.326\,410$ ( $0.312\,919$ - $0.336\,699$ )	$0.085\,546$ ( $0.079\,406$ - $0.092\,623$ )	$0.972\,475$ ( $0.966\,877$ - $0.978\,334$ )	$0.263\,485$ ( $0.247\,436$ - $0.280\,532$ )	$0.696\,457$ ( $0.669\,228$ - $0.719\,506$ )
Amazon Nova Pro	$0.323\,336$ ( $0.305\,534$ - $0.339\,461$ )	$0.073\,091$ ( $0.067\,001$ - $0.080\,140$ )	$0.912\,533$ ( $0.901\,385$ - $0.922\,636$ )	$0.321\,422$ ( $0.303\,225$ - $0.339\,696$ )	$0.428\,436$ ( $0.409\,311$ - $0.449\,238$ )
Gemini 2.0 Flash	$0.536\,523$ ( $0.517\,146$ - $0.553\,442$ )	$0.209\,974$ ( $0.193\,961$ - $0.223\,207$ )	$0.999\,639$ ( $0.999\,083$ - $1.000\,000$ )	$0.773\,993$ ( $0.755\,541$ - $0.791\,069$ )	$0.983\,469$ ( $0.972\,644$ - $0.991\,451$ )
Gemini 2.5 Flash	$0.569\,297$ ( $0.549\,526$ - $0.591\,401$ )	$0.270\,373$ ( $0.240\,171$ - $0.300\,947$ )	$0.999\,278$ ( $0.998\,197$ - $1.000\,000$ )	$0.784\,940$ ( $0.762\,777$ - $0.803\,900$ )	$0.975\,588$ ( $0.964\,304$ - $0.987\,218$ )
Gemini 2.5 Pro	$0.573\,162$ ( $0.557\,505$ - $0.588\,030$ )	$0.319\,334$ ( $0.298\,040$ - $0.340\,375$ )	$0.999\,639$ ( $0.998\,902$ - $1.000\,000$ )	$0.781\,429$ ( $0.764\,232$ - $0.801\,249$ )	$0.974\,158$ ( $0.961\,319$ - $0.983\,906$ )
Gemma 3 27B IT	$0.325\,991$ ( $0.304\,070$ - $0.341\,046$ )	$0.065\,951$ ( $0.060\,024$ - $0.072\,509$ )	$0.996\,756$ ( $0.994\,535$ - $0.998\,555$ )	$0.222\,238$ ( $0.200\,774$ - $0.240\,810$ )	$0.988\,028$ ( $0.978\,534$ - $0.994\,253$ )

Table 5: Phase 1 field-wise performance (F1 with abstention penalty) across available structured output fields: primary diagnosis, detailed diagnosis, imaging modality, specialized MRI sequence (MRI-only), and imaging plane (when annotated). Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best point estimate per column

Model	Avg In. Tokens	Avg Out. Tokens	Total Tokens	Avg Latency (ms)	Avg In. Cost (est.) **	Avg Out. Cost (est.) **	Avg Cost
Amazon Nova Lite	2435.07	104.69	2539.75	6162.08	$0.000146	$0.000025	$0.000171
Amazon Nova Pro	2435.07	97.29	2532.36	9945.20	$0.001948	$0.000311	$0.002259
Gemini 2.0 Flash	2157.14	100.64	2257.78	716.31	$0.000216	$0.000040	$0.000256
Gemini 2.5 Flash	1190.59	97.18	1287.77	1708.56	$0.000357	$0.000243	$0.000593
Gemini 2.5 Pro	1237.99	1096.03	2334.02	12084.35	$0.001547	$0.010960	$0.012507
Gemma 3 27B IT	1199.61	103.64	1303.25	2120.09	$0.000078	$0.000027	$0.000148

Table 6: Computational efficiency and economic cost per inference across evaluated models. Metrics include average input tokens, output tokens, total tokens, latency (ms), input cost, output cost, and total cost. All values reflect per-request averages under the standardized evaluation protocol.

Model	Macro-F1	Macro-F1-weighted	Micro-F1	Accuracy	ECE	Brier	Valid JSON (%)	Abstention Rate (%)
Gemini 2.0 Flash	$0.526\,876$ ( $0.517\,764$ – $0.535\,468$ )	$0.703\,259$ ( $0.693\,847$ – $0.711\,630$ )	$0.702\,572$ ( $0.693\,491$ – $0.710\,126$ )	$0.595\,323$ ( $0.578\,274$ – $0.615\,156$ )	$0.141\,067$	$0.204\,196$	$99.99$	$0.007\,532\,389$
Gemini 2.5 Flash	$0.549\,482$ ( $0.539\,249$ – $0.558\,867$ )	$0.726\,365$ ( $0.720\,278$ – $0.733\,964$ )	$0.715\,556$ ( $0.707\,780$ – $0.721\,971$ )	$0.604\,645$ ( $0.585\,461$ – $0.624\,355$ )	$0.235\,836$	$0.249\,878$	$99.99$	$0.015\,064\,779$
Gemini 2.5 Pro	$0.572\,630$ ( $0.561\,292$ – $0.582\,589$ )	$0.742\,264$ ( $0.733\,512$ – $0.749\,708$ )	$0.741\,059$ ( $0.734\,669$ – $0.748\,405$ )	$0.606\,472$ ( $0.585\,268$ – $0.623\,530$ )	$0.225\,877$	$0.239\,935$	$99.84$	$0.030\,175\,015$
MedGemma 1 4B	$0.382\,149$ ( $0.375\,870$ – $0.389\,581$ )	$0.560\,298$ ( $0.552\,317$ – $0.568\,108$ )	$0.589\,024$ ( $0.582\,845$ – $0.595\,436$ )	$0.390\,705$ ( $0.383\,006$ – $0.397\,545$ )	$0.340\,421$	$0.349\,931$	$100$	$0$
MedGemma 1.5 4B	$0.391\,436$ ( $0.382\,782$ – $0.398\,700$ )	$0.575\,142$ ( $0.567\,013$ – $0.583\,442$ )	$0.626\,659$ ( $0.618\,534$ – $0.636\,394$ )	$0.429\,966$ ( $0.418\,509$ – $0.441\,206$ )	$0.314\,875$	$0.316\,461$	$99.34$	$0$
LLaMA 4 Maverick	$0.451\,750$ ( $0.442\,477$ – $0.461\,160$ )	$0.649\,502$ ( $0.641\,486$ – $0.660\,500$ )	$0.648\,933$ ( $0.640\,434$ – $0.657\,732$ )	$0.476\,909$ ( $0.457\,741$ – $0.495\,820$ )	$0.252\,955$	$0.279\,326$	$99.80$	$1.962\,264\,151$

Table 7: Phase 2 (stability validation) - Model performance on the primary diagnostic task on the basis of abstention penalized discriminative performance metrics (Macro-F1, weighted Macro-F1, Micro-F1, Accuracy), calibration metrics (ECE and Brier score), and structured output reliability (valid JSON rate and abstention rate). Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best point estimate per column.

Model	Diagnosis	Detailed diagnosis	Modality	Specialized sequence	Plane
Gemini 2.0 Flash	$0.526\,876$ ( $0.517\,760$ - $0.538\,607$ )	$0.225\,503$ ( $0.218\,818$ - $0.234\,093$ )	$0.999\,149$ ( $0.998\,721$ - $0.999\,678$ )	$0.856\,756$ ( $0.848\,492$ - $0.864\,091$ )	$0.987\,911$ ( $0.984\,357$ - $0.991\,396$ )
Gemini 2.5 Flash	$0.549\,482$ ( $0.538\,719$ - $0.561\,245$ )	$0.254\,855$ ( $0.240\,930$ - $0.265\,130$ )	$0.998\,767$ ( $0.998\,295$ - $0.999\,282$ )	$0.783\,426$ ( $0.776\,044$ - $0.790\,260$ )	$0.976\,856$ ( $0.970\,304$ - $0.981\,130$ )
Gemini 2.5 Pro	$0.572\,630$ ( $0.565\,093$ - $0.582\,741$ )	$0.317\,199$ ( $0.302\,265$ - $0.327\,772$ )	$0.999\,117$ ( $0.998\,595$ - $0.999\,598$ )	$0.785\,220$ ( $0.776\,886$ - $0.795\,365$ )	$0.967\,637$ ( $0.961\,826$ - $0.972\,394$ )
MedGemma 1 4B	$0.382\,149$ ( $0.375\,870$ - $0.389\,581$ )	$0.104\,548$ ( $0.098\,274$ - $0.112\,408$ )	$1.000\,000$ ( $1.000\,000$ - $1.000\,000$ )	$0.172\,935$ ( $0.165\,919$ - $0.180\,135$ )	$1.000\,000$ ( $1.000\,000$ - $1.000\,000$ )
MedGemma 1.5 4B	$0.391\,436$ ( $0.382\,782$ - $0.398\,700$ )	$0.071\,375$ ( $0.062\,771$ - $0.079\,284$ )	$0.951\,620$ ( $0.948\,323$ - $0.954\,847$ )	$0.238\,126$ ( $0.228\,457$ - $0.245\,943$ )	$0.497\,918$ ( $0.484\,018$ - $0.509\,302$ )
LLaMA 4 Maverick	$0.451\,750$ ( $0.444\,314$ - $0.460\,447$ )	$0.230\,175$ ( $0.217\,764$ - $0.243\,939$ )	$0.989\,329$ ( $0.987\,690$ - $0.990\,855$ )	$0.624\,250$ ( $0.613\,660$ - $0.634\,017$ )	$0.980\,946$ ( $0.976\,507$ - $0.984\,497$ )

Table 8: Phase 2 (stability validation) field-wise performance (F1 with abstention penalty) across structured output fields: primary diagnosis, detailed diagnosis, imaging modality, specialized MRI sequence (MRI-only), and imaging plane (when annotated). Values in parentheses denote 95% confidence intervals.

Figure 8: Illustrative multi-dimensional summary of selected output fields and evaluation dimensions on the 45% development split, including diagnosis, detailed diagnosis, metadata recognition, schema validity, and calibration-related behavior.

Model	Macro-F1	Macro-F1-weighted	Micro-F1	Accuracy	AUC	ECE	Brier	Valid JSON (%)	Abstention Rate (%)
Gemini 2.0 Flash	$0.537\,340$ ( $0.524\,054$ – $0.552\,802$ )	$0.707\,602$ ( $0.696\,702$ – $0.722\,090$ )	$0.705\,579$ ( $0.694\,348$ – $0.717\,336$ )	$0.618\,429$ ( $0.594\,302$ – $0.645\,080$ )	$0.756\,031$	$0.141\,528$	$0.201\,407$	$100$	$0.040\,667$
Gemini 2.5 Flash	$0.563\,861$ ( $0.551\,675$ – $0.577\,894$ )	$0.731\,160$ ( $0.721\,402$ – $0.740\,368$ )	$0.720\,233$ ( $0.710\,306$ – $0.730\,275$ )	$0.616\,075$ ( $0.593\,375$ – $0.644\,100$ )	$0.609\,192$	$0.231\,183$	$0.246\,289$	$99.96$	$0$
Gemini 2.5 Pro	$0.576\,743$ ( $0.562\,861$ – $0.589\,617$ )	$0.741\,138$ ( $0.731\,185$ – $0.751\,152$ )	$0.740\,826$ ( $0.732\,587$ – $0.750\,730$ )	$0.604\,887$ ( $0.580\,062$ – $0.626\,921$ )	$0.595\,499$	$0.226\,294$	$0.240\,720$	$99.74$	$0$
MedGemma 1 4B	$0.379\,716$ ( $0.366\,078$ – $0.392\,120$ )	$0.555\,574$ ( $0.544\,528$ – $0.566\,576$ )	$0.585\,197$ ( $0.574\,326$ – $0.594\,493$ )	$0.388\,109$ ( $0.377\,134$ – $0.398\,588$ )	$0.592\,396$	$0.344\,750$	$0.353\,848$	$100$	$0$
MedGemma 1.5 4B	$0.418\,824$ ( $0.408\,453$ – $0.429\,961$ )	$0.602\,401$ ( $0.591\,439$ – $0.614\,896$ )	$0.635\,695$ ( $0.623\,764$ – $0.645\,187$ )	$0.458\,969$ ( $0.441\,839$ – $0.473\,547$ )	$0.565\,540$	$0.318\,432$	$0.321\,627$	$99.50$	$0$
LLaMA 4 Maverick	$0.453\,718$ ( $0.441\,742$ – $0.468\,640$ )	$0.658\,688$ ( $0.647\,990$ – $0.668\,694$ )	$0.657\,445$ ( $0.645\,098$ – $0.670\,222$ )	$0.490\,331$ ( $0.467\,494$ – $0.519\,745$ )	$0.610\,189$	$0.253\,382$	$0.279\,175$	$100$	$0.081\,334$

Table 9: Phase 3 (final evaluation under zero-shot prompting) - Model performance on the primary diagnostic task evaluated for abstentions penalized discriminative performance metrics (Macro-F1, weighted Macro-F1, Micro-F1, Accuracy, AUC), calibration (ECE and Brier score), and structured output reliability (valid JSON rate, abstention rate). Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best observed point estimate per column.

Model	Avg In. Tokens	Avg Out. Tokens	Total Tokens	Avg Latency (ms)	Avg In. Cost (est.) **	Avg Out. Cost (est.) **	Avg Cost
Gemini 2.0 Flash	2430.91	105.07	2535.98	1607.12	$0.000243	$0.000042	$0.000285
Gemini 2.5 Flash	1475.00	103.57	1578.57	1775.04	$0.000443	$0.000259	$0.000701
Gemini 2.5 Pro	1475.00	1134.59	2609.59	13423.05	$0.001844	$0.011346	$0.013190
MedGemma 1.5 4B *	1493.00	519.31	2012.31	N/A	$0.000184	$0.000237	$0.000420
MedGemma 1 4B *	300.00	1000.00	1300.00	N/A	$0.000037	$0.000456	$0.000493
LLaMA 4 Maverick	1827.77	86.05	1913.82	937.65	$0.000274	$0.000052	$0.000326

Table 10: Computational efficiency and economic cost per inference across evaluated models in Phase 3 (zero-shot). Metrics include average input tokens, output tokens, total tokens, estimated input cost, output cost, and total cost. All values reflect per-request averages under the standardized evaluation protocol.

Model	Macro-F1	Macro-F1-weighted	Micro-F1	Accuracy	AUC	ECE	Brier	Valid JSON (%)	Abstention Rate (%)
Gemini 2.0 Flash	$0.577\,833$ ( $0.565\,670$ – $0.590\,448$ )	$0.739\,939$ ( $0.732\,649$ – $0.749\,215$ )	$0.721\,996$ ( $0.713\,058$ – $0.731\,351$ )	$0.679\,066$ ( $0.648\,518$ – $0.707\,649$ )	$0.719\,120$	$0.173\,139$	$0.213\,237$	$99.97$	$0.027\,118\,644$
Gemini 2.5 Flash	$0.612\,046$ ( $0.596\,073$ – $0.629\,174$ )	$0.769\,355$ ( $0.760\,352$ – $0.776\,717$ )	$0.758\,083$ ( $0.749\,387$ – $0.767\,663$ )	$0.683\,918$ ( $0.659\,943$ – $0.708\,097$ )	$0.601\,556$	$0.178\,346$	$0.211\,461$	$100.00$	$0.013\,555\,646$
Gemini 2.5 Pro	$0.594\,458$ ( $0.575\,786$ – $0.610\,713$ )	$0.753\,350$ ( $0.744\,725$ – $0.763\,074$ )	$0.747\,117$ ( $0.738\,176$ – $0.757\,258$ )	$0.698\,083$ ( $0.674\,420$ – $0.722\,946$ )	$0.654\,758$	$0.212\,496$	$0.228\,542$	$99.97$	$0.108\,474\,576$
MedGemma 1.5 4B	$0.557\,444$ ( $0.542\,215$ – $0.572\,004$ )	$0.723\,397$ ( $0.712\,873$ – $0.733\,629$ )	$0.724\,549$ ( $0.715\,453$ – $0.732\,839$ )	$0.591\,174$ ( $0.568\,623$ – $0.616\,303$ )	$0.610\,520$	$0.260\,842$	$0.281\,554$	$100.00$	$0.0000$
MedGemma 1 4B	$0.408\,694$ ( $0.399\,107$ – $0.418\,075$ )	$0.598\,584$ ( $0.590\,701$ – $0.609\,730$ )	$0.608\,177$ ( $0.600\,034$ – $0.617\,884$ )	$0.444\,607$ ( $0.433\,912$ – $0.455\,936$ )	$0.492\,620$	$0.357\,827$	$0.365\,184$	$100.00$	$0.067\,778\,230$
LLaMA 4 Maverick	$0.520\,425$ ( $0.507\,162$ – $0.535\,534$ )	$0.708\,752$ ( $0.699\,067$ – $0.720\,006$ )	$0.700\,229$ ( $0.689\,034$ – $0.710\,503$ )	$0.573\,159$ ( $0.541\,754$ – $0.592\,668$ )	$0.716\,861$	$0.228\,507$	$0.248\,076$	$97.76$	$0.041\,597\,338$

Table 11: Phase 3: final evaluation under few-shot prompting - Model performance on the primary diagnostic task evaluated for abstentions penalized discriminative performance metrics (Macro-F1, weighted Macro-F1, Micro-F1, Accuracy, AUC), calibration (ECE and Brier score), and structured output reliability (valid JSON rate, abstention rate). Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best observed point estimate per column.

Model	Avg In. Tokens	Avg Out. Tokens	Total Tokens	Avg Latency (ms)	Avg In. * Cost (est.)	Avg Out. * Cost (est.)	Avg Cost
Gemini 2.0 Flash	6869.00	104.95	6973.95	1752.21	$0.000687	$0.000042	$0.000729
Gemini 2.5 Flash	6868.57	103.68	6972.25	2091.15	$0.002061	$0.000259	$0.002320
Gemini 2.5 Pro	6867.03	1018.23	7885.26	11689.04	$0.008584	$0.010182	$0.018766
MedGemma 1.5 4B *	8081.00	57.07	8138.07	N/A	$0.000994	$0.000026	$0.001020
MedGemma 1 4B *	1068.00	1000.00	1300.00	N/A	$0.000037	$0.000456	$0.000493
LLaMA 4 Maverick	16595.01	92.87	16687.88	1696.96	$0.002489	$0.000056	$0.002545

Table 12: Computational efficiency and economic cost per inference across evaluated models in Phase 3 (few-shot prompting). Metrics include average input tokens, output tokens, total tokens, estimated input cost, output cost, and total cost. All values reflect per-request averages under the standardized evaluation protocol.

Figure 9: Comparison between zero-shot and few-shot Macro-F1 performance with confidence intervals for the primary diagnostic task. Points indicate the model’s Macro-F1 score under zero-shot and few-shot prompting, and horizontal whiskers represent 95% confidence intervals. Models are grouped by provider to facilitate comparison across model families.

Model	Diagnosis	Detailed diagnosis	Modality	Specialized sequence	Plane
Gemini 2.0 Flash	$0.537\,340$ ( $0.522\,433$ – $0.553\,087$ )	$0.223\,363$ ( $0.213\,574$ – $0.234\,630$ )	$0.999\,368$ ( $0.998\,793$ – $0.999\,802$ )	$0.851\,496$ ( $0.841\,198$ – $0.862\,019$ )	$0.986\,122$ ( $0.979\,958$ – $0.990\,671$ )
Gemini 2.5 Flash	$0.563\,861$ ( $0.549\,318$ – $0.577\,596$ )	$0.258\,661$ ( $0.242\,944$ – $0.270\,849$ )	$0.999\,278$ ( $0.998\,623$ – $0.999\,856$ )	$0.783\,816$ ( $0.775\,571$ – $0.794\,867$ )	$0.976\,080$ ( $0.969\,695$ – $0.982\,821$ )
Gemini 2.5 Pro	$0.576\,743$ ( $0.564\,799$ – $0.591\,176$ )	$0.322\,712$ ( $0.305\,275$ – $0.337\,935$ )	$0.999\,422$ ( $0.998\,914$ – $0.999\,931$ )	$0.792\,360$ ( $0.780\,181$ – $0.802\,454$ )	$0.970\,052$ ( $0.963\,191$ – $0.977\,253$ )
MedGemma 1 4B	$0.379\,716$ ( $0.366\,975$ – $0.392\,677$ )	$0.101\,338$ ( $0.091\,438$ – $0.111\,571$ )	$1.000\,000$ ( $1.000\,000$ – $1.000\,000$ )	$0.169\,878$ ( $0.159\,937$ – $0.183\,787$ )	$1.000\,000$ ( $1.000\,000$ – $1.000\,000$ )
MedGemma 1.5 4B	$0.418\,824$ ( $0.408\,301$ – $0.429\,015$ )	$0.071\,423$ ( $0.063\,003$ – $0.079\,391$ )	$0.947\,443$ ( $0.942\,685$ – $0.951\,259$ )	$0.251\,111$ ( $0.238\,627$ – $0.264\,367$ )	$0.502\,978$ ( $0.488\,302$ – $0.518\,245$ )
LLaMA 4 Maverick	$0.453\,718$ ( $0.439\,437$ – $0.467\,487$ )	$0.225\,677$ ( $0.210\,270$ – $0.240\,989$ )	$0.996\,611$ ( $0.995\,264$ – $0.997\,969$ )	$0.641\,746$ ( $0.629\,325$ – $0.653\,629$ )	$0.984\,528$ ( $0.978\,004$ – $0.990\,612$ )

Table 13: Phase 3 — Detailed abstention-aware Macro-F1 performance by model across output fields under zero-shot prompting. Performance is reported for the primary diagnosis, detailed diagnosis, imaging modality, specialized imaging sequence, and anatomical plane. Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best observed point estimate per column.

Model	Diagnosis	Detailed diagnosis	Modality	Specialized sequence	Plane
Gemini 2.0 Flash	$0.577\,833$ ( $0.564\,862$ - $0.591\,770$ )	$0.241\,156$ ( $0.228\,556$ - $0.253\,517$ )	$0.999\,279$ ( $0.998\,627$ - $0.999\,730$ )	$0.867\,514$ ( $0.853\,118$ - $0.878\,966$ )	$0.990\,003$ ( $0.984\,686$ - $0.995\,080$ )
Gemini 2.5 Flash	$0.612\,046$ ( $0.597\,689$ - $0.630\,406$ )	$0.284\,182$ ( $0.266\,233$ - $0.307\,984$ )	$0.999\,567$ ( $0.999\,133$ - $1.000\,000$ )	$0.786\,989$ ( $0.776\,063$ - $0.797\,023$ )	$0.977\,916$ ( $0.971\,668$ - $0.984\,430$ )
Gemini 2.5 Pro	$0.594\,458$ ( $0.578\,917$ - $0.606\,029$ )	$0.329\,969$ ( $0.309\,375$ - $0.346\,206$ )	$0.999\,513$ ( $0.998\,981$ - $1.000\,000$ )	$0.806\,477$ ( $0.796\,803$ - $0.817\,593$ )	$0.982\,505$ ( $0.977\,767$ - $0.987\,512$ )
MedGemma 1 4B	$0.408\,694$ ( $0.398\,150$ - $0.418\,592$ )	$0.093\,754$ ( $0.084\,582$ - $0.101\,230$ )	$1.000\,000$ ( $1.000\,000$ - $1.000\,000$ )	$0.273\,237$ ( $0.261\,182$ - $0.289\,666$ )	$1.000\,000$ ( $1.000\,000$ - $1.000\,000$ )
MedGemma 1.5 4B	$0.557\,444$ ( $0.542\,496$ - $0.570\,977$ )	$0.137\,000$ ( $0.126\,743$ - $0.149\,338$ )	$0.987\,522$ ( $0.984\,421$ - $0.990\,213$ )	$0.405\,988$ ( $0.387\,703$ - $0.419\,932$ )	$0.643\,121$ ( $0.617\,029$ - $0.669\,541$ )
LLaMA 4 Maverick	$0.520\,425$ ( $0.503\,816$ - $0.533\,775$ )	$0.251\,110$ ( $0.235\,960$ - $0.262\,830$ )	$0.995\,727$ ( $0.993\,954$ - $0.997\,099$ )	$0.692\,319$ ( $0.679\,338$ - $0.705\,375$ )	$0.994\,857$ ( $0.990\,883$ - $0.997\,956$ )

Table 14: Phase 3 — Detailed abstention-aware Macro-F1 performance by model across structured output fields under few-shot prompting (4 examples per class). Reported are Macro-F1 scores with abstention for the primary diagnosis, detailed diagnosis, imaging modality, specialized imaging sequence, and anatomical plane. Values in parentheses denote 95% confidence intervals. Bold underlined values indicate the best observed point estimate per column (ties highlighted equally).

Panel of Figure 10 — Figure 10: Comparison between zero-shot and few-shot Macro-F1 performance with confidence intervals across additional structured output fields: (a) detailed diagnosis, (b) imaging modality, (c) specialized imaging sequence, and (d) anatomical plane. Points represent the model Macro-F1 scores under zero-shot and few-shot prompting, while horizontal whiskers denote 95% confidence intervals. Models are grouped by provider to facilitate comparison across model families.

Figure 11: Illustrative multidimensional overview of model performance across selected output fields and evaluation dimensions in Phase 3 of the benchmark with zero-shot prompting. Each horizontal bar summarizes diagnostic performance, detailed diagnosis, modality prediction, specialized sequence recognition, anatomical plane detection, structured schema validity, model certainty, and calibration (ECE), enabling qualitative comparison of model behaviour across tasks.

Figure 12: Illustrative multidimensional overview of model performance across selected output fields and evaluation dimensions in Phase 3 of the benchmark with few-shot prompting. Each horizontal bar summarizes diagnostic performance, detailed diagnosis, modality prediction, specialized sequence recognition, anatomical plane detection, structured schema validity, model certainty, and calibration (ECE), enabling qualitative comparison of model behaviour across tasks.

Model	Tumor	Stroke	Multiple sclerosis	Normal	Other abnormalities
Gemini 2.0 Flash	$0.908\,219$ ( $0.900\,553$ - $0.914\,694$ )	$0.471\,398$ ( $0.442\,212$ - $0.495\,413$ )	$0.536\,941$ ( $0.501\,028$ - $0.568\,862$ )	$0.654\,589$ ( $0.638\,119$ - $0.666\,007$ )	$0.115\,556$ ( $0.074\,838$ - $0.153\,718$ )
Gemini 2.5 Flash	$0.897\,496$ ( $0.887\,594$ - $0.905\,480$ )	$0.525\,682$ ( $0.498\,729$ - $0.547\,017$ )	$0.601\,527$ ( $0.563\,463$ - $0.638\,285$ )	$0.696\,400$ ( $0.680\,566$ - $0.709\,531$ )	$0.098\,200$ ( $0.072\,488$ - $0.125\,956$ )
Gemini 2.5 Pro	$0.912\,138$ ( $0.904\,834$ - $0.917\,815$ )	$0.642\,009$ ( $0.622\,978$ - $0.657\,167$ )	$0.623\,244$ ( $0.588\,971$ - $0.658\,554$ )	$0.622\,989$ ( $0.609\,343$ - $0.637\,637$ )	$0.083\,333$ ( $0.040\,738$ - $0.130\,400$ )
MedGemma 1 4B	$0.771\,649$ ( $0.761\,765$ - $0.780\,759$ )	$0.348\,624$ ( $0.321\,967$ - $0.370\,727$ )	$0.305\,913$ ( $0.273\,853$ - $0.337\,421$ )	$0.472\,393$ ( $0.452\,272$ - $0.493\,033$ )	$0.000\,000$ ( $0.000\,000$ - $0.000\,000$ )
MedGemma 1.5 4B	$0.822\,663$ ( $0.813\,278$ - $0.835\,302$ )	$0.152\,486$ ( $0.130\,070$ - $0.176\,538$ )	$0.415\,385$ ( $0.374\,093$ - $0.454\,666$ )	$0.679\,195$ ( $0.665\,810$ - $0.695\,915$ )	$0.024\,390$ ( $0.007\,622$ - $0.045\,744$ )
LLaMA 4 Maverick	$0.877\,331$ ( $0.868\,646$ - $0.886\,318$ )	$0.347\,092$ ( $0.324\,999$ - $0.371\,123$ )	$0.327\,986$ ( $0.267\,886$ - $0.367\,615$ )	$0.660\,917$ ( $0.650\,511$ - $0.676\,694$ )	$0.055\,263$ ( $0.033\,897$ - $0.077\,939$ )

Table 15: Per-class F1-scores with abstention per class (Diagnosis name) in the case of zero-shot prompting.

Model	Tumor	Stroke	Multiple Sclerosis	Normal	Other Abnormalities
Gemini 2.0 Flash	$0.896\,709$ ( $0.888\,945$ - $0.904\,404$ )	$0.630\,435$ ( $0.608\,170$ - $0.648\,687$ )	$0.550\,839$ ( $0.524\,096$ - $0.589\,630$ )	$0.656\,636$ ( $0.643\,297$ - $0.671\,855$ )	$0.154\,545$ ( $0.114\,341$ - $0.197\,080$ )
Gemini 2.5 Flash	$0.915\,019$ ( $0.907\,731$ - $0.922\,671$ )	$0.684\,354$ ( $0.662\,001$ - $0.701\,114$ )	$0.560\,088$ ( $0.520\,866$ - $0.588\,889$ )	$0.685\,491$ ( $0.667\,954$ - $0.700\,792$ )	$0.215\,278$ ( $0.161\,149$ - $0.293\,269$ )
Gemini 2.5 Pro	$0.936\,805$ ( $0.928\,676$ - $0.942\,683$ )	$0.704\,992$ ( $0.683\,908$ - $0.721\,429$ )	$0.549\,738$ ( $0.515\,886$ - $0.573\,715$ )	$0.592\,836$ ( $0.577\,359$ - $0.609\,845$ )	$0.187\,919$ ( $0.128\,329$ - $0.243\,217$ )
MedGemma 1 4B	$0.807\,424$ ( $0.795\,881$ - $0.816\,121$ )	$0.287\,776$ ( $0.259\,758$ - $0.311\,961$ )	$0.347\,511$ ( $0.314\,240$ - $0.380\,330$ )	$0.600\,759$ ( $0.585\,374$ - $0.615\,285$ )	$0.000\,000$ ( $0.000\,000$ - $0.000\,000$ )
MedGemma 1.5 4B	$0.869\,508$ ( $0.861\,415$ - $0.878\,810$ )	$0.568\,455$ ( $0.542\,565$ - $0.590\,776$ )	$0.531\,868$ ( $0.495\,637$ - $0.575\,416$ )	$0.687\,390$ ( $0.673\,213$ - $0.698\,854$ )	$0.130\,000$ ( $0.065\,190$ - $0.177\,645$ )
LLaMA 4 Maverick	$0.904\,655$ ( $0.898\,227$ - $0.911\,171$ )	$0.541\,299$ ( $0.519\,670$ - $0.563\,907$ )	$0.390\,501$ ( $0.348\,263$ - $0.437\,405$ )	$0.635\,671$ ( $0.620\,125$ - $0.652\,771$ )	$0.130\,000$ ( $0.090\,852$ - $0.176\,391$ )

Table 16: Per-class F1-scores with abstention per class (Diagnosis name) in the case of few-shot prompting (4 examples per class).

Explore similar work

Mar 30, 2026

Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

Trustworthy clinical AI must use real evidence and avoid relying on surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on two clinical neuroimaging cohorts for binary classification of affective disorders and cognitive decline. Both cohorts include structural magnetic resonance imaging (MRI) acquired under their original research protocols. Prior work does not establish the included neuroimaging inputs as reliable stand-alone diagnostic evidence for the present tasks. Nevertheless, when neuroimaging context is introduced, smaller VLMs gain up to 0.66 F1 under the evaluated augmented conditions, becoming competitive with models an order of magnitude larger. Confidence estimation shows that most of the calibration improvement for the analyzed smaller models occurs after the MRI reference is added to the prompt, before any image is supplied. Our preliminary expert case study finds that faithfulness remains low in every condition examined, with the reviewed model introducing unverified clinical details. Finally, in our single-model intervention, preference alignment suppresses MRI-referencing behavior but reduces the augmented-condition advantage, leaving the underlying issue unresolved. These results caution against reading surface metric gains as evidence of true multimodal integration, with direct implications for clinical VLM deployment.

Doan Nam Long Vu, Simone Balloccu

Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt

Medical VLMs Prompt Sensitivity Faithfulness in VLMs VLM Calibration Counterfactual Evaluation of VLMs

Ask PDF

Apr 14, 2026

Medical thinking with multiple images

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Zonghai Yao, Benlu Wang, Yifan Zhang +8

Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University +2

Medical VQA Multimodal Reasoning Medical Image Benchmarks Medical Image Grounding

Ask PDF

Apr 10, 2026

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Vision-language models (VLMs) are increasingly adapted through domain-specific fine-tuning, yet it remains unclear whether this improves reasoning beyond superficial visual cues, particularly in high-stakes domains like medicine. We evaluate four paired open-source VLMs (LLaVA vs. LLaVA-Med; Gemma vs. MedGemma) across four medical imaging tasks of increasing difficulty: brain tumor, pneumonia, skin cancer, and histopathology classification. We find that performance degrades toward near-random levels as task difficulty increases, indicating limited clinical reasoning. Medical fine-tuning provides no consistent advantage, and models are highly sensitive to prompt formulation, with minor changes causing large swings in accuracy and refusal rates. To test whether closed-form VQA suppresses latent knowledge, we introduce a description-based pipeline where models generate image descriptions that a text-only model (GPT-5.1) uses for diagnosis. This recovers a limited additional signal but remains bounded by task difficulty. Analysis of vision encoder embeddings further shows that failures stem from both weak visual representations and downstream reasoning. Overall, medical VLM performance is fragile, prompt-dependent, and not reliably improved by domain-specific fine-tuning.

Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3

Brown University · University of Washington · University of Tübingen +1

Medical VLMs VLM Reasoning VLM Robustness Medical Image Classification Prompt Sensitivity

Ask PDF