Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models

Patrick Wienholt; Frederik Hauke; Christiane Kuhl; Jakob Nikolas Kather; Sven Nebelung; Daniel Truhn

cs.CVOct 14, 2026

Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models

Authors: Patrick Wienholt, Frederik Hauke, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel Truhn

Organizations: Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. · Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany. ·

Abstract

Multimodal large language models (MLLMs) answer binary clinical questions with a yes or no token, leaving the decision threshold implicit and fixed. We derive a continuous score from a single forward pass by renormalizing the probabilities of the yes and no tokens, allowing the threshold to be adapted to clinical needs without retraining. We evaluated this approach with MedGemma and Gemma-3 (4B and 27B) on six medical imaging datasets. The default threshold was frequently misaligned with the task: for identifying normal chest radiographs, MedGemma-27B labeled every image as abnormal, whereas a validation-selected threshold yielded a sensitivity of 0.945 and a specificity of 0.696. Normalization achieved the highest AUROC on four of six datasets (CheXpert: 0.819 vs. 0.777 for raw and 0.743 for text-elicited probabilities), and a single decision required 53-79 ms on one GPU. This simple post-hoc method turns medical MLLMs into fast classifiers with adjustable operating points. Code: https://github.com/TruhnLab/llm-threshold-calibration

Figures & tables

Figure 1: Workflow for threshold shifting in LLM-based binary classification. (a) An image-conditioned prompt is provided to the LLM. (b) From the first decoding step, the model assigns probabilities to all vocabulary tokens, including the verbalizers yes and no . (c) We normalize the probabilities over { yes , no } and vary the decision threshold, which has an implicit default of 0.5. (d) Sweeping the threshold yields different operating points on the ROC curve, trading true positive rate (TPR) against false positive rate (FPR). The optimal threshold was determined on a validation set disjoint from the test set.

Figure 2: Overview of the experimental setup. We evaluated the threshold-shifting method on six medical imaging datasets spanning chest radiography, dermatology, ophthalmology, and histopathology. Notably, CheXpert was explicitly excluded from the training data of the MedGemma models [ 18 ] , making it well suited for external validation. Each dataset was split into a validation set, used exclusively to select the decision threshold, and a test set, used only for the final performance evaluation. We compared four models, namely Gemma-3 and MedGemma each in a 4B and a 27B variant, using three scoring methods: the normalized yes / no token probability, the raw yes token probability, and a text-elicited probability generated directly as output. The threshold optimized on the validation set was compared against the model’s implicit default threshold of 0.5 on the test set.

Figure 3: Class-specific AUROC values for the CheXpert dataset using MedGemma-27B. The three bars for each class represent normalized token probabilities, raw token probabilities (“Unnormalized” in the legend), and text-elicited probability estimates (“Text”). Error bars indicate 95% confidence intervals. The “average” bars show the mean across classes.

Dataset	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
CheXpert	0.819 [0.794, 0.845]	0.777 [0.746, 0.807]	0.743 [0.714, 0.772]
DERM12345	0.701 [0.682, 0.719]	0.637 [0.613, 0.662]	0.547 [0.530, 0.566]
G1020	0.494 [0.437, 0.551]	0.489 [0.423, 0.555]	0.487 [0.455, 0.519]
HAM10000	0.654 [0.642, 0.666]	0.607 [0.592, 0.621]	0.542 [0.531, 0.554]
MIMIC	0.658 [0.647, 0.669]	0.666 [0.655, 0.676]	0.625 [0.616, 0.635]
NCT-CRC-HE	0.783 [0.777, 0.788]	0.796 [0.790, 0.801]	0.642 [0.638, 0.647]

Table 1: Average AUROC values computed over the test data with corresponding 95% confidence intervals for MedGemma-27B.

		Raw		Text-elicited
Dataset	Normalized	$\Delta$ (95% CI)	$p$	$\Delta$ (95% CI)	$p$
CheXpert	0.819	-0.043	$\mathbf{<.001}$	-0.077	$\mathbf{<.001}$
CheXpert	0.819	[-0.057, -0.029]	$\mathbf{<.001}$	[-0.095, -0.059]	$\mathbf{<.001}$
DERM12345	0.701	-0.064	$\mathbf{<.001}$	-0.154	$\mathbf{<.001}$
DERM12345	0.701	[-0.086, -0.041]	$\mathbf{<.001}$	[-0.175, -0.133]	$\mathbf{<.001}$
G1020	0.494	-0.005	$.873$	-0.007	$.783$

Table 2: Pairwise comparison of average AUROC values computed over the test data using MedGemma-27B from Table 1 . The table shows the difference to the normalized approach. Statistically significant (

p<.0042

) values are in bold .

Panel of Figure 4 — Figure 4: ROC curves for selected datasets and classes using MedGemma-27B. The default threshold of 0.5, corresponding to greedy sampling, is compared to the “Optimal Threshold,” which maximizes the sum of sensitivity and specificity, computed over the validation set.

Dataset	MedGemma-27B	MedGemma-4B	Gemma-3-27B	Gemma-3-4B
CheXpert	0.819	0.790	0.535	0.519
CheXpert	[0.794, 0.845]	[0.750, 0.827]	[0.496, 0.573]	[0.483, 0.557]
DERM12345	0.701	0.636	0.749	0.680
DERM12345	[0.682, 0.719]	[0.610, 0.671]	[0.730, 0.777]	[0.660, 0.704]
G1020	0.494	0.541	0.507	0.480
G1020	[0.437, 0.551]	[0.480, 0.603]	[0.446, 0.567]	[0.422, 0.538]

Table 3: Average AUROC values with corresponding 95% confidence intervals for different models using the normalized probabilities

Figure 5: Distribution of optimal classification thresholds across CheXpert classes for MedGemma-27B. Optimal thresholds, defined as those maximizing sensitivity plus specificity, fall below 0.5 for 11 of 13 findings. Bars show the distance from the default 0.5 threshold, terminating at the optimal threshold value. The bar labeled “average” shows the mean across classes.

Original label	Phrase in prompt
No Finding	no finding
Enlarged Cardiomediastinum	enlarged cardiomediastinum
Cardiomegaly	cardiomegaly
Lung Opacity	lung opacity
Lung Lesion	lung lesion
Edema	edema

Table S1: Labels of CheXpert (14 labels).

Original label	Phrase in prompt
Atelectasis	atelectasis
Cardiomegaly	cardiomegaly
Consolidation	consolidation
Edema	edema
Enlarged Cardiomediastinum	enlarged cardiomediastinum
Fracture	fracture

Table S2: Labels of MIMIC-CXR (14 labels).

Level	Original label	Phrase in prompt
1	Benign	a benign skin lesion
1	Indeterminate	a skin lesion with an indeterminate diagnosis
1	Malignant	a malignant skin lesion
2	Benign epidermal proliferations	benign epidermal proliferation
2	Benign melanocytic proliferations	benign melanocytic proliferation
2	Benign soft tissue proliferations - Fibro-histiocytic	benign fibro-histiocytic proliferation

Table S3: Labels of HAM10000 (18 labels).

Level	Original label	Phrase in prompt
1	Benign	a benign skin lesion
1	Indeterminate	a skin lesion with an indeterminate diagnosis
1	Malignant	a malignant skin lesion
2	Benign epidermal proliferations	benign epidermal proliferation
2	Benign melanocytic proliferations	benign melanocytic proliferation
2	Benign soft tissue proliferations - Fibro-histiocytic	benign fibro-histiocytic soft tissue proliferation

Table S4: Labels of DERM12345 (37 labels).

Original label	Phrase in prompt
ADI	adipose tissue
BACK	background elements
DEB	debris
LYM	lymphocytes
MUC	mucus
MUS	smooth muscle

Table S5: Labels of NCT-CRC-HE-100K (9 labels).

Original label	Phrase in prompt
glaucoma	glaucoma
normal	a normal eye

Table S6: Labels of G1020 (2 labels).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
No Finding	0.921 [0.892, 0.945]	0.909 [0.881, 0.934]	0.882 [0.852, 0.909]
Enlarged Cardiomediastinum	0.809 [0.773, 0.842]	0.769 [0.730, 0.805]	0.733 [0.700, 0.764]
Cardiomegaly	0.848 [0.812, 0.882]	0.820 [0.781, 0.858]	0.824 [0.787, 0.858]
Lung Opacity	0.808 [0.773, 0.840]	0.659 [0.616, 0.699]	0.731 [0.702, 0.761]
Lung Lesion	0.863 [0.763, 0.947]	0.824 [0.697, 0.934]	0.695 [0.552, 0.821]
Edema	0.839 [0.795, 0.880]	0.839 [0.793, 0.881]	0.830 [0.786, 0.872]

Table S7 : Per-class AUROC values for CheXpert (MedGemma-27B, test set).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
Atelectasis	0.511 [0.488, 0.533]	0.535 [0.513, 0.557]	0.557 [0.538, 0.576]
Cardiomegaly	0.710 [0.691, 0.729]	0.695 [0.676, 0.713]	0.704 [0.687, 0.721]
Consolidation	0.681 [0.646, 0.715]	0.684 [0.649, 0.718]	0.668 [0.632, 0.703]
Edema	0.741 [0.721, 0.760]	0.731 [0.712, 0.750]	0.710 [0.692, 0.728]
Enlarged Cardiomediastinum	0.434 [0.386, 0.482]	0.474 [0.425, 0.524]	0.459 [0.417, 0.502]
Fracture	0.669 [0.614, 0.722]	0.652 [0.598, 0.704]	0.526 [0.489, 0.566]

Table S8 : Per-class AUROC values for MIMIC-CXR (MedGemma-27B, test set).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
Benign	0.610 [0.590, 0.629]	0.586 [0.567, 0.606]	0.636 [0.618, 0.653]
Indeterminate	0.427 [0.359, 0.495]	0.555 [0.472, 0.636]	0.590 [0.534, 0.643]
Malignant	0.573 [0.551, 0.596]	0.546 [0.524, 0.569]	0.524 [0.504, 0.544]
Benign epidermal proliferations	0.664 [0.641, 0.687]	0.585 [0.559, 0.611]	0.435 [0.415, 0.455]
Benign melanocytic proliferations	0.513 [0.496, 0.531]	0.480 [0.463, 0.498]	0.522 [0.516, 0.529]
Benign soft tissue proliferations - Fibro-histiocytic	0.624 [0.571, 0.677]	0.603 [0.545, 0.660]	0.434 [0.375, 0.492]

Table S9 : Per-class AUROC values for HAM10000 (MedGemma-27B, test set).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
Benign	0.835 [0.819, 0.849]	0.793 [0.775, 0.810]	0.839 [0.817, 0.859]
Indeterminate	0.545 [0.425, 0.665]	0.643 [0.541, 0.737]	0.709 [0.614, 0.800]
Malignant	0.492 [0.461, 0.523]	0.476 [0.446, 0.506]	0.500 [0.469, 0.531]
Benign epidermal proliferations	0.642 [0.606, 0.677]	0.604 [0.571, 0.638]	0.488 [0.460, 0.516]
Benign melanocytic proliferations	0.744 [0.724, 0.763]	0.637 [0.616, 0.658]	0.585 [0.571, 0.599]
Benign soft tissue proliferations - Fibro-histiocytic	0.583 [0.529, 0.638]	0.512 [0.455, 0.567]	0.393 [0.340, 0.448]

Table S10 : Per-class AUROC values for DERM12345 (MedGemma-27B, test set).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
ADI	0.968 [0.960, 0.976]	0.974 [0.967, 0.980]	0.831 [0.821, 0.842]
BACK	0.932 [0.922, 0.942]	0.909 [0.897, 0.921]	0.497 [0.495, 0.498]
DEB	0.769 [0.746, 0.791]	0.707 [0.683, 0.729]	0.673 [0.651, 0.694]
LYM	0.956 [0.950, 0.961]	0.929 [0.922, 0.935]	0.882 [0.876, 0.888]
MUC	0.739 [0.717, 0.761]	0.784 [0.763, 0.805]	0.502 [0.500, 0.506]
MUS	0.543 [0.521, 0.565]	0.641 [0.619, 0.661]	0.578 [0.561, 0.594]

Table S11 : Per-class AUROC values for NCT-CRC-HE-100K (MedGemma-27B, test set).

Label	Normalized (95% CI)	Raw (95% CI)	Text-elicited (95% CI)
glaucoma	0.522 [0.438, 0.603]	0.506 [0.419, 0.591]	0.500 [0.500, 0.500]
normal	0.467 [0.383, 0.549]	0.471 [0.388, 0.556]	0.473 [0.410, 0.538]

Table S12 : Per-class AUROC values for G1020 (MedGemma-27B, test set).

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 396 papers

MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging
Apr 15, 2026Zhijie Bao, Fangke Chen, Licheng Bao +4Multimodal Model Evaluation Medical VLMs
School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China +2
MediX-R1: Open Ended Medical Reinforcement Learning
Feb 26, 2026Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5RL for VLMs Medical VLMs
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College
Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis
Sep 30, 2026Tian Xia, Minghao Liu, Yiqing Liang +2Medical Diagnosis Multimodal Large Language Models
Harvard University · University of California, Santa Cruz · Brown University +2

Apr 15, 2026J/K move · Enter open · S save

MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging

Zhijie Bao, Fangke Chen, Licheng Bao, Chenhui Zhang+3

School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China+2

Multimodal Model Evaluation Medical VLMs Medical Image Benchmarks

Open paper AskPDF arXiv

The potential of Multimodal Large Language Models (MLLMs) in domain of medical imaging raise the demands of systematic and rigorous evaluation frameworks that are aligned with the real-world medical imaging practice. Existing practices that report single or coarse-grained metrics are lack the granularity required for specialized clinical support and fail to assess the reliability of reasoning mechanisms. To address this, we propose a paradigm shift toward multidimensional, fine-grained and in-depth evaluation. Based on a two-stage systematic construction pipeline designed for this paradigm, we instantiate it with MedRCube. We benchmark 33 MLLMs, Lingshu-32B achieve top-tier performance. Crucially, MedRCube exposes a series of pronounced insights inaccessible under prior evaluation settings. Furthermore, we introduce a credibility evaluation subset to quantify reasoning credibility, uncover a highly significant positive association between shortcut behavior and diagnostic task performance, raising concerns for clinically trustworthy deployment. The resources of this work can be found at https://github.com/F1mc/MedRCube.

Abstract

Medical LLMs

Explore similar work

MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging

MediX-R1: Open Ended Medical Reinforcement Learning

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis