Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Tian Xia; Minghao Liu; Yiqing Liang; Laixi Shi; Jiayun Wang

cs.LGSep 30, 2026

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Authors: Tian Xia, Minghao Liu, Yiqing Liang, Laixi Shi, Jiayun Wang

Organizations: Harvard University · University of California, Santa Cruz · Brown University · Johns Hopkins University · Georgia Institute of Technology

Abstract

Multimodal large language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores matrix with one row per evaluation instance and one column per candidate prompt; cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the prompt evolution search reads from - the scores matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via vision-encoder-tuned SFT or medical pretraining - is a prerequisite that prompt search cannot replace - our recipe extends reflective prompt evolution from text-only data to multimodal clinical decision-making.

Figures & tables

Figure 1: Why clinical diagnosis needs ranking metrics. a) : multimodal diagnosis combines medical images (e.g. chest X-rays), clinical profile, lab tests, and prompts to produce calibrated disease scores. b) : under heavy class imbalance common in real-world clinical datasets, threshold accuracy can look strong even when the score ranking is clinically useless, while AUROC distinguishes models across all operating points. c) : our recipe improves ranking by combining a base MLLM, SFT with vision-encoder tuning, and ranking-aware prompt evolution, moving from threshold-centric prediction to AUROC-driven optimization for reliable multimodal clinical diagnosis.

Figure 2: Ranking-aware prompt evolution. (left) Pair-level Pareto scores matrix: rows are

(x^{+},x^{-})

pairs, cells are

1

for correct order,

0.5

for ties, and

0

for reversed order. The column average over a candidate prompt

\Phi_{k}

is its validation AUROC. (right top) Reflective pipeline: at each step we select a

\Phi_{k}

from the pool, run it on a mini-batch of

(P,N)

, collect ranking-shaped feedback (clinical error type, confidence magnitude, cross-example rank context), and let the reflector propose

\Phi_{k+1}

, which is pushed back into the pool. (right bottom) Final selection:

\Phi^{*}=\arg\max_{k}\mathrm{AUROC}(\Phi_{k})

, which reduces to a column-mean argmax once the matrix records pairwise ordering events.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B	65.4	64.0	61.3	51.2	74.0	64.4	66.9	59.9
Gemma3-4B	38.2	48.9	55.1	49.7	48.6	54.3	47.3	51.0
GPT-4o-mini	60.1	60.2	54.7	50.5	70.5	53.0	61.8	54.6
GPT-5-mini	50.6	58.2	71.5	66.8	75.9	64.6	66.0	63.2
Gemini-2.5-flash	46.5	54.3	69.0	58.3	75.1	66.4	63.5	59.7

Table 1: Test AUROC and balanced accuracy (%) on three MIMIC disease tasks. Ranking-PE outperforms Accuracy-PE on AUROC across both backbones and all diseases. Open-source PE rows report means; per-disease SDs are in Appendix Table 5 . Avg is the unweighted disease average. Closed-source rows are zero-shot. Endpoints without top-

k

logprobs use a three-level verbal score, which limits comparison with logprob-based scores (Appendix A.1 ).

Model	Configuration	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	Base	65.4	61.3	74.0	66.9
	$+$ Ranking-PE	63.5	65.8	76.0	68.4
	$+$ SFT	63.3	68.0	72.6	68.0
	$+$ SFT $+$ Ranking-PE	70.7	71.3	79.3	73.8
Gemma3-4B	Base	38.2	55.1	48.6	47.3
	$+$ Ranking-PE	50.5	55.1	49.3	51.6

Table 2: Test AUROC (%) across model adaptation settings. Qwen3-VL-8B uses a

2\times 2

comparison of SFT and Ranking-PE; SFT uses a tuned vision encoder. Gemma3-4B and medically pretrained MedGemma-4B are evaluated before and after Ranking-PE. PE entries are means; their SDs are reported in Appendix Table 7 . Avg is the unweighted average of the three disease values.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Accuracy-PE	63.6	45.4	68.1	58.0	72.4	61.8	68.0	55.1
+ AUROC final selection	43.7	51.5	68.3	56.5	72.0	63.9	61.3	57.3
+ pair-level Pareto	63.8	58.6	70.6	64.9	77.6	64.6	70.7	62.7
+ ranking feedback (full)	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
Ranking-PE (full) $-$ pair-level Pareto	56.0	51.1	68.2	61.4	73.3	60.0	65.8	57.5

Table 3: Component ablation on Qwen3-VL-8B + SFT (VE-tuned). Entries are mean test AUROC and balanced accuracy (%). The first four rows form a cumulative ablation; the last row replaces pair-level Pareto with an instance-level matrix while retaining ranking feedback. Avg averages the three disease means. SDs appear in Appendix Table 6 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

max_pairs	AUROC	Bal. Acc.
$300$	$71.2$	$67.3$
$1{,}000$	$70.6$	$61.4$
$3{,}000$	$70.6$	$65.4$
$10{,}000$ (default)	$71.3$	$66.5$

Appendix

Table 4: Sensitivity to the pair cap on Cardiomegaly with Qwen3-VL-8B + SFT (VE-tuned) and Ranking-PE. Values are test AUROC and balanced accuracy (%).

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B + SFT (VE-tuned)
Accuracy-PE	$63.6\pm 0.5$	$45.4\pm 0.6$	$68.1\pm 1.2$	$58.0\pm 1.4$	$72.4\pm 0.2$	$61.8\pm 1.0$	68.0	55.1
BAcc-Select	$63.9\pm 0.4$	$43.7\pm 0.4$	$67.6\pm 1.0$	$60.6\pm 0.7$	$71.4\pm 1.3$	$63.3\pm 0.6$	67.6	55.9
Class-Weighted PE	$64.3\pm 0.2$	$44.2\pm 0.4$	$68.2\pm 1.7$	$61.3\pm 1.4$	$71.5\pm 0.7$	$61.8\pm 3.2$	68.0	55.8
Scalar-AUROC PE	$66.5\pm 3.6$	$65.5\pm 0.8$	$69.3\pm 1.6$	$63.1\pm 1.1$	$72.9\pm 1.6$	$61.2\pm 3.9$	69.6	63.3

Appendix

Table 5: Variability of the PE results in Table 1 . Per-disease test AUROC and balanced accuracy (%) are mean

\pm

SD. Avg is the unweighted average of the reported disease means; no aggregate SD is reported.

Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
	Atelectasis		Cardiomegaly		Consolidation		Avg
Accuracy-PE	$63.6\pm 0.5$	$45.4\pm 0.6$	$68.1\pm 1.2$	$58.0\pm 1.4$	$72.4\pm 0.2$	$61.8\pm 1.0$	68.0	55.1
+ AUROC final selection	$43.7\pm 2.8$	$51.5\pm 0.7$	$68.3\pm 1.9$	$56.5\pm 1.7$	$72.0\pm 0.0$	$63.9\pm 2.6$	61.3	57.3
+ pair-level Pareto	$63.8\pm 3.4$	$58.6\pm 2.1$	$70.6\pm 0.9$	$64.9\pm 0.4$	$77.6\pm 2.7$	$64.6\pm 2.1$	70.7	62.7
+ ranking feedback (full)	$70.7\pm 0.9$	$70.9\pm 4.3$	$71.3\pm 1.0$	$66.5\pm 2.1$	$79.3\pm 1.2$	$68.5\pm 1.2$	73.8	68.6
Ranking-PE (full) $-$ pair-level Pareto	$56.0\pm 4.4$	$51.1\pm 1.8$	$68.2\pm 0.1$	$61.4\pm 0.4$	$73.3\pm 0.5$	$60.0\pm 4.3$	65.8	57.5

Appendix

Table 6: Variability of the component ablation in Table 3 . Per-disease AUROC and balanced accuracy (%) are mean

\pm

SD. Avg averages the disease means.

Model	SFT setting	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	None	$63.5\pm 3.6$	$65.8\pm 0.4$	$76.0\pm 3.8$	68.4
	VE-frozen	$65.9\pm 0.8$	$67.3\pm 1.2$	$77.6\pm 0.8$	70.3
	VE-tuned	$70.7\pm 0.9$	$71.3\pm 1.0$	$79.3\pm 1.2$	73.8
Gemma3-4B	None	$50.5\pm 4.3$	$55.1\pm 0.0$	$49.3\pm 1.3$	51.6
MedGemma-4B	None	$69.1\pm 3.1$	$63.0\pm 4.5$	$77.1\pm 3.8$	69.7

Appendix

Table 7: Test AUROC (%) across SFT settings and model backbones. All configurations use Ranking-PE. On Qwen, average AUROC increases with each SFT stage, with vision-encoder tuning providing the largest increment. Disease entries report mean

\pm

SD; Avg is the unweighted average of the reported disease means.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Self-reflector	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
GPT-5-nano reflector	59.9	55.7	68.0	61.8	72.1	57.5	66.7	58.3
GPT-5-mini reflector	58.3	50.0	68.0	61.8	77.3	58.6	67.9	56.8

Appendix

Table 8: Reflector substitution on Qwen3-VL-8B + SFT (VE-tuned). Self-reflection outperforms both GPT reflectors on all three diseases in AUROC and balanced accuracy. Values are test AUROC and balanced accuracy (%). Avg is the unweighted average across diseases.

Cross-model transfer: prompt optimized on MedGemma-4B, evaluated on Qwen3-VL-8B + SFT
	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B + SFT (VE-tuned)	63.3	45.7	68.0	59.5	72.6	62.5	68.0	55.9
+ Accuracy-PE	63.6	45.4	68.1	58.0	72.4	61.8	68.0	55.1
+ Ranking-PE (ours)	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
+ Transferred Ranking-PE prompt	64.4	58.2	68.1	63.5	78.1	66.1	70.2	62.6

Appendix

Table 9: Cross-model prompt transfer from MedGemma-4B to Qwen3-VL-8B + SFT without further prompt optimization. On average, the transferred prompt outperforms Accuracy-PE run on the target model, suggesting that the discovered instructions are not entirely model-specific. Values are test AUROC and balanced accuracy (%); Avg is the unweighted average across diseases.

Method	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	$97.9$	$82.9$	$79.0$	$86.6$
Qwen3-VL-8B + SFT (VE-tuned)	$98.2$	$85.4$	$81.4$	$88.3$
+ Accuracy-PE	$98.2\pm 0.1$	$84.0\pm 1.3$	$81.3\pm 0.2$	$87.8$
+ BAcc-Select	$98.2\pm 0.1$	$84.0\pm 2.8$	$76.7\pm 4.3$	86.3
+ Class-Weighted PE	$98.3\pm 0.0$	$84.8\pm 1.5$	$79.5\pm 1.6$	87.5
+ Scalar-AUROC PE	$98.1\pm 0.2$	$84.0\pm 1.3$	$79.2\pm 1.1$	87.1

Appendix

Table 10: Test AUPRC (%). For each disease, PE methods report mean

\pm

SD; models without PE report means only. Avg is the unweighted average of the reported disease means.

$e$	String appended to $\mu_{f}$
$\mathrm{TP}$	TRUE POSITIVE --- correctly identified $d$ as present.
$\mathrm{TN}$	TRUE NEGATIVE --- correctly ruled out $d$ .
$\mathrm{FN}$	FALSE NEGATIVE (clinically dangerous) --- missed $d$ . Re-examine the image for subtle evidence.
$\mathrm{FP}$	FALSE POSITIVE --- over-called $d$ . Require converging evidence before committing.

Appendix

Table 11: Exact feedback string appended to

\mu_{f}

for each value of

e(\hat{y}_{n,k},y_{n})

. Here

d

is the target disease for the instance (§ 3 ). Only

e=\mathrm{FN}

carries the “(clinically dangerous)” annotation.

Feedback level	AUROC	Bal. Acc.
Basic (correctness only)	77.6	64.6
+ clinical error type	78.2	65.5
+ confidence $\|s(x)\|$	79.8	63.8
+ cross-example rank context (full)	79.3	68.5

Appendix

Table 12: Feedback-signal ablation on Consolidation with Qwen3-VL-8B + SFT (VE-tuned). Signals are added cumulatively with pair-level Pareto and AUROC final selection. Values are test AUROC and balanced accuracy (%).

Panel of Figure 4 — Figure 4: Ranking-PE produces qualitatively different prompts. Both prompts are evolved from the same seed prompt under identical infrastructure, rollout budget, and reflector; they differ in the three Ranking-PE components (§ 4.2.3 ): AUROC-based selection, the pair-indexed scores matrix driving Pareto dominance, and ranking-shaped feedback. Under Accuracy-PE the reflector converges to a hard-threshold flowchart that pushes asymmetrically toward “Yes”, licenses “No” on degraded images, and lets labs override imaging. Under Ranking-PE, the same reflector converges to a deliberative reasoning surface: a calibration guard, two-sided error remedies, anti-dismissal under visual ambiguity, labs as calibration rather than override, and a structured etiology section. Excerpts are from Qwen3-VL-8B SFT on Cardiomegaly and are illustrative.

Explore similar work

Jun 5, 2026cs.CL

LLM-Guided Evolution for Medical Decision Pipelines

Adapting large language models (LLMs) to clinical workflows often requires costly fine-tuning or manual prompt and pipeline engineering. We study LLM-guided MAP-Elites evolution as an inference-time alternative for discovering medical decision strategies and provide an implementation repository at https://github.com/univanxx/llm_guided_evo_medical. We formulate urgency triage, interactive consultation, and medical image classification as evolutionary searches over executable artifacts optimized by task-specific fitness functions. Across all three settings, evolution improves over manually designed baselines under practical constraints. In triage, evolved programs increase Semigran accuracy from

77.3\%

87.1\%

and emergency recall from

0.60

0.97

, while improving safety-weighted held-out MIMIC-ESI performance. In interactive consultation, evolved policies improve the accuracy--cost frontier across Llama-3, Qwen-3.5, and Gemma-4 and transfer to held-out iCRAFTMD. In PneumoniaMNIST, prompt-only evolution improves frozen MedGemma VLMs while preserving strict JSON outputs. Qualitative analysis shows that the gains come from interpretable program-level mechanisms, calibrated triage boundaries, targeted evidence acquisition, selective commitment, and finding-oriented visual decision rules, rather than superficial prompt rewording alone.

Ivan Sviridov, Artem Oskin, Ivan Panin +4

1Sber AI Lab · 2AIRI

Triage Alphaevolve Evolution

Ask PDF

Jul 28, 2026cs.CL

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning. However, existing evaluations of multimodal large language models (MLLMs) typically rely on single-turn or isolated tasks, making it difficult to fully capture the complexity of real-world clinical diagnosis. To bridge this gap, we developed ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date. ClinMM-Bench contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties. We systematically evaluated 15 representative MLLMs using a two-level evaluation framework that assessed both diagnostic accuracy and diagnostic reasoning quality. Results showed that proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited across all models. In terms of diagnostic reasoning quality, current models can identify plausible diagnostic directions but still have considerable limitations in generating reliable diagnostic reasoning. Error analysis further identified five representative failure modes: information synthesis failure, knowledge mapping error, perception error, premature closure, and visual hallucination.

Rui Yang, Weihao Xuan, Yi Lin +23

Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24

Multimodal Clinical Data Clinical Reasoning Diagnostic Benchmark

Ask PDF

Sep 21, 2026cs.AI

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards

Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.

Kalash Shah, Kunal Singh, Snehan J +1

Fractal AI Research

Topic pending

Ask PDF