Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Tian Xia; Minghao Liu; Yiqing Liang; Laixi Shi; Jiayun Wang

cs.LGSep 30, 2026

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Authors: Tian Xia, Minghao Liu, Yiqing Liang, Laixi Shi, Jiayun Wang

Organizations: Harvard University · University of California, Santa Cruz · Brown University · Johns Hopkins University · Georgia Institute of Technology

Abstract

Multimodal large language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores matrix with one row per evaluation instance and one column per candidate prompt; cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the prompt evolution search reads from - the scores matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via vision-encoder-tuned SFT or medical pretraining - is a prerequisite that prompt search cannot replace - our recipe extends reflective prompt evolution from text-only data to multimodal clinical decision-making.

Figures & tables

Figure 1: Why clinical diagnosis needs ranking metrics. a) : multimodal diagnosis combines medical images (e.g. chest X-rays), clinical profile, lab tests, and prompts to produce calibrated disease scores. b) : under heavy class imbalance common in real-world clinical datasets, threshold accuracy can look strong even when the score ranking is clinically useless, while AUROC distinguishes models across all operating points. c) : our recipe improves ranking by combining a base MLLM, SFT with vision-encoder tuning, and ranking-aware prompt evolution, moving from threshold-centric prediction to AUROC-driven optimization for reliable multimodal clinical diagnosis.

Figure 2: Ranking-aware prompt evolution. (left) Pair-level Pareto scores matrix: rows are

(x^{+},x^{-})

pairs, cells are

1

for correct order,

0.5

for ties, and

0

for reversed order. The column average over a candidate prompt

\Phi_{k}

is its validation AUROC. (right top) Reflective pipeline: at each step we select a

\Phi_{k}

from the pool, run it on a mini-batch of

(P,N)

, collect ranking-shaped feedback (clinical error type, confidence magnitude, cross-example rank context), and let the reflector propose

\Phi_{k+1}

, which is pushed back into the pool. (right bottom) Final selection:

\Phi^{*}=\arg\max_{k}\mathrm{AUROC}(\Phi_{k})

, which reduces to a column-mean argmax once the matrix records pairwise ordering events.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B	65.4	64.0	61.3	51.2	74.0	64.4	66.9	59.9
Gemma3-4B	38.2	48.9	55.1	49.7	48.6	54.3	47.3	51.0
GPT-4o-mini	60.1	60.2	54.7	50.5	70.5	53.0	61.8	54.6
GPT-5-mini	50.6	58.2	71.5	66.8	75.9	64.6	66.0	63.2
Gemini-2.5-flash	46.5	54.3	69.0	58.3	75.1	66.4	63.5	59.7

Table 1: Test AUROC and balanced accuracy (%) on three MIMIC disease tasks. Ranking-PE outperforms Accuracy-PE on AUROC across both backbones and all diseases. Open-source PE rows report means; per-disease SDs are in Appendix Table 5 . Avg is the unweighted disease average. Closed-source rows are zero-shot. Endpoints without top-

k

logprobs use a three-level verbal score, which limits comparison with logprob-based scores (Appendix A.1 ).

Model	Configuration	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	Base	65.4	61.3	74.0	66.9
	$+$ Ranking-PE	63.5	65.8	76.0	68.4
	$+$ SFT	63.3	68.0	72.6	68.0
	$+$ SFT $+$ Ranking-PE	70.7	71.3	79.3	73.8
Gemma3-4B	Base	38.2	55.1	48.6	47.3
	$+$ Ranking-PE	50.5	55.1	49.3	51.6

Table 2: Test AUROC (%) across model adaptation settings. Qwen3-VL-8B uses a

2\times 2

comparison of SFT and Ranking-PE; SFT uses a tuned vision encoder. Gemma3-4B and medically pretrained MedGemma-4B are evaluated before and after Ranking-PE. PE entries are means; their SDs are reported in Appendix Table 7 . Avg is the unweighted average of the three disease values.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Accuracy-PE	63.6	45.4	68.1	58.0	72.4	61.8	68.0	55.1
+ AUROC final selection	43.7	51.5	68.3	56.5	72.0	63.9	61.3	57.3
+ pair-level Pareto	63.8	58.6	70.6	64.9	77.6	64.6	70.7	62.7
+ ranking feedback (full)	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
Ranking-PE (full) $-$ pair-level Pareto	56.0	51.1	68.2	61.4	73.3	60.0	65.8	57.5

Table 3: Component ablation on Qwen3-VL-8B + SFT (VE-tuned). Entries are mean test AUROC and balanced accuracy (%). The first four rows form a cumulative ablation; the last row replaces pair-level Pareto with an instance-level matrix while retaining ranking feedback. Avg averages the three disease means. SDs appear in Appendix Table 6 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

max_pairs	AUROC	Bal. Acc.
$300$	$71.2$	$67.3$
$1{,}000$	$70.6$	$61.4$
$3{,}000$	$70.6$	$65.4$
$10{,}000$ (default)	$71.3$	$66.5$

Appendix

Table 4: Sensitivity to the pair cap on Cardiomegaly with Qwen3-VL-8B + SFT (VE-tuned) and Ranking-PE. Values are test AUROC and balanced accuracy (%).

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B + SFT (VE-tuned)
Accuracy-PE	$63.6\pm 0.5$	$45.4\pm 0.6$	$68.1\pm 1.2$	$58.0\pm 1.4$	$72.4\pm 0.2$	$61.8\pm 1.0$	68.0	55.1
BAcc-Select	$63.9\pm 0.4$	$43.7\pm 0.4$	$67.6\pm 1.0$	$60.6\pm 0.7$	$71.4\pm 1.3$	$63.3\pm 0.6$	67.6	55.9
Class-Weighted PE	$64.3\pm 0.2$	$44.2\pm 0.4$	$68.2\pm 1.7$	$61.3\pm 1.4$	$71.5\pm 0.7$	$61.8\pm 3.2$	68.0	55.8
Scalar-AUROC PE	$66.5\pm 3.6$	$65.5\pm 0.8$	$69.3\pm 1.6$	$63.1\pm 1.1$	$72.9\pm 1.6$	$61.2\pm 3.9$	69.6	63.3

Appendix

Table 5: Variability of the PE results in Table 1 . Per-disease test AUROC and balanced accuracy (%) are mean

\pm

SD. Avg is the unweighted average of the reported disease means; no aggregate SD is reported.

Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
	Atelectasis		Cardiomegaly		Consolidation		Avg
Accuracy-PE	$63.6\pm 0.5$	$45.4\pm 0.6$	$68.1\pm 1.2$	$58.0\pm 1.4$	$72.4\pm 0.2$	$61.8\pm 1.0$	68.0	55.1
+ AUROC final selection	$43.7\pm 2.8$	$51.5\pm 0.7$	$68.3\pm 1.9$	$56.5\pm 1.7$	$72.0\pm 0.0$	$63.9\pm 2.6$	61.3	57.3
+ pair-level Pareto	$63.8\pm 3.4$	$58.6\pm 2.1$	$70.6\pm 0.9$	$64.9\pm 0.4$	$77.6\pm 2.7$	$64.6\pm 2.1$	70.7	62.7
+ ranking feedback (full)	$70.7\pm 0.9$	$70.9\pm 4.3$	$71.3\pm 1.0$	$66.5\pm 2.1$	$79.3\pm 1.2$	$68.5\pm 1.2$	73.8	68.6
Ranking-PE (full) $-$ pair-level Pareto	$56.0\pm 4.4$	$51.1\pm 1.8$	$68.2\pm 0.1$	$61.4\pm 0.4$	$73.3\pm 0.5$	$60.0\pm 4.3$	65.8	57.5

Appendix

Table 6: Variability of the component ablation in Table 3 . Per-disease AUROC and balanced accuracy (%) are mean

\pm

SD. Avg averages the disease means.

Model	SFT setting	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	None	$63.5\pm 3.6$	$65.8\pm 0.4$	$76.0\pm 3.8$	68.4
	VE-frozen	$65.9\pm 0.8$	$67.3\pm 1.2$	$77.6\pm 0.8$	70.3
	VE-tuned	$70.7\pm 0.9$	$71.3\pm 1.0$	$79.3\pm 1.2$	73.8
Gemma3-4B	None	$50.5\pm 4.3$	$55.1\pm 0.0$	$49.3\pm 1.3$	51.6
MedGemma-4B	None	$69.1\pm 3.1$	$63.0\pm 4.5$	$77.1\pm 3.8$	69.7

Appendix

Table 7: Test AUROC (%) across SFT settings and model backbones. All configurations use Ranking-PE. On Qwen, average AUROC increases with each SFT stage, with vision-encoder tuning providing the largest increment. Disease entries report mean

\pm

SD; Avg is the unweighted average of the reported disease means.

	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Self-reflector	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
GPT-5-nano reflector	59.9	55.7	68.0	61.8	72.1	57.5	66.7	58.3
GPT-5-mini reflector	58.3	50.0	68.0	61.8	77.3	58.6	67.9	56.8

Appendix

Table 8: Reflector substitution on Qwen3-VL-8B + SFT (VE-tuned). Self-reflection outperforms both GPT reflectors on all three diseases in AUROC and balanced accuracy. Values are test AUROC and balanced accuracy (%). Avg is the unweighted average across diseases.

Cross-model transfer: prompt optimized on MedGemma-4B, evaluated on Qwen3-VL-8B + SFT
	Atelectasis		Cardiomegaly		Consolidation		Avg
Method	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.	AUROC	Bal. Acc.
Qwen3-VL-8B + SFT (VE-tuned)	63.3	45.7	68.0	59.5	72.6	62.5	68.0	55.9
+ Accuracy-PE	63.6	45.4	68.1	58.0	72.4	61.8	68.0	55.1
+ Ranking-PE (ours)	70.7	70.9	71.3	66.5	79.3	68.5	73.8	68.6
+ Transferred Ranking-PE prompt	64.4	58.2	68.1	63.5	78.1	66.1	70.2	62.6

Appendix

Table 9: Cross-model prompt transfer from MedGemma-4B to Qwen3-VL-8B + SFT without further prompt optimization. On average, the transferred prompt outperforms Accuracy-PE run on the target model, suggesting that the discovered instructions are not entirely model-specific. Values are test AUROC and balanced accuracy (%); Avg is the unweighted average across diseases.

Method	Atelectasis	Cardiomegaly	Consolidation	Avg
Qwen3-VL-8B	$97.9$	$82.9$	$79.0$	$86.6$
Qwen3-VL-8B + SFT (VE-tuned)	$98.2$	$85.4$	$81.4$	$88.3$
+ Accuracy-PE	$98.2\pm 0.1$	$84.0\pm 1.3$	$81.3\pm 0.2$	$87.8$
+ BAcc-Select	$98.2\pm 0.1$	$84.0\pm 2.8$	$76.7\pm 4.3$	86.3
+ Class-Weighted PE	$98.3\pm 0.0$	$84.8\pm 1.5$	$79.5\pm 1.6$	87.5
+ Scalar-AUROC PE	$98.1\pm 0.2$	$84.0\pm 1.3$	$79.2\pm 1.1$	87.1

Appendix

Table 10: Test AUPRC (%). For each disease, PE methods report mean

\pm

SD; models without PE report means only. Avg is the unweighted average of the reported disease means.

$e$	String appended to $\mu_{f}$
$\mathrm{TP}$	TRUE POSITIVE --- correctly identified $d$ as present.
$\mathrm{TN}$	TRUE NEGATIVE --- correctly ruled out $d$ .
$\mathrm{FN}$	FALSE NEGATIVE (clinically dangerous) --- missed $d$ . Re-examine the image for subtle evidence.
$\mathrm{FP}$	FALSE POSITIVE --- over-called $d$ . Require converging evidence before committing.

Appendix

Table 11: Exact feedback string appended to

\mu_{f}

for each value of

e(\hat{y}_{n,k},y_{n})

. Here

d

is the target disease for the instance (§ 3 ). Only

e=\mathrm{FN}

carries the “(clinically dangerous)” annotation.

Feedback level	AUROC	Bal. Acc.
Basic (correctness only)	77.6	64.6
+ clinical error type	78.2	65.5
+ confidence $\|s(x)\|$	79.8	63.8
+ cross-example rank context (full)	79.3	68.5

Appendix

Table 12: Feedback-signal ablation on Consolidation with Qwen3-VL-8B + SFT (VE-tuned). Signals are added cumulatively with pair-level Pareto and AUROC final selection. Values are test AUROC and balanced accuracy (%).

Panel of Figure 4 — Figure 4: Ranking-PE produces qualitatively different prompts. Both prompts are evolved from the same seed prompt under identical infrastructure, rollout budget, and reflector; they differ in the three Ranking-PE components (§ 4.2.3 ): AUROC-based selection, the pair-indexed scores matrix driving Pareto dominance, and ranking-shaped feedback. Under Accuracy-PE the reflector converges to a hard-threshold flowchart that pushes asymmetrically toward “Yes”, licenses “No” on degraded images, and lets labs override imaging. Under Ranking-PE, the same reflector converges to a deliberative reasoning surface: a calibration guard, two-sided error remedies, anti-dismissal under visual ambiguity, labs as calibration rather than override, and a structured etiology section. Excerpts are from Qwen3-VL-8B SFT on Cardiomegaly and are illustrative.

Explore similar work

LLM-Guided Evolution for Medical Decision Pipelines
Jun 5, 2026Ivan Sviridov, Artem Oskin, Ivan Panin +4Triage Alphaevolve
1Sber AI Lab · 2AIRI
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Jul 28, 2026Rui Yang, Weihao Xuan, Yi Lin +23Multimodal Clinical Data Clinical Reasoning
Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24
Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Sep 21, 2026Kalash Shah, Kunal Singh, Snehan J +1
Fractal AI Research

Jun 5, 2026, cs.CLJ/K move · Enter open · S save

LLM-Guided Evolution for Medical Decision Pipelines

Ivan Sviridov, Artem Oskin, Ivan Panin, Iaroslav Bespalov+3

1Sber AI Lab · 2AIRI

Triage Alphaevolve Evolution

Open paper AskPDF arXiv

Adapting large language models (LLMs) to clinical workflows often requires costly fine-tuning or manual prompt and pipeline engineering. We study LLM-guided MAP-Elites evolution as an inference-time alternative for discovering medical decision strategies and provide an implementation repository at https://github.com/univanxx/llm_guided_evo_medical. We formulate urgency triage, interactive consultation, and medical image classification as evolutionary searches over executable artifacts optimized by task-specific fitness functions. Across all three settings, evolution improves over manually designed baselines under practical constraints. In triage, evolved programs increase Semigran accuracy from

77.3\%

87.1\%

and emergency recall from

0.60

0.97

, while improving safety-weighted held-out MIMIC-ESI performance. In interactive consultation, evolved policies improve the accuracy--cost frontier across Llama-3, Qwen-3.5, and Gemma-4 and transfer to held-out iCRAFTMD. In PneumoniaMNIST, prompt-only evolution improves frozen MedGemma VLMs while preserving strict JSON outputs. Qualitative analysis shows that the gains come from interpretable program-level mechanisms, calibrated triage boundaries, targeted evidence acquisition, selective commitment, and finding-oriented visual decision rules, rather than superficial prompt rewording alone.

Abstract

Explore similar work

LLM-Guided Evolution for Medical Decision Pipelines

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards