BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models

Kaushitha Silva; Mansitha Eashwara; Sanduni Ubayasiri; Ruwan Tennakoon; Damayanthi Herath

cs.CVOct 17, 2025

BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models

Authors: Kaushitha Silva, Mansitha Eashwara, Sanduni Ubayasiri, Ruwan Tennakoon, Damayanthi Herath

Organizations: University of Peradeniya Peradeniya, Sri Lanka · RMIT University Melbourne, Australia

Abstract

The clinical adoption of biomedical vision-language models is hindered by prompt optimization techniques that produce either uninterpretable latent vectors or single textual prompts. This lack of transparency and failure to capture the multi-faceted nature of clinical diagnosis, which relies on integrating diverse observations, limits their trustworthiness in high-stakes settings. To address this, we introduce BiomedXPro, an evolutionary framework that leverages a large language model as both a biomedical knowledge extractor and an adaptive optimizer to automatically generate a diverse ensemble of interpretable, natural-language prompt pairs for disease diagnosis. Experiments on multiple biomedical benchmarks show that BiomedXPro consistently outperforms state-of-the-art prompt-tuning methods, particularly in data-scarce few-shot settings. Furthermore, our analysis demonstrates a strong semantic alignment between the discovered prompts and statistically significant clinical features, grounding the model's performance in verifiable concepts. By producing a diverse ensemble of interpretable prompts, BiomedXPro provides a verifiable basis for model predictions, representing a critical step toward the development of more trustworthy and clinically-aligned AI systems.

Figures & tables

Dataset	Method	1-Shot	2-Shot	4-Shot	8-Shot	16-Shot
Camelyon17-WILDS	Zero-shot	41.93
	CoOp	78.79	70.76	74.15	84.25	88.47
	CoCoOp	76.43	66.13	75.19	85.13	86.88
	BiomedCoOp	53.15	61.06	58.69	63.46	56.94
	XCoOp	66.90	35.60	45.00	64.20	84.50
	BiomedXPro(Ours)	72.06	86.95	90.20	90.87	90.38

Table 1: F1-macro percentage results of BiomedXPro compared with baseline methods. Camelyon17-WILDS evaluates domain generalization (train on 3 hospitals, test on unseen). Derm7pt evaluates binary melanoma classification. WBCAtt extends to multiclass via one-vs-rest classifiers.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Absence or Normal Appearance ( $p_{j}^{-}$ )	Disease-Indicative Observations ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
Pigmentation is limited to brown tones with sharp borders.	Pigmentation is variegated with shades of brown, black, blue, and grey, with ill-defined borders.	0.6741
Pigmentation is homogenous and limited to the epidermis.	Pigmentation is heterogeneous and extends into the dermis, causing a bluish hue.	0.6724
Blue-whitish veil is absent.	Blue-whitish veil is present and associated with ulceration.	0.6667
Pigmentation is limited to the periphery with a normal skin color center.	Pigmentation extends to the center of the lesion with shades of black, grey, and blue.	0.6565
Dots and globules have a smooth, rounded appearance.	Dots and globules are angular, irregular, and exhibit pseudopods.	0.6543
Pigmentation is light brown or tan.	Pigmentation is dark brown or black.	0.6535

Appendix

Table 2: Prompts Learned for the Derm7pt full dataset: F1-macro utilized as the fitness function

Absence or Normal Appearance ( $p_{j}^{-}$ )	Disease-Indicative Observations ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
Blue whitish veil is absent or minimal.	Blue whitish veil is pronounced and covers a significant portion of the lesion.	1.0
Blue whitish veil is absent or very subtle.	Blue whitish veil is prominent and widespread, obscuring the underlying structures.	1.0
Dots are evenly distributed and small.	Dots are clustered and large, with irregular shapes.	1.0
Dots and globules are small, round, and uniformly pigmented.	Dots and globules are large, irregular, and exhibit color variation.	1.0
Blue-whitish veil is absent or minimal, with no associated changes.	Blue-whitish veil is dense and associated with surrounding inflammation.	1.0

Appendix

Table 3: Prompts Learned for the 1-Shot Derm7pt Case: Inverted BCE is utilized for the fitness function

Negative Class Observation ( $p_{j}^{-}$ )	Neutrophil-Indicative Observation ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
The cell shape is consistently round with a smooth perimeter.	The cell shape is often irregular and slightly distorted due to the segmented nucleus.	0.7945
Cytoplasm is clear and homogenous, lacking distinct features.	Cytoplasm is abundant and contains fine, pale lilac granules.	0.7710
The nuclear shape is predominantly spherical with a smooth outline.	The nuclear shape is irregular and lobulated, with a clearly indented outline.	0.7565
Granules, if present, are sparse and difficult to discern.	Granules are abundant, easily visible, and fill a significant portion of the cytoplasm.	0.7487
The cell shape is generally round or slightly irregular.	The cell exhibits a more polymorphic shape, often displaying pseudopods or extensions.	0.7467
The cell shape is consistently oval or spherical, maintaining a regular form.	The cell shape is highly variable and often irregular, exhibiting pseudopod extensions.	0.7418

Appendix

Table 4: Prompts Learned for Neutrophil Classification (Full-Shot Scenario): F1-macro as Fitness Function

Negative Class Observation ( $p_{j}^{-}$ )	Neutrophil-Indicative Observation ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
Basophils granules contain histamine and heparin, giving them a metachromatic staining property.	Neutrophils granules contain enzymes like myeloperoxidase and lysozyme, staining specifically with neutral dyes.	1.0000
Lymphocytes have a condensed, hyperchromatic nucleus.	Neutrophils have a segmented nucleus with varying degrees of chromatin condensation in each lobe.	1.0000
Basophils often have a U- or S-shaped nucleus due to the abundance of granules.	Neutrophils maintain a consistently segmented nuclear shape, even with granule presence.	1.0000
Eosinophils have granules that are rich in major basic protein and eosinophil cationic protein.	Neutrophils have granules containing elastase and collagenase.	1.0000
Basophils have granules that stain intensely with basic dyes.	Neutrophils have granules that stain weakly with both acidic and basic dyes.	1.0000
Basophils exhibit a high nuclear-cytoplasmic ratio, with a relatively small amount of cytoplasm.	Neutrophils have a low nuclear-cytoplasmic ratio, with a significant amount of cytoplasm.	0.9942

Appendix

Table 5: Prompts Learned for Neutrophil Classification (1-Shot Scenario): Inverted BCE as Fitness Function

Absence or Normal Appearance ( $p_{j}^{-}$ )	Disease-Indicative Observations ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
Cells arranged in a normal lymphatic pattern.	Cells arranged in sheets or solid nests.	0.9998
No infiltration of atypical cells.	Infiltration of atypical cells disrupting normal lymph node architecture.	0.9998
No plasma cells are identified.	Numerous plasma cells with eccentric nuclei and abundant cytoplasm.	0.9994
Few scattered reactive lymphocytes.	Sheets of atypical lymphocytes with brisk proliferation.	0.9992
Immunocytes are rare.	Abundant immunocytes are present, forming rosettes.	0.9988
Mature lymphocytes.	Immunoblasts and plasmablasts.	0.9987

Appendix

Table 6: Prompts Learned for Camelyon17-WILDS (1-Shot Scenario): Inverted BCE as Fitness Function

Absence or Normal Appearance ( $p_{j}^{-}$ )	Disease-Indicative Observations ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
No increase in mitotic activity.	Increased mitotic activity.	0.9193
No increase in eosinophilic cytoplasm.	Increased eosinophilic cytoplasm in tumor cells.	0.8982
Normal follicular architecture is preserved.	Disrupted follicular architecture with loss of polarity.	0.8904
Lymphocytes are small and round with dense nuclei.	Lymphocytes are large and atypical with prominent nucleoli.	0.8859
Benign inflammatory infiltrate.	Aggressive, atypical inflammatory infiltrate.	0.8837
No prominent nucleolus.	Large and prominent nucleoli within tumor cells.	0.8822

Appendix

Table 7: Prompts Learned for Camelyon17-WILDS (16-Shot Scenario): F1-macro as Fitness Function

Clinical Observation	P(Melanoma $\|$ Obs.)
Vascular Structures: Linear Irregular	0.80
Blue-Whitish Veil: Present	0.59
Regression Structures: Combinations	0.58
Pigment Network: Atypical	0.55
Streaks: Irregular	0.53
Pigmentation: Diffuse Irregular	0.48

Appendix

Table 8: Conditional probabilities of melanoma given specific clinical observations, calculated on the Derm7pt training dataset (N=412).

Cytological Observation	P(Neutrophil $\|$ Obs.)
Granule Type: Small	1.00
Granule Colour: Pink	1.00
Nucleus Shape: Unsegmented-band	0.85
Granularity: Yes	0.43
Cytoplasm Colour: Light blue	0.43
Cytoplasm Texture: Clear	0.40

Appendix

Table 9: Conditional probabilities of a cell being a Neutrophil given specific cytological observations, calculated on the WBCAtt training dataset.

Negative Class Observation ( $p_{j}^{-}$ )	Eosinophil-Indicative Observation ( $p_{j}^{+}$ )	Fitness Score ( $s_{j}$ )
Neutrophils exhibit granules that stain neutrally, appearing a faint lilac color.	Eosinophil granules are intensely stained with eosin, resulting in a bright red-orange color.	0.8311
Monocytes exhibit a dull, greyish cytoplasm.	Eosinophils display a cytoplasm with a pale pink or slightly orange hue.	0.8247
Basophils have granules that stain metachromatically, appearing purple-black.	Eosinophil granules stain specifically with eosin, resulting in a bright orange-red color.	0.8179
Basophils display dark blue or purple granules that often overwhelm the cytoplasm.	Eosinophil granules are characteristically bright pink or red-orange, contrasting with the basophilic granules of basophils.	0.8092
Monocytes display a folded or indented nuclear membrane.	Eosinophil nuclei are generally round or oval with a smooth, regular membrane.	0.8079
Monocytes exhibit a ”cloud-like” or ”ground glass” appearance to their cytoplasm.	Eosinophil cytoplasm is filled with distinct, refractile granules, lacking the diffuse appearance of monocytes.	0.8015

Appendix

Table 10: Prompts Learned for Eosinophil Classification (Full-Shot Scenario)

Figure 1: Effect of prompt selection strategy on convergence. Our proposed Roulette Wheel selection achieves the best balance of exploration and exploitation, leading to more stable and consistent performance gains compared to Best-N and Random selection.

Figure 2: Impact of generation size per iteration on performance. The 10-pair setting (our choice) provides the optimal balance, outperforming 5 pairs and 50 pairs.

Figure 3: Effect of Chain-of-Thought (CoT) prompting, which consistently outperforms a direct-instruction baseline.

Figure 4: Effect of initial population size on performance. A larger and more diverse initial population (50) consistently outperforms smaller sets

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1VLM Robustness Prompt Learning
Wenzhou University · Wenzhou Business College
Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning
Mar 9, 2026Mohamed Harmanani, Bining Long, Zhuoxin Guo +6Explainable Medical Image Analysis Medical Reasoning
Queen’s University · Vector Institute · McMaster University +1
Uncertainty Reasoning with Large Language Models for Explainable Disease Diagnosis
May 25, 2026Xiaoyang Fan, Yufan Cai, Zhe Hou +1Clinical Decision Support Clinical Reasoning
National University of Singapore · Griffith University

May 15, 2026J/K move · Enter open · S save

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

Huanyang Tong, Kai Liu, Fangjun Kuang, Huiling Chen

Wenzhou University · Wenzhou Business College

VLM Robustness Prompt Learning Healthcare VLM Adaptation Medical VLMs Vision-Language Alignment

Open paper AskPDF arXiv

Biomedical Vision--Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: fragility to prompt variations.Existing adaptation frameworks typically optimize visual and textual prompts as independent streams, relying on ideal ``Golden Prompts''. In clinical reality, where descriptions are often noisy and heterogeneous, this modality isolation leads to unstable cross-modal alignment. To address this, we propose BiomedAP, a vision-informed dual-anchor framework with gated cross-modal fusion.BiomedAP enforces synergistic alignment through two mechanisms: (1) Gated Cross-Modal Fusion, which enables layer-wise interaction between modalities, acting as a dynamic noise regulator to suppress irrelevant textual cues; and (2) a Dual-Anchor Constraint that regularizes learnable prompts toward stable semantic centroids derived from both expert templates (High Anchors) and few-shot visual prototypes (Low Anchors). Extensive experiments across 11 benchmarks demonstrate that BiomedAP consistently surpasses baselines, achieving competitive few-shot accuracy and markedly enhanced robustness under prompt perturbations. Our code is available at: https://github.com/tongdiedie/BiomedAP. Keywords: Vision-Language Models; Prompt Learning; Parameter-Efficient Fine-Tuning; Few-shot Learning

Abstract

Medical VLMs

Explore similar work

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

Uncertainty Reasoning with Large Language Models for Explainable Disease Diagnosis