MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence

Yingshu Li; Shaoyang Zhou; Zhanyu Wang; Yunyi Liu; Xinyu Liang; Xi Zhang; Lingqiao Liu; Lei Wang

cs.CVNov 2, 2026

MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence

Authors: Yingshu Li, Shaoyang Zhou, Zhanyu Wang, Yunyi Liu, Xinyu Liang, Xi Zhang, Lingqiao Liu, Lei Wang,

Organizations: The University of Sydney, Sydney, Australia · College of Medical Imaging Shandong Medical and Pharmaceutical University Yantai, China · University of Glasgow Glasgow, UK · Adelaide University Adelaide, Australia ·

Abstract

Clinical evidence is inherently relational: medical images are interpreted with prior studies, cross-modal examinations, reports, and localized findings, and retrieving relevant evidence is therefore central to clinical interpretation. Recent foundation models increasingly encode these heterogeneous sources into shared embedding spaces, yet existing embedding evaluation often focuses on isolated matches, leaving unclear whether current models retrieve relevant evidence across changes in modality, appearance, and granularity. We introduce MedHEB, a unified benchmark with 107 retrieval tasks from 54 datasets, including both in-distribution and held-out-source evaluation, across three axes: evidence form (2D images, 3D volumes, text), pairing type (image--image, image--text, text--text, and modality-bridging visual matching), and relevance granularity (global, question-conditioned, region-level). Under a single ranking protocol, MedHEB varies clinically relevant evidence definitions to test whether embedding proximity remains meaningful across heterogeneous evidence, including modality-bridging visual retrieval and region-level text-to-region retrieval. Across representative models, performance is uneven across retrieval families: models strong on conventional image--text or text--text retrieval are less reliable on modality-bridging and region-level tasks. Current embedding spaces capture some clinical relationships well but do not yet provide balanced retrieval across heterogeneous evidence. We further provide MedEmb, an MLLM-based reference embedding model trained on the MedHEB ID split, as a reproducible baseline for unified 2D--3D--text medical retrieval. We release MedHEB with preprocessing scripts, evaluation code, access manifests, and MedEmb to support research on clinically meaningful medical embeddings.

Figures & tables

Figure 1: MedHEB design across clinical evidence axes and anatomical domains. The schematic organizes 54 datasets and 107 tasks by evidence form, pairing type, and relevance granularity.

Benchmark	Year	Evidence Form			Retrieval Setting
		2D	3D	Text	I–T	T–T	I–I	MB-VR	Region
BioASQ [ 16 ]	2015			✓		✓
NFCorpus [ 15 ]	2016			✓		✓
3D-MIR [ 19 ]	2023		✓				✓
BIMCV-R [ 20 ]	2024		✓	✓	✓
M3Retrieve [ 18 ]	2025	✓		✓	✓	✓

Table 1: Coverage of representative medical retrieval benchmarks. I–T, T–T, I–I, and MB-VR denote image–text, text–text, image–image, and modality-bridging visual retrieval.

Figure 2: MedHEB task composition and representative retrieval examples. A: Hierarchical distribution of 107 tasks and 75,233 queries by evidence form, retrieval family, and source task, with side bars for family–modality and anatomy–modality counts. B: Query–candidate examples across modality-bridging, region-level, image–text, VQA, and biomedical text retrieval.

Evidence Form	ID Tasks	Held-out-source Tasks	Total Tasks
2D images	60	9	69
3D volumes	30	1	31
Biomedical text	6	1	7
Total	96	11	107

Table 2: Composition of MedHEB retrieval tasks by evidence form and split.

Family	Query	Candidates & positive criterion	Representative sources
Image–image (20 tasks)	2D image or 3D volume	Same-source studies; positive if sharing the query’s class, diagnosis, or anatomy label.	e.g., Kvasir, LC25000, RFMiD [ 30 , 31 , 32 ]
Modality-bridging visual (10 tasks)	Image or volume from one modality	Different-modality studies paired across MRI T1 $\leftrightarrow$ T2, CT $\leftrightarrow$ MRI, or X-ray $\leftrightarrow$ CT. Identifiers define positives only; not given to the model.	e.g., BraTS, SynthRAD2023, HaN-Seg, CT-RATE [ 33 , 34 , 35 , 36 , 37 ]
Image-to-label (22 tasks)	Image or volume	Textual class labels; positive if matching the source label.	e.g., CheXpert-Plus, MedMNIST [ 38 , 39 ]
Image-to-report (7 tasks)	Image or volume	Reports or captions; positive is the source-paired report.	e.g., MIMIC-CXR, M3D, RadGen-CT [ 40 , 38 , 41 , 42 , 43 , 37 , 44 , 45 , 46 ]
Text-to-image (23 tasks)	Class label or short clinical description	Same-source images matching the queried label or description.	Reuses image-to-label and image-to-report image sources.
Question-conditioned (11 tasks)	Image–question pair	Dataset-specific answer set; positive if matching the ground-truth answer.	e.g., MIMIC-CXR-VQA, OmniMedVQA, Path-VQA [ 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ]

Table 3: MedHEB retrieval families. Each family follows the same ranking protocol but differs in query form, candidate pool. Full per-family dataset listings are provided in Appendix A.2 .

Figure 3: MedEmb reference model. Text, 2D images, 3D volumes, and mixed image–text inputs are serialized into one MLLM, and the final-token hidden state is used as the retrieval embedding.

Model	Retrieval Families								Summary
	I-I	MB-VR	I-Lab	I-Rep	T-I	VQA	VG	T-T	ID	Held-out- source	All
# of Tasks $\rightarrow$	20	10	22	7	23	11	7	7	96	11	107
SigLIP [ 23 ]	31.39 (48.13)	6.78 (11.19)	12.19 (22.66)	5.00 (8.05)	20.91 (36.13)	2.64 (4.86)	2.65 (4.15)	9.45 (13.88)	14.53 (24.09)	18.01 (29.78)	14.89 (24.67)
BLIP-2 [ 24 ]	29.68 (46.88)	7.23 (12.02)	7.05 (16.51)	5.65 (11.41)	12.85 (27.70)	3.38 (5.46)	2.70 (4.16)	7.91 (11.75)	11.27 (20.73)	16.91 (29.04)	11.85 (21.58)
MedImageInsight [ 26 ]	44.74 (59.48)	10.71 (18.51)	14.25 (25.55)	9.90 (16.71)	26.13 (41.11)	10.22 (13.88)	7.05 (9.14)	26.84 (34.57)	21.28 (31.47)	26.57 (39.77)	21.83 (32.32)
MedSigLIP [ 25 ]	19.73 (37.22)	3.71 (7.34)	13.50 (23.35)	4.86 (9.63)	24.18 (39.72)	6.58 (9.62)	0.46 (1.05)	11.42 (17.24)	13.47 (23.04)	16.53 (30.36)	13.78 (23.80)

Table 4: Main MedHEB retrieval results. Entries are Hit@1 with MRR@5 in parentheses (%). Family columns average retrieval families; summary columns average ID, held-out-source, and all tasks. Abbreviations: I = image, T = text, MB = modality-bridging, VR = visual retrieval, Lab = label, Rep = report, VQA = question-conditioned retrieval, VG = region-level text-to-region.

Figure 4: Benchmark difficulty and cross-relation evaluation. A: Tasks are grouped by public-baseline mean Hit@1 into Collapse, Weak, Medium, and Tractable regimes. B: Paired model scores compare established families (I-I, I-Lab, I-Rep, T-I, T-T) with underexplored relation families (MB-VR, VG).

Figure 5: Retrieval difficulty across anatomical domains and evidence forms.

Figure 6: Capability, model-size ablation, and consistency profiles. A: Family-level Hit@1 for a strong non-MLLM baseline, Qwen3-VL-8B, and MedEmb. B: Per-task MedEmb-2B vs MedEmb-8B scores for ID and held-out-source tasks. C: Mean family Hit@1 vs cross-family CV for accuracy–balance trade-off.

Model	Severity	Laterality	Diagnosis	Cell/tissue	All
BLIP-2	67.31	32.02	48.85	32.61	47.47
MedSigLIP	70.06	29.78	51.40	40.49	50.72
SigLIP	69.63	34.82	52.37	40.21	51.55
RzenEmbed	67.62	31.66	51.97	45.92	51.85
VLM2Vec-V2.0	71.29	30.40	52.04	44.45	52.23
MedImageInsight	70.61	36.56	61.76	43.64	56.90

Table 5: Graded-relevance retrieval performance (NDCG@5, %).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\mu$	$\sigma$	CV	I-I	MB-VR	I-Lab	I-Rep	T-I	VQA	VG	T-T
Baseline Models
siglip-so400m-patch14-384	11.34	9.45	0.833	31.38	6.78	12.19	5.01	20.91	2.36	2.65	9.45
blip2-itm-vit-g	9.54	8.16	0.855	29.67	7.23	7.05	5.65	12.85	3.24	2.70	7.91
MedImageInsight	18.68	12.12	0.649	44.74	10.71	14.25	9.90	26.13	9.81	7.05	26.84
MedSigLIP-448	10.55	7.72	0.732	19.73	3.72	13.51	4.86	24.18	6.49	0.46	11.41
RzenEmbed	18.71	16.93	0.905	31.04	8.62	13.86	7.99	17.25	4.49	7.96	58.47

Appendix

Table 7: Cross-family consistency statistics.

\mu

and

\sigma

are computed from family-level Hit@1 across the eight retrieval families; lower CV

=\sigma/\mu

indicates more balanced performance.

Model	Retrieval Families								Summary
	I-I	MB-VR	I-Lab	I-Rep	T-I	VQA	VG	T-T	ID	Held-out- source	All
# of Tasks $\rightarrow$	20	10	22	7	23	11	7	7	96	11	107
Baseline Models
SigLIP [ 23 ]	77.43	20.22	42.01	14.10	64.23	9.23	7.00	21.74	41.61	50.83	42.56
BLIP2 [ 24 ]	77.53	21.65	33.17	22.59	57.28	9.22	6.76	18.29	38.33	51.75	39.71
MedImageInsight [ 26 ]	84.23	33.63	42.95	31.03	67.58	20.51	12.58	47.39	48.68	64.48	50.31

Appendix

Table 8: Task-macro Hit@5 (%) on MedHEB. Family and summary columns follow Table 4 , providing a top-5 complement to Hit@1 and MRR@5.

Figure 8: Target granularity effects in text-conditioned retrieval. Tasks are ordered by average target length and grouped into short labels, VQA answers, and long-text targets; lower panels compare Hit@1 trajectories for the best public baseline, MedEmb-2B, and MedEmb-8B.

Task(s)	qry_text
MIMIC-CXR-LT, ChestMNIST, CheXpert-Plus	What is the radiological abnormal findings observed in this chest X-ray?
ISIC-2019	What is the dermatological condition observed in this image?
Brain-Tumor-MRI	What is the brain tumor diagnostic category observed in this MRI scan?
BloodMNIST	What is the blood cell category observed in this image?
DermaMNIST	What is the dermatological diagnostic category observed in this image?
OCTMNIST	What is the retinal diagnostic category observed in this image?

Appendix

Table 9: 2D image-to-label query texts. All rows share qry_inst = “ Represent the given image with the following question: ”; the table lists task-specific qry_text prompts.

Task(s)	qry_inst	qry_text
Volume classification
CT-RATE-cls	Represent the given list of images with the following question:	What is the pathological finding represented in this list of images?
CirrMRI600-cls	Represent the given MRI series with the following question:	What is the degree of liver cirrhosis shown in the picture?
MRNet-cls	Represent the given MRI series with the following question:	What type of knee disease is shown in this image?
NoduleMNIST-cls	Represent the given CT series with the following question:	What is the nature of the tumor in the chest?
Organ3dMNIST-cls	Represent the given CT series with the following question:	What anatomical structure is shown in this image?

Appendix

Table 10: 3D instruction templates by retrieval family. “per row” denotes instance-specific question or report text; <image> denotes the sampled 3D volume or image sequence used as query content.

Task(s)	qry_inst
MedMCQA, MedicalQA, PublicHealthQA	Find the answer of the given question:
MIMIC findings-to-impression, CheXpert-Plus f2i	Identifying impressions based on given radiology findings.
PubMedQA	Summarize the content of the given context:
MMDental	Find the treatment plan of the given diagnosis:

Appendix

Table 11: Biomedical text–text instruction templates. Rows list the shared qry_inst for each source; qry_text is populated per instance with the corresponding question, finding, context, or diagnosis.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 132 papers

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
Mar 5, 2026Dannong Xu, Zhongyu Yang, Jun Chen +6Multimodal IR Multimodal Model Evaluation
INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3
CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
Feb 28, 2026Yannian Gu, Zhongzhen Huang, Linjie Mu +3Clinical Language Model Evaluation Multimodal Model Evaluation
Shanghai Jiao Tong University, Shanghai, China · SenseTime Research, China · Shanghai Innovation Institute, Shanghai, China
MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
Apr 5, 2026Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra +1Clinical Information Extraction Clinical Language Model Evaluation
Optum AI

Mar 5, 2026J/K move · Enter open · S save

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

Dannong Xu, Zhongyu Yang, Jun Chen, Yingfang Yuan+5

INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology+3

Multimodal IR Multimodal Model Evaluation Cross-Modal Retrieval Multimodal Large Language Models Multimodal Dataset Multimodal Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.

Abstract

Multimodal IR

Explore similar work

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction