FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

David Restrepo; Chenwei Wu; Luis Filipe Nakayama; Miguel L. Martins; Stergios Christodoulidis; Maria Vakalopoulou; Enzo Ferrante

cs.CVSep 27, 2026

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

Authors: David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante

Organizations: CentraleSupélec, Université Paris-Saclay, France · Cancer Data Science Unit, IHU PRISM, Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, France · University of Michigan, USA · Federal University of São Paulo, Brazil · University of Porto, Portugal · Aalborg University, Denmark · Universidad de Buenos Aires, Argentina

Abstract

Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards

Figures & tables

Figure 1: Multimetric arena view. Spider plots summarize the top 6 VM, VLM-encoders, and MLLM configurations across AUROC, AUPRC, accuracy, calibration, fairness diagnostics, and image-quality robustness. Colors distinguish models; LP = Linear probing, ZS = Zero-shot, ZS Prompt = Zero-shot prompting. This view makes visible that strong ranking performance need not imply strong calibration or balanced subgroup/quality behavior.

Resource	Type	DR	Ref. DR	Glauc.	multi- reg.	Fair.	Qual.	VM	VLM	MLLM	FT trans.
IDRiD [ 24 ]	DR challenge	Y	Y	–	–	–	–	D	–	–	–
RFMiD / RFMiD 2.0 [ 22 , 23 ]	Multi-disease fundus datasets	Y	D	D	–	–	–	D	–	–	–
BRSET / mBRSET [ 17 , 16 , 18 , 41 ]	Brazilian fundus datasets	Y	Y	Y	D	D	D	D	–	–	–
PAPILA / G1020 / REFUGE [ 9 , 2 , 21 ]	Glaucoma datasets/challenges	–	–	Y	D	D	–	D	–	–	–
FunBench [ 39 ]	Fundus MLLM VQA benchmark	VQA	VQA	VQA	Y	–	–	–	–	Y	–
LMOD [ 25 ]	Ophthalmology LVLM benchmark	VQA	VQA	VQA	Y	D	–	–	D	Y	–

Table 1: Coverage comparison with related fundus datasets, challenges, and ophthalmic benchmarks.

Family	Domain	Architectures	Interface
VM	Generalist	ViT-L/16, DINOv2-L, DINOv3-ViT-L	Image embeddings
VM	Ophthalmic	RETFound variants, VisionFM-Fundus	Image embeddings
VLM	Generalist	CLIP, SigLIP2, MedSigLIP	Joint image-text embeddings
VLM	Ophthalmic	EyeCLIP, RET-CLIP, FLAIR	Joint image-text embeddings
MLLM	Generalist	Qwen3-VL, Gemma-3, LLaVA-NeXT	Text output and token logits
MLLM	Medical	MedGemma variants	Text output and token logits

Table 2: Model families represented in the current FOCUS result bundle.

Binary DR
Task	Model type	Method	Accuracy	AUROC	AUPRC	ECE	Fairness	Quality	Shift
	General VM	Linear probing	0.813	0.867	0.782	0.188	0.932	0.913	0.739
	Ophthalmic VM	Linear probing	0.720	0.690	0.537	0.206	0.922	0.910	0.703
	General VLM-encoders	Linear probing	0.819	0.865	0.784	0.179	0.936	0.874	0.746
	General VLM-encoders	Zero-shot	0.477	0.547	0.346	0.311	0.948	0.897	0.840
	Ophthalmic VLM-encoders	Linear probing	0.780	0.814	0.702	0.151	0.916	0.911	0.801

Table 3: Method-aware mean metrics by model type and task. Higher is better except for ECE.

Dataset	Country	Device	Gen.	Oph.	$\Delta$
mBRSET	Brazil	mobile	0.754	0.592	+0.162
IDRiD	India	desktop	0.825	0.692	+0.133
BRSET	Brazil	desktop	0.822	0.700	+0.122
JSIEC1000	China	desktop	0.958	0.851	+0.106
Messidor-2	France	desktop	0.836	0.759	+0.077
RFMiD	India	desktop	0.914	0.849	+0.066

Table 4: Per-dataset mean AUROC for VM encoders.

\Delta=\mathrm{AUROC}_{\mathrm{gen}}-\mathrm{AUROC}_{\mathrm{oph}}

Figure 3: Performance vs. reliability and fairness, by model family. Each point is one (model, method) configuration; rows index axis (top: age equalised-odds gap; bottom: ECE), columns index family (CV / VLM / MLLM). Marker colour encodes model type, marker shape encodes adaptation method (circle = linear probe, triangle = zero-shot, square = prompted). Both axes are inverted so that “up” is desirable on every panel. The dotted line in each panel is a least-squares fit on that panel’s points; red denotes a positive AUROC–gap slope (a ranking–axis trade-off ) and green a negative slope ( aligned ). Pearson’s $r$ is reported in each panel ( * $p<0.05$ , ** $p<0.01$ , *** $p<0.001$ ).

Figure 4: Glaucoma is the clearest task-level failure in FOCUS.

Test domain	AUROC	$\Delta$ AUROC	$q$	$\Delta$ Accuracy	$q$	$\Delta$ ECE	$q$
In-domain	0.776	0.004	0.477	0.033	0.004	-0.040	0.004
OOD	0.724	0.002	0.521	0.012	0.005	-0.018	0.004

Table 5: Language-model fine-tuning transfer summary. Deltas are SFT metric minus matching base MLLM metric; negative

\Delta

ECE indicates improved calibration. Bootstrap

q

-values apply Benjamini–Hochberg correction within each metric family.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Binary DR	Referable DR	Glaucoma
BRSET	Yes	Yes	Yes
mBRSET	Yes	Yes	Yes
PAPILA	No	No	Yes
RFMiD	Yes	Yes	No
RFMiD 2.0	Yes	Yes	No
IDRiD	Yes	Yes	No

Appendix

Table 6: Configured dataset-task availability. A model is evaluated on a dataset-task pair only when the task label is available after harmonization.

Dataset	Source	Tasks	Extra metadata
BRSET	Brazil	Any DR, ref. DR, GON	Age, sex, image quality
mBRSET	Brazil	Any DR, ref. DR, GON	Age, sex, image quality
PAPILA	Spain	GON	Age, sex
RFMiD	India	Any DR, ref. DR	–
RFMiD 2.0	India	Any DR, ref. DR	–
IDRiD	India	Any DR, ref. DR	–

Appendix

Table 7: Datasets configured in FOCUS. Tasks use the harmonized definitions in Sec. 3.1 ; Fairness and robustness analyses are computed only when metadata are available.

Experiment stage	Worker type	Memory	Batch size	Time limit
Dataset downloads	CPU, 4 cores	system memory	–	8h
Model downloads	CPU, 4 cores	system memory	–	2h
VM base evaluation	1 H100 GPU, 24 CPU cores	80GB GPU	16	20h
VLM base evaluation	1 H100 GPU, 24 CPU cores	80GB GPU	16	20h
MLLM base prompting	1 H100 GPU, 24 CPU cores	80GB GPU	16	20h
SFT LoRA training	4 H100 GPUs, 48 CPU cores	4 $\times$ 80GB GPU	1–4	18h

Appendix

Table 8: Compute resources used by the benchmark jobs. The SFT LoRA batch size is 1 with gradient accumulation 8 for 27B-class models and 4 with gradient accumulation 4 for smaller models. VM/VLM/MLLM base evaluation jobs iterate over the configured model–dataset–task–method grid for each family.

Model type	Binary DR	Glaucoma	Referable DR
General Vision Encoder Model (VM)	0.867	0.742	0.850
Ophthalmic Vision Encoder Model (VM)	0.690	0.591	0.697
General Multimodal LLM (MLLM)	0.685	0.630	0.651
Medical Multimodal LLM (MLLM)	0.868	0.656	0.844
General Dual Encoder Vision Language Model (VLM-encoders)	0.706	0.616	0.724
Ophthalmic Dual Encoder Vision Language Model (VLM-encoders)	0.756	0.630	0.721

Appendix

Table 9: Mean AUROC by task and model type in the current benchmark result bundle.

Domain	AUROC	AUPRC	Accuracy	F1	ECE
In-domain	0.776	0.619	0.826	0.511	0.164
OOD	0.724	0.513	0.675	0.402	0.290

Appendix

Table 10: Mean SFT metrics by test domain. The corresponding delta estimates are summarized in the main paper.

Domain	Metric	Mean delta	95% CI low	95% CI high	$q$
In-domain	AUROC	0.004	-0.002	0.010	0.521
In-domain	Accuracy	0.033	0.003	0.076	0.004
In-domain	ECE	-0.040	-0.085	-0.008	0.004
OOD	AUROC	0.002	-0.002	0.006	0.521
OOD	Accuracy	0.012	0.005	0.021	0.005
OOD	ECE	-0.018	-0.026	-0.009	0.004

Appendix

Table 11: Bootstrap uncertainty for domain-level SFT deltas. Positive ECE deltas indicate worse calibration, while negative ECE deltas indicate improved calibration. Intervals are estimated by paired resampling over matched base and fine-tuned predictions.

Explore similar work

Towards Interpretable Foundation Models for Retinal Fundus Images
Mar 19, 2026Samuel Ofosu Mensah, Camila Roa, Kerol Djoumessi +1Medical Imaging Medical Imaging Foundation Models
Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany
Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening
May 25, 2026Durjoy Dey, Aymane Ajbar, Yuhong YanHybrid CNN-Transformer Architectures Vision Transformer
Department of Computer Science and Software Engineering Concordia University Montreal, QC, Canada · Ebovir Biotechnologie Inc. Montreal, QC, Canada
Bridging Restoration and Diagnosis: A Comprehensive Benchmark for Retinal Fundus Enhancement
Apr 4, 2026Xuanzhao Dong, Wenhui Zhu, Xiwen Chen +8Image Enhancement Medical Image Benchmarks
Arizona State University · Clemson University · LinkedIn Corporation +1

Mar 19, 2026J/K move · Enter open · S save

Towards Interpretable Foundation Models for Retinal Fundus Images

Samuel Ofosu Mensah, Camila Roa, Kerol Djoumessi, Philipp Berens

Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany

Medical Imaging Medical Imaging Foundation Models Fundus Imaging Explainable Medical Image Analysis Medical Image Representation Learning Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Foundation models are used to extract transferable representations from large amounts of unlabeled data, typically via self-supervised learning (SSL). However, many of these models rely on architectures that offer limited interpretability, a critical issue in high-stakes domains such as medical imaging. We propose DualIFM, a foundation model that is interpretable-by-design via a BagNet backbone whose small receptive fields generate class evidence maps that are faithful to the model's decision-making process. Additionally, DualIFM incorporates a

2D

projection layer during pretraining that enables direct visualization of the representation space, providing a dataset-level view of the learned structure including meaningful clinical clusters as well as potential spurious correlations. We trained DualIFM on over 800,000 color fundus photographs from various sources to learn generalizable representations for different downstream tasks. Our model achieves performance comparable to RETFound, which has

16\times

more parameters, while providing interpretable predictions on out-of-distribution data. These results suggest that large-scale SSL pretraining paired with inherent interpretability can lead to robust representations for retinal imaging. Code and pretrained models are available at github.com/berenslab/interpretable_FM.

Abstract

Explore similar work

Towards Interpretable Foundation Models for Retinal Fundus Images

Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening

Bridging Restoration and Diagnosis: A Comprehensive Benchmark for Retinal Fundus Enhancement