FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
Authors: David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante
Organizations: CentraleSupélec, Université Paris-Saclay, France · Cancer Data Science Unit, IHU PRISM, Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, France · University of Michigan, USA · Federal University of São Paulo, Brazil · University of Porto, Portugal · Aalborg University, Denmark · Universidad de Buenos Aires, Argentina
Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards
Figures & tables
Figure 1: Multimetric arena view. Spider plots summarize the top 6 VM, VLM-encoders, and MLLM configurations across AUROC, AUPRC, accuracy, calibration, fairness diagnostics, and image-quality robustness. Colors distinguish models; LP = Linear probing, ZS = Zero-shot, ZS Prompt = Zero-shot prompting. This view makes visible that strong ranking performance need not imply strong calibration or balanced subgroup/quality behavior.
Resource
Type
DR
Ref. DR
Glauc.
multi- reg.
Fair.
Qual.
VM
VLM
MLLM
FT trans.
IDRiD [ 24 ]
DR challenge
Y
Y
–
–
–
–
D
–
–
–
RFMiD / RFMiD 2.0 [ 22 , 23 ]
Multi-disease fundus datasets
Y
D
D
–
–
–
D
–
–
–
BRSET / mBRSET [ 17 , 16 , 18 , 41 ]
Brazilian fundus datasets
Y
Y
Y
D
D
D
D
–
–
–
PAPILA / G1020 / REFUGE [ 9 , 2 , 21 ]
Glaucoma datasets/challenges
–
–
Y
D
D
–
D
–
–
–
FunBench [ 39 ]
Fundus MLLM VQA benchmark
VQA
VQA
VQA
Y
–
–
–
–
Y
–
LMOD [ 25 ]
Ophthalmology LVLM benchmark
VQA
VQA
VQA
Y
D
–
–
D
Y
–
Table 1: Coverage comparison with related fundus datasets, challenges, and ophthalmic benchmarks.
Family
Domain
Architectures
Interface
VM
Generalist
ViT-L/16, DINOv2-L, DINOv3-ViT-L
Image embeddings
VM
Ophthalmic
RETFound variants, VisionFM-Fundus
Image embeddings
VLM
Generalist
CLIP, SigLIP2, MedSigLIP
Joint image-text embeddings
VLM
Ophthalmic
EyeCLIP, RET-CLIP, FLAIR
Joint image-text embeddings
MLLM
Generalist
Qwen3-VL, Gemma-3, LLaVA-NeXT
Text output and token logits
MLLM
Medical
MedGemma variants
Text output and token logits
Table 2: Model families represented in the current FOCUS result bundle.
Task
Model type
Method
Accuracy
AUROC
AUPRC
ECE
Fairness
Quality
Shift
Binary DR
General VM
Linear probing
0.813
0.867
0.782
0.188
0.932
0.913
0.739
Ophthalmic VM
Linear probing
0.720
0.690
0.537
0.206
0.922
0.910
0.703
General VLM-encoders
Linear probing
0.819
0.865
0.784
0.179
0.936
0.874
0.746
General VLM-encoders
Zero-shot
0.477
0.547
0.346
0.311
0.948
0.897
0.840
Ophthalmic VLM-encoders
Linear probing
0.780
0.814
0.702
0.151
0.916
0.911
0.801
Table 3: Method-aware mean metrics by model type and task. Higher is better except for ECE.
Dataset
Country
Device
Gen.
Oph.
Δ
mBRSET
Brazil
mobile
0.754
0.592
+0.162
IDRiD
India
desktop
0.825
0.692
+0.133
BRSET
Brazil
desktop
0.822
0.700
+0.122
JSIEC1000
China
desktop
0.958
0.851
+0.106
Messidor-2
France
desktop
0.836
0.759
+0.077
RFMiD
India
desktop
0.914
0.849
+0.066
Table 4: Per-dataset mean AUROC for VM encoders. Δ=AUROCgen−AUROCoph .
Figure 3: Performance vs. reliability and fairness, by model family. Each point is one (model, method) configuration; rows index axis (top: age equalised-odds gap; bottom: ECE), columns index family (CV / VLM / MLLM). Marker colour encodes model type, marker shape encodes adaptation method (circle = linear probe, triangle = zero-shot, square = prompted). Both axes are inverted so that “up” is desirable on every panel. The dotted line in each panel is a least-squares fit on that panel’s points; red denotes a positive AUROC–gap slope (a ranking–axis trade-off ) and green a negative slope ( aligned ). Pearson’s r is reported in each panel ( * p<0.05 , ** p<0.01 , *** p<0.001 ).
Figure 4: Glaucoma is the clearest task-level failure in FOCUS.
Test domain
AUROC
Δ AUROC
q
Δ Accuracy
q
Δ ECE
q
In-domain
0.776
0.004
0.477
0.033
0.004
-0.040
0.004
OOD
0.724
0.002
0.521
0.012
0.005
-0.018
0.004
Table 5: Language-model fine-tuning transfer summary. Deltas are SFT metric minus matching base MLLM metric; negative Δ ECE indicates improved calibration. Bootstrap q -values apply Benjamini–Hochberg correction within each metric family.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Binary DR
Referable DR
Glaucoma
BRSET
Yes
Yes
Yes
mBRSET
Yes
Yes
Yes
PAPILA
No
No
Yes
RFMiD
Yes
Yes
No
RFMiD 2.0
Yes
Yes
No
IDRiD
Yes
Yes
No
Appendix
Table 6: Configured dataset-task availability. A model is evaluated on a dataset-task pair only when the task label is available after harmonization.
Dataset
Source
Tasks
Extra metadata
BRSET
Brazil
Any DR, ref. DR, GON
Age, sex, image quality
mBRSET
Brazil
Any DR, ref. DR, GON
Age, sex, image quality
PAPILA
Spain
GON
Age, sex
RFMiD
India
Any DR, ref. DR
–
RFMiD 2.0
India
Any DR, ref. DR
–
IDRiD
India
Any DR, ref. DR
–
Appendix
Table 7: Datasets configured in FOCUS. Tasks use the harmonized definitions in Sec. 3.1 ; Fairness and robustness analyses are computed only when metadata are available.
Experiment stage
Worker type
Memory
Batch size
Time limit
Dataset downloads
CPU, 4 cores
system memory
–
8h
Model downloads
CPU, 4 cores
system memory
–
2h
VM base evaluation
1 H100 GPU, 24 CPU cores
80GB GPU
16
20h
VLM base evaluation
1 H100 GPU, 24 CPU cores
80GB GPU
16
20h
MLLM base prompting
1 H100 GPU, 24 CPU cores
80GB GPU
16
20h
SFT LoRA training
4 H100 GPUs, 48 CPU cores
4 × 80GB GPU
1–4
18h
Appendix
Table 8: Compute resources used by the benchmark jobs. The SFT LoRA batch size is 1 with gradient accumulation 8 for 27B-class models and 4 with gradient accumulation 4 for smaller models. VM/VLM/MLLM base evaluation jobs iterate over the configured model–dataset–task–method grid for each family.
Model type
Binary DR
Glaucoma
Referable DR
General Vision Encoder Model (VM)
0.867
0.742
0.850
Ophthalmic Vision Encoder Model (VM)
0.690
0.591
0.697
General Multimodal LLM (MLLM)
0.685
0.630
0.651
Medical Multimodal LLM (MLLM)
0.868
0.656
0.844
General Dual Encoder Vision Language Model (VLM-encoders)
0.706
0.616
0.724
Ophthalmic Dual Encoder Vision Language Model (VLM-encoders)
0.756
0.630
0.721
Appendix
Table 9: Mean AUROC by task and model type in the current benchmark result bundle.
Domain
AUROC
AUPRC
Accuracy
F1
ECE
In-domain
0.776
0.619
0.826
0.511
0.164
OOD
0.724
0.513
0.675
0.402
0.290
Appendix
Table 10: Mean SFT metrics by test domain. The corresponding delta estimates are summarized in the main paper.
Domain
Metric
Mean delta
95% CI low
95% CI high
q
In-domain
AUROC
0.004
-0.002
0.010
0.521
In-domain
Accuracy
0.033
0.003
0.076
0.004
In-domain
ECE
-0.040
-0.085
-0.008
0.004
OOD
AUROC
0.002
-0.002
0.006
0.521
OOD
Accuracy
0.012
0.005
0.021
0.005
OOD
ECE
-0.018
-0.026
-0.009
0.004
Appendix
Table 11: Bootstrap uncertainty for domain-level SFT deltas. Positive ECE deltas indicate worse calibration, while negative ECE deltas indicate improved calibration. Intervals are estimated by paired resampling over matched base and fine-tuned predictions.
Foundation models are used to extract transferable representations from large amounts of unlabeled data, typically via self-supervised learning (SSL). However, many of these models rely on architectures that offer limited interpretability, a critical issue in high-stakes domains such as medical imaging. We propose DualIFM, a foundation model that is interpretable-by-design via a BagNet backbone whose small receptive fields generate class evidence maps that are faithful to the model's decision-making process. Additionally, DualIFM incorporates a 2D projection layer during pretraining that enables direct visualization of the representation space, providing a dataset-level view of the learned structure including meaningful clinical clusters as well as potential spurious correlations. We trained DualIFM on over 800,000 color fundus photographs from various sources to learn generalizable representations for different downstream tasks. Our model achieves performance comparable to RETFound, which has 16× more parameters, while providing interpretable predictions on out-of-distribution data. These results suggest that large-scale SSL pretraining paired with inherent interpretability can lead to robust representations for retinal imaging. Code and pretrained models are available at github.com/berenslab/interpretable_FM.
Samuel Ofosu Mensah, Camila Roa, Kerol Djoumessi +1
Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany
Modern deep learning offers powerful tools for automated retinal screening, but it remains unclear how different visual model families compare in realistic multi-disease settings and under domain shift. In this work, we benchmark twelve architectures across four model families: convolutional neural networks, vision transformers, hybrid CNN-transformer backbones, and vision-language models, using the Retinal Fundus Multi-disease Image Dataset (RFMiD). We evaluate two tasks: binary screening for any retinal disease and multi-label classification across 28 disease classes. Using standardized training, calibration, and evaluation protocols, we report AUC, F1, precision, recall, and sensitivity at a clinically relevant operating point with specificity near 80%. On RFMiD, all architectures perform well on binary screening, with AUC above 84%, but attention-based models perform best. SwinTiny and the hybrid CoAtNet0 and MaxViTTiny models achieve the strongest binary screening results and improve macro and micro F1 in the multi-label setting. Vision-language models, including CLIP ViT-B/16 and SigLIP-Base384, are competitive with CNN baselines but do not surpass the best transformer and hybrid backbones. In external validation on Messidor-2 for referable diabetic retinopathy, AUC ranges from 66.8% to 84.7%, with hybrid and transformer models again showing strong performance. These results provide a reproducible reference for model selection in multi-disease retinal screening and guide future automated screening tools for clinical deployment.
Durjoy Dey, Aymane Ajbar, Yuhong Yan
Department of Computer Science and Software Engineering Concordia University Montreal, QC, Canada · Ebovir Biotechnologie Inc. Montreal, QC, Canada
Over the past decade, generative models have demonstrated success in enhancing fundus images. However, the evaluation of these models remains a challenge. A benchmark for fundus image enhancement is needed for three main reasons:(1) Conventional denoising metrics such as PSNR and SSIM fail to capture clinically relevant features, such as lesion preservation and vessel morphology consistency, limiting their applicability in real-world settings; (2) There is a lack of unified evaluation protocols that address both paired and unpaired enhancement methods, particularly those guided by clinical expertise; and (3) An evaluation framework should provide actionable insights to guide future advancements in clinically aligned enhancement models. To address these gaps, we introduce EyeBench-V2, a benchmark designed to bridge the gap between enhancement model performance and clinical utility. Our work offers three key contributions:(1) Multi-dimensional clinical-alignment through downstream evaluations: Beyond standard enhancement metrics, we assess performance across clinically meaningful tasks including vessel segmentation, diabetic retinopathy (DR) grading, generalization to unseen noise patterns, and lesion segmentation. (2) Expert-guided evaluation design: We curate a novel dataset enabling fair comparisons between paired and unpaired enhancement methods, accompanied by a structured manual assessment protocol by medical experts, which evaluates clinically critical aspects such as lesion structure alterations, background color shifts, and the introduction of artificial structures. (3) Actionable insights: Our benchmark provides a rigorous, task-oriented analysis of existing generative models, equipping clinical researchers with the evidence needed to make informed decisions, while also identifying limitations in current methods to inform the design of next-generation enhancement models.
Xuanzhao Dong, Wenhui Zhu, Xiwen Chen +8
Arizona State University · Clemson University · LinkedIn Corporation +1