VOLMO: Versatile and Open Large Models for Ophthalmology

Zhenyue Qin; Younjoon Chung; Elijah Lee; Wanyue Feng; Xuguang Ai; Serina Applebaum; Minjie Zou; Yang Liu

cs.CVMar 25, 2026

VOLMO: Versatile and Open Large Models for Ophthalmology

Authors: Zhenyue Qin, Younjoon Chung, Elijah Lee, Wanyue Feng, Xuguang Ai, Serina Applebaum, Minjie Zou, Yang Liu, +11 more

Organizations: Department of Biomedical Informatics & Data Science, Yale University · Ray and Stephanie Lane Computational Biology Department, Carnegie Mellon University · Yong Loo Lin School of Medicine, National University of Singapore · Department of Radiology, Washington University in Saint Louis · Harvard Medical School, Harvard University · National Eye Institute, National Institutes of Health · National Library of Medicine, National Institutes of Health

Abstract

Vision impairment affects millions globally, and early detection is critical to preventing irreversible vision loss. Ophthalmology workflows require clinicians to integrate medical images, structured clinical data, and free-text notes to determine disease severity and management, which is time-consuming and burdensome. Recent multimodal large language models (MLLMs) show promise, but existing general and medical MLLMs perform poorly in ophthalmology, and few ophthalmology-specific MLLMs are openly available. We present VOLMO (Versatile and Open Large Models for Ophthalmology), a model-agnostic, data-open framework for developing ophthalmology-specific MLLMs. VOLMO includes three stages: ophthalmology knowledge pretraining on 86,965 image-text pairs from 26,569 articles across 82 journals; domain task fine-tuning on 26,929 annotated instances spanning 12 eye conditions for disease screening and severity classification; and multi-step clinical reasoning on 913 patient case reports for assessment, planning, and follow-up care. Using this framework, we trained a compact 2B-parameter MLLM and compared it with strong baselines, including InternVL-2B, LLaVA-Med-7B, MedGemma-4B, MedGemma-27B, and RETFound. We evaluated these models on image description generation, disease screening and staging classification, and assessment-and-management generation, with additional manual review by two healthcare professionals and external validation on three independent cohorts for age-related macular degeneration and diabetic retinopathy. Across settings, VOLMO-2B consistently outperformed baselines, achieving stronger image description performance, an average F1 of 87.4% across 12 eye conditions, and higher scores in external validation.

Figures & tables

Figure 1: Overview of VOLMO development pipeline and multi-round clinical reasoning. The framework consists of three components: (Left) Dataset Extraction from publicly available sources including ophthalmological literature (PubMed Central), benchmarking datasets (Hugging Face, GitHub, etc.), and patient clinical profiles with diagnosis reports. (Center) Model Training through a three-stage progressive framework: Stage 1 - Ophthalmology knowledge pretraining using 86,965 image-text pairs to inject foundational domain knowledge; Stage 2 - Domain task fine-tuning on 26,929 disease-labeled instances across 12 conditions and signs for disease screening and staging; Stage 3 - Reasoning and synthesis training on 913 comprehensive case reports to enable clinical assessment generation. Snowflake icons indicate frozen components during training. (Right) Multi-round Conversation Example demonstrating VOLMO’s clinical reasoning workflow, where the model sequentially generates differential diagnoses, determines the most likely diagnosis, formulates clinical assessments and treatment plans, recommends specific treatments, and provides follow-up care guidance based on patient clinical profiles and multi-modal ophthalmological imaging.

Figure 2: Process of assessment and management generation. Given a patient clinical profile including medical history, ocular history, family history, presenting symptoms, and multi-modal ophthalmology imaging data (left panel), VOLMO generates diagnostic reports through five sequential reasoning stages (right panel): (1) Differential Diagnoses - listing potential conditions that could explain the patient’s clinical presentation, (2) Most Likely Diagnosis - the single condition determined to be the most probable explanation with justification, (3) Assessments and Plans - evaluating the patient’s current clinical status including disease severity and stage, (4) Treatment - recommending therapeutic interventions tailored to address the diagnosed condition, and (5) Follow-up Care - providing ongoing management plans including monitoring parameters and anticipated outcomes.

Dataset	Images	Modality	Population
BRSET	15,201	CFP	Brazilian
OIMHS	2,933	OCT	Multi-ethnic
FIVES	240	CFP	Asian
EyePACS	8,555	CFP	United States

Table 1: Composition of the stage 2 ophthalmological training cohort. The stage 2 dataset comprises 26,929 image–text instances drawn from four independent ophthalmological datasets, covering 12 common eye conditions and signs across diverse patient populations and imaging modalities.

Figure 3: Visual interpretation of VOLMO’s attention patterns for the query word scar . Top two rows show fundus images with retinal scars (first row) and corresponding attention heatmaps (second row). Bottom two rows display fundus images without scars (third row) and their attention heatmaps (fourth row). Attention intensity is visualized using a color scale ranging from blue (low attention) through green and yellow to red (high attention).

Model	BLEU-1	ROUGE-L F1	BERTScore F1	SBERT Similarity
InternVL-2B	$0.0796\pm 0.0044$	$0.1070\pm 0.0042$	$0.6464\pm 0.0032$	$0.4409\pm 0.0092$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )
LLaVA-Med	$0.1181\pm 0.0068$	$0.1644\pm 0.0037$	$0.6867\pm 0.0028$	$0.4468\pm 0.0117$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )
MedGemma-4B	$0.1083\pm 0.0068$	$0.1212\pm 0.0052$	$0.6428\pm 0.0034$	$0.4669\pm 0.0090$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )

Table 2: Statistical comparison of baseline models against VOLMO-2B for image description tasks. Values show mean

\pm

standard deviation. P-values from Wilcoxon signed-rank test are shown in parentheses.

Model	F1	Sensitivity	Specificity	F1	Sensitivity	Specificity	F1	Sensitivity	Specificity
	Glaucoma			AMD			DR
RETFound	$\mathbf{92.33}$	$92.33$	$92.71$	$87.91$	$87.93$	$87.87$	$84.94$	$84.93$	$85.15$
	$(p=0.3172)$	$(p=0.2324)$	$(p=0.3814)$	$(p=0.0027)$	$(p<0.0001)$	$(p=0.0008)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$
InternVL-2B	$31.79$	$48.37$	$51.63$	$35.46$	$50.90$	$50.77$	$33.11$	$49.23$	$50.77$
	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$
LLaVA-Med	$34.44$	$50.76$	$49.24$	$33.78$	$50.10$	$49.90$	$39.24$	$54.67$	$45.33$

Table 3: Statistical comparison of baseline models against VOLMO-2B for binary condition and sign classification tasks. Values show mean

\pm

standard deviation. All values are percentages (0–100) without the percent sign. P-values from Wilcoxon signed-rank test vs. VOLMO-2B are shown in parentheses. The best F1 score in each column is highlighted in bold.

	AMD			DR
Model	F1	Sensitivity	Specificity	F1	Sensitivity	Specificity
RETFound	$46.39\pm 9.95$	$55.33\pm 7.93$	$56.33\pm 7.79$	$53.36\pm 9.77$	$56.83\pm 8.79$	$67.30\pm 7.99$
	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p=0.0004)$	$(p=0.0025)$	$(p=0.0130)$
InternVL-2B	$41.47\pm 10.10$	$53.07\pm 8.36$	$52.06\pm 7.59$	$24.69\pm 8.21$	$41.37\pm 8.28$	$58.63\pm 8.28$
	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$	$(p<0.0001)$
LLaVA-Med	$34.41\pm 8.69$	$50.63\pm 7.90$	$49.37\pm 7.90$	$43.77\pm 9.85$	$58.71\pm 8.19$	$41.29\pm 8.19$

Table 4: External validation performance on UK Biobank cohort for AMD and DR screening. Values show mean F1-score, sensitivity, and specificity (

\pm

standard deviation) across bootstrap resampling. P-values from Wilcoxon signed-rank tests compare each baseline model against VOLMO-2B.

Method	Overall			Stage 0			Stage 1			Stage 2			Stage 3			Stage 4
	F1	Sens	Spec	F1	Sens	Spec	F1	Sens	Spec	F1	Sens	Spec	F1	Sens	Spec	F1	Sens	Spec
Macular Hole
InternVL-2B	5.47	3.67	94.83	–	–	–	–	–	–	13.81	9.67	84.83	2.61	1.33	99.67	0.00	0.00	100.00
LLaVA-Med	16.67	33.33	66.67	–	–	–	–	–	–	0.00	0.00	100.00	0.00	0.00	100.00	50.00	100.00	0.00
MedGemma-27B	22.00	30.57	66.31	–	–	–	–	–	–	11.11	6.96	88.72	48.78	81.08	16.10	6.11	3.67	94.12
MedGemma-4B	19.84	18.04	82.32	–	–	–	–	–	–	25.25	21.33	76.13	34.27	32.78	70.83	0.00	0.00	100.00

Table 5: Disease stage grading performance across multiple datasets. We report F1, Sensitivity, and Specificity for each disease stage. In-domain datasets were used during training, while cross-domain datasets evaluate generalization to unseen data sources. Dashes indicate stages not present in the dataset. Best Overall F1 in each dataset block is bolded.

Figure 9

Panel of Figure 4 — Figure 4: Comparison of medical AI model responses on ophthalmological image description tasks.

Panel of Figure 5 — Figure 5: Comparison of medical AI model responses on ophthalmological image description tasks (continued).

Panel of Figure 6 — Figure 6: Performance comparison across different clinical stages. Each column shows Assessment & Plan, Treatments and Follow-up Care performance with BERTScore-F1 and SBERT scores. Each subplot compares VOLMO-2B against InternVL-2B, LLaVA-Med, MedGemma-4B and MedGemma-27B. Error bars represent standard deviations, and statistical significance markers (***) indicate p-values compared to VOLMO-2B baseline.

Panel of Figure 7 — Figure 7: Average manual evaluation scores across three quality dimensions. Two ophthalmology residents independently rated generated ophthalmological image descriptions on a 5-point scale (1 = poor, 5 = excellent) across 40 test samples for conciseness, accuracy, and readability. Model identities were anonymized and evaluation order randomized to minimize bias.

Panel of Figure 8 — Figure 8: Representative examples of ophthalmological image descriptions generated by different models, manually evaluated by ophthalmologists.

Panel of Figure 9 — Figure 9: Representative examples of ophthalmological image descriptions generated by different models, manually evaluated by ophthalmologists (continued).

Panel of Figure 10 — Figure 10: Representative examples of ophthalmological image descriptions generated by different models, manually evaluated by ophthalmologists (continued).

Panel of Figure 11 — Figure 11: Representative examples of ophthalmological image descriptions generated by different models, manually evaluated by ophthalmologists (continued).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	ROUGE-L F1	BLEU-1	BLEU-2	BLEU-3	BLEU-4	BERTScore F1	SBERT Similarity
InternVL-2B	$0.1070\pm 0.0042$	$0.0796\pm 0.0044$	$0.0273\pm 0.0017$	$0.0095\pm 0.0007$	$0.0046\pm 0.0004$	$0.6464\pm 0.0032$	$0.4409\pm 0.0092$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )
LLaVA-Med	$0.1644\pm 0.0037$	$0.1181\pm 0.0068$	$0.0410\pm 0.0030$	$0.0170\pm 0.0014$	$0.0094\pm 0.0008$	$0.6867\pm 0.0028$	$0.4468\pm 0.0117$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )
MedGemma-4B	$0.1212\pm 0.0052$	$0.1083\pm 0.0068$	$0.0440\pm 0.0033$	$0.0187\pm 0.0017$	$0.0092\pm 0.0010$	$0.6428\pm 0.0034$	$0.4669\pm 0.0090$
	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )	( $p<0.0001$ )

Appendix

Table 8: Statistical comparison of baseline models against VOLMO-2B for image description tasks. Values show mean

\pm

standard deviation. P-values from Wilcoxon signed-rank test are shown in parentheses.

	Glaucoma			AMD			DR
Model	F1	Sensitivity	Specificity	F1	Sensitivity	Specificity	F1	Sensitivity	Specificity
RETFound (frozen)	$\mathbf{92.33}$	$92.33$	$92.71$	$87.91$	$87.93$	$87.87$	$84.94$	$84.93$	$85.15$
RETFound (unfrozen)	$91.65$	$86.67$	$96.67$	$\mathbf{90.71}$	$85.19$	$96.30$	$78.73$	$76.92$	$80.54$

Appendix

Table 9: Comparison of frozen versus unfrozen RETFound fine-tuning strategies for binary condition classification tasks. The best F1 score in each column is highlighted in bold.

Hyperparameter	Value
Training Configuration
Number of GPUs	4
Total batch size	1
Per-device batch size	1
Gradient accumulation steps	1
Mixed precision	bfloat16

Appendix

Table 10: Training hyperparameters

Explore similar work

May 27, 2026

OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models

The advancement of general medical Multimodal Large Language Models (MLLMs) has shown great potential for building conversational assistants to support clinical diagnosis. However, their adaptation to highly specialized domains such as ophthalmology remains underexplored, primarily due to the scarcity of large-scale, domain-specific instruction-tuning data. Existing ophthalmic datasets for conversational agents are often limited in scale and largely rely on images from established public benchmarks, limiting the scalability of ophthalmic MLLMs and their ability to capture real-world clinical complexity. To address this gap, we propose

\textbf{OphIn-Engine}

, an ophthalmology-specific instruction data curation pipeline that constructs high-quality instruction data from open-access ophthalmology web-scale videos. The pipeline integrates multimodal transcription for extracting image-transcript pairs, visual cue separation and scoring for identifying clinically relevant visual descriptions, and instruction synthesis with quality control for generating accurate and diverse clinical dialogues. Using this engine, we introduce

\textbf{OphIn-500K}

, a large-scale multimodal ophthalmology instruction-tuning dataset containing over 500,000 instruction instances and more than 151,000 unique images from over 29,000 video clips, formatted as visual question answering (VQA), multi-turn conversational interactions, and chain-of-thought (CoT) reasoning. Built upon this dataset, we further develop

\textbf{OphIn-VL}

, an ophthalmology-specific MLLM with advanced visual understanding and conversational capabilities. Comprehensive experiments and case studies demonstrate that OphIn-VL achieves superior performance compared with state-of-the-art general medical and domain-specific MLLMs.

Xuanzhao Dong, Wenhui Zhu, Xiwen Chen +13

1Arizona State University · 2Clemson University · 3Washington University in St. Louis +5

Medical Image Analysis Medical VQA Medical VLMs Instruction Tuning Multimodal Instruction Tuning

Ask PDF

May 4, 2026

Scientific Domain Knowledge Improves Vision-Language Fundus Models

Vision-language models hold considerable promise for ophthalmology, but it remains unclear which training data source best conveys expert domain knowledge. Existing ophthalmic models are trained on fixed text templates, medical reports, or general biomedical literature, sources that have never been compared under matched conditions. To include domain-specific literature in this comparison, we present PubMed-Ophtha, a hierarchical dataset with high domain density of 102,023 panels with their subcaptions from 15,842 open-access articles in PubMed Central. We then finetuned identical CLIP models on each source, using a general biomedical literature model as baseline, and found that domain-specific literature achieved the best average performance across 110 clinical tasks, reaching a mean linear probing AUROC of 88.63% ahead of medical reports (85.68%). Restricting the dataset to fundus images, to the image count of the medical report dataset, or to articles unrelated to the evaluation datasets did not reduce performance, indicating that the gains likely stem from domain density. We release the dataset, the finetuned models, and the full generation pipeline.

Verena Jasmin Hallitschke, Carsten Eickhoff, Philipp Berens

1Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Institute for Applied Medical Informatics, University of Tübingen, Tübingen, Germany · 2Tübingen AI Center, Tübingen, Germany

Healthcare Medical Diagnosis Medical VLMs Fundus Imaging Medical Image Benchmarks Vision-Language Pretraining

Ask PDF

Sep 26, 2026

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized. Existing ophthalmic datasets are typically designed for individual diseases or specialized tasks, making it difficult to systematically evaluate whether VLMs can move beyond disease recognition toward comparative reasoning and fine-grained spatial grounding. We introduce EyeVQA, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs. EyeVQA is constructed from 21 available ophthalmic datasets and contains 20,000 question-answer pairs spanning six disease groups and seven question types: Single-Choice, Multi-Select, Variable-Select, True-False, Ranking, Point Location, and Bounding Box. Gold answers are deterministically derived from source-provided diagnoses, severity grades, clinical findings, segmentation masks, bounding boxes, and anatomical landmarks, enabling reproducible evaluation without relying on model-generated annotations. Notably, 44.5% of the questions require reasoning across multiple images, extending evaluation beyond conventional single-image medical VQA. We benchmark fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol. The best-performing model only achieves an overall score of 62.8, while substantial gaps remain in spatial grounding and cross-task generalization. These results highlight the limitations of current VLMs in comprehensive ophthalmic visual understanding and establish EyeVQA as a diagnostic benchmark for developing more reliable and spatially grounded ophthalmic multimodal models. The project page is available at https://github.com/PKUTHM/EyeVQA.

Gujie Shao, Zixun Xie, Xuechun Xing +7

VLM Evaluation Medical VQA Vision-Language Grounding Medical Image Benchmarks

Ask PDF