Fundus-R1: Training a Fundus-Reading MLLM with Knowledge-Aware Reasoning on Public Data

Yuchuan Deng; Qijie Wei; Kaiheng Qian; Jiazhen Liu; Zijie Xin; Bangxiang Lan; Jingyu Liu; Jianfeng Dong

cs.CVApr 9, 2026

Fundus-R1: Training a Fundus-Reading MLLM with Knowledge-Aware Reasoning on Public Data

Authors: Yuchuan Deng, Qijie Wei, Kaiheng Qian, Jiazhen Liu, Zijie Xin, Bangxiang Lan, Jingyu Liu, Jianfeng Dong, +1 more

Organizations: Renmin University of China China · The Hong Kong University of Science and Technology Hong Kong SAR, China · Zhejiang Gongshang University China

Abstract

Fundus imaging such as CFP, OCT and UWF is crucial for the early detection of retinal anomalies and diseases. Fundus image understanding, due to its knowledge-intensive nature, poses a challenging vision-language task. An emerging approach to addressing the task is to post-train a generic multimodal large language model (MLLM), either by supervised finetuning (SFT) or by reinforcement learning with verifiable rewards (RLVR), on a considerable amount of in-house samples paired with high-quality clinical reports. However, these valuable samples are not publicly accessible, which not only hinders reproducibility but also practically limits research to few players. To overcome the barrier, we make a novel attempt to train a reasoning-enhanced fundus-reading MLLM, which we term Fundus-R1, using exclusively public datasets, wherein over 94% of the data are annotated with only image-level labels. Our technical contributions are two-fold. First, we propose a RAG-based method for composing image-specific, knowledge-aware reasoning traces. Such auto-generated traces link visual findings identified by a generic MLLM to the image labels in terms of ophthalmic knowledge. Second, we enhance RLVR with a process reward that encourages self-consistency of the generated reasoning trace in each rollout. Extensive experiments on three fundus-reading benchmarks, i.e., FunBench, Omni-Fundus and GMAI-Fundus, show that Fundus-R1 clearly outperforms multiple baselines, including its generic counterpart (Qwen2.5-VL) and a stronger edition post-trained without using the generated traces. This work paves the way for training powerful fundus-reading MLLMs with publicly available data.

Figures & tables

Figure 1. Showcases of multimodal large language model (MLLM) based fundus image reading. Our proposed Fundus-R1 is a reasoning -enhanced model trained exclusively on public data. Best viewed on screen.

Model	Training Data (Raw annotations)	Reasoning Trace	RL Reward
DeepDR-LLM ( Li et al., 2019 )	private (clinical reports)	✗	–
VisionFM ( Qiu et al., 2024 )	public + private (clinical reports)	✗	–
VisionUnite ( Li et al., 2025b )	public + private (clinical reports)	✓	–
RetinalGPT ( Zhu et al., 2025 )	public + private (retinal images)	✓	–
EyeCareGPT ( Li et al., 2025a )	public + private (clinical reports)	✓	–
FundusExpert* ( Liu and Song, 2025 )	public + private (clinical reports)	✓	–

Table 1. Fundus-R1 versus existing fundus-reading MLLMs. The star symbol (*) indicates models publicly accessible and thus evaluated in our experiments.

Dataset	#Samples	Pixel labeled	Primary tasks
Image modality: CFP
FGADR-143-9 ( Kheradfallah et al., 2022 )	143	143	DR-specific lesion segmentation
IDRiD ( Porwal et al., 2018 )	413	54	DR grading
GRAPE ( Huang et al., 2023 )	631	0	Glaucoma progression prediction
JSIEC ( Cen et al., 2021 )	800	0	35-class disease classification
Retinal-Lesions ( Wei et al., 2021 )	1,264	1,264	DR grading

Table 2. Training data sources for Fundus-R1 . To avoid data leakage, images included in our test sets, i.e. FunBench ( Wei et al., 2025 ) , Omni-Fundus ( Hu et al., 2024 ) and GMAI-Fundus ( Ye et al., 2024 ) , are excluded. Over 94% of our training samples are with image-level labels only.

Panel of Figure 2 — Figure 2. Proposed method for generating visual-finding-embedded reasoning traces. Given a task label $y$ and fundus modality $m$ , we first emplopy a generic LLM $\mathcal{L}$ ( Qwen3-Max ) to obtain structured label-and-modality-specific domain knowledge $DK[y,m]$ and a visual-finding vocabulary from authoritative ophthalmic references. For each training image $I$ of modality $m$ and labeled with $y$ , we extract from the image visual findings $VF[I]$ by prompting a generic MLLM $\mathcal{M}_{vfe}$ ( Qwen2.5-VL-32B ). Lastly, a reasoning trace is generated for the given image by prompting $\mathcal{L}$ with question $q$ , answer $y$ , $DK[y,m]$ , and $VF[I]$ .

Panel of Table 3 — Table 3. Reasoning traces generated for given VQA triplets. Free-form CoT generation tends to introduce over-interpretation and logically inconsistent reasoning, while our method produces evidence-aligned and logically consistent reasoning.

Panel of Figure 3 — Figure 3. Our process reward $r_{pro}$ . Best viewed digitally.

Test set	Modalities (#Images)	#VQA triplets
FunBench ( Wei et al., 2025 )	CFP (7,608) / OCT (6,076) / UWF (2,664)	91,810
Omni-Fundus ( Hu et al., 2024 )	CFP (8,209) / OCT (3,791)	12,000
GMAI-Fundus ( Ye et al., 2024 )	CFP (1,327) / OCT (311) /UWF(63)	1,718

Table 4. Three test sets used in our experiments.

Training without reasoning trace:
#	Training Setup	Overall				FunBench				Omni-Fundus				GMAI-Fundus
#	Training Setup	Avg	FunBench	Omni-Fundus	GMAI-Fundus	L1	L2	L3	L4	MR	AI	LG	DD	AR	NT	BVR	DD	SG
A :	Base (Qwen2.5-VL-3B)	48.3	41.0	66.4	37.4	88.1	40.8	21.7	13.6	96.4	87.4	44.6	37.3	31.3	25.3	74.7	37.1	18.6
B :	A + SFT	53.8	58.9	71.1	31.4	69.3	95.8	20.4	50.0	94.2	91.9	51.7	46.4	24.7	16.0	65.3	33.4	17.6
C :	A + GRPO	55.5	53.9	68.1	44.6	76.8	71.5	27.2	40.2	92.4	92.8	36.2	50.9	38.7	27.3	85.3	32.0	39.7
D :	A + SFT + GRPO	52.5	53.1	71.3	33.0	88.4	57.7	30.5	35.7	95.4	93.8	43.4	52.7	28.0	19.3	37.3	43.3	37.1

Table 5. Performance of varied training setups. Bold and underline denote the best and second-best, respectively.

Visual finding	Ours			Qwen3-VL-Plus
Visual finding	Sen.	Spe.	S2	Sen.	Spe.	S2
Microaneurysm (MA)	78.3	49.5	60.7	58.4	23.3	33.3
Retinal hemorrhage (RH)	73.7	44.4	55.4	45.5	51.0	48.1
Hard exudate (HE)	78.7	58.1	66.9	49.9	65.8	56.7
Cotton-wool spot (CWS)	48.2	94.4	63.8	31.9	85.3	46.4
Vitreous hemorrhage (VH)	62.5	75.4	68.3	37.5	81.0	51.3

Table 6. Evaluation of visual finding extraction. Test set: Retinal-Lesions ( Wei et al., 2021 ) . Performance metrics: Sensitivity, Specificity and their harmonic mean (S2).

Generic MLLMs:
Model	Vision Encoder	LLM	Post-training Data Scale	FunBench	Omni-Fundus	GMAI-Fundus	Avg.
Qwen2.5-VL-3B ( Team, 2025a )	Qwen2.5-ViT	Qwen2.5-3B	–	41.0	66.4	37.4	48.3
InternVL2.5-8B ( Chen et al., 2024b )	InternViT-300M	InternLM2.5-7B	–	51.0	55.6	48.2	51.6
Qwen2.5-VL-7B	Qwen2.5-ViT	Qwen2.5-7B	–	46.1	68.1	42.1	52.1
Medical MLLMs (SFT)
HealthGPT-M3-7B ( Lin et al., 2025 )	CLIP ViT-L/14-336	Phi-3-mini	1.6M	52.4	64.0	46.3	54.2

Table 7. Fundus-R1 versus others. Models within each group are sorted in descending order by their overall performance.

Panel of Figure 4 — Figure 4. The use of the process reward $r_{pro}$ makes the model achieve larger answer rewards with shorter output length.

Explore similar work

VOLMO: Versatile and Open Large Models for Ophthalmology
Mar 25, 2026Zhenyue Qin, Younjoon Chung, Elijah Lee +16Medical VLMs Medical LLMs
Department of Biomedical Informatics & Data Science, Yale University · Ray and Stephanie Lane Computational Biology Department, Carnegie Mellon University · Yong Loo Lin School of Medicine, National University of Singapore +4
Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning
Aug 22, 2025Ruiqi Wu, Yuang Yao, Na Su +11Medical Image Analysis Multimodal Reasoning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China
FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
Sep 27, 2026David Restrepo, Chenwei Wu, Luis Filipe Nakayama +4Medical Image Analysis Medical Image Benchmarks
CentraleSupélec, Université Paris-Saclay, France · Cancer Data Science Unit, IHU PRISM, Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, France · University of Michigan, USA +4

Mar 25, 2026J/K move · Enter open · S save

VOLMO: Versatile and Open Large Models for Ophthalmology

Zhenyue Qin, Younjoon Chung, Elijah Lee, Wanyue Feng+15

Department of Biomedical Informatics & Data Science, Yale University · Ray and Stephanie Lane Computational Biology Department, Carnegie Mellon University · Yong Loo Lin School of Medicine, National University of Singapore+4

Medical VLMs Medical LLMs

Open paper AskPDF arXiv

Vision impairment affects millions globally, and early detection is critical to preventing irreversible vision loss. Ophthalmology workflows require clinicians to integrate medical images, structured clinical data, and free-text notes to determine disease severity and management, which is time-consuming and burdensome. Recent multimodal large language models (MLLMs) show promise, but existing general and medical MLLMs perform poorly in ophthalmology, and few ophthalmology-specific MLLMs are openly available. We present VOLMO (Versatile and Open Large Models for Ophthalmology), a model-agnostic, data-open framework for developing ophthalmology-specific MLLMs. VOLMO includes three stages: ophthalmology knowledge pretraining on 86,965 image-text pairs from 26,569 articles across 82 journals; domain task fine-tuning on 26,929 annotated instances spanning 12 eye conditions for disease screening and severity classification; and multi-step clinical reasoning on 913 patient case reports for assessment, planning, and follow-up care. Using this framework, we trained a compact 2B-parameter MLLM and compared it with strong baselines, including InternVL-2B, LLaVA-Med-7B, MedGemma-4B, MedGemma-27B, and RETFound. We evaluated these models on image description generation, disease screening and staging classification, and assessment-and-management generation, with additional manual review by two healthcare professionals and external validation on three independent cohorts for age-related macular degeneration and diabetic retinopathy. Across settings, VOLMO-2B consistently outperformed baselines, achieving stronger image description performance, an average F1 of 87.4% across 12 eye conditions, and higher scores in external validation.

Abstract

Explore similar work

VOLMO: Versatile and Open Large Models for Ophthalmology

Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs