Model Confidence-Guided Multi-Image Fusion of Fundus Images for Diabetic Retinopathy Diagnosis
Authors: Ananya Raghu, Anisha Raghu, Alice S. Tang, Yannis M. Paulus, Tyson N. Kim, Tomiko T. Oskotsky
Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA · Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA, USA · Wilmer Eye Institute, Department of Ophthalmology, Johns Hopkins University, Baltimore, MD, USA · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA · Department of Ophthalmology, University of California San Francisco, San Francisco, CA, USA · Division of Clinical Informatics and Digital Transformation, University of California San Francisco, San Francisco, CA, USA
Purpose: Early screening for eye diseases is critical in low- and middle-income countries where access to care is limited. We investigate whether a confidence-guided, multi-image diabetic retinopathy diagnosis framework can integrate image filtering with confidence-aware predictions for reliable screening at capture. Methods: We develop a multi-image fusion method that aggregates retinal views to improve confidence and balanced accuracy. Our method uses confidence to identify unreliable predictions, prompting retakes when needed. We compare: (1) a cascaded image-quality and disease diagnosis pipeline using a single image per patient, (2) confidence-based prediction, and (3) our confidence-based multi-image fusion pipeline. All methods are evaluated using a RETFoundGreen backbone on the mBRSET (n = 1,234) and BRSET (n = 7,599) datasets. Results: At 70% coverage, our method achieves 91% balanced accuracy on mBRSET and 97% on BRSET, improvements of ~12% and ~6%, respectively, over cascade filtering. The image-quality cascade reaches sensitivities of 61% on mBRSET and 86% on BRSET, whereas our framework reaches 94% and 96%, respectively, at 50% coverage. Conclusions: Human-annotated quality labels are weakly associated with diagnostic performance, and confidence-based filtering consistently outperforms image quality-based cascaded pipelines. Translational Relevance: Using confidence-based multi-image fusion, patients receive more reliable predictions, reducing incorrect diagnoses during screening. The lightweight backbone and single inference pass per image make the framework compatible with low-latency mobile screening systems in resource-limited settings.
Diabetic retinopathy (DR) is a leading cause of vision impairment worldwide, highlighting the need for accurate and accessible screening tools. Optical Coherence Tomography (OCT) provides high-resolution structural information of the retina, whereas OCT angiography (OCTA) offers complementary vascular information that is highly relevant for DR diagnosis. In this study, we propose a cross-modal fusion of OCT B-scans with single-channel en face OCTA using a bidirectional cross-modal attention network for automated DR classification. Two independent datasets, OCT500 and UIC, comprising 730 subjects in total, were utilized to evaluate performance under within-dataset, combined-dataset, and cross-dataset generalization settings. A ConvNeXt V2 model trained solely on OCT images served as the unimodal baseline. In addition to ground-truth (GT) OCTA, we explored the use of translated (TR) OCTA generated from OCT scans, eliminating the requirement for dedicated OCTA hardware. Experimental results demonstrate that cross-modal fusion consistently outperforms unimodal OCT classification across all evaluation scenarios. Fusion with GT OCTA improved classification accuracy and discriminative performance, while TR OCTA achieved comparable or superior results in most settings. Furthermore, TR OCTA improved sensitivity and cross-dataset generalization, indicating enhanced robustness to domain shifts. These findings demonstrate that attention-based OCT-OCTA en face fusion provides clinically meaningful improvements for DR detection and suggest that computationally generated OCTA can serve as a practical, low-cost alternative to hardware-acquired OCTA, enabling broader deployment of high-performance retinal screening systems in resource-limited clinical environments.
Rashadul Hasan Badhon, Atalie Carina Thompson, Jennifer I. Lim +2
Diabetic Retinopathy (DR) is one of the leading causes of preventable blindness, yet rural regions often lack the specialists and infrastructure needed for early detection. Although cloud-based deep learning systems offer high accuracy, they face significant challenges in these settings due to high latency, limited bandwidth, and high data transmission costs. To address these challenges, we propose a two-tier edge-cloud cascade on the public APTOS 2019 Blindness Detection dataset. Tier 1 runs a lightweight MobileNetV3-small model on a local clinic device to perform a binary triage between Referable DR (Classes 2-4) and Non-referable DR (Classes 0-1). Tier 2 runs a RETFoundDINOv2 model in the cloud for ordinal severity grading, but only on the subset of images flagged as referable by Tier 1. On a stratified APTOS test split of 733 images, Tier 1 reaches 98.99% sensitivity and 84.37% specificity at a validation-tuned high-sensitivity threshold. The default cascade forwards 49.52% of test images to Tier 2, reducing cloud calls by 50.48% relative to using a cloud-based model for all images. In the deployed 4-class output space (Class 0-1 / Class 2 / Class 3 / Class 4), the cascade obtains 80.49% accuracy and 0.8167 quadratic weighted kappa; the cloud-only baseline obtains 80.76% accuracy and 0.8184 quadratic weighted kappa. On APTOS, the cascade cuts cloud use by about half with a modest drop in grading performance. Index Terms: Diabetic Retinopathy, Edge-Cloud Cascade, MobileNetV3-small, RETFound-DINOv2, Retinal Screening, tele-ophthalmology
Diabetic retinopathy is a leading cause of preventable blindness; its early lesions are small, low contrast, and easily missed in manual screening. Most automated detectors handle the four non-proliferative DR lesions: microaneurysms, hemorrhages, hard exudates, and soft exudates, with a single multi-class model, even though these lesions differ sharply in size, color, morphology, and prevalence, so a shared model favors common, easy classes over rare, difficult ones. We present PRISM-DR, a lesion-specific pipeline that trains one single-class detector per lesion, each with its own configuration. From a raw fundus image, the pipeline applies region of interest cropping, fundus-specific preprocessing, four parallel YOLO detectors, tiling, per-lesion ensembling of five cross-validation folds, and an inter-lesion suppression step that resolves overlaps by physical lesion size and clinical priority rather than confidence. Per lesion, the best of five YOLO generations is selected, and augmentation is tuned by Bayesian optimization. Trained on IDRiD with stratified five-fold cross-validation, the system reaches a test mAP50 of 0.527 and F1 of 0.529, highest AP50 on hard exudates with 0.561. Without fine-tuning, the models transfer well where the imaging scale is close to IDRiD and degrade as field of view and resolution depart. These modest absolute results reflect a small single-source training set and a difficult task; however, treating each lesion as a separate detection problem is a practical alternative to a single multi-class model.