RetinaGuard: Risk-Controlled Agentic Triage for Multi-Disease Retinal Screening
Authors: Sohaib Sulman, Muhammad Atif Saeed, Akhtar Jamil, Alaa Ali Hameed, Roland Aydin
Organizations: Department of Data Science and Artificial Intelligence, National University of Computer and Emerging Sciences, Islamabad, Pakistan · Department of Computer Science, National University of Computer and Emerging Sciences, Islamabad, Pakistan · Department of Computer Engineering, Istinye University, Istanbul, Turkey · Department of Materials Science, Saarland University, Saarbrücken, Germany · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
Retinal screening programmes need to know which automated decisions can be accepted without review. This paper presents RetinaGuard, an agentic triage framework in which quality, diagnostic, uncertainty, verification and explanation agents gather evidence for each fundus photograph, a learned failure predictor estimates the probability that the decision is unsafe, auxiliary tools are invoked only when their expected value of information exceeds their cost, and the acceptance threshold is certified by conformal risk control. On the RFMiD test split, RetinaGuard was compared with confidence and calibrated-uncertainty referral at matched referral budgets and equal certified safety, using two RETFound primary models of different strength. With the weaker model (macro-AUROC 0.896), verifier disagreement added significant information, RetinaGuard captured 119 of 280 unsafe decisions at a 20% budget against 88 for lowest-confidence referral, and at a nominal 5% unsafe level it accepted 37.0% of cases against 10.3%. With the stronger model (0.925) the verifier added nothing beyond uncertainty and the advantage shrank to a few percentage points. In both settings its coverage at the nominal 5% level stayed near 37.7%, gating cut computation by about 30%, and a quality agent reduced unsafe acceptances under most image corruptions. The benefit of agentic evidence gathering thus depends on how much the primary model's own uncertainty already reveals about its errors.
Figures & tables
Dataset
Partition
Images
Use
Labels
RFMiD [ 7 ]
official training, 80%
1536
model training
screening + 24 diseases
official training, 20%
384
conformal calibration only
same
official validation
640
model selection and calibration
same
official test
640
evaluation only
same
ODIR-5K [ 8 ]
labelled eyes
6690
external evaluation only
screening + 12 diseases
IDRiD [ 25 ]
segmentation set
81
explanation validation only
lesion masks
Table 1: Datasets, partitions and their use. Abnormal prevalence: 79.1% in the RFMiD test split. Lesion masks: microaneurysms, haemorrhages, hard and soft exudates, optic disc.
Figure 1: Example images after field-of-view cropping: RFMiD test images with single labels, ODIR-5K images, and an IDRiD image with its lesion masks overlaid.
Figure 2: RetinaGuard agent graph. The quality agent refers acquisition outliers before diagnosis; the diagnostic, verification and evidence agents run in sequence, and after each stage a failure predictor estimates the probability rk(x) that the automated decision is unsafe. The next agent is invoked only when its expected value of information exceeds its cost μc . The decision agent accepts a case only when its final risk is below the threshold λ^α certified by conformal risk control, and a reporting agent summarises the evidence without authority over the action.
probability difference, label disagreement, Jensen–Shannon divergence; risk r1
19
Evidence
rollout, Grad-CAM
map entropy and concentration, stability, cross-model agreement; risk r2
81
Decision
certified threshold
accept, verify or refer
–
Table 2: Agents, tools and evidence. Latency is the measured per-image cost of each tool on one T4 GPU (batch size 1).
Method
Referral score
B1
lowest confidence (raw screening confidence or minimum label margin)
B2
calibrated uncertainty (entropy, label margin, Monte Carlo mutual information, variance or flip rate)
B3
unlearned sum of standardised uncertainty and verifier disagreement
B4
learned failure predictor without explanation features, verifier invoked for every case
B5
RetinaGuard: learned failure predictor with value-of-information tool invocation
B6
learned failure predictor with all evidence computed for every case
Table 3: Referral methods compared at matched budgets.
Metric
Definition
Interpretation
Coverage
∣A∣/n
share of cases decided automatically
Unsafe acceptance (selective risk)
∑i∈AEi/∣A∣
error rate among accepted cases
Marginal unsafe rate
∑i∈AEi/n
quantity bounded by conformal risk control
Error-detection rate
∑i∈REi/∑iEi
share of unsafe decisions that are referred
Error-detection AUROC
AUROC of the score for Ei
ranking quality of a failure predictor
Macro-AUROC / macro-F1
mean over labels
classification quality (all or accepted cases)
Table 4: Evaluation metrics. A : accepted (not referred) cases; R : referred cases; Ei : unsafe automated decision; n : test cases.
Backbone
Macro-AUROC [95% CI]
Screening AUROC
Macro-F1
ResNet-50
0.897 [0.870, 0.920]
0.964
0.408
EfficientNet-B3
0.874 [0.841, 0.901]
0.971
0.446
ConvNeXt-B (verifier)
0.952 [0.939, 0.963]
0.981
0.508
Swin-B
0.949 [0.935, 0.961]
0.983
0.513
RETFound, conservative schedule
0.896 [0.869, 0.919]
0.958
0.432
RETFound, original recipe (primary)
0.925 [0.909, 0.941]
0.974
0.477
Table 5: Backbones on the RFMiD test split ( n=640 ). Macro-AUROC over 25 labels with 95% bootstrap confidence interval.
Evidence used by the failure predictor
Error-detection AUROC
EDR@20 (%)
A0 minimum label margin (no learning)
0.685
29.7
A1 uncertainty (calibrated and Monte Carlo)
0.747
37.1
A2 A1 + verifier disagreement
0.751
35.4
A3 A2 + image quality
0.757
33.2
A4 A3 + explanation features
0.766
38.4
Table 6: Cumulative evidence ablation on the test split; failure predictors are fitted on validation data only. EDR@20: share of the 229 unsafe decisions referred at a 20% referral budget.
Figure 3: Comparison at matched referral budgets on the RFMiD test split. (a) Share of unsafe automated decisions captured by referral; (b) unsafe-acceptance rate among the cases left to the system.
Budget 10%
Budget 20%
Budget 30%
Method
Caught
Unsafe (%)
Caught
Unsafe (%)
Caught
Unsafe (%)
B1 lowest confidence
33
34.0
68
31.4
99
29.0
B2 calibrated uncertainty
35
33.7
70
31.1
103
28.1
B4 verifier always on
43
32.3
76
29.9
115
25.4
B6 all evidence always on
48
31.4
88
27.5
122
23.9
B5 RetinaGuard
48
31.4
82
28.7
119
24.6
Table 7: Errors captured (out of 229) and unsafe acceptance among accepted cases at matched referral budgets; without referral the unsafe-acceptance rate is 35.8%.
α=2%
α=5%
α=10%
Method
Coverage
Unsafe
Coverage
Unsafe
Coverage
Unsafe
B1 lowest confidence
17.5
2.5
33.1
5.3
50.2
11.1
B2 calibrated uncertainty
20.9
2.3
38.4
7.8
59.1
14.2
B4 verifier always on
22.2
2.2
37.2
5.0
52.5
9.5
B5 RetinaGuard
20.0
2.2
37.7
5.5
46.4
8.0
Table 8: Operating points certified by conformal risk control on the calibration set: coverage and realised marginal unsafe rate on the test split (both in %).
Quantity
Conservative schedule
Original recipe
Primary macro-AUROC
0.896
0.925
Unsafe automated decisions (%)
43.8 (280)
35.8 (229)
Error-detection AUROC, A0 / A1 / A2
0.614 / 0.790 / 0.848
0.685 / 0.747 / 0.751
Gain from verifier disagreement [95% CI]
0.058 [0.027, 0.089]
0.004 [-0.027, 0.033]
EDR gain over B1 at 20% ( p )
11.1 (<0.001)
6.1 (0.150)
EDR gain over B2 at 20% ( p )
12.9 (<0.001)
5.2 (0.200)
Table 9: The same analysis with a weaker and a stronger RETFound primary model. EDR gain: difference in error-detection rate between RetinaGuard (B5) and lowest-confidence (B1) or calibrated-uncertainty (B2) referral at a 20% budget, in percentage points, with Holm-adjusted p-value. Coverage: share of test cases accepted at the certified α=5% level.
Figure 4: (a) Macro-AUROC under six corruptions (severity 0: clean). (b) Marginal unsafe rate at the α=5% operating point with (solid) and without (dashed) the quality agent. (c) Distribution of predicted risk on the RFMiD test split and on ODIR-5K.
Figure 5: Explanation validation on IDRiD lesion masks (union of microaneurysms, haemorrhages and exudates). CAM: Grad-CAM (ConvNeXt-B); Roll: attention rollout and IG: Integrated Gradients (RETFound); Ctr: centre prior; Rnd: random map. Bars show means with 95% confidence intervals.
This paper presents a safety-centered empirical evaluation of uncertainty-aware last-layer adaptation for referable diabetic retinopathy screening using RETFound, a self-supervised vision-transformer retinal foundation model used here as a frozen feature encoder, and the public APTOS 2019 and DDR diabetic retinopathy fundus image datasets. We compare a cached-feature softmax head, post-hoc temperature scaling, variational Bayesian last-layer heads, a diagonal Laplace last-layer approximation, and an SNGP-style cached-feature head. On APTOS, uncertainty-aware operating points improved sensitivity and selective-referral behavior. The strongest APTOS selective-referral result deferred approximately 20 percent of cases and reduced accepted-case false negatives to zero while preserving high accepted-case specificity. However, threshold tuning also reduced false negatives at high false-positive cost, so false-negative reduction alone was not unique to Bayesian modeling. On DDR, native Bayesian heads qualitatively reproduced the APTOS direction but with weaker tradeoffs, while the APTOS-trained SNGP checkpoint transferred poorly and failed to provide useful external selective-referral behavior. These results highlight the value of safety-centered evaluation beyond aggregate accuracy: uncertainty-aware last-layer heads can improve internal safety-oriented operating points, but trustworthy retinal screening claims require explicit safety-coverage evaluation and second-dataset validation under shift.
Hallucinations in medical large language models (LLMs) remain a safety-critical issue, particularly when available evidence is insufficient or conflicting. We study this problem in diabetic retinopathy (DR) decision settings and introduce RETINA-SAFE, an evidence-grounded benchmark aligned with retinal grading records, comprising 12,522 samples. RETINA-SAFE is organized into three evidence-relation tasks: E-Align (evidence-consistent), E-Conflict (evidence-conflicting), and E-Gap (evidence-insufficient). We further propose ECRT (Evidence-Conditioned Risk Triage), a two-stage white-box detection framework: Stage 1 performs Safe/Unsafe risk triage, and Stage 2 refines unsafe cases into contradiction-driven versus evidence-gap risks. ECRT leverages internal representation and logit shifts under CTX/NOCTX conditions, with class-balanced training for robust learning. Under evidence-grouped (not patient-disjoint) splits across multiple backbones, ECRT provides strong Stage-1 risk triage and explicit subtype attribution, improves Stage-1 balanced accuracy by +0.15 to +0.19 over external uncertainty and self-consistency baselines and by +0.02 to +0.07 over the strongest adapted supervised baseline, and consistently exceeds a single-stage white-box ablation on Stage-1 balanced accuracy. These findings support white-box internal signals grounded in retinal evidence as a practical route to interpretable medical LLM risk triage.
Zhe Yu, Wenpeng Xing, Meng Han
Binjiang Institute of Zhejiang University, Hangzhou, China · Communication University of Zhejiang, Hangzhou, China · Zhejiang University, Hangzhou, China
Diabetic Retinopathy (DR) is a leading cause of preventable blindness among working-age adults worldwide, yet most automated screening systems are limited to image-level classification and lack clinically structured reporting. We propose Retina-RAG, a low-cost modular framework that jointly performs DR severity grading, macular edema (ME) detection, and report generation. The architecture decouples a high-performance retinal classifier and a parameter-efficient vision-language model (Qwen2.5-VL-7B-Instruct) adapted via Low-Rank Adaptation (LoRA), enabling flexible component integration. A retrieval-augmented generation (RAG) module injects curated ophthalmic knowledge together with structured classifier outputs at inference time to improve diagnostic consistency and reduce hallucinations. Retina-RAG achieves an F1-score of 0.731 for DR grading and 0.948 for ME detection, substantially outperforming zero-shot Qwen (0.096, 0.732) and MMed-RAG (0.541, 0.641) on a retinal disease detection dataset with captions. For report generation, Retina-RAG attains ROUGE-L 0.438 and SBERT similarity 0.884, exceeding all baselines. The full framework operates on a single consumer-grade GPU, demonstrating that clinically structured retinal AI can be achieved with modest computational resources.