Hybrid Approach for Enhancing Lesion Segmentation in Fundus Images

Mohammadmahdi Eshragh; Emad A. Mohammed; Behrouz Far; Ezekiel Weis; Carol L Shields; Sandor R Ferenczy; Trafford Crump

cs.CVSep 29, 2025

Hybrid Approach for Enhancing Lesion Segmentation in Fundus Images

Authors: Mohammadmahdi Eshragh, Emad A. Mohammed, Behrouz Far, Ezekiel Weis, Carol L Shields, Sandor R Ferenczy, Trafford Crump

Organizations: Department of Electrical & Software Engineering, University of Calgary, Canada. · Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, Canada. · Dept of Ophthalmology & Visual Sciences, University of Alberta, Edmonton, Canada. · Wills Eye Hospital, Philadelphia, USA. · Department of Surgery, University of Calgary, Canada.

Abstract

Choroidal nevi are common benign pigmented lesions in the eye, with a small risk of transforming into melanoma. Early detection is critical to improving survival rates, but misdiagnosis or delayed diagnosis can lead to poor outcomes. Despite advancements in AI-based image analysis, diagnosing choroidal nevi in colour fundus images remains challenging, particularly for clinicians without specialized expertise. Existing datasets often suffer from low resolution and inconsistent labelling, limiting the effectiveness of segmentation models. This paper addresses the challenge of achieving precise segmentation of fundus lesions, a critical step toward developing robust diagnostic tools. While deep learning models like U-Net have demonstrated effectiveness, their accuracy heavily depends on the quality and quantity of annotated data. Previous mathematical/clustering segmentation methods, though accurate, required extensive human input, making them impractical for medical applications. This paper proposes a novel approach that combines mathematical/clustering segmentation models with insights from U-Net, leveraging the strengths of both methods. This hybrid model improves accuracy, reduces the need for large-scale training data, and achieves significant performance gains on high-resolution fundus images. The proposed model achieves a Dice coefficient of 89.7% and an IoU of 80.01% on 1024*1024 fundus images, outperforming the Attention U-Net model, which achieved 51.3% and 34.2%, respectively. It also demonstrated better generalizability on external datasets. This work forms a part of a broader effort to develop a decision support system for choroidal nevus diagnosis, with potential applications in automated lesion annotation to enhance the speed and accuracy of diagnosis and monitoring.

Figures & tables

Ref#	Method	Segmented Part	Dataset
( Eshragh et al., 2024 )	Ensemble UNet	CN	Private
( Biglarbeiki et al., 2024a )	Application (YOLO)	CN	Private
( Jiang et al., 2020 )	Multi-Path Recurrent UNet	Vessels & Optic	DRIVE ( Niemeijer et al., 2009 )
( Zhao et al., 2021 )	UNet & Transfer Learning	Disk & Cup	Drishti-GS ( Sivaswamy et al., 2014 )

Table 1 : Studies on Fundus image segmentation

Figure 1 : Proposed architecture. Blue sections represent this study’s contributions, orange sections denote ML/DL-based tool, and green sections indicate traditional segmentation model.

Figure 2 : Eye fundus image, segmented with proposed method

Component	Setting	How chosen
Guide network input	$128\times 128$ RGB; scaled to $[0,1]$	Design (Sec. 7.5)
Guide output threshold	0.5 on sigmoid map	Fixed
Data split	90/10 train/test; 80/20 train/validation; seed 101112	Fixed
$\mathrm{Coef}$ (Eq. 13)	5	Largest training lesion
Calibration factor $c$ (Eq. 14)	0.85	Validation grid search
Default $K$ for empty mask	700	Smallest training lesion

Table 2 : Constants and settings of the Hybrid pipeline used for all reported results.

Figure 3 : Effect of image size on the standard UNet: (a) IoU and Dice decrease, while (b) energy (bars) and training time (line) rise sharply.

Model	IOU	Dice	Hausdorff (px)	Specificity	Volumetric Similarity
Attention UNet	47.05%	60.57%	77.91	98.77%	72.37%
Swin UNet (Transformer-based)	62.17%	72.97%	104.30	99.42%	88.90%
nnU-Net	73.04%	83.23%	49.22	98.89%	90.03%
Hybrid Model	87.89%	90.93%	21.6	99.92%	95.32%

Table 3 : Segmentation quality comparison of UNet variants, nnU-Net, and the Hybrid Method on 1024×1024 fundus images.

Model	Train Time (min)	Energy (Wh)
Attention UNet	134.2	671.0
Swin UNet (Transformer-based)	160.7	803.5
nnU-Net	102.0	510.0
Hybrid Model (one training run of the 128 $\times$ 128 guide network)	3.16	15.8

Table 4 : Computational cost comparison for training segmentation models.

Figure 4 : Per-image Dice coefficients for the four models on the internal test set ( $n=25$ ). Each grey line connects the scores obtained for a single test image across the four models; solid bars indicate the medians, while dashed bars indicate the means. The Hybrid Model scores are concentrated near 99%, with four lower-performing cases, two of which correspond to the failures discussed in Section 7.10 .

Figure 5 : Paired per-image differences in Dice scores (Hybrid Model minus baseline, expressed in percentage points), sorted within each panel. Red bars indicate differences favouring the Hybrid Model, while the dashed line denotes the median paired difference. Reported $p$ values are from exact two-sided Wilcoxon signed-rank tests with Holm–Bonferroni correction.

Model	Mean Dice (%)	95% CI (bootstrap)	Median (%)	SD	Images with Dice $\geq$ 80%
Attention UNet	60.57	[50.63, 69.22]	65.89	24.07	6/25
Swin UNet	72.97	[63.06, 81.60]	85.14	24.60	14/25
nnU-Net	83.23	[77.73, 87.72]	86.96	13.20	20/25
Hybrid Model	90.93	[81.27, 97.68]	98.96	21.88	21/25

Table 5 : Per-image Dice coefficients on the internal test set (

n=25

, 1024

\times

1024). Confidence intervals correspond to percentile bootstrap 95% intervals for the mean, based on 10,000 resamples.

Hybrid Model vs.	Mean $\Delta$ Dice [95% CI]	Median $\Delta$ Dice	Wins / losses	Wilcoxon $W$ ; $p$ (Holm)	Permutation $p$	Effect size $r_{rb}$ ; $d_{z}$
nnU-Net	+7.70 [0.93, 14.06]	+8.66	21 / 4	51; $0.0018$	$0.0329$	0.69; 0.45
Swin UNet	+17.96 [8.89, 27.63]	+10.60	21 / 4	41; $0.0011$	$6.1\times 10^{-4}$	0.75; 0.74
Att UNet	+30.36 [20.30, 40.42]	+31.45	23 / 2	18; $4.5\times 10^{-5}$	$1.5\times 10^{-5}$	0.89; 1.16

Table 6 : Pairwise comparisons of per-image Dice scores between the Hybrid Model and each baseline on the internal test set (

n=25

\Delta

Dice represents the Hybrid Model minus the baseline, expressed in percentage points; the corresponding 95% CI is a percentile bootstrap interval based on 10,000 resamples.

W

denotes the exact two-sided Wilcoxon signed-rank statistic, with

p

values adjusted across the three comparisons using the Holm–Bonferroni procedure. The permutation

p

value is obtained from a sign-flip permutation test of the mean paired difference using 200,000 resamples.

r_{rb}

denotes the matched-pairs rank-biserial correlation, and

d_{z}

denotes Cohen’s

d

for paired differences.

Panel of Figure 6 — Figure 6 : Representative segmentation results across performance groups.

Prediction	Measurement Method	Results
Attention UNet	IOU	42.41%
Attention UNet	Dice	54.07%
Swin UNet	IOU	47.27%
Swin UNet	Dice	59.28%
nnU-Net	IOU	59.49%
nnU-Net	Dice	69.31%

Table 7 : Segmentation accuracy of the four models on the external Wills Eye Hospital dataset (

n=90

, 1024

\times

1024; IoU and Dice are means of per-image scores).

Model	Mean Dice (%)	95% CI (bootstrap)	Median (%)	SD	Images with Dice $\geq$ 80%
Attention UNet	54.07	[48.19, 59.88]	57.87	28.84	22/90
Swin UNet	59.28	[53.56, 64.93]	63.19	27.54	26/90
nnU-Net	69.31	[63.14, 75.06]	81.46	29.07	46/90
Hybrid Model	76.79	[69.94, 82.99]	91.95	31.63	64/90

Table 8 : Per-image Dice coefficient on the external Wills Eye Hospital dataset (

n=90

, 1024

\times

1024). Confidence intervals are percentile bootstrap intervals for the mean (10,000 resamples).

Hybrid Model vs.	Mean $\Delta$ Dice [95% CI]	Median $\Delta$ Dice	Wins / losses / ties	Wilcoxon $W$ ; $p$ (Holm)	Permutation $p$	Effect size $r_{rb}$ ; $d_{z}$
nnU-Net	+7.47 [2.81, 12.16]	+3.26	72 / 15 / 3	725; $4.8\times 10^{-7}$	$0.0021$	0.62; 0.32
Swin UNet	+17.51 [11.92, 22.95]	+14.83	73 / 13 / 4	508; $8.9\times 10^{-9}$	$5.0\times 10^{-6}$	0.73; 0.66
Att UNet	+22.72 [16.35, 29.16]	+18.67	75 / 12 / 3	467; $2.7\times 10^{-9}$	$5.0\times 10^{-6}$	0.76; 0.73

Table 9 : Pairwise comparisons of per-image Dice scores between the Hybrid Model and each baseline on the external Wills Eye Hospital dataset (

n=90

\Delta

Dice represents the Hybrid Model minus the baseline, expressed in percentage points, with the corresponding percentile bootstrap 95% CI based on 10,000 resamples. Ties correspond to images for which both models returned an empty mask and are excluded prior to ranking.

W

denotes the two-sided Wilcoxon signed-rank statistic, computed using the normal approximation (

n>50

), with

p

values adjusted across the three comparisons using the Holm–Bonferroni procedure. The permutation

p

value is obtained from a sign-flip permutation test of the mean paired difference using 200,000 resamples.

r_{rb}

denotes the matched-pairs rank-biserial correlation, and

d_{z}

denotes Cohen’s

d

for paired differences.

Figure 7 : Per-image Dice coefficients for the four models on the external Wills Eye Hospital dataset ( $n=90$ ). Each grey line connects the scores obtained for a single image across the four models; solid bars indicate the medians, while dashed bars indicate the means. The Hybrid Model’s scores are concentrated above 90%, with a failure tail comprising 15 images with Dice $<0.5$ , including 7 empty predictions.

Figure 8 : Paired per-image differences in Dice scores on the external dataset (Hybrid Model minus baseline, expressed in percentage points), sorted within each panel. Red bars indicate differences favouring the Hybrid Model, while the dashed line denotes the median paired difference. Reported $p$ values are from two-sided Wilcoxon signed-rank tests with Holm–Bonferroni correction.

Covariate	Internal test ( $n=25$ )	External ( $n=90$ )	Mann–Whitney $p$	SMD
Brightness ( $L^{*}$ mean)	31.43 $\pm$ 4.96	32.09 $\pm$ 5.25	$0.732$	$+0.13$
Contrast ( $L^{*}$ SD)	8.36 $\pm$ 1.35	8.15 $\pm$ 2.05	$0.279$	$-0.12$
Colour cast ( $a^{*}$ mean)	26.69 $\pm$ 7.13	27.03 $\pm$ 6.80	$0.662$	$+0.05$
Colour cast ( $b^{*}$ mean)	27.28 $\pm$ 6.47	27.92 $\pm$ 5.91	$0.697$	$+0.10$
Sharpness (Laplacian variance)	1.18 $\pm$ 0.48	1.27 $\pm$ 0.48	$0.358$	$+0.18$
Lesion area (% of FOV)	1.64 $\pm$ 2.03	4.97 $\pm$ 8.52	$0.0016$	$+0.54$

Table 10 : Image-level covariates for the internal test set and the external Wills Eye Hospital dataset (mean

\pm

SD), computed within the fundus field of view (FOV). SMD denotes the standardized mean difference (external minus internal, expressed in pooled-SD units), and

p

values are from two-sided Mann–Whitney

U

tests. Lesion size was the only covariate that differed between the two datasets.

Covariate	Failures (Dice $<$ 0.5, $n=15$ )	Non-failures ( $n=75$ )	Mann–Whitney $p$	Spearman $\rho$ with Dice
Brightness ( $L^{*}$ mean)	31.47	31.86	$0.770$	$+0.03$ ( $0.78$ )
Contrast ( $L^{*}$ SD)	7.72	7.76	$0.618$	$-0.07$ ( $0.49$ )
Colour cast ( $a^{*}$ mean)	26.28	27.97	$0.509$	$+0.20$ ( $0.054$ )
Colour cast ( $b^{*}$ mean)	26.69	27.66	$0.665$	$+0.12$ ( $0.28$ )
Sharpness (Laplacian variance)	1.13	1.18	$0.914$	$+0.07$ ( $0.54$ )
Lesion area (% of FOV)	8.36	2.00	$0.0012$	$+0.04$ ( $0.72$ )

Table 11 : Failure-mode analysis on the external dataset. The median value of each covariate is reported for images on which the Hybrid Model failed (per-image Dice

<0.5

) and for the remaining successfully segmented images, together with the corresponding two-sided Mann–Whitney

U

test

p

values. Spearman rank correlations, with associated

p

values, quantify the relationship between each covariate and the Hybrid Model’s per-image Dice across all 90 images. Lesion size was the only covariate that significantly distinguished failures from successful cases.

Variant	Configuration	Mean Dice (%) [95% CI]	Median (%)	Full model better on	Wilcoxon $p$ (Holm)
A	Full model: automated $K$ + probability selection	90.93 [81.27, 97.68]	98.96	–	–
B	Fixed $K=100$ + probability selection	69.12 [57.84, 79.86]	85.94	21 / 25	$4.0\times 10^{-4}$
B	Fixed $K=500$ + probability selection	45.70 [34.99, 56.80]	39.42	23 / 25	$6.9\times 10^{-5}$
B	Fixed $K=1000$ + probability selection	27.66 [20.25, 35.68]	22.21	24 / 25	$5.5\times 10^{-5}$
B ∗	Oracle: best fixed $K$ per image	80.37	90.34	–	–
C	Automated $K$ + centroid selection	90.93 [81.27, 97.68]	98.96	0 / 25 (identical)	–

Table 12 : Ablation analysis of the Hybrid Model on the 25 internal test images (1024

\times

1024). Confidence intervals are percentile bootstrap intervals for the mean based on 10,000 resamples. Reported

p

values are from exact two-sided Wilcoxon signed-rank tests comparing each variant with the full model, with Holm–Bonferroni correction across the four comparisons. Variant C produced the same segmentation mask as the full model for every image; therefore, no statistical test was applicable. B ∗ represents an oracle upper bound over the fixed values

K\in{100,500,1000}

and is not a directly executable method.

Figure 9 : Comparison of automated and fixed $K$ values on the 24 test images with a non-empty guide mask. (a) Per-image Dice scores for the full model and the fixed- $K$ variants plotted against the automated $K$ value for each image (Eq. 14) on a logarithmic axis; dotted lines indicate the three fixed values. (b) Dice scores obtained with a fixed $K$ plotted against the mismatch between that value and the corresponding image-specific automated $K$ , pooled across the three fixed settings (72 image– $K$ pairs).

Model	Stage	1024 $\times$ 1024 (s per image)	3900 $\times$ 3900 (s per image)
Hybrid Model	Guide network: read + resize to 128 $\times$ 128	0.07 $\pm$ 0.01	0.07 $\pm$ 0.01
	Guide network: forward pass (CPU)	0.49 $\pm$ 0.03	0.49 $\pm$ 0.03
	Resize guide mask, compute $K$ (Eqs. 11–14)	$<$ 0.01	0.19 $\pm$ 0.01
	SLIC (Eq. 15)	1.00 $\pm$ 0.06	14.16 $\pm$ 1.12
	Superpixel selection (Eqs. 17–18)	0.01	0.14 $\pm$ 0.01
	Morphological closing	$<$ 0.01	0.02

Table 13 : Per-image CPU inference time (mean

\pm

SD over the 25 internal test images) on a laptop with an AMD Ryzen 7 5700U (8 cores/16 threads, 15 GB RAM) and no GPU. Guide network: TensorFlow 2.21; superpixel stage: scikit-image SLIC; nnU-Net: nnU-Net v2 with PyTorch 2.14 (CPU). The 3900

\times

3900 measurements use the test images upsampled from 1024

\times

1024. The guide network used for timing is architecturally identical to the one used for all accuracy results.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Test image	Attention UNet	Swin UNet	nnU-Net	Hybrid Model
0	74.81	55.31	85.86	99.67
1	73.12	84.08	82.56	79.36
2	58.23	92.19	91.62	98.16
3	19.18	46.58	79.75	98.96
4	46.89	92.38	79.19	78.39
5	87.81	74.51	80.05	49.80

Appendix

Table 14 : Per-image Dice coefficient (%) of all four models on the 25 internal test images (1024

\times

1024). Test-image indices follow the fixed data split.

Img	Att	Swin	nnU	Hyb	Img	Att	Swin	nnU	Hyb	Img	Att	Swin	nnU	Hyb
0	49.1	20.2	51.6	0.0	30	54.3	49.0	48.5	50.8	60	62.0	74.5	93.9	97.6
1	63.4	96.5	75.2	98.7	31	95.5	88.4	94.9	94.9	61	5.1	51.2	55.2	87.5
2	5.4	1.2	1.2	21.2	32	61.3	49.7	75.6	88.1	62	52.3	60.2	80.5	96.8
3	62.7	62.1	95.6	97.9	33	89.1	94.7	95.3	97.8	63	42.7	77.1	84.0	73.6
4	76.6	94.1	86.9	84.5	34	93.4	95.3	96.8	97.1	64	82.5	61.7	84.5	90.6
5	45.2	59.4	91.3	97.0	35	68.8	91.6	94.5	97.0	65	12.5	35.3	63.5	91.0

Appendix

Table 15 : Per-image Dice coefficient (%) of all four models on the 90 external Wills Eye Hospital images (Att = Attention UNet, Swin = Swin UNet, nnU = nnU-Net, Hyb = Hybrid Model). Means: 54.07, 59.28, 69.31 and 76.79.

Explore similar work

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images
May 12, 2024Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath +2Vessel Segmentation Fundus Images
aDepartment of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh
HSQ-VLM: A Novel Spatially-Constrained Quadrant Segmentation VLM Model for Explainability in Diabetic Retinopathy
Jun 11, 2026Shivum TelangMedical Vision-Language Models Ava-Vlm
Pittsburgh, Pennsylvania
ELEMENT: Multi-Modal Retinal Vessel Segmentation Based on a Coupled Region Growing and Machine Learning Approach
May 19, 2026Erick O. Rodrigues, Aura Conci, Panos LiatsisVessel Segmentation Fundus Images
Academic Department of Informatics, Universidade Tecnologica Federal do Parana (UTFPR), Pato Branco 80230-901, Brazil · Department of Computer Science, Universidade Federal Fluminense (UFF), Niteroi 24220-900, Brazil · Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi 127788, United Arab Emirates

May 12, 2024, eess.IVJ/K move · Enter open · S save

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images

Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath, Asif Iftekher Fahim+1

aDepartment of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh

Vessel Segmentation Fundus Images Retinal Imaging Semantic Segmentation Convolutional

Open paper AskPDF arXiv

Early detection of vision-threatening conditions such as diabetic retinopathy, glaucoma, and age-related macular degeneration depends on retinal fundus image analysis, but manual assessment is slow and expert-dependent. Automated convolutional neural networks classify fundus images accurately yet act as black boxes, and existing retinal vessel segmentation methods lose discriminative power under pathology and seldom exploit attention or transformer backbones. Using the FIVES and DRIVE fundus datasets, we develop a two-pipeline framework that pairs four-class disease classification with attention- and transformer-based vessel segmentation, organised in three stages: (1) FIVES images are augmented by rotation and horizontal and vertical flips and used to fine-tune eight ImageNet-pretrained CNNs: ResNet101, DenseNet169, Xception, InceptionV3, DenseNet121, InceptionResNetV2, ResNet50, and EfficientNetB0. (2) Five gradient-based explanation methods, Grad-CAM, Grad-CAM++, Score-CAM, Faster Score-CAM, and Layer-CAM, are computed on the final convolutional block of each classifier and compared qualitatively across architectures. (3) Ten U-Net variants are benchmarked for vessel segmentation: TransUNet (hybrid CNN--Transformer encoder) and Attention U-Net (gated skip connections), evaluated with ResNet50V2, ResNet101V2, and ResNet152V2 backbones, along with additional Attention U-Net configurations using DenseNet backbones, and the fully transformer-based Swin-UNet. ResNet101 gives the highest classification accuracy: 94.17% (F1 0.942)

>

88.33% for EfficientNetB0. For segmentation, the architecture ranking is consistent on both datasets: Attention U-Net

>

TransUNet

>

Swin-UNet. The strongest configuration is Attention U-Net with a ResNet101V2 backbone: FIVES IoU 0.722, Dice 0.838; DRIVE IoU 0.648, Dice 0.787, lifting DRIVE IoU 60.80

\rightarrow

64.83 over a prior custom U-Net.

Abstract

Explore similar work

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images

HSQ-VLM: A Novel Spatially-Constrained Quadrant Segmentation VLM Model for Explainability in Diabetic Retinopathy

ELEMENT: Multi-Modal Retinal Vessel Segmentation Based on a Coupled Region Growing and Machine Learning Approach