Hybrid Approach for Enhancing Lesion Segmentation in Fundus Images

Mohammadmahdi Eshragh; Emad A. Mohammed; Behrouz Far; Ezekiel Weis; Carol L Shields; Sandor R Ferenczy; Trafford Crump

cs.CVSep 29, 2025

Hybrid Approach for Enhancing Lesion Segmentation in Fundus Images

Authors: Mohammadmahdi Eshragh, Emad A. Mohammed, Behrouz Far, Ezekiel Weis, Carol L Shields, Sandor R Ferenczy, Trafford Crump

Organizations: Department of Electrical & Software Engineering, University of Calgary, Canada. · Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, Canada. · Dept of Ophthalmology & Visual Sciences, University of Alberta, Edmonton, Canada. · Wills Eye Hospital, Philadelphia, USA. · Department of Surgery, University of Calgary, Canada.

Abstract

Choroidal nevi are common benign pigmented lesions in the eye, with a small risk of transforming into melanoma. Early detection is critical to improving survival rates, but misdiagnosis or delayed diagnosis can lead to poor outcomes. Despite advancements in AI-based image analysis, diagnosing choroidal nevi in colour fundus images remains challenging, particularly for clinicians without specialized expertise. Existing datasets often suffer from low resolution and inconsistent labelling, limiting the effectiveness of segmentation models. This paper addresses the challenge of achieving precise segmentation of fundus lesions, a critical step toward developing robust diagnostic tools. While deep learning models like U-Net have demonstrated effectiveness, their accuracy heavily depends on the quality and quantity of annotated data. Previous mathematical/clustering segmentation methods, though accurate, required extensive human input, making them impractical for medical applications. This paper proposes a novel approach that combines mathematical/clustering segmentation models with insights from U-Net, leveraging the strengths of both methods. This hybrid model improves accuracy, reduces the need for large-scale training data, and achieves significant performance gains on high-resolution fundus images. The proposed model achieves a Dice coefficient of 89.7% and an IoU of 80.01% on 1024*1024 fundus images, outperforming the Attention U-Net model, which achieved 51.3% and 34.2%, respectively. It also demonstrated better generalizability on external datasets. This work forms a part of a broader effort to develop a decision support system for choroidal nevus diagnosis, with potential applications in automated lesion annotation to enhance the speed and accuracy of diagnosis and monitoring.

Figures & tables

Ref#	Method	Segmented Part	Dataset
( Eshragh et al., 2024 )	Ensemble UNet	CN	Private
( Biglarbeiki et al., 2024a )	Application (YOLO)	CN	Private
( Jiang et al., 2020 )	Multi-Path Recurrent UNet	Vessels & Optic	DRIVE ( Niemeijer et al., 2009 )
( Zhao et al., 2021 )	UNet & Transfer Learning	Disk & Cup	Drishti-GS ( Sivaswamy et al., 2014 )

Table 1 : Studies on Fundus image segmentation

Figure 1 : Proposed architecture. Blue sections represent this study’s contributions, orange sections denote ML/DL-based tool, and green sections indicate traditional segmentation model.

Figure 2 : Eye fundus image, segmented with proposed method

Component	Setting	How chosen
Guide network input	$128\times 128$ RGB; scaled to $[0,1]$	Design (Sec. 7.5)
Guide output threshold	0.5 on sigmoid map	Fixed
Data split	90/10 train/test; 80/20 train/validation; seed 101112	Fixed
$\mathrm{Coef}$ (Eq. 13)	5	Largest training lesion
Calibration factor $c$ (Eq. 14)	0.85	Validation grid search
Default $K$ for empty mask	700	Smallest training lesion

Table 2 : Constants and settings of the Hybrid pipeline used for all reported results.

Figure 3 : Effect of image size on the standard UNet: (a) IoU and Dice decrease, while (b) energy (bars) and training time (line) rise sharply.

Model	IOU	Dice	Hausdorff (px)	Specificity	Volumetric Similarity
Attention UNet	47.05%	60.57%	77.91	98.77%	72.37%
Swin UNet (Transformer-based)	62.17%	72.97%	104.30	99.42%	88.90%
nnU-Net	73.04%	83.23%	49.22	98.89%	90.03%
Hybrid Model	87.89%	90.93%	21.6	99.92%	95.32%

Table 3 : Segmentation quality comparison of UNet variants, nnU-Net, and the Hybrid Method on 1024×1024 fundus images.

Model	Train Time (min)	Energy (Wh)
Attention UNet	134.2	671.0
Swin UNet (Transformer-based)	160.7	803.5
nnU-Net	102.0	510.0
Hybrid Model (one training run of the 128 $\times$ 128 guide network)	3.16	15.8

Table 4 : Computational cost comparison for training segmentation models.

Figure 4 : Per-image Dice coefficients for the four models on the internal test set ( $n=25$ ). Each grey line connects the scores obtained for a single test image across the four models; solid bars indicate the medians, while dashed bars indicate the means. The Hybrid Model scores are concentrated near 99%, with four lower-performing cases, two of which correspond to the failures discussed in Section 7.10 .

Figure 5 : Paired per-image differences in Dice scores (Hybrid Model minus baseline, expressed in percentage points), sorted within each panel. Red bars indicate differences favouring the Hybrid Model, while the dashed line denotes the median paired difference. Reported $p$ values are from exact two-sided Wilcoxon signed-rank tests with Holm–Bonferroni correction.

Model	Mean Dice (%)	95% CI (bootstrap)	Median (%)	SD	Images with Dice $\geq$ 80%
Attention UNet	60.57	[50.63, 69.22]	65.89	24.07	6/25
Swin UNet	72.97	[63.06, 81.60]	85.14	24.60	14/25
nnU-Net	83.23	[77.73, 87.72]	86.96	13.20	20/25
Hybrid Model	90.93	[81.27, 97.68]	98.96	21.88	21/25

Table 5 : Per-image Dice coefficients on the internal test set (

n=25

, 1024

\times

1024). Confidence intervals correspond to percentile bootstrap 95% intervals for the mean, based on 10,000 resamples.

Hybrid Model vs.	Mean $\Delta$ Dice [95% CI]	Median $\Delta$ Dice	Wins / losses	Wilcoxon $W$ ; $p$ (Holm)	Permutation $p$	Effect size $r_{rb}$ ; $d_{z}$
nnU-Net	+7.70 [0.93, 14.06]	+8.66	21 / 4	51; $0.0018$	$0.0329$	0.69; 0.45
Swin UNet	+17.96 [8.89, 27.63]	+10.60	21 / 4	41; $0.0011$	$6.1\times 10^{-4}$	0.75; 0.74
Att UNet	+30.36 [20.30, 40.42]	+31.45	23 / 2	18; $4.5\times 10^{-5}$	$1.5\times 10^{-5}$	0.89; 1.16

Table 6 : Pairwise comparisons of per-image Dice scores between the Hybrid Model and each baseline on the internal test set (

n=25

\Delta

Dice represents the Hybrid Model minus the baseline, expressed in percentage points; the corresponding 95% CI is a percentile bootstrap interval based on 10,000 resamples.

W

denotes the exact two-sided Wilcoxon signed-rank statistic, with

p

values adjusted across the three comparisons using the Holm–Bonferroni procedure. The permutation

p

value is obtained from a sign-flip permutation test of the mean paired difference using 200,000 resamples.

r_{rb}

denotes the matched-pairs rank-biserial correlation, and

d_{z}

denotes Cohen’s

d

for paired differences.

Panel of Figure 6 — Figure 6 : Representative segmentation results across performance groups.

Prediction	Measurement Method	Results
Attention UNet	IOU	42.41%
Attention UNet	Dice	54.07%
Swin UNet	IOU	47.27%
Swin UNet	Dice	59.28%
nnU-Net	IOU	59.49%
nnU-Net	Dice	69.31%

Table 7 : Segmentation accuracy of the four models on the external Wills Eye Hospital dataset (

n=90

, 1024

\times

1024; IoU and Dice are means of per-image scores).

Model	Mean Dice (%)	95% CI (bootstrap)	Median (%)	SD	Images with Dice $\geq$ 80%
Attention UNet	54.07	[48.19, 59.88]	57.87	28.84	22/90
Swin UNet	59.28	[53.56, 64.93]	63.19	27.54	26/90
nnU-Net	69.31	[63.14, 75.06]	81.46	29.07	46/90
Hybrid Model	76.79	[69.94, 82.99]	91.95	31.63	64/90

Table 8 : Per-image Dice coefficient on the external Wills Eye Hospital dataset (

n=90

, 1024

\times

1024). Confidence intervals are percentile bootstrap intervals for the mean (10,000 resamples).

Hybrid Model vs.	Mean $\Delta$ Dice [95% CI]	Median $\Delta$ Dice	Wins / losses / ties	Wilcoxon $W$ ; $p$ (Holm)	Permutation $p$	Effect size $r_{rb}$ ; $d_{z}$
nnU-Net	+7.47 [2.81, 12.16]	+3.26	72 / 15 / 3	725; $4.8\times 10^{-7}$	$0.0021$	0.62; 0.32
Swin UNet	+17.51 [11.92, 22.95]	+14.83	73 / 13 / 4	508; $8.9\times 10^{-9}$	$5.0\times 10^{-6}$	0.73; 0.66
Att UNet	+22.72 [16.35, 29.16]	+18.67	75 / 12 / 3	467; $2.7\times 10^{-9}$	$5.0\times 10^{-6}$	0.76; 0.73

Table 9 : Pairwise comparisons of per-image Dice scores between the Hybrid Model and each baseline on the external Wills Eye Hospital dataset (

n=90

\Delta

Dice represents the Hybrid Model minus the baseline, expressed in percentage points, with the corresponding percentile bootstrap 95% CI based on 10,000 resamples. Ties correspond to images for which both models returned an empty mask and are excluded prior to ranking.

W

denotes the two-sided Wilcoxon signed-rank statistic, computed using the normal approximation (

n>50

), with

p

values adjusted across the three comparisons using the Holm–Bonferroni procedure. The permutation

p

value is obtained from a sign-flip permutation test of the mean paired difference using 200,000 resamples.

r_{rb}

denotes the matched-pairs rank-biserial correlation, and

d_{z}

denotes Cohen’s

d

for paired differences.

Figure 7 : Per-image Dice coefficients for the four models on the external Wills Eye Hospital dataset ( $n=90$ ). Each grey line connects the scores obtained for a single image across the four models; solid bars indicate the medians, while dashed bars indicate the means. The Hybrid Model’s scores are concentrated above 90%, with a failure tail comprising 15 images with Dice $<0.5$ , including 7 empty predictions.

Figure 8 : Paired per-image differences in Dice scores on the external dataset (Hybrid Model minus baseline, expressed in percentage points), sorted within each panel. Red bars indicate differences favouring the Hybrid Model, while the dashed line denotes the median paired difference. Reported $p$ values are from two-sided Wilcoxon signed-rank tests with Holm–Bonferroni correction.

Covariate	Internal test ( $n=25$ )	External ( $n=90$ )	Mann–Whitney $p$	SMD
Brightness ( $L^{*}$ mean)	31.43 $\pm$ 4.96	32.09 $\pm$ 5.25	$0.732$	$+0.13$
Contrast ( $L^{*}$ SD)	8.36 $\pm$ 1.35	8.15 $\pm$ 2.05	$0.279$	$-0.12$
Colour cast ( $a^{*}$ mean)	26.69 $\pm$ 7.13	27.03 $\pm$ 6.80	$0.662$	$+0.05$
Colour cast ( $b^{*}$ mean)	27.28 $\pm$ 6.47	27.92 $\pm$ 5.91	$0.697$	$+0.10$
Sharpness (Laplacian variance)	1.18 $\pm$ 0.48	1.27 $\pm$ 0.48	$0.358$	$+0.18$
Lesion area (% of FOV)	1.64 $\pm$ 2.03	4.97 $\pm$ 8.52	$0.0016$	$+0.54$

Table 10 : Image-level covariates for the internal test set and the external Wills Eye Hospital dataset (mean

\pm

SD), computed within the fundus field of view (FOV). SMD denotes the standardized mean difference (external minus internal, expressed in pooled-SD units), and

p

values are from two-sided Mann–Whitney

U

tests. Lesion size was the only covariate that differed between the two datasets.

Covariate	Failures (Dice $<$ 0.5, $n=15$ )	Non-failures ( $n=75$ )	Mann–Whitney $p$	Spearman $\rho$ with Dice
Brightness ( $L^{*}$ mean)	31.47	31.86	$0.770$	$+0.03$ ( $0.78$ )
Contrast ( $L^{*}$ SD)	7.72	7.76	$0.618$	$-0.07$ ( $0.49$ )
Colour cast ( $a^{*}$ mean)	26.28	27.97	$0.509$	$+0.20$ ( $0.054$ )
Colour cast ( $b^{*}$ mean)	26.69	27.66	$0.665$	$+0.12$ ( $0.28$ )
Sharpness (Laplacian variance)	1.13	1.18	$0.914$	$+0.07$ ( $0.54$ )
Lesion area (% of FOV)	8.36	2.00	$0.0012$	$+0.04$ ( $0.72$ )

Table 11 : Failure-mode analysis on the external dataset. The median value of each covariate is reported for images on which the Hybrid Model failed (per-image Dice

<0.5

) and for the remaining successfully segmented images, together with the corresponding two-sided Mann–Whitney

U

test

p

values. Spearman rank correlations, with associated

p

values, quantify the relationship between each covariate and the Hybrid Model’s per-image Dice across all 90 images. Lesion size was the only covariate that significantly distinguished failures from successful cases.

Variant	Configuration	Mean Dice (%) [95% CI]	Median (%)	Full model better on	Wilcoxon $p$ (Holm)
A	Full model: automated $K$ + probability selection	90.93 [81.27, 97.68]	98.96	–	–
B	Fixed $K=100$ + probability selection	69.12 [57.84, 79.86]	85.94	21 / 25	$4.0\times 10^{-4}$
B	Fixed $K=500$ + probability selection	45.70 [34.99, 56.80]	39.42	23 / 25	$6.9\times 10^{-5}$
B	Fixed $K=1000$ + probability selection	27.66 [20.25, 35.68]	22.21	24 / 25	$5.5\times 10^{-5}$
B ∗	Oracle: best fixed $K$ per image	80.37	90.34	–	–
C	Automated $K$ + centroid selection	90.93 [81.27, 97.68]	98.96	0 / 25 (identical)	–

Table 12 : Ablation analysis of the Hybrid Model on the 25 internal test images (1024

\times

1024). Confidence intervals are percentile bootstrap intervals for the mean based on 10,000 resamples. Reported

p

values are from exact two-sided Wilcoxon signed-rank tests comparing each variant with the full model, with Holm–Bonferroni correction across the four comparisons. Variant C produced the same segmentation mask as the full model for every image; therefore, no statistical test was applicable. B ∗ represents an oracle upper bound over the fixed values

K\in{100,500,1000}

and is not a directly executable method.

Figure 9 : Comparison of automated and fixed $K$ values on the 24 test images with a non-empty guide mask. (a) Per-image Dice scores for the full model and the fixed- $K$ variants plotted against the automated $K$ value for each image (Eq. 14) on a logarithmic axis; dotted lines indicate the three fixed values. (b) Dice scores obtained with a fixed $K$ plotted against the mismatch between that value and the corresponding image-specific automated $K$ , pooled across the three fixed settings (72 image– $K$ pairs).

Model	Stage	1024 $\times$ 1024 (s per image)	3900 $\times$ 3900 (s per image)
Hybrid Model	Guide network: read + resize to 128 $\times$ 128	0.07 $\pm$ 0.01	0.07 $\pm$ 0.01
	Guide network: forward pass (CPU)	0.49 $\pm$ 0.03	0.49 $\pm$ 0.03
	Resize guide mask, compute $K$ (Eqs. 11–14)	$<$ 0.01	0.19 $\pm$ 0.01
	SLIC (Eq. 15)	1.00 $\pm$ 0.06	14.16 $\pm$ 1.12
	Superpixel selection (Eqs. 17–18)	0.01	0.14 $\pm$ 0.01
	Morphological closing	$<$ 0.01	0.02

Table 13 : Per-image CPU inference time (mean

\pm

SD over the 25 internal test images) on a laptop with an AMD Ryzen 7 5700U (8 cores/16 threads, 15 GB RAM) and no GPU. Guide network: TensorFlow 2.21; superpixel stage: scikit-image SLIC; nnU-Net: nnU-Net v2 with PyTorch 2.14 (CPU). The 3900

\times

3900 measurements use the test images upsampled from 1024

\times

1024. The guide network used for timing is architecturally identical to the one used for all accuracy results.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Test image	Attention UNet	Swin UNet	nnU-Net	Hybrid Model
0	74.81	55.31	85.86	99.67
1	73.12	84.08	82.56	79.36
2	58.23	92.19	91.62	98.16
3	19.18	46.58	79.75	98.96
4	46.89	92.38	79.19	78.39
5	87.81	74.51	80.05	49.80

Appendix

Table 14 : Per-image Dice coefficient (%) of all four models on the 25 internal test images (1024

\times

1024). Test-image indices follow the fixed data split.

Img	Att	Swin	nnU	Hyb	Img	Att	Swin	nnU	Hyb	Img	Att	Swin	nnU	Hyb
0	49.1	20.2	51.6	0.0	30	54.3	49.0	48.5	50.8	60	62.0	74.5	93.9	97.6
1	63.4	96.5	75.2	98.7	31	95.5	88.4	94.9	94.9	61	5.1	51.2	55.2	87.5
2	5.4	1.2	1.2	21.2	32	61.3	49.7	75.6	88.1	62	52.3	60.2	80.5	96.8
3	62.7	62.1	95.6	97.9	33	89.1	94.7	95.3	97.8	63	42.7	77.1	84.0	73.6
4	76.6	94.1	86.9	84.5	34	93.4	95.3	96.8	97.1	64	82.5	61.7	84.5	90.6
5	45.2	59.4	91.3	97.0	35	68.8	91.6	94.5	97.0	65	12.5	35.3	63.5	91.0

Appendix

Table 15 : Per-image Dice coefficient (%) of all four models on the 90 external Wills Eye Hospital images (Att = Attention UNet, Swin = Swin UNet, nnU = nnU-Net, Hyb = Hybrid Model). Means: 54.07, 59.28, 69.31 and 76.79.

Explore similar work

May 12, 2024eess.IV

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images

Early detection of vision-threatening conditions such as diabetic retinopathy, glaucoma, and age-related macular degeneration depends on retinal fundus image analysis, but manual assessment is slow and expert-dependent. Automated convolutional neural networks classify fundus images accurately yet act as black boxes, and existing retinal vessel segmentation methods lose discriminative power under pathology and seldom exploit attention or transformer backbones. Using the FIVES and DRIVE fundus datasets, we develop a two-pipeline framework that pairs four-class disease classification with attention- and transformer-based vessel segmentation, organised in three stages: (1) FIVES images are augmented by rotation and horizontal and vertical flips and used to fine-tune eight ImageNet-pretrained CNNs: ResNet101, DenseNet169, Xception, InceptionV3, DenseNet121, InceptionResNetV2, ResNet50, and EfficientNetB0. (2) Five gradient-based explanation methods, Grad-CAM, Grad-CAM++, Score-CAM, Faster Score-CAM, and Layer-CAM, are computed on the final convolutional block of each classifier and compared qualitatively across architectures. (3) Ten U-Net variants are benchmarked for vessel segmentation: TransUNet (hybrid CNN--Transformer encoder) and Attention U-Net (gated skip connections), evaluated with ResNet50V2, ResNet101V2, and ResNet152V2 backbones, along with additional Attention U-Net configurations using DenseNet backbones, and the fully transformer-based Swin-UNet. ResNet101 gives the highest classification accuracy: 94.17% (F1 0.942)

>

88.33% for EfficientNetB0. For segmentation, the architecture ranking is consistent on both datasets: Attention U-Net

>

TransUNet

>

Swin-UNet. The strongest configuration is Attention U-Net with a ResNet101V2 backbone: FIVES IoU 0.722, Dice 0.838; DRIVE IoU 0.648, Dice 0.787, lifting DRIVE IoU 60.80

\rightarrow

64.83 over a prior custom U-Net.

Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath +2

aDepartment of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh

Vessel Segmentation Fundus Images Retinal Imaging Semantic Segmentation Convolutional

Ask PDF

Jun 11, 2026cs.CV

HSQ-VLM: A Novel Spatially-Constrained Quadrant Segmentation VLM Model for Explainability in Diabetic Retinopathy

Diabetic Retinopathy (DR) is an aggressive retinal disease and a leading cause of global blindness, yet its clinical management is currently hindered by the black-box nature of diagnostic AI. While deep learning models achieve high classification accuracy, there is a critical lack of explainability methods capable of detailing the exact anatomical landmarks and lesion distributions that lead to a clinical decision for DR. Therefore, we propose HSQ-VLM, a novel quadrant segmentation pipeline on fundus images that utilizes a Landmark-Anchored Cartesian Cross-Attention mechanism to unify visual feature extraction with structured clinical reasoning. Unlike traditional methods that rely on arbitrary image partitioning, our pipeline implements 4-quadrant Topological Latent Partitioning (TLP) to dynamically align retinal features with a fovea-centered coordinate system. This allows the Vision-Language Model to generate natural language reports that quantify pathology with anatomical precision. On a dataset of 3,500 high-resolution fundus images, this innovative methodology achieved a lesion detection sensitivity of 99.6% for hemorrhages and 96.4% for microaneurysms, while demonstrating a significant reduction in boundary-ambiguity errors compared to standard segmentation baselines.

Shivum Telang

Pittsburgh, Pennsylvania

Medical Vision-Language Models Ava-Vlm Recent Vision-Language Models Explainability Cross-Attention

Ask PDF

May 19, 2026cs.CV

ELEMENT: Multi-Modal Retinal Vessel Segmentation Based on a Coupled Region Growing and Machine Learning Approach

Vascular structures in the retina contain important information for the detection and analysis of ocular diseases, including age-related macular degeneration, diabetic retinopathy and glaucoma. Commonly used modalities in diagnosis of these diseases are fundus photography, scanning laser ophthalmoscope (SLO) and fluorescein angiography (FA). Typically, retinal vessel segmentation is carried out either manually or interactively, which makes it time consuming and prone to human errors. In this research, we propose a new multi-modal framework for vessel segmentation called ELEMENT (vEsseL sEgmentation using Machine lEarning and coNnecTivity). This framework consists of feature extraction and pixel-based classification using region growing and machine learning. The proposed features capture complementary evidence based on grey level and vessel connectivity properties. The latter information is seamlessly propagated through the pixels at the classification phase. ELEMENT reduces inconsistencies and speeds up the segmentation throughput. We analyze and compare the performance of the proposed approach against state-of-the-art vessel segmentation algorithms in three major groups of experiments, for each of the ocular modalities. Our method produced higher overall performance, with an overall accuracy of 97.40%, compared to 25 of the 26 state-of-the-art approaches, including six works based on deep learning, evaluated on the widely known DRIVE fundus image dataset. In the case of the STARE, CHASE-DB, VAMPIRE FA, IOSTAR SLO and RC-SLO datasets, the proposed framework outperformed all of the state-of-the-art methods with accuracies of 98.27%, 97.78%, 98.34%, 98.04% and 98.35%, respectively.

Erick O. Rodrigues, Aura Conci, Panos Liatsis

Academic Department of Informatics, Universidade Tecnologica Federal do Parana (UTFPR), Pato Branco 80230-901, Brazil · Department of Computer Science, Universidade Federal Fluminense (UFF), Niteroi 24220-900, Brazil · Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi 127788, United Arab Emirates

Vessel Segmentation Fundus Images Multimodal Model

Ask PDF