Learning to Fuse and Reconstruct Multi-View Graphs for Diabetic Retinopathy Grading
Authors: Haoran Li, Yuxin Lin, Huan Wang, Xiaoling Luo, Qi Zhu, Jiahua Shi, Huaming Chen, Bo Du, Johan Barthelemy, Zongyan Xue, Jun Shen, Yong Xu
Organizations: Department of Data Science and Artificial Intelligence, Monash University, Melbourne, Australia · ARC Centre of Excellence for the Weather of the 21st Century · Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology (Shenzhen), Shenzhen, China · School of Computing and Information Technology, University of Wollongong, Wollongong, Australia · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · Centre for Nutrition and Food Sciences, The University of Queensland, Brisbane, Australia · School of Electrical and Computer Engineering, University of Sydney, Sydney, Australia · Department of Management, Griffith University, Brisbane, Australia · NVIDIA, Santa Clara, USA · The University of New South Wales, Sydney, Australia
Diabetic retinopathy (DR) is one of the leading causes of vision loss worldwide, making early and accurate DR grading critical for timely intervention. Recent clinical practices leverage multi-view fundus images for DR detection with a wide coverage of the field of view (FOV), motivating deep learning methods to explore the potential of multi-view learning for DR grading. However, existing methods often overlook the inter-view correlations when fusing multi-view fundus images, failing to fully exploit the inherent consistency across views originating from the same patient. In this work, we present MVGFDR, an end-to-end Multi-View Graph Fusion framework for DR grading. Different from existing methods that directly fuse visual features from multiple views, MVGFDR is equipped with a novel Multi-View Graph Fusion (MVGF) module to explicitly disentangle the shared and view-specific visual features. Specifically, MVGF comprises three key components: (1) Multi-view Graph Initialization, which constructs visual graphs via residual-guided connections and employs Discrete Cosine Transform (DCT) coefficients as frequency-domain anchors; (2) Multi-view Graph Fusion, which integrates selective nodes across multi-view graphs based on frequency-domain relevance to capture complementary view-specific information; and (3) Masked Cross-view Reconstruction, which leverages masked reconstruction of shared information across views to facilitate view-invariant representation learning. Extensive experimental results on MFIDDR, by far the largest multi-view fundus image dataset, demonstrate the superiority of our proposed approach over existing state-of-the-art approaches in diabetic retinopathy grading.
Figures & tables
Fig. 1: Illustration of our motivation. (a) Semantic information captured by different DCT frequency components in fundus images. (b) Comparison between (i) existing multi-view methods and (ii) our approach. Unlike prior methods, our approach disentangles the feature embeddings into shared and unique components, applying feature fusion and masked reconstruction learning separately to each. (c) Comparison of state-of-the-art multi-view DR methods and our proposed MVGFDR on seven evaluation metrics.
Fig. 2: Overall framework of MVGFDR. (a) Multi-view Graph Initialization (MVGI) for each view based on DCT frequency. (b) Graph nodes selection from each view according to their corresponding high-, mid- and low-frequency DCT components. (c) Multi-view Graph Fusion (MGF) of high-frequency information using a GCN. (d) Masked Cross-View Reconstruction (MCVR) on low- and mid-frequency information with the random node masking rate η .
Fig. 3: Visualization of view-specific representations extracted by MVGI from different views. A representative Grade-4 sample is shown for illustration. The visualization focuses on the high-frequency (DCT) representations, which are associated with lesion-related patterns.
Fig. 4: Proposed Masked Cross-View Reconstructor (MCVR) (a) We set a simple cross-graph reconstructor as the comparison baseline. (b) The proposed MCVR adopts a decoder-only transformer to perform masked view reconstruction guided by view-positional embeddings (VP, ) and frequency-positional embeddings (FP, ).
Grade 0
Grade 1
Grade 2
Grade 3
Grade 4
Method
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Inception_ResNet_k_2_BSV
74.83
97.15
84.54
46.28
12.53
19.72
48.98
39.34
43.64
63.38
60.81
62.07
66.67
15.38
25.00
Mobile_Net_k2_BSV
76.60
96.70
85.49
48.55
14.99
22.91
56.85
45.36
50.46
64.16
75.00
69.16
60.00
15.38
24.49
ResNet50_BSV
79.38
94.68
86.36
48.34
22.82
31.00
51.48
47.54
49.43
62.94
72.30
67.30
70.00
17.95
28.57
ResNet50_32x4d_BSV
79.51
95.13
86.62
46.64
27.96
34.97
55.04
38.80
45.51
71.23
70.27
70.75
75.00
23.08
35.29
ConvNeXt-B_BSV
82.62
95.13
88.43
57.14
34.00
42.64
55.83
49.73
52.60
64.57
76.35
69.97
81.82
23.08
36.00
TABLE I: Comparison of precision (Prec.), sensitivity (Sens.), and F 1 scores between our method and single-view approaches in different DR grades. Best results are highlighted in bold.
Method
Acc.
Spe.
Kappa
F 1
AUC
Inception_ResNet_k_2_BSV [AAAI17] [ 59 ]
70.66
67.13
38.69
65.43
85.55
Mobile_Net_k2_BSV [CVPR18] [ 60 ]
72.38
68.73
43.61
67.28
86.88
ResNet50_BSV [CVPR16] [ 61 ]
73.22
73.20
45.29
69.35
86.76
ResNext50_32x4d_BSV [CVPR17] [ 62 ]
73.36
73.04
47.15
70.37
88.22
ConvNeXt-B_BSV [CVPR22] [ 63 ]
75.96
77.81
53.72
73.65
89.77
Swin-B_BSV [CVPR21] [ 64 ]
75.08
75.53
51.32
72.42
88.83
TABLE II: Comparison results against single-view approaches with our proposed multi-view approach. Best results are in bold font.
Grade 0
Grade 1
Grade 2
Grade 3
Grade 4
Method
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
Pre.
Sens.
F 1
RETFound
80.11
96.33
87.47
50.20
27.96
35.92
54.41
40.44
46.39
65.79
67.57
66.67
90.00
23.08
36.73
MVCINN
86.71
96.33
91.26
68.25
48.10
56.43
57.44
61.20
59.26
70.00
66.22
68.06
68.42
33.33
44.83
ETMC
86.79
97.53
91.82
73.26
56.38
63.72
66.41
47.54
55.41
64.41
77.03
70.15
0.12
90.15
0.87
LFMVDR (with lesion)
89.69
95.20
92.36
69.53
63.31
66.28
62.05
56.28
59.03
69.48
72.30
70.86
54.55
61.54
57.83
MVTSM (with lesion)
89.23
95.65
92.33
73.86
54.36
62.63
61.08
67.76
64.25
72.67
73.65
73.15
64.10
64.10
64.10
TABLE III: Comparison of precision (Prec.), sensitivity (Sens.), and F 1 scores between our method and multi-view approaches in different DR grades. The best results are highlighted in bold.
Method (End-to-end)
Acc.
Spe.
Kappa
F 1
AUC
RETFound [Nature23] [ 66 ]
74.06
73.83
48.44
70.91
89.04
MVCINN [AAAI23] [ 12 ]
80.10
83.32
62.45
78.86
91.07
ETMC [TPAMI22] [ 14 ]
81.54
83.44
64.76
79.74
93.53
LFMVDR (w/o lesion) [JBHI24] [ 32 ]
80.38
85.90
64.04
79.39
-
MVTSM (w/o lesion) [PR2025] [ 13 ]
71.73
82.76
-
70.93
-
SMVDR (w/o prompts) [AAAI25] [ 33 ]
81.64
89.92
66.94
81.38
-
TABLE IV: Comparison of experimental results between multi-view approaches and our proposed method. For fair comparison, we divide existing methods into two types: End-to-end and Expert-guided. The best results are in bold font.
DRTiD
Acc.
Spe.
Kappa
F 1
MVCINN [AAAI23] [ 12 ]
85.87
85.08
74.14
84.28
PLReMix [WACV25] [ 71 ]
87.94
88.93
76.70
87.64
Qmix [TMI25] [ 72 ]
89.29
91.15
80.80
88.31
MVGFDR (Ours)
92.68
92.74
83.99
90.71
CheXpert
Acc.
Spe.
Kappa
F 1
MVCINN [AAAI23] [ 12 ]
73.04
79.61
56.90
71.73
TABLE V: Comparison of experimental results on DRTiD and CheXpert datasets.
Fig. 5: Examples of V1-V4 views of different datasets. (a) MFIDDR. (b) The generated MVG-DDR.
MVG-DDR (Single-view)
Acc.
Pre.
Spe.
Kappa
F 1
ResNet-50_BSV [CVPR16] [ 61 ]
76.59
74.63
78.80
65.00
74.12
DenseNet-121_BSV [CVPR17] [ 78 ]
73.57
71.85
72.58
60.53
69.54
SE-BN-Inception_BSV [CVPR18] [ 79 ]
76.39
74.83
80.94
64.72
74.62
Swin-B_BSV [CVPR21] [ 64 ]
78.75
77.09
79.17
68.88
76.47
PVT-M_BSV [ICCV21] [ 49 ]
79.08
77.72
78.52
69.08
76.62
PVT-L_BSV [ICCV21] [ 49 ]
80.85
79.74
81.96
73.43
79.25
TABLE VI: Comparison of experimental results on the generated MVG-DDR dataset. We re-conduct experiments on single-veiw methods and multi-view methods. The best results are in bold font.
Fig. 6: (a) t-SNE [ 80 ] visualization of feature representations learned by the model. (b-c) Evaluation of the hyper-parameter settings. (b) Joint accuracy under different masking rates η and reconstruction loss control rate α . (c) Joint F 1 score under varying numbers of clustering center M and frequency range thresholds kθ .
Candidate
MFIDDR
PVT+GF
Node Select.
View Recon.
Acc.
Pre.
Spe.
Kappa
F 1
No.1
✔
79.96
78.63
80.17
61.28
77.91
No.2
✔
✔
81.10
79.87
81.93
63.52
78.76
No.3
✔
✔
82.52
80.74
85.11
67.39
80.86
Ours
✔
✔
✔
85.87
85.13
93.00
73.14
84.49
TABLE VII: Ablation study of the key components on MFIDDR.
Department of Computer Science and Engineering, Dhaka International University, Dhaka, Bangladesh · KUET, Khulna, Bangladesh · Bangladesh Bank, Dhaka, Bangladesh