GLEAM: A Multimodal Imaging Dataset and HAMM for Glaucoma Classification

Jiao Wang; Chi Liu; Yiying Zhang; Hongchen Luo; Zhifen Guo; Ying Hu; Ke Xu; Jing Zhou

eess.IVMar 13, 2026

GLEAM: A Multimodal Imaging Dataset and HAMM for Glaucoma Classification

Authors: Jiao Wang, Chi Liu, Yiying Zhang, Hongchen Luo, Zhifen Guo, Ying Hu, Ke Xu, Jing Zhou, +3 more

Organizations: College of the Information Science and Engineering, Northeastern University, Shenyang 110819, China · Department of Ophthalmology, Shenyang Fourth People’s Hospital, Shenyang 110000, China

Abstract

We propose glaucoma lesion evaluation and analysis with multimodal imaging (GLEAM), the first publicly available tri-modal glaucoma dataset comprising scanning laser ophthalmoscopy fundus images, circumpapillary OCT images, and visual field pattern deviation maps, annotated with four disease stages, enabling effective exploitation of multimodal complementary information and facilitating accurate diagnosis and treatment across disease stages. To effectively integrate cross-modal information, we propose hierarchical attentive masked modeling (HAMM) for multimodal glaucoma classification. Our framework employs hierarchical attentive encoders and light decoders to focus cross-modal representation learning on the encoder.

Figures & tables

Dataset name	Year	Quantity	Imaging modalities	Classification	Public
ORIGA [ 7 ]	2010	650	Fundus	NG/G	✓
Drishti-GS [ 8 ]	2014	101	Fundus	NG/G	✓
Refuge [ 9 ]	2018	1200	Fundus	NG/G	✓
HDV1 [ 10 ]	2018	1542	Fundus	NG/EaG/PG	✓
ACRIMA [ 11 ]	2019	705	Fundus	NG/G	✓
Harvard-GDP [ 12 ]	2023	1000	RNFL	NG/G	✓

TABLE I : Overview of major glaucoma datasets for classification. (NG: Non-glaucoma, EaG: Early Glaucoma, InG: Intermediate Glaucoma, AdG: Advanced Glaucoma , PG: Progressive Glaucoma, NPG: Non-progressive Glaucoma. )

Fig. 1 : The detail of ophthalmic data, including (a) SLO, OCT, and VF examinations (b) EMR documented by ophthalmologists.

Agreement Type	Expert 1	Expert 2	Expert 3
Intra-grader Agreement	98.25	97.68	97.42
Expert 1 vs. Expert 2	95.81	-	-
Expert 1 vs. Expert 3	96.37	-	-
Expert 2 vs. Expert 3	-	95.59	-

TABLE II : Inter- and Intra-grader Agreement via Quadratic Weighted Cohen’s Kappa (%)

Fig. 2 : GLEAM dataset character, including (a) distributions of GLEAM, and (b) examples of GLEAM. We add lines for splitting the optic disc/optic cup and the thickness of the RNFL to more visually show the gaps in the images of different progressions of glaucoma.

Fig. 3 : The architecture of HAMM. Stage one is the masked autoencoder pretraining task to train a better feature representation, and stage two is the classification task to train the final classification model. The MCGA module is designed to model the multimodal information between different images when pretraining and classification. For clarity, token notations (e.g $c^{f}$ , $\hat{v}^{f}$ , $v^{f}$ ) are shown for a single modality, though the same applies to all.

Method	Pre-train	Acc	F1	AUROC	QWK	Param	FLOPs
ResNet50 [ 50 ]	-	76.75 $\pm$ 1.47	66.84 $\pm$ 2.60	89.95 $\pm$ 0.27	85.88 $\pm$ 0.76	83.12	12.41
ResNet50 [ 50 ]	TL	77.67 $\pm$ 0.86	70.19 $\pm$ 0.93	92.14 $\pm$ 1.81	87.00 $\pm$ 0.55	83.12	12.41
ViT-S [ 54 ]	TL	77.75 $\pm$ 1.52	69.62 $\pm$ 3.79	91.79 $\pm$ 0.48	88.03 $\pm$ 0.64	65.21	12.75
ConvNeXt-T [ 55 ]	TL	79.00 $\pm$ 0.76**	71.58 $\pm$ 1.32	91.87 $\pm$ 0.77	87.83 $\pm$ 0.72	85.20	13.39
MHCA [ 40 ]	TL	78.16 $\pm$ 0.63	69.97 $\pm$ 3.20	92.28 $\pm$ 0.27	87.14 $\pm$ 0.75	248.29	17.63
DRIFA-Net [ 41 ]	TL	77.83 $\pm$ 0.86*	69.70 $\pm$ 1.96	92.42 $\pm$ 0.10	86.75 $\pm$ 0.57	931.31	88.48

TABLE III : Comparison of the results obtained by HAMM and several representative methods on the GLEAM dataset. Statistical significance of accuracy differences relative to the baseline method is evaluated using two-tailed paired t-tests. (*:

p<0.05

; **:

p<0.01

, where

p

denotes p-value. TL: Transfer Learning, SCL: Supervised Contrastive Learning, SSL: Self-supervised Learning.

Fig. 4 : Visualization of regions of interest identified by the model using Guided Grad-CAM

Modality	Acc	F1	AUROC	QWK	Acc-C1	Acc-C2	Acc-C3	Acc-C4
SLO	60.25**	37.25	74.72	56.79	94.83	3.00	2.00	72.00
OCT	61.75**	42.39	76.70	62.84	93.33	8.00	10.50	72.00
VF	74.25**	59.85	90.42	83.03	98.00	6.00	57.50	88.00
SLO+OCT	64.42**	46.47	67.22	79.72	94.00	11.00	13.50	80.00
SLO+VF	77.67*	68.36	91.87	86.84	96.33	26.00	61.50	89.50
OCT+VF	77.08*	67.38	92.24	86.25	96.50	22.50	61.00	89.50

TABLE IV : Experimental results of different modalities. Statistical significance of accuracy differences relative to the tri-modal method is evaluated using two-tailed paired t-tests. (Acc-Cn: accuracy for class n.)

Fig. 5 : Comprehensive evaluation of model feature representations and classification performance under three input settings: (a), (b), and (c) are t-SNE visualizations of feature embeddings corresponding to the input settings of VF, SLO+VF, and SLO+OCT+VF respectively; (e), (f), and (g) are confusion matrices corresponding to the identical set of input combinations.

Modality	Acc	F1	AUROC	QWK
Corolla	69.87	60.42	84.65	73.15
ETSCL	70.35	61.76	84.89	73.22
MultiMAE	71.62	64.15	85.76	75.25
URFound	72.48	64.89	86.83	76.28
HAMM(Ours)	74.59	66.73	88.92	77.05

TABLE V : Model performance under missing-modality scenarios.

Fig. 6 : Reliability diagram of models, including: (a) reliability diagram for different modalities VF, SLO+VF, and SLO+OCT+VF, and (b) reliability diagram for SOTA methods ConvNeXt, ETSCL, and HAMM.

MCGA	Pre-train	Acc	F1	AUROC	QWK
		77.67	70.19	92.14	87.00
✓		79.17	71.93	92.89	89.52
	✓	79.67	73.68	92.83	89.57
✓	✓	81.08	75.90	93.03	90.07

TABLE VI : The ablation study results of HAMM.

Pooling Scheme	Attention Scheme	Fusion Scheme	Acc	F1	AUROC	QWK	Param	FLOPs
-	CBAM [ 59 ]	Late	78.00	70.89	92.14	87.93	125.59	14.27
-	CBAM [ 59 ]	Hierarchical	78.62	71.46	92.51	88.42	139.53	19.83
GAP only	Gating only	Hierarchical	77.75	70.61	92.49	87.02	131.68	12.49
GAP only	Graph only	Hierarchical	78.59	71.15	92.41	87.92	166.70	12.60
GAP only	Full Attention	Hierarchical	78.75	71.82	92.66	88.76	233.55	12.66
Full Pooling	Full Attention	Late	78.50	71.25	92.36	88.30	118.80	12.46

TABLE VII : Ablation study results showing the contribution of different components in the MCGA module.

Fig. 7 : Visualization of averaged feature maps from the 3rd and 4th backbone layers with and without the MCGA module.

Fig. 8 : Impact of masking ratio on model accuracy, with the optimum at 0.7 when trained for 20 epochs.

Method	Add Info.	Add Mod.	Ensemble	QWK
SmartDSP [ 24 ]	✗	✗	✓	85.49
COROLLA [ 27 ]	✗	✗	✓	85.50
ETSCL [ 29 ]	✗	✓	✗	88.44
Mstnet (ResNet50) [ 26 ]	✓	✗	✗	86.00
GeCoM-Net [ 28 ]	✗	✗	✗	86.00
GeCoM-Net [ 28 ]	✗	✗	✓	88.10

TABLE VIII : Comparison of Quadratic Weighted Kappa Results Among HAMM and SOTA Methods on the GAMMA dataset. (Add Info.: Use additional ophthalmic data for training, Add Mod.: Use additional modality)

Explore similar work

Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis
Mar 15, 2026Zhiwei Wang, Yuxing Li, Meilu Zhu +2Medical Image Segmentation Cross-Modality Medical Image Segmentation
Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou, China
An Automated Multimodal Glaucoma Detection Framework Using ViT and a Stacking-Based Ensemble
Jul 2, 2026Ishrat Jahan, Muhammad E. H Chowdhury, Murugappan Murugappan +7Multimodal Classification Medical Image Classification
Department of Computer Science and Engineering, Shahjalal University of Science and Technology, Sylhet, Bangladesh · Department of Electrical Engineering, Qatar University, Doha, Qatar · Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5
GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification
Aug 3, 2025Cheng Huang, Zeyu Han, Weizheng Xie +4Multimodal Fusion Clinical Risk Prediction
Department of Computer Science, Southern Methodist University, Dallas, TX 75205, USA · Department of Ophthalmology, UT Southwestern Medical Center, Dallas, TX 75390, USA · High Performance Computing Program, National Aeronautics and Space Administration, Washington, DC 20546, USA

Mar 15, 2026J/K move · Enter open · S save

Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis

Zhiwei Wang, Yuxing Li, Meilu Zhu, Defeng He+1

Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou, China

Medical Image Segmentation Cross-Modality Medical Image Segmentation Multimodal Optimization

Open paper AskPDF arXiv

Accurate diagnosis of glaucoma is challenging, as early-stage changes are subtle and often lack clear structural or appearance cues. Most existing approaches rely on a single modality, such as fundus or optical coherence tomography (OCT), capturing only partial pathological information and often missing early disease progression. In this paper, we propose an iterative multimodal optimization model (IMO) for joint segmentation and grading. IMO integrates fundus and OCT features through a mid-level fusion strategy, enhanced by a cross-modal feature alignment (CMFA) module to reduce modality discrepancies. An iterative refinement decoder progressively optimizes the multimodal features through a denoising diffusion mechanism, enabling fine-grained segmentation of the optic disc and cup while supporting accurate glaucoma grading. Extensive experiments show that our method effectively integrates multimodal features, providing a comprehensive and clinically significant approach to glaucoma assessment. Source codes are available at https://github.com/warren-wzw/IMO.git.

Abstract

Explore similar work

Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis

An Automated Multimodal Glaucoma Detection Framework Using ViT and a Stacking-Based Ensemble

GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification