Learning Pathology-Sensitive Representations for Unsupervised Retinal OCT Anomaly Detection
Authors: Tania Haghighi, Sina Gholami, Behafarin Emam, Jennifer I Lim, Theodore Leng, Atalie C Thompson, Hamed Tabkhi, Minhaj Nur Alam
Organizations: University of North Carolina at Charlotte, Charlotte, North Carolina, USA · University of Illinois Chicago, Chicago, IL, USA · Byers Eye Institute at Stanford, Stanford University School of Medicine, Stanford, CA, USA · Surgical Ophthalmology, Atrium Health Wake Forest Baptist, Winston–Salem, NC, USA
Deep learning approaches for retinal optical coherence tomography (OCT) predominantly rely on predefined disease categories, limiting their ability to detect rare or previously unseen abnormalities. While visual foundation models provide transferable representations, their effectiveness in identifying pathological deviations from normal retinal anatomy remains insufficiently understood. In this work, we introduce an unsupervised framework for retinal OCT anomaly detection and localization that adapts pretrained visual representations to normal retinal anatomy through anatomy-aware reconstruction and contrastive triplet learning guided by synthetic structural perturbations, without requiring disease-specific annotations. We systematically investigate the influence of pretraining strategies, anatomy-aware representation adaptation, and anomaly scoring mechanisms on the detection and localization of unseen retinal abnormalities. Our findings demonstrate that adapting pretrained representations to normal retinal structure improves anomaly sensitivity, while pixel-space reconstruction errors and feature-space deviations provide complementary signals for identifying pathological changes. These findings underscore the importance of anatomy-aware representation learning and offer insights into developing generalizable retinal anomaly detection frameworks beyond predefined disease categories.
Figures & tables
Figure 1: Overview of the proposed representation-learning framework. A shared ViT encoder processes a clean anchor OCT B-scan, a different clean normal positive, and a perturbed version of the anchor as the negative. CLS tokens are mapped to L2-normalized embeddings and optimized using triplet loss, while patch tokens are reconstructed through a linear patch decoder using the reconstruction objective.
Method
AUROC
F1 (%)
Accuracy (%)
ConvNeXt
0.716
69.7
62.8
VAE
0.685
69.6
60.0
VQ-VAE
0.737
71.4
65.5
VQGAN
0.747
71.3
67.6
f-AnoGAN
0.763
71.1
68.4
Anatomy-aware VQGAN
0.799
75.2
71.7
Table 1: Image-level anomaly detection on the Kermany test set. F1 score and accuracy are computed at the operating threshold. Best values are shown in bold.
Method
Dice
mIoU
ConvNeXt
0.181
0.100
VAE
0.123
0.066
VQ-VAE
0.185
0.102
VQGAN
0.145
0.078
Anatomy-aware VQGAN
0.200
0.111
DINOv3
0.140
0.075
Table 2: Unsupervised anomaly localization on RETOUCH. All models were trained on Kermany only and applied to RETOUCH without further training. mIoU is foreground-only (background excluded from the class average). Best values are shown in bold.
Encoder
Configuration
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
DINOv3
Frozen
0.440
13.0
52.9
86.0
7.0
0.012
0.006
+ Reconstruction
0.510
21.0
54.8
83.0
12.0
0.023
0.012
+ Triplet
0.834
71.6
76.9
91.7
58.7
0.074
0.038
+ ROI
0.719
63.2
70.5
83.0
51.0
0.140
0.075
RETFound
Frozen
0.598
13.4
52.6
71.2
7.4
0.030
0.015
+ Reconstruction
0.640
26.4
55.3
75.0
16.0
0.051
0.026
Table 3: Effect of the training objective. Frozen: the pretrained encoder is fixed and only the linear reconstruction head is trained. + Reconstruction: the encoder and head are trained jointly using reconstruction loss only. + Triplet: the encoder and head are trained jointly with the reconstruction and triplet objectives. + ROI: retinal region-of-interest (ROI) weighting is added to the pixel loss (complete model). Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average). Reconstruction-only values are illustrative placeholders, not measured results.
Training data
Encoder
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
0.5%
DINOv3
0.605
47.8
60.8
70.6
36.1
0.090
0.047
RETFound
0.766
72.7
77.1
89.2
62.3
0.141
0.076
ImageNet ViT
0.709
63.0
70.9
85.3
49.9
0.109
0.058
1%
DINOv3
0.588
45.8
60.5
72.0
33.5
0.081
0.042
RETFound
0.740
67.4
74.2
90.1
53.8
0.145
0.078
ImageNet ViT
0.753
69.4
75.2
89.3
56.8
0.125
0.067
Table 4: Effect of the amount of normal training data on the complete models. Percentages refer to the fraction of the normal training set used for training. Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average). Bold values indicate the best performance for each metric at a given training-data percentage.
Figure 2: Effect of the amount of normal training data on the complete models. a , AUROC and b , F1 score for image-level detection on Kermany. c , Dice for anomaly localization on RETOUCH. The x axis shows the percentage of the normal training set used for training, on a logarithmic scale.
Scoring
Encoder
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
Pixel level
DINOv3
0.719
63.2
70.5
83.0
51.0
0.140
0.075
RETFound
0.859
82.7
84.2
90.4
76.3
0.220
0.124
ImageNet ViT
0.921
85.9
87.3
89.1
84.7
0.242
0.138
Feature level
DINOv3
0.871
54.0
68.3
96.7
37.4
0.273
0.158
RETFound
0.91
74.4
79.3
97.8
60.5
0.397
0.248
ImageNet ViT
0.910
73.5
79.0
98.8
58.8
0.318
0.189
Table 5: Comparison of reconstruction-based (pixel-level) and feature-level anomaly scoring for the complete models. Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average).
Figure 3: Perturbation strategy used to build training triplets from healthy (normal) OCT B-scans. A clean normal image x is used as the anchor. Its retinal band is detected automatically and used to place and scale the synthetic lesions.
Perturbation Strategy
Image-Level Anomaly Detection
Pixel-Level Anomaly Segmentation
Accuracy
F1
AUROC
AUPRC
Dice
mIoU
Thinning
58.00
33.75
50.91
60.31
6.80
3.52
Thickening
71.27
63.84
70.01
77.88
5.97
3.08
Subretinal fluid
64.59
52.62
62.68
70.23
6.85
3.55
PED
65.28
50.60
63.95
72.26
6.52
3.37
Foci
73.60
66.96
71.63
79.48
7.94
4.13
Table 6: Comparison of different synthetic perturbation strategies for OCT anomaly detection and segmentation. The best result for each metric is highlighted in bold. All values are reported as percentages. mIoU is foreground-only (background excluded from the class average).
Parameter
Setting
Model architecture and preprocessing
Pretrained encoders
DINOv3, RETFound, and supervised ImageNet ViT-L/16
Input resolution
256×256 pixels
Image preprocessing
Bicubic resizing; ImageNet normalization
Patch size and grid
16×16 pixels; 16×16 patches (256 tokens)
Patch embedding dimension
1,024
Table S1: Model architecture, training hyperparameters, and optimization settings used for the OCT anomaly detection framework.
Reliable automated analysis of Optical Coherence Tomography (OCT) imaging is crucial for diagnosing retinal disorders but faces a critical barrier: the need for expensive, labor-intensive expert annotations. Supervised deep learning models struggle to generalize across diverse pathologies, imaging devices, and patient populations due to their restricted vocabulary of annotated abnormalities. We propose an unsupervised anomaly detection framework that learns the normative distribution of healthy retinal anatomy without lesion annotations, directly addressing annotation efficiency challenges in clinical deployment. Our approach leverages a discrete latent model trained on normal B-scans to capture OCT-specific structural patterns. To enhance clinical robustness, we incorporate retinal layer-aware supervision and structured triplet learning to separate healthy from pathological representations, improving model reliability across varied imaging conditions. During inference, anomalies are detected and localized via reconstruction discrepancies, enabling both image and pixel-level identification without requiring disease-specific labels. On the Kermany dataset (AUROC: 0.799), our method substantially outperforms VAE, VQVAE, VQGAN, and f-AnoGAN baselines. Critically, cross-dataset evaluation on Srinivasan achieves AUROC 0.884 with superior generalization, demonstrating robust domain adaptation. On the external RETOUCH benchmark, unsupervised anomaly segmentation achieves competitive Dice (0.200) and mIoU (0.117) scores, validating reproducibility across institutions.
Tania Haghighi, Sina Gholami, Hamed Tabkhi +1
University of North Carolina at Charlotte Charlotte, NC, USA
Optical Coherence Tomography (OCT) has become one of the most used imaging modality in ophthalmology. It provides high-resolution, non-invasive visualization of retinal microarchitecture. The automated analysis of OCT images through representation learning has emerged as a central research frontier. This has mainly been driven by the clinical need to process large acquisition volumes. The objective is to reduce the reliance on expert annotation, and improve diagnostic consistency across devices and populations. This survey provides a comprehensive and structured review of representation learning methods for retinal OCT image analysis. It covers the period from early deep learning approaches to the most recent developments in foundation models and vision-language systems. We organize the literature along a principled taxonomy of learning paradigms, encompassing supervised learning with CNN-based and transformer-based architectures, self-supervised and semi-supervised methods, generative approaches, as well as 3D volumetric modeling, multimodal representation learning, and large-scale pretrained foundation models. For each paradigm, we analyze the core methodological contributions, identify persistent limitations, and trace the connections between successive approaches. We further provide a structured overview of publicly available OCT datasets, discuss evaluation protocol considerations, and present a unified problem formulation that situates each learning paradigm within a common mathematical framework. Building on this analysis, we identify and discuss the most pressing open research directions emerging in the literature. This includes volumetric foundation model pretraining, uncertainty-aware representation learning, federated and privacy-preserving training, fairness and bias mitigation, concept-based interpretability,...
Hedi Tabia, Désiré Sidibé, Nawres Khlifa +6
IBISC Univ. Evry Université Paris-Saclay · University of Tunis El Manar · ESIEE Paris - Université Gustave Eiffel +2
Optical coherence tomography (OCT), a commonly used retinal imaging modality, plays a central role in retinal disease diagnosis by providing high-resolution visualization of retinal layers. While deep learning (DL) has achieved expert-level accuracy in OCT-based retinal disease detection, its "black box" nature poses challenges for clinical adoption, where explainability is essential for clinical trust and regulatory approval. Existing post-hoc explainable AI (XAI) methods often struggle to delineate fine-grained lesion structures, respect anatomical boundaries, or suppress noise, limiting the trustworthiness of their explanations. To bridge these gaps, we propose a Structure-Aware Interpretable Learning (SAIL) framework that integrates retinal anatomical priors at the representation level and couples them with semantic features via a fusion design. Without modifying standard post-hoc explainability methods, this representation yields sharper and more anatomically aligned attribution maps. Comprehensive experiments on diverse OCT datasets demonstrate that our structure-aware method consistently enhances interpretability, producing clinically meaningful and anatomy-aware explanations. Ablation studies further show that strong interpretability requires both structural priors and semantic features, and that properly fusing the two is critical to achieve the best explanation quality. Together, these results highlight structure-aware representations as a key step toward reliable explainability in OCT.
Tienyu Chang, Tianhao Li, Ruogu Fang +2
Dept. of BioHealth Informatics, Indiana University Indianapolis, Indiana, USA · School of Information, University of Texas at Austin Austin, Texas, USA · Dept. of Biomedical Engineering, University of Florida Gainesville, Florida, USA +1