Learning Pathology-Sensitive Representations for Unsupervised Retinal OCT Anomaly Detection
Authors: Tania Haghighi, Sina Gholami, Behafarin Emam, Jennifer I Lim, Theodore Leng, Atalie C Thompson, Hamed Tabkhi, Minhaj Nur Alam
Organizations: University of North Carolina at Charlotte, Charlotte, North Carolina, USA · University of Illinois Chicago, Chicago, IL, USA · Byers Eye Institute at Stanford, Stanford University School of Medicine, Stanford, CA, USA · Surgical Ophthalmology, Atrium Health Wake Forest Baptist, Winston–Salem, NC, USA
Deep learning approaches for retinal optical coherence tomography (OCT) predominantly rely on predefined disease categories, limiting their ability to detect rare or previously unseen abnormalities. While visual foundation models provide transferable representations, their effectiveness in identifying pathological deviations from normal retinal anatomy remains insufficiently understood. In this work, we introduce an unsupervised framework for retinal OCT anomaly detection and localization that adapts pretrained visual representations to normal retinal anatomy through anatomy-aware reconstruction and contrastive triplet learning guided by synthetic structural perturbations, without requiring disease-specific annotations. We systematically investigate the influence of pretraining strategies, anatomy-aware representation adaptation, and anomaly scoring mechanisms on the detection and localization of unseen retinal abnormalities. Our findings demonstrate that adapting pretrained representations to normal retinal structure improves anomaly sensitivity, while pixel-space reconstruction errors and feature-space deviations provide complementary signals for identifying pathological changes. These findings underscore the importance of anatomy-aware representation learning and offer insights into developing generalizable retinal anomaly detection frameworks beyond predefined disease categories.
Figures & tables
Figure 1: Overview of the proposed representation-learning framework. A shared ViT encoder processes a clean anchor OCT B-scan, a different clean normal positive, and a perturbed version of the anchor as the negative. CLS tokens are mapped to L2-normalized embeddings and optimized using triplet loss, while patch tokens are reconstructed through a linear patch decoder using the reconstruction objective.
Method
AUROC
F1 (%)
Accuracy (%)
ConvNeXt
0.716
69.7
62.8
VAE
0.685
69.6
60.0
VQ-VAE
0.737
71.4
65.5
VQGAN
0.747
71.3
67.6
f-AnoGAN
0.763
71.1
68.4
Anatomy-aware VQGAN
0.799
75.2
71.7
Table 1: Image-level anomaly detection on the Kermany test set. F1 score and accuracy are computed at the operating threshold. Best values are shown in bold.
Method
Dice
mIoU
ConvNeXt
0.181
0.100
VAE
0.123
0.066
VQ-VAE
0.185
0.102
VQGAN
0.145
0.078
Anatomy-aware VQGAN
0.200
0.111
DINOv3
0.140
0.075
Table 2: Unsupervised anomaly localization on RETOUCH. All models were trained on Kermany only and applied to RETOUCH without further training. mIoU is foreground-only (background excluded from the class average). Best values are shown in bold.
Encoder
Configuration
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
DINOv3
Frozen
0.440
13.0
52.9
86.0
7.0
0.012
0.006
+ Reconstruction
0.510
21.0
54.8
83.0
12.0
0.023
0.012
+ Triplet
0.834
71.6
76.9
91.7
58.7
0.074
0.038
+ ROI
0.719
63.2
70.5
83.0
51.0
0.140
0.075
RETFound
Frozen
0.598
13.4
52.6
71.2
7.4
0.030
0.015
+ Reconstruction
0.640
26.4
55.3
75.0
16.0
0.051
0.026
Table 3: Effect of the training objective. Frozen: the pretrained encoder is fixed and only the linear reconstruction head is trained. + Reconstruction: the encoder and head are trained jointly using reconstruction loss only. + Triplet: the encoder and head are trained jointly with the reconstruction and triplet objectives. + ROI: retinal region-of-interest (ROI) weighting is added to the pixel loss (complete model). Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average). Reconstruction-only values are illustrative placeholders, not measured results.
Training data
Encoder
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
0.5%
DINOv3
0.605
47.8
60.8
70.6
36.1
0.090
0.047
RETFound
0.766
72.7
77.1
89.2
62.3
0.141
0.076
ImageNet ViT
0.709
63.0
70.9
85.3
49.9
0.109
0.058
1%
DINOv3
0.588
45.8
60.5
72.0
33.5
0.081
0.042
RETFound
0.740
67.4
74.2
90.1
53.8
0.145
0.078
ImageNet ViT
0.753
69.4
75.2
89.3
56.8
0.125
0.067
Table 4: Effect of the amount of normal training data on the complete models. Percentages refer to the fraction of the normal training set used for training. Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average). Bold values indicate the best performance for each metric at a given training-data percentage.
Figure 2: Effect of the amount of normal training data on the complete models. a , AUROC and b , F1 score for image-level detection on Kermany. c , Dice for anomaly localization on RETOUCH. The x axis shows the percentage of the normal training set used for training, on a logarithmic scale.
Scoring
Encoder
AUROC
F1 (%)
Acc. (%)
Prec. (%)
Rec. (%)
Dice
mIoU
Pixel level
DINOv3
0.719
63.2
70.5
83.0
51.0
0.140
0.075
RETFound
0.859
82.7
84.2
90.4
76.3
0.220
0.124
ImageNet ViT
0.921
85.9
87.3
89.1
84.7
0.242
0.138
Feature level
DINOv3
0.871
54.0
68.3
96.7
37.4
0.273
0.158
RETFound
0.91
74.4
79.3
97.8
60.5
0.397
0.248
ImageNet ViT
0.910
73.5
79.0
98.8
58.8
0.318
0.189
Table 5: Comparison of reconstruction-based (pixel-level) and feature-level anomaly scoring for the complete models. Image-level metrics are computed on Kermany and localization metrics on RETOUCH. mIoU is foreground-only (background excluded from the class average).
Figure 3: Perturbation strategy used to build training triplets from healthy (normal) OCT B-scans. A clean normal image x is used as the anchor. Its retinal band is detected automatically and used to place and scale the synthetic lesions.
Perturbation Strategy
Image-Level Anomaly Detection
Pixel-Level Anomaly Segmentation
Accuracy
F1
AUROC
AUPRC
Dice
mIoU
Thinning
58.00
33.75
50.91
60.31
6.80
3.52
Thickening
71.27
63.84
70.01
77.88
5.97
3.08
Subretinal fluid
64.59
52.62
62.68
70.23
6.85
3.55
PED
65.28
50.60
63.95
72.26
6.52
3.37
Foci
73.60
66.96
71.63
79.48
7.94
4.13
Table 6: Comparison of different synthetic perturbation strategies for OCT anomaly detection and segmentation. The best result for each metric is highlighted in bold. All values are reported as percentages. mIoU is foreground-only (background excluded from the class average).
Parameter
Setting
Model architecture and preprocessing
Pretrained encoders
DINOv3, RETFound, and supervised ImageNet ViT-L/16
Input resolution
256×256 pixels
Image preprocessing
Bicubic resizing; ImageNet normalization
Patch size and grid
16×16 pixels; 16×16 patches (256 tokens)
Patch embedding dimension
1,024
Table S1: Model architecture, training hyperparameters, and optimization settings used for the OCT anomaly detection framework.
Dept. of BioHealth Informatics, Indiana University Indianapolis, Indiana, USA · School of Information, University of Texas at Austin Austin, Texas, USA · Dept. of Biomedical Engineering, University of Florida Gainesville, Florida, USA +1