Where to Predict in Retinal OCT? Anatomy-Guided Target Selection for I-JEPA
Organizations: University of Pennsylvania · Columbia University
Abstract
Self-supervised learning aims to learn meaningful representations from unlabeled data, potentially allowing models to capture subtle anatomical variations across patients and reduce the amount of labeled data needed. Image-based Joint-Embedding Predictive Architecture (I-JEPA) learns by predicting target-region representations from visible context rather than reconstructing pixels. However, its uniform target placement does not account for the anatomical structure of retinal OCT, where retinal tissue occupies only part of each B-scan. We investigate anatomy-guided target selection using image intensity and predicted retinal segmentation while retaining the I-JEPA architecture and latent prediction objective. We compare anatomy-guided rectangular targets, anatomy-shaped targets and coverage-constrained placement with uniform masking, and evaluate the learned representations using linear probing for glaucoma classification on FairVision. We found that rectangular targets guided by image intensity achieved the highest linear-probe AUC in our comparisons. We show that anatomically meaningful target selection is promising for retinal OCT representation learning, while greater anatomical specificity did not consistently improve performance.
Figures & tables
| policy | guide | target shape | placement | target length |
|---|---|---|---|---|
| random | none | rectangles | uniform | variable |
| envelope | hard MIRAGE envelope | rectangles | rejection-sampled on envelope | variable |
| centroid | intensity-centroid band | rectangles | weighted by band overlap | variable |
| anatomy-v1/v2 | soft MIRAGE scores | connected components | grown on scores | fixed ( ) |
| cover | soft MIRAGE scores | rectangles | greedy coverage | variable |
| epoch 50 | epoch 75 | epoch 100 | |||||
| strategy | guidance signal | AUC | (pp) | AUC | (pp) | AUC | (pp) |
| random | none (null) | 0.8641 | — | 0.8723 | — | 0.8746 | — |
| envelope | segmenter (location only) | 0.8761 | 0.8803 | 0.8807 | |||
| centroid | intensity centroid, no segmenter | 0.8740 | 0.8836 | 0.8855 | |||
| anatomy-v2 | segmenter (target shape) | 0.8654 | 0.8612 | — | — | ||
| cover | segmenter (coverage) | 0.8643 | 0.8639 | 0.8577 | |||
| strategy | tissue hidden | purity | unique targets | loss slots | context (% of grid) | AUC (ep. 50) |
|---|---|---|---|---|---|---|
| random | 53.2% | 30.9% | 113.1 | 160.2 | 0.8641 | |
| centroid | 59.7% | 38.2% | 102.5 | 160.2 | 0.8740 | |
| envelope | 70.7% | 42.1% | 110.2 | 160.2 | 0.8761 | |
| cover | 73.3% | 40.1% | 120.1 | 160.2 | 0.8643 | |
| anatomy-v2 | 79.7% | 96.9% | 54.0 | 64.0 | 0.8654 |
| random | centroid | envelope | (pp) centroid | (pp) envelope | |
|---|---|---|---|---|---|
| original (re-probed) | 0.8641 | 0.8752 | 0.8755 | ||
| seed A | 0.8624 | 0.8721 | 0.8716 | ||
| seed B | 0.8635 | 0.8754 | 0.8710 |
| policy | test AUC | (pp) (interval) | U | L | D | C | purity (%) |
|---|---|---|---|---|---|---|---|
| random | 0.8624 | — | 113.1 | 160.2 | 47.1 | 70.5 | 36.0 |
| random-cb | 0.8650 | 102.2 | 159.2 | 57.1 | 79.5 | 36.7 | |
| centroid | 0.8721 | 102.5 | 160.2 | 57.7 | 80.8 | 46.0 | |
| centroid random-cb | placement at matched budgets | ||||||
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| per image | delivered | |||||
|---|---|---|---|---|---|---|
| strategy | tissue hidden | purity | mask ratio | context | loss slots | context |
| random | 53.0 | 30.9 | 44.0 | 42.1 | 159.3 | 27.5 |
| centroid | 59.7 | 38.3 | 39.9 | 45.7 | 159.3 | 31.5 |
| envelope | 70.5 | 42.1 | 42.9 | 43.7 | 159.3 | 29.2 |
| cover | 73.2 | 40.1 | 46.7 | 39.9 | 159.3 | 25.2 |
| anatomy-v2 | 79.7 | 96.8 | 21.1 | 68.0 | 64.0 | 63.3 |
| policy | run | test AUC | head s.d. | val. AUC | (pp) test (interval) | (pp) val. | analysis |
|---|---|---|---|---|---|---|---|
| random | original | 0.8641 | 0.0005 | 0.8467 | — | — | original |
| random | seed A | 0.8624 | 0.0005 | 0.8495 | — | — | descriptive |
| random | seed B | 0.8635 | 0.0008 | 0.8443 | — | — | descriptive |
| centroid | original | 0.8752 | 0.0006 | 0.8560 | original | ||
| centroid | seed A | 0.8721 | 0.0007 | 0.8547 | descriptive | ||
| centroid | seed B | 0.8754 | 0.0005 | 0.8566 | descriptive |
| pooling | random | centroid | envelope | centroid | envelope |
|---|---|---|---|---|---|
| mean, mean (primary) | 0.8624 | 0.8721 | 0.8716 | ||
| max, mean | 0.8483 | 0.8625 | 0.8553 | ||
| mean, max | 0.8449 | 0.8637 | 0.8633 | ||
| mean, mean and max | 0.8553 | 0.8709 | 0.8719 |
| policy | run | MAE | RMSE | Spearman | MAE (interval) | Spearman |
|---|---|---|---|---|---|---|
| random | seed A | 3.55 | 4.72 | 0.52 | — | — |
| random | seed B | 3.49 | 4.64 | 0.53 | — | — |
| centroid | original | 3.45 | 4.57 | 0.53 | — | — |
| centroid | seed A | 3.47 | 4.60 | 0.53 | ||
| centroid | seed B | 3.46 | 4.58 | 0.54 | ||
| envelope | original | 3.46 | 4.60 | 0.54 | — | — |
| Measurement | COVER |
|---|---|
| Views losing tissue targets during equalization | 300 |
| Valid-guide context-criterion misses | 358 |
| Views with zero tissue context | 55 |
| Scored hidden guide mass (%) | 78.51 |
| Final hidden guide mass (%) | 74.57 |
| Mean encoder-visible tissue cells | 10.13 |
| contrast | epoch | (pp) | 95% CI (pp) | ||
|---|---|---|---|---|---|
| envelope random | 50 | +1.20 | [+0.68, +1.73] | 0.0001 | 0.0001 |
| envelope random | 75 | +0.80 | [+0.28, +1.32] | 0.0025 | 0.0045 |
| envelope random | 100 | +0.62 | [+0.10, +1.13] | 0.0199 | 0.0299 |
| centroid random | 50 | +0.99 | [+0.51, +1.49] | 0.0001 | 0.0001 |
| centroid random | 75 | +1.13 | [+0.63, +1.64] | 0.0001 | 0.0001 |
| centroid random | 100 | +1.09 | [+0.57, +1.60] | 0.0001 | 0.0001 |
| policy | epoch | fp16 AUC | fp32 AUC | ||
|---|---|---|---|---|---|
| envelope | 50 | 0.876064 | 0.876063 | -0.000001 | 0.964 |
| envelope | 75 | 0.880307 | 0.880305 | -0.000002 | 0.806 |
| envelope | 100 | 0.880743 | 0.880761 | +0.000018 | 0.167 |
| centroid | 50 | 0.874030 | 0.874015 | -0.000015 | 0.412 |
| centroid | 100 | 0.885485 | 0.885293 | -0.000192 | 0.767 |
| random | 50 | 0.864097 | 0.864121 | +0.000024 | 0.128 |
Medical Image Representation Learning
Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.
See all 281 papers