Paper ID: 2501.05933

Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2

Olivier Morelle (1 and 2), Justus Bisten (1), Maximilian W. M. Wintergerst (2 and 5), Robert P. Finger (2 and 4), Thomas Schultz (1 and 3) ((1) B-IT and Department of Computer Science, University of Bonn, (2) Department of Ophthalmology, University Hospital Bonn, (3) Lamarr Institute for Machine Learning and Artificial Intelligence, (4) Department of Ophthalmology, University Medical Center Mannheim, Heidelberg University, (5) Augenzentrum Grischun, Chur, Switzerland)

Weakly supervised segmentation has the potential to greatly reduce the annotation effort for training segmentation models for small structures such as hyper-reflective foci (HRF) in optical coherence tomography (OCT). However, most weakly supervised methods either involve a strong downsampling of input images, or only achieve localization at a coarse resolution, both of which are unsatisfactory for small structures. We propose a novel framework that increases the spatial resolution of a traditional attention-based Multiple Instance Learning (MIL) approach by using Layer-wise Relevance Propagation (LRP) to prompt the Segment Anything Model (SAM~2), and increases recall with iterative inference. Moreover, we demonstrate that replacing MIL with a Compact Convolutional Transformer (CCT), which adds a positional encoding, and permits an exchange of information between different regions of the OCT image, leads to a further and substantial increase in segmentation accuracy.

Submitted: Jan 10, 2025