CAMEL: Cross-Architecture Masked Embedding Learning for Ultra-Lightweight Retinal Foundation Models

Andre Guarnier De Mitri; Giovani Decico Lucafo; Miller Matheus Lima Anacleto Rocha; Diego Furtado Silva

cs.LGOct 29, 2026

CAMEL: Cross-Architecture Masked Embedding Learning for Ultra-Lightweight Retinal Foundation Models

Authors: Andre Guarnier De Mitri, Giovani Decico Lucafo, Miller Matheus Lima Anacleto Rocha, Diego Furtado Silva

Organizations: ICMC, University of São Paulo São Carlos, São Paulo, Brazil

Abstract

Recent progress in deep learning has substantially advanced automated grading of ophthalmologic pathologies, supported by large annotated datasets and improved representation learning. However, a significant gap remains between state-of-the-art performance and clinical scalability. In particular, foundation models have reshaped image-based medical screening and diagnosis. These large-scale architectures trained on extensive datasets learn rich, generalizable feature representations that can be adapted to a wide range of downstream tasks. In ophthalmology, RETFound, one of the most relevant foundation models for retinal images, performs strongly across classification, segmentation, and risk prediction tasks. Despite their effectiveness, foundation models are often computationally prohibitive, hindering their deployment in real-world clinical environments where hardware constraints and latency are critical. To bridge this gap, we introduce CAMEL (Cross-Architecture Masked Embedding Learning), a novel cross-architecture distillation framework that uses an asymmetric masking strategy during embedding learning to effectively transfer knowledge from massive teacher foundation models to compact, lightweight student models. Our experimental evaluation across four datasets shows that CAMEL preserves accuracy while drastically reducing computational complexity, enabling lightweight yet highly accurate models suitable for resource-limited clinical settings.

Figures & tables

Figure 1. Comparison of parameter count, computational cost, and AUROC between RETFound, a large Transformer-based foundation model, and a series of lightweight models for the dataset IDRID. The suffix “D” represents models distilled from RETFound.

Figure 2. Classification task, IDRiD sample images illustrating diabetic retinopathy severity levels (DR0–DR4). Diabetic retinopathy is a diabetes complication that progressively damages retinal blood vessels and can lead to vision loss

Figure 3. CAMEL framework: distillation process for training the student backbone using RETFound as the teacher model. In this configuration, the ViM student processes a masked version of the input image, whereas the teacher receives the full, unmasked image; an MLP projection layer approximates the student’s and teacher’s feature embeddings.

Model	Params (M)	GFLOPs	Latency (ms)	Throughput (img/s)	Energy ( $10^{-6}$ kWh)	CO 2 ( $10^{-6}$ kg)
MobileNet	1.52	0.111	1.18 $\pm$ 0.02	11611.9 $\pm$ 82.5	0.95 $\pm$ 1.95	0.09 $\pm$ 0.19
EfficientNet	4.01	0.769	1.83 $\pm$ 0.04	2649.6 $\pm$ 3.2	8.50 $\pm$ 2.10	0.84 $\pm$ 0.21
UNet	4.85	6.561	0.71 $\pm$ 0.00	2348.2 $\pm$ 30.9	8.60 $\pm$ 1.85	0.84 $\pm$ 0.18
Vim	5.02	0.476	17.46 $\pm$ 0.75	1691.2 $\pm$ 77.8	18.15 $\pm$ 1.70	1.78 $\pm$ 0.17
TinyViT	5.07	2.510	1.92 $\pm$ 0.03	1695.6 $\pm$ 7.4	14.40 $\pm$ 2.20	1.41 $\pm$ 0.21
Foundation Teacher

Table 1. Computational complexity, inference efficiency, and environmental footprint across evaluated base architectures using a NVIDIA RTX A5000 GPU. Latency is measured per image at batch size 1, while throughput represents images processed per second at batch size 32. Energy and CO 2 metrics are scaled by

10^{-6}

for readability. Results are aggregated across all runs (mean

\pm

std).

Figure 4. Average loss during the distillation of the RETFound teacher and the masked/unmasked Vim-based students.

In-Domain IDRiD
Model	Acc	F1	AUROC	AUPR
Vim (Masked)	0.412 $\pm$ 0.033	0.318 $\pm$ 0.023	0.610 $\pm$ 0.020	0.350 $\pm$ 0.008
Vim (Unmasked)	0.454 $\pm$ 0.021	0.330 $\pm$ 0.011	0.626 $\pm$ 0.027	0.356 $\pm$ 0.020
RETFound (Teacher)	0.516 $\pm$ 0.047	0.390 $\pm$ 0.024	0.665 $\pm$ 0.029	0.393 $\pm$ 0.020
Cross: Train APTOS $\rightarrow$ Eval IDRiD
Vim (Masked)	0.213 $\pm$ 0.051	0.146 $\pm$ 0.029	0.554 $\pm$ 0.021	0.240 $\pm$ 0.009

Table 2. Masked vs. unmasked variants across validation and cross-dataset evaluations (mean

\pm

std). The best results in each column per dataset are bolded , and the second best are underlined .

Figure 5. Mean validation AUROC by model and dataset. Bar height shows the mean AUROC, and error bars show the standard deviation. Each dot is the result of a single model run.

Model	Acc	F1	AUROC	AUPR
IDRID
Vim	0.254 $\pm$ 0.056	0.203 $\pm$ 0.027	0.569 $\pm$ 0.026	0.296 $\pm$ 0.021
MobileNet	0.208 $\pm$ 0.059	0.168 $\pm$ 0.055	0.580 $\pm$ 0.015	0.317 $\pm$ 0.031
MobileNet-D	0.204 $\pm$ 0.027	0.190 $\pm$ 0.026	0.623 $\pm$ 0.035	0.347 $\pm$ 0.036
EfficientNet	0.235 $\pm$ 0.033	0.189 $\pm$ 0.040	0.587 $\pm$ 0.025	0.304 $\pm$ 0.009
EfficientNet-D	0.365 $\pm$ 0.037	0.322 $\pm$ 0.059	0.750 $\pm$ 0.065	0.421 $\pm$ 0.053

Table 3. Comprehensive Benchmark Performance on validation across IDRiD, APTOS, mBRSET, and MESSIDOR datasets (mean

\pm

std). The best results in each column per dataset are bolded , and the second best are underlined .

Figure 6. Critical difference (CD) diagram for AUROC. Models connected by a horizontal bar are not significantly different according to the Friedman test with Nemenyi post-hoc correction (

\alpha=0.05

Medical Imaging Foundation Models

Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.

See all 240 papers

Compression of Vision Foundation Models for Retinal OCT via Growing-Neural-Gas Codebooks and Graph-Guided Masked Image Modeling
Oct 23, 2026Ali Yavari, Tilman Schmoll, Rainer A. Leitgeb +2Model Compression Vision Foundation Models
Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Währinger Gürtel 18-20, Vienna, 1090, Austria · Carl Zeiss Meditec AG, Rudolf-Eber-Straße 11, Oberkochen, 73447, Germany
Towards Interpretable Foundation Models for Retinal Fundus Images
Mar 19, 2026Samuel Ofosu Mensah, Camila Roa, Kerol Djoumessi +1Fundus Imaging Medical Imaging Foundation Models
Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany
Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging
Aug 1, 2026Mingya Alexa Gong, Da Ma, Lovre Antonio Budimir +7Medical Imaging Foundation Models Diabetic Retinopathy Grading
Institute of Ophthalmology, University College London, London, United Kingdom · Wake Forest University School of Medicine, Winston-Salem, NC, USA · Virginia Tech-Wake Forest University School of Biomedical Engineering and Sciences, Blacksburg, VA, USA +5

Oct 23, 2026J/K move · Enter open · S save

Compression of Vision Foundation Models for Retinal OCT via Growing-Neural-Gas Codebooks and Graph-Guided Masked Image Modeling

Ali Yavari, Tilman Schmoll, Rainer A. Leitgeb, Wolfgang Drexler+1

Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Währinger Gürtel 18-20, Vienna, 1090, Austria · Carl Zeiss Meditec AG, Rudolf-Eber-Straße 11, Oberkochen, 73447, Germany

Model Compression Vision Foundation Models Ophthalmic Image Analysis Masked Image Modeling Optical Coherence Tomography

Open paper AskPDF arXiv

Vision foundation models for retinal imaging are large and computationally demanding, limiting point-of-care deployment. We ask how much of such a model's downstream utility survives compression into a smaller network. We propose a topology-aware discretization framework that distills a continuous foundation model into a discrete codebook, then uses it as the reconstruction target for masked image modeling (MIM). It combines gradient-free codebook learning via Growing Neural Gas, which yields an adjacency graph over the codes, with a graph-distance soft cross-entropy loss that replaces one-hot MIM targets with topologically graded supervision. With student pretraining on 383 public Duke iAMD optical coherence tomography (OCT) volumes, the pipeline distills an 86M-parameter teacher into a 22M-parameter student with at least 3.6x fewer FLOPs. The student performs close to, and in some settings above, its teacher across three classification and two segmentation benchmarks, with the magnitude and direction of the difference depending on the teacher, task, metric, and student capacity. On the external OLIVES benchmark, it surpasses its MIRAGE teacher on every metric (98.06% vs. 95.22% balanced accuracy). It works with two Vision Transformer (ViT-Base) teachers, retinal (MIRAGE) and natural-image (DINOv3); with the out-of-domain teacher, the student outperforms it on retinal-layer segmentation, indicating that the added value of retina-specific pretraining is task-dependent: largest for segmentation, while a general-domain teacher is competitive for classification. These results yield compact retinal models potentially suitable for deployment, transferring across OCT datasets, devices, and clinical tasks, including three datasets not used in the student's pretraining.

Abstract

Medical Imaging Foundation Models

Explore similar work

Compression of Vision Foundation Models for Retinal OCT via Growing-Neural-Gas Codebooks and Graph-Guided Masked Image Modeling

Towards Interpretable Foundation Models for Retinal Fundus Images

Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging