Sparse Autoencoders for Interpretable Medical Image Representation Learning
Authors: Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios Gatidis
Organizations: Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Chair of AI in Healthcare and Medicine, Technical University of Munich, Munich, Germany · TUM University Hospital, Munich, Germany
Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D image slices from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R2 up to 0.941) and recover up to 87.8% of downstream performance using only 10 features (99.4% dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation. (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: https://github.com/pwesp/sail.
Figures & tables
Figure 1: (A) A Sparse Autoencoder replaces opaque dense FM embeddings with a sparse feature space. (B) Sparse fingerprint retrieval matches images by cosine similarity over k top-activated features. (C) A VLM generates a concept description for each feature from its top-activating images and metadata. (D) An LLM maps a clinical text query to matching feature concepts for zero-shot image retrieval.
Figure 2: SAE quality and performance recovery across 96 configurations per FM (DINOv3: blue, BiomedParse: orange, random baseline: grey). (A–D) Reconstruction fidelity (R 2 ), downstream ROC-AUC, alive features, and monosemanticity score vs. L0 sparsity. (E–G) Performance recovery using only the top- N features ( N=1,3,10,50 ).
Model
Dict Sizes
Top-K Values
# Mono
# Perf
# Comb
BiomedParse
128, 512, 2048, 8192
20, 40, 80, 160
2
3
1
128, 512, 2048, 8192
30, 60, 120, 240
3
6
2
128, 512, 2048, 8192
10, 20, 40, 80
1
12
3
DINOv3
128, 512, 2048, 8192
5, 10, 20, 40
1
11
1
64, 256, 1024, 4096
5, 10, 20, 40
4
10
2
128, 512, 2048, 8192
10, 20, 40, 80
2
14
3
Table 1: Top-3 SAE configurations per FM ranked by combined monosemanticity and performance recovery score (96 configurations each). Bold: selected optimal configuration. # Mono / Perf / Comb : monosemanticity/performance/combined rank.
Figure 3: Sparse fingerprint retrieval at k=5 for five reference cases (A–E) spanning CT and MRI across multiple anatomical regions. Row 1: reference images with BiomedParse (orange) and DINOv3 (blue) fingerprint insets. Rows 2–3: top-2 BiomedParse retrievals. Rows 4–5: top-2 DINOv3 retrievals.
Model
k=1
k=5
k=10
k=20
Dense
BiomedParse
0.929
0.954
0.964
0.967
0.976
DINOv3
0.752
0.831
0.852
0.857
0.895
Table 2: Sparse fingerprint retrieval quality (mean cosine similarity to the reference image in the dense embedding space) as a function of fingerprint size k , averaged over N=1,000 test images. Dense : full dense retrieval quality (upper bound).
Figure 4: Zero-shot language-driven retrieval for “Axial CT of the abdomen and retroperitoneum in an elderly patient.” An LLM selects matching feature concepts (left), determining a sparse fingerprint (center) for cosine retrieval (right). BiomedParse selects mixed MRI/CT concepts and retrieves thoracic images. DINOv3 selects CT-specific abdomen features and retrieves correct axial abdominal CT.