Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders

Jacob Beattie; Samuel Stevens; Neil Rosser; Yu Su; Tanya Berger-Wolf

cs.CVNov 21, 2025

Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders

Authors: Jacob Beattie, Samuel Stevens, Neil Rosser, Yu Su, Tanya Berger-Wolf

Organizations: The Ohio State University

Abstract

Foundation models in several scientific domains, including visual domains, learn representations that capture complex semantics from their respective fields. Despite this, most existing applications focus on pre-specified concepts or targets, excelling at confirmation but not open-ended discovery of unknown patterns. We study whether sparse autoencoders (SAEs) with heuristic feature ranking can provide a practical methodology for surfacing candidate visual features from vision foundation models for scientific discovery without concept-specific supervision. We evaluate this approach in three stages: (1) general-purpose concept rediscovery, (2) domain-specific concept rediscovery, and (3) question-driven feature ranking. On ADE20K, SAEs outperform baseline methods ( $k$ -means, PCA, SemiNMF) across most downstream metrics for concept rediscovery. On FishVista, the same approach recovers fine-grained fish anatomical features without concept supervision. Finally, on Heliconius butterflies, a heuristic based on the Mann-Whitney $U$ statistic consistently surfaces features corresponding to real diagnostic keys between mimic subspecies. Together, these results show that SAEs with heuristic ranking can efficiently learn and surface semantic structure in vision foundation model representations. We view this as a methodological prerequisite for open-ended visual scientific discovery: the present experiments validate feature surfacing and ranking through rediscovery, while prospective discovery of previously unknown scientific phenomena remains future work.

Figures & tables

Figure 1: Validating an instrument for open-ended feature discovery. Left: Typical use of a foundation model in science: an input image is passed through a pretrained encoder and a task-specific head to yield class scores; the dense representation remains an opaque embedding, so unnamed factors are inaccessible. Right: Our procedure composes a foundation model with a sparse autoencoder, producing a library of interpretable semantic concepts with per-example activation maps. We find that these concepts align with and localize anatomical parts (e.g., head, dorsal fin) without seeing part-of-body labels. The learned concepts are inherently quantitative, making them easy to rank and search using heuristics.

Panel of Figure 2 — Figure 2: Comparison of methods for extracting knowledge from vision foundation models. Most approaches require researchers to specify concepts or tasks in advance.

Panel of Figure 3 — Figure 3: Top images for lowest-loss probes for “person” (left) and “toilet” (right) classes for each method. (a): $k$ -means does not recover the “person” class; the best “person” cluster fires on roads. (b): PCA learns a “person” component, but does not consistently activate on the entire person. Furthermore, PCA does not recover the “toilet” class; the best “toilet” doesn’t have an obvious semantic concept. (c): Semi-nonnegative matrix factorization (SemiNMF) is a strong baseline and learns both “person” and “toilet” concepts. (d): SAEs reliably recover both “person” and “toilet” and activate on the entire object. Takeaway : While $k$ -means and PCA report strong dictionary learning metrics, concepts recovered by SAEs are both more visually consistent and salient.

Method	NMSE	L 0	NMSE	L 0	Probe $R$	mAP	Pur@ $k$	Cov@ $\tau$
	Source Recon. $\downarrow$		Target Recon. $\downarrow$		Downstream $\uparrow$
$k$ -means	0.508	1	0.618	1	0.001	0.031	0.795	0.026
PCA	0.288	256	0.352	256	0.259	0.093	0.612	0.079
SemiNMF	0.298	256	0.363	256	0.209	0.088	0.696	0.079
SAE	0.166	379.2	0.211	379.2	0.441	0.224	0.748	0.264

Table 1: SAEs vs. unsupervised baselines for rediscovering ADE20K’s semantic segmentation classes from DINOv3 ViT-L/16. For each method, we choose the best layer from DINOv3 based on probe loss ( Equation 10 ) on ADE20K’s training split and report ADE20K validation metrics. Probe

R

is normalized probe loss, as described in Section 4 . Purity@

k

is precision over the top-

k

highest-scoring patches for a feature, where

k=16

. Coverage@

\tau

is the fraction of ADE20K classes with best AP

\geq\tau

, where

\tau=0.3

. Takeaway: Even a vanilla ReLU SAE improves downstream concept rediscovery over traditional unsupervised methods, despite comparable reconstruction metrics.

	Source Recon. $\downarrow$		Target Recon. $\downarrow$		Downstream $\uparrow$
Method	NMSE	L 0	NMSE	L 0	Probe $R$	mAP	Pur@ $k$	Cov@ $\tau$
ReLU	0.166	379.2	0.211	379.2	0.441	0.224	0.748	0.264
+Matryoshka	0.591	6.5	0.702	6.5	0.402	0.319	0.808	0.405
+TopK	0.207	64	0.274	64	0.479	0.389	0.788	0.576
PE ViT-L/14	0.120	128	0.149	128	0.123	0.110	0.557	0.092

Table 2: SAE variant and backbone ablation on ADE20K. Top rows compare SAE variants on DINOv3 ViT-L/16 (final layer); bottom row switches to a pretrained Perception Encoder ViT-L/14 checkpoint, using the Matryoshka+TopK SAE training protocol. Best SAE per configuration selected by training probe loss.

Dataset	NMSE	L 0	NMSE	L 0	Probe $R$	mAP	Pur@ $k$	Cov@ $\tau$
	Source Recon. $\downarrow$		Target Recon. $\downarrow$		Downstream $\uparrow$
ADE20K	0.207	64	0.274	64	0.479	0.389	0.788	0.576
FishVista	0.184	16	0.184	16	0.438	0.543	0.501	0.7

Table 3: Matryoshka TopK SAEs for concept rediscovery on general domain (ADE20K) and ecological imagery (FishVista) under the same DINOv3 ViT-L/16 backbone. Source Recon. measures reconstruction on the SAE’s training distribution; Target Recon. measures transfer to the evaluation domain.

Panel of Figure 4 — Figure 4: Example TopK SAE features (left) and patch-level segmentation masks (right) for (a) “Head” , (b) “Dorsal fin” and (c) “Eye” from DINOv3 ViT-L/16. Features were picked by minimizing cross entropy on binary classification for each body part, as described in Sections 4.2 and 10 . Takeaway : TopK SAEs accurately learn scientific concepts (anatomical body parts) without labels.

Panel of Figure 5 — Figure 5: (Left) Prevalence vs rediscovery on FishVista. Each point is a body part; x-axis: number of patch-level samples in the validation split, y-axis: AP of the best-matching (lowest probe loss) TopK SAE latent. More common body parts are rediscovered more reliably. Takeaway: Higher concept prevalence is associated with more rediscovery. (Right) Reconstruction–sparsity tradeoff for SAE variants (ReLU, Matryoshka, TopK) on DINOv3 ViT-L/16 ImageNet-1K activations. TopK achieves the best reconstruction at fixed sparsity.

Panel of Figure 6 — Figure 6: For a single Matryoshka TopK SAE trained on DINOv3 ViT-L/16 patch activations, the highest selectivity features for each studied H. erato (top)/ H. melpomene (bottom) mimic pair. (a) Feature 1,662 corresponds to hindwing fringe, (b) Feature 104 corresponds to hindwing ray shape, and (c) Feature 192 corresponds to the forewing red patch placement. Note that the highest-ranked H. melpomene image for (b) Feature 104 is actually a mislabeled H. erato . Similar features were ranked very highly for all 3 pairs among other trained SAEs. Takeaway : SAE features can be easily ranked using relevant heuristics to identify the most promising discovery candidates.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparam	Sweep Values
Learning Rate	[ $3\text{\times}{10}^{-4}$ , $1\text{\times}{10}^{-3}$ , $3\text{\times}{10}^{-3}$ , $1\text{\times}{10}^{-2}$ , $3\text{\times}{10}^{-2}$ , $1\text{\times}{10}^{-1}$ ]
Sparsity Coeff.	[ $1\text{\times}{10}^{-4}$ , $1\text{\times}{10}^{-3}$ , $1\text{\times}{10}^{-2}$ , $1\text{\times}{10}^{-1}$ ]

Appendix

Table B1: Sparse autoencoder hyperparameter sweep values.

Panel of Figure 11 — Figure C1: We compare $k$ -means, PCA, vanilla SAEs and Matryoshka SAEs against sparsity (L 0 ) and downstream metrics (Probe $R$ , mAP, Purity@ $k$ and Coverage@ $\tau$ ) for ADE20K.

Panel of Figure 12 — Figure D2: Some additional qualitative examples for (a) $k$ -means: “waterfall” (left) and “road” (right), (b) PCA: “sky” (left) and “pool table” (right), (c) vanilla SAEs: “dishwasher” (left) and “bed” (right) and (d) Matryoshka SAEs: “towel” (left) and “chandelier” (right).

Hyperparam	Sweep Values
Learning Rate	[ $3\text{\times}{10}^{-4}$ , $1\text{\times}{10}^{-3}$ , $3\text{\times}{10}^{-3}$ ]
$k$	[16, 32, 64, 128]

Appendix

Table E2: Sparse autoencoder hyperparameter sweep values for SAEs trained on Heliconius images.

Panel of Figure 14 — Figure F3: (Left) The reconstruction–sparsity tradeoff for SAEs trained on different sized ViTs. Larger backbones are harder to reconstruct at a given sparsity (ViT-L/16 $>$ ViT-B/16 $>$ ViT-S/16), though all improve as $L_{0}$ increases. Note that the $y$ -axis is raw rather than normalized MSE. (Right) Layer-wise reconstruction–sparsity trade-off (DINOv3 ViT-L/16) on ImageNet-1K. Later layers consistently incur higher MSE at matched sparsity, indicating greater representational complexity.

Figure F4: Matryoshka SAE probe quality as measured by mean average precision (mAP) for DINOv3 ViT-S/16, ViT-B/16 and ViT-L/16 checkpoints over different transformer layers. Takeaway: Despite larger ViTs and later layers being harder to reconstruct at a given sparsity (see Figure F3 for ViT size ablation), larger ViTs and later layers lead to better SAE concept alignment.

Figure G5: We compare

k

-means, PCA, SemiNMF and SAEs along the reconstruction–sparsity tradeoff for learning to decompose ViT patch activations. We use final layer activations from DINOv3 ViT-L/16 on ImageNet-1K; we fit all methods on the training split and measure normalized MSE (see Equation 5 ) and L 0 on the validation split. For

k

-means, we “reconstruct” every patch with its nearest centroid; thus, L 0 is always

1

. For PCA, we sweep the number of components. We sweep

\lambda

for the SAE and show the Pareto frontier of reconstruction–sparsity. Takeaway: Reconstruction–sparsity does not indicate an optimal method.

Panel of Figure 17 — Figure H6: Layer-depth ablation for all four downstream metrics on ADE20K, for DINOv3 ViT-S/16, ViT-B/16, and ViT-L/16. The trend from Figure F4 (later layers improve concept alignment) holds across all metrics.

Model		Dict. $\downarrow$		Downstream $\uparrow$
Size	Layer	NMSE	L 0	Probe $R$	mAP	Purity@ $k$	Coverage@ $\tau$
ViT-S/16	7	0.182	70.9	0.165	0.044	0.744	0.053
ViT-S/16	8	0.143	135.1	0.209	0.062	0.623	0.086
ViT-S/16	9	0.133	159.8	0.232	0.071	0.511	0.086
ViT-S/16	10	0.299	84.1	0.251	0.099	0.626	0.093
ViT-S/16	11	0.438	23.6	0.310	0.160	0.645	0.205

Appendix

Table H3: Probe results for Matryoshka SAEs trained on ImageNet-1K and evaluated on ADE20K. For each layer in each size of ViT, we choose the best SAE based on training probe loss and report all dictionary learning and downstream metrics.

Figure I7: 5 highest selectivity features for H. erato cyrbia / H. melpomene cythera . Features 1-3 correspond to hindwing fringe in H. erato cyrbia , a known diagnostic trait. Features 4 and 5 may correspond to the shape of the forewing red patch, a known diagnostic trait, but these are not counted as corresponding to a known trait due to difficulty interpreting diffuse activation patterns.

Figure I8: 5 highest selectivity features for H. erato lativitta / H. melpomene malleti . All 5 features correspond to the black bar or striping on the hindwing, a known diagnostic trait in this mimic pair.

Figure I9: 5 highest selectivity features for H. erato notabilis / H. melpomene plesseni . Features 1, 2, 3, and 5 correspond to the red forewing patch placement in H. erato notabilis , a known diagnostic trait. Feature 4 corresponds to the upper forewing white patch in H. erato notabilis . No diagnostic trait for this portion of the wing is well-documented in this mimic pair.

Explore similar work

Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?
Jun 22, 2026Nils Grandien, David Steinmann, Felix Friedrich +1Hierarchical Unsupervised
Computer Science Department, TU Darmstadt · Hessian Center for AI (hessian.AI) · Black Forest Labs +1
Position: Use Sparse Autoencoders to Discover Unknowns
Jun 30, 2025Kenny Peng, Rajiv Movva, Jon Kleinberg +2Improving Sparse Autoencoders Model Discovery
1Cornell University · 2UC Berkeley
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniImproving Sparse Autoencoders Sparsity
Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?

Nils Grandien, David Steinmann, Felix Friedrich, Kristian Kersting

Computer Science Department, TU Darmstadt · Hessian Center for AI (hessian.AI) · Black Forest Labs+1

Hierarchical Unsupervised

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchical structure, either explicitly or as an implicit effect of training constraints, yet rigorous comparison remains difficult. There are no agreed-upon requirements for what a meaningful feature hierarchy should satisfy, and evaluation has largely relied on qualitative illustrations with fragmented quantitative protocols. To address this, we derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery, drawing on semantic net and taxonomy research alongside recent SAE work, and use them to derive a concrete evaluation protocol. Applying this protocol to current SAE approaches trained on visual data, we find that while feature spaces generally provide a basis for sensible hierarchies, establishing good hierarchical structure remains challenging. In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality, pointing to a fundamental tension that future approaches will need to navigate.

Abstract

Explore similar work

Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?

Position: Use Sparse Autoencoders to Discover Unknowns

Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders