Sparse autoencoders (SAEs) decompose LLM activations into sparse dictionary atoms, so that each distinct concept gets its own feature. One recurring behavior complicates this premise: feature absorption, in which a parent concept and its children--fruit and {apple, banana, pear}, say--collapse into a shared family direction. Prior work documents absorption empirically; missing is a closed-form prediction of when the shared direction is the cost-optimal representation of an active semantic family. This paper closes that gap. For a hierarchical Bernoulli generator with k active children and residual scale α, the L0-penalized reconstruction objective admits a closed-form phase boundary λc(k,α)=α2k/(k−1): above it, pure parent absorption is strictly cheaper than pure child coding. Building on this boundary, we introduce HiPACE, an evaluation protocol that tests the boundary's structural consequence in real SAE dictionaries--measuring parent--child decoder structure over WordNet families, freezing the discovery-selected statistic before testing on unseen families, and contrasting genuine families against randomized sibling nulls. The boundary proves sharp in its native regime, predicting the synthetic transition within ±15 on all 30 tested cells. In Pythia-160m SAEs, the parent--child decoder gap recovers the predicted ordering with partial correlations up to −0.93 that sustain on the locked holdout and exclude sibling nulls (p=0.002). Controlled activation composition connects the theory's active-child count to the recovered family directions, and residual-stream interventions show that signed family directions increase parent-category logits, reversing under sign flip and vanishing under random controls--establishing causal sufficiency at the family-subspace level.
Figures & tables
Figure 1: Motivation for family-level evaluation. Hierarchical feature competition predicts a phase boundary in dictionary structure. The empirical analysis therefore measures parent–child structure over semantic families, validates the selected statistic on a frozen holdout, and tests the recovered family directions mechanistically. Here k denotes active-child count, whereas K denotes family size.
Figure 2: Overview of HiPACE. The framework connects a hierarchical family model and its phase-boundary analysis to family-level SAE dictionary measurement, frozen validation on unseen semantic families, and controlled mechanistic follow-up. Lowercase k denotes sample-level active-child count; uppercase K denotes total family size.
Figure 3: Phase-boundary validation (RQ1). Predicted and observed 50% absorption onsets across the hierarchical generator grid.
Split
Frozen statistic
Partial correlation
95% CI
Sibling-null p
Discovery
L6 mean gap
−0.681
[−0.816,−0.448]
0.005
Discovery
L11 PC1 gap
−0.933
[−0.955,−0.875]
0.005
Holdout
L6 mean gap
−0.605
–
0.002
Holdout
L11 PC1 gap
−0.914
–
0.002
Table 1: HiPACE structural results (RQ2–RQ3). Partial Spearman correlations control family size K . Discovery statistics are frozen before holdout evaluation.
Figure 4: Structural recovery and grouping specificity (RQ2–RQ3). Discovery and frozen-holdout correlations compared with randomized sibling-group nulls.
Intervention
Median shift
Parent-signed inject
+0.0523
Child-signed inject
+0.0523
Sign-flipped parent
−0.0474
Random
−0.0003
Orthogonal random
−0.0009
Parent-signed ablate
−0.0086
Table 2: L6 intervention controls (RQ4). Median true-parent logit-margin shift at strength 1.0.
Figure 6: 507-row synthetic stress sweep: the bound predicts the absorption regions in all four corner regimes.
Figure 7: Dictionary-structural evaluation recovers the hierarchy signal that the firing proxy missed. (a) Parent-child decoder gap correlations across layers after controlling for family size K . Middle layers recover the family-mean parent direction; deep layer L11 recovers the PC1 parent direction. (b) The two main structural claims survive bootstrap intervals and are outside randomized-sibling correlation nulls.
Figure 8: Detailed plots for the real-LLM singleton top- 1 firing proxy and its external replication.
Figure 9: Three v^ estimators on the same 50 families; ρ remains positive for all three (spread 0.37 ).
Notion
Operational meaning
Predicted by Theorem 1 ?
Structural
Does the trained dictionary contain a parent-aligned atom?
Yes (pure-strategy bound)
Per-sample firing
Does the SAE fire a child atom on a k -active sample?
No (mixed-strategy escape)
Subset skipping
Does a child atom fire on all concept members, or skip subsets?
Learning hierarchical features in Sparse Autoencoders (SAEs) is essential for capturing the structured nature of real-world data and mitigating issues like feature absorption or splitting. Existing works attempt to identify hierarchical relationships within independent feature sets by relying on activation coverage, the assumption that child feature should only activate when its parent feature activates. However, we demonstrate that this condition alone is insufficient; that is, it often produces false positives where parent and child concepts are semantically unrelated. To address this, we introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels. By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set. Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks. Finally, we demonstrate the practical utility of our Tree SAE in mapping the geometry of child feature subspaces and uncovering the complex hierarchical concept structures encoded within large language models.
Tue M. Cao, Hoang X. Nhat, Raed Alharbi +2
Hanoi University of Science and Technology, Hanoi, Vietnam · University of Florida, Florida, USA · Computer Science Department, Saudi Electronic University
Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchical structure, either explicitly or as an implicit effect of training constraints, yet rigorous comparison remains difficult. There are no agreed-upon requirements for what a meaningful feature hierarchy should satisfy, and evaluation has largely relied on qualitative illustrations with fragmented quantitative protocols. To address this, we derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery, drawing on semantic net and taxonomy research alongside recent SAE work, and use them to derive a concrete evaluation protocol. Applying this protocol to current SAE approaches trained on visual data, we find that while feature spaces generally provide a basis for sensible hierarchies, establishing good hierarchical structure remains challenging. In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality, pointing to a fundamental tension that future approaches will need to navigate.
Nils Grandien, David Steinmann, Felix Friedrich +1
Computer Science Department, TU Darmstadt · Hessian Center for AI (hessian.AI) · Black Forest Labs +1
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.