Sparse autoencoders (SAEs) expose interpretable features in large language model activations, yet existing structured SAEs impose single-parent trees or forests, while post-hoc graphs permit multiple parents but neither guide feature learning nor ensure reliable relation recovery. We introduce the Adaptive Graph Sparse Autoencoder (AG-SAE), a structure-guided training paradigm that treats each feature's complete parent set as an atomic structural hypothesis and lets evidence select zero, one, or multiple parents. By competing complete parent sets against null, subset, and alternative explanations, AG-SAE identifies jointly necessary multi-parent relations while rejecting redundant or spurious alternatives and verifying that each child contributes beyond its parents. The induced topology over SAE features then defines a differentiable structural loss that guides SAE training, while topology-guided refinement mitigates feature absorption and uses persistent reconstruction gaps exposed by the learned structure to initialize new features. The entire graph is then induced again from the revised dictionary by reassessing every feature's complete parent set, closing the dictionary-graph self-consistency cycle. Experiments demonstrate exact mixed-topology recovery in a controlled toy model, greater relational reliability and semantic validity than structured and post-hoc baselines on real LLM activations, and stronger feature-level causal interventions than conventional SAE features. AG-SAE thereby turns recovered mixed-topology feature structure into an unsupervised training signal that improves the dictionary, enables reliable feature organization beyond the topological limitations of trees, and exhibits stronger causal control beyond reconstruction.
Figures & tables
Figure 1: Overview of AG-SAE. The induced topology guides SAE training and dictionary refinement; Full Re-Induction closes the cycle, which repeats until the dictionary–graph pair stabilizes. This example is illustrative.
Table 1: AG-SAE structure recovery in toy and real LLM settings. A. Dictionary quality and structure recovery on the ground-truth mixed-topology toy model. In A, † denotes N/A for methods without a native parent graph; Vanilla SAE’s eight no-parent matches reflect the default empty parent set. B. Final AG-SAE feature graph on Gemma-2-2B layer-13 activations from MiniPile. Panels C–D use the same setting. Category percentages use the 30,718 active features ( 99.99% of 30,720 ); Multi / parented is the fraction of parented features with multiple parents. C. Dictionary and graph statistics across refinement cycles. D. Representation quality and relation validation across methods (Abs., Feature Absorption; AutoInt., AutoInterp). PSV, NR, and SV denote Predictive Structural Validity, Non-Redundancy, and Semantic Validity; Joint denotes their intersection. † marks Tree SAE’s lower L0 from its native sparsification, which also gives its Abs. score a sparsity advantage.
Figure 2: Learned three-parent relation for appellant references. Left. Feature 7707 and its complete parent set in final AG-SAE trained on MiniPile Gemma-2-2B layer-13 activations. Feature IDs are global across dictionary banks. Right. Activation examples selected per feature from profiling data and report contexts. Orange shading shows per-token activation on the same raw scale.
Figure 3: Individual and joint parent coverage of feature 7707. Counts use 4,910,199 report token positions with frozen activation thresholds. The first three panels show individual parents; the rightmost requires all three parents to be active. Orange denotes parent-only activation, green coactivation with the child, and pink child activation with the parent condition unmet. Coverage uses the 484 child activations as its denominator. Panels use separate visual scales.
Figure 4: AG-SAE consolidates bidirectional causal control into its leading features. Left. Mean causal effects of adding and ablating the leading feature. Center. AG-SAE improves both intervention directions for 21/26 concepts. Right. The gain in causal yield per active feature is concentrated at Top-1 and narrows as additional features are included. Together, the three panels establish stronger magnitude, cross-concept consistency, and leading-feature concentration.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: AG-SAE refinement dynamics and joint stability. Results use Gemma-2-2B layer-13 activations from MiniPile. Left. Counts of single- and multi-parent features across graph generations, together with accepted Residual Completion updates between successive refinement cycles. Middle. Normalized feature-contribution change dF on held-out activations and graph change dG in feature membership and complete parent-set assignments between successive refinement cycles. Both decrease after G(2) , reaching 9.25% and 3.06% , respectively, in the final transition. Right. Relation gain is the held-out fraction of residual representation explained by parents with the child withheld, averaged over all selected relations, increasing from 4.56% at G(0) to 9.12% at G(5) .
A. Coverage by child width
Width
Parented
Cov. %
Single
Multi
Multi %
2,048
422
20.61
218
204
48.34
4,096
1,197
29.22
847
350
29.24
8,192
2,118
25.85
1,636
482
22.76
16,384
3,354
20.47
2,720
634
18.90
B. Validity by multi-parent arity
Appendix
Table 2: Multi-parent topology spans all four dictionary banks, and Joint validity changes by only 0.75 percentage points between two- and three-parent sets. A. Cov. % is the number of parented features divided by bank width, and Multi % is the number of multi-parent features divided by the number of parented features. B. Structures denotes the total number of recovered relations in each arity cohort. Panel B applies the frozen report-only protocols of Table 1 to fixed arity-specific relation cohorts. PSV, NR, and SV are evaluated on the same relations, while Joint reports the percentage satisfying all three criteria. All rates are reported as percentages.
Figure 6: Dictionary scale organizes graph roles, while graph depth extends beyond the four bank levels. Left. Root and leaf shares within each dictionary bank. Roots fall from 27.29% to 5.94% of bank width, while leaves rise from 6.49% to 18.85% . Right. A feature’s graph depth is the maximum number of edges among all directed paths that begin at a root and end at that feature. The dashed line marks the maximum depth of three permitted by a strict progression from narrower to wider banks. The recovered graph places 394 features beyond this bound and reaches depth eight.
Comparison
Parent set
Held-out fit ↑
Complete learned set
{5963,14785,3242}
0.5497
Remove 5963
{14785,3242}
0.4543
Remove 14785
{5963,3242}
0.4409
Remove 3242
{5963,14785}
0.3460
Best single parent
{14785}
0.2452
Best matched alternative
{5295,14785,3242}
0.4627
Appendix
Table 3: Complete parent-set comparison for feature 7707. The fit score used in PSV is one minus the squared prediction error for the child’s decoder contribution, normalized by its total squared norm over the same 484 report positions where the child is active. Nonnegative coefficients are fitted separately for each parent set on a disjoint split and frozen for evaluation. The matched alternative is the strongest replacement set in the evaluated candidate pool. The null hypothesis has score 0 .
A. Dictionary Quality and Graph Composition
Dictionary
Graph
Setting
Width
EV
L0
Parented
Cov. %
Single %
Multi %
Root %
Leaf %
Gemma / PubMed
2,048
0.660
50.29
328
16.02
50.61
49.39
34.28
4.74
4,096
0.686
50.12
1,082
26.42
70.15
29.85
23.07
7.08
8,192
0.710
49.74
2,349
28.67
74.37
25.63
17.19
14.64
16,384
0.727
50.36
3,700
22.58
79.35
20.65
4.72
19.46
Appendix
Table 4: AG-SAE dictionary quality, graph composition, and relation validity across models and datasets. Gemma/PubMed uses Gemma-2-2B layer 13 with four dictionary scales; Qwen/MiniPile uses Qwen3.5-2B-Base layer 18 with two scales and 49,152 features in total. A. EV and L0 are report-only measurements with graph routing disabled at each dictionary scale. Cov. % divides parented features by bank width. Single % and Multi % divide the respective feature counts by parented features; Root % and Leaf % divide their counts by bank width. Roots have children and no parents; leaves have parents and no children. B. Validation follows the frozen report-only protocols of Table 1 , Panel D. Structures denotes the total number of recovered relations in each cohort. PSV, NR, SV, and Joint are evaluated under the same frozen report-only protocol within each cohort, with Joint reporting simultaneous satisfaction of all three criteria.
Condition
Change
95% CI
Wins
Full AG-SAE
+0.867
[0.298,1.506]
10/13
Normalized continuation
−0.215
[−0.597,0.184]
3/13
Free-scale continuation
−0.160
[−0.546,0.235]
5/13
Appendix
Table 5: Matched changes in bidirectional causal control under full AG-SAE training and two conventional SAE continuation controls.
Metric
Gain
95% CI
Wins
p -value
Addition
+0.573
[0.308,0.893]
24/26
1.05×10−5
Ablation
+0.685
[0.260,1.240]
21/26
0.0025
Bidirectional
+0.674
[0.281,1.210]
21/26
0.0025
Top-1 concentration
+0.681
[0.259,1.241]
20/26
0.0094
Appendix
Table 6: Paired AG-SAE gains over Vanilla SAE across the 26 first-letter concepts.
Figure 7: Complete parent-set induction improves structural reliability, and realignment reduces feature absorption. Left. Multi-parent PSV and PSV∩NR under Pairwise Assembly, Complete Parent-Set Induction, and Dictionary Optimization. Dictionary Optimization denotes the full post-induction dictionary update, including Structure-Guided SAE Training and Mixed-Topology-Guided Dictionary Refinement, followed by one Full Re-Induction on the updated dictionary to obtain the evaluated graph. The first two configurations share a fixed dictionary and candidate space. Set Induction denotes Complete Parent-Set Induction. Middle. Realignment coverage and resolution over 15,306 benchmark-diagnosed absorption cases. Of these, 12,607 receive an Absorption Realignment update and 10,817 are subsequently resolved; percentages use all diagnosed cases as the denominator. Right. Absorption measurements before and after realignment. Gray and green mark the before and after values, respectively; reductions are in percentage points.
Figure 8: Ground-truth mixed topology of the toy model. Gray, orange, and green nodes have zero, one, and two immediate parents, respectively. Child activation requires all immediate parents to be active.
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.
Sparse autoencoders (SAEs) have become central to mechanistic interpretability, decomposing transformer activations into monosemantic features. Yet existing analyses characterise features almost exclusively through top-activating token lists or decoder weight vectors, leaving the higher-order co-occurrence structure shared across features largely unexamined. We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph: nodes are the tokens most frequent near strong activations, and edges connect pairs that co-occur within local context windows. A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space. Applied as a proof of concept to features from a large SAE trained on GPT-2 Small and probed with a synthetic mixed-domain corpus, our clustering recovers heuristic motif families (punctuation-heavy patterns, language and script clusters, and code-like templates) that are not recovered by clustering on decoder cosine similarity. A token-histogram baseline achieves higher overall purity, so the contribution of the graph view is complementary rather than dominant: it surfaces structural relationships that token-frequency and decoder-weight views alone do not capture. Cluster assignments are stable across graph-construction hyperparameters and random seeds.
Ruben Fernandez-Boullon, Pablo Magariños-Docampo, Javier Perez-Robles
Sparse autoencoders (SAEs) extract millions of interpretable features from a language model, but flat feature inventories aren't very useful on their own. Domain concepts get mixed with generic and weakly grounded features, while related ideas are scattered across many units, and there's no way to understand relationships between features. We address this by first constructing a strict domain-specific concept universe from a large SAE inventory using contrastive activations and a multi-stage filtering process. Next, we build two aligned graph views on the filtered set: a co-occurrence graph for corpus-level conceptual structure, organized at multiple levels of granularity, and a transcoder-based mechanism graph that links source-layer and target-layer features through sparse latent pathways. Automated edge labeling then turns these graph views into readable knowledge graphs rather than unlabeled layouts. In a case study on a biology textbook, these graphs recover coherent chapter and subchapter-level structure, reveal concepts that bridge neighboring topics, and transform messy sentence-level activity containing thousands of features into compact, readable views that illustrate the model's local activity. Taken together, this reframes a flat SAE inventory as an internal knowledge graph that converts feature-level interpretability into a global map of model knowledge and enables audits of reasoning faithfulness.
John Winnicki, Abeynaya Gnanasekaran, Eric Darve
Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA