When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoders

Xiaozuo Shen; Yifei Cai; Tian Tan; Rui Ning; Chunsheng Xin; Hongyi Wu

cs.LGSep 28, 2026

When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoders

Authors: Xiaozuo Shen, Yifei Cai, Tian Tan, Rui Ning, Chunsheng Xin, Hongyi Wu

Organizations: University of Arizona · Iowa State University · Old Dominion University

Abstract

Sparse autoencoders (SAEs) expose interpretable features in large language model activations, yet existing structured SAEs impose single-parent trees or forests, while post-hoc graphs permit multiple parents but neither guide feature learning nor ensure reliable relation recovery. We introduce the Adaptive Graph Sparse Autoencoder (AG-SAE), a structure-guided training paradigm that treats each feature's complete parent set as an atomic structural hypothesis and lets evidence select zero, one, or multiple parents. By competing complete parent sets against null, subset, and alternative explanations, AG-SAE identifies jointly necessary multi-parent relations while rejecting redundant or spurious alternatives and verifying that each child contributes beyond its parents. The induced topology over SAE features then defines a differentiable structural loss that guides SAE training, while topology-guided refinement mitigates feature absorption and uses persistent reconstruction gaps exposed by the learned structure to initialize new features. The entire graph is then induced again from the revised dictionary by reassessing every feature's complete parent set, closing the dictionary-graph self-consistency cycle. Experiments demonstrate exact mixed-topology recovery in a controlled toy model, greater relational reliability and semantic validity than structured and post-hoc baselines on real LLM activations, and stronger feature-level causal interventions than conventional SAE features. AG-SAE thereby turns recovered mixed-topology feature structure into an unsupervised training signal that improves the dictionary, enables reliable feature organization beyond the topological limitations of trees, and exhibits stronger causal control beyond reconstruction.

Figures & tables

Figure 1: Overview of AG-SAE. The induced topology guides SAE training and dictionary refinement; Full Re-Induction closes the cycle, which repeats until the dictionary–graph pair stabilizes. This example is illustrative.

A. Toy-model evaluation Dictionary quality Structure recovery Method Test Feat. Exact PS /24

\uparrow

HN Rejected

R^{2}

\uparrow

/24

\uparrow

0/1/2-Parent /32

\uparrow

Vanilla SAE 1.00 24

8,(8/0/0)

—

\dagger

Matryoshka SAE 0.92 19

8,(8/0/0)

—

\dagger

Tree SAE 0.91 10

8,(6/2/0)

28 HSAE 0.99 23

10,(8/2/0)

25 AG-SAE (ours) 1.00 24

\mathbf{24},(\mathbf{8}/\mathbf{8}/\mathbf{8})

32 B. Final mixed-topology graph C. Refinement across cycles Metric Result Metric

G^{(0)}

G^{(2)}

G^{(5)}

Total features 30,720 Single-parent 2,343 5,142 5,421 Isolated 19,912 (64.82%) Multi-parent 782 1,637 1,670 Roots 3,715 (12.09%) Absorption realignment — 5,973 7,068 Single-parent 5,421 (17.65%) Residual completion — 161 12 Multi-parent 1,670 (5.44%) Relation gain

\uparrow

0.0456 0.0875 0.0912 Multi / parented 23.55% Graph change

\bm{d_{G}}

(%)

\downarrow

— 18.20% 3.06% Longest path 8 edges Feature change

\bm{d_{F}}

(%)

\downarrow

— 68.45% 9.25%

D. Real LLM validation Representation quality Recovered-structure reliability Global dictionary Single-parent (%)

\uparrow

Multi-parent (%)

\uparrow

Joint (%)

\uparrow

Method EV

\uparrow

\bm{L_{0}}

Abs.

\downarrow

AutoInt.

\uparrow

PSV NR SV PSV NR SV PSV

\cap

\cap

SV AG-SAE 0.670 50.13 9.16 86.44 99.75 97.00 76.88 97.00 91.88 75.00 69.56 Vanilla SAE Post-hoc 0.668 50.00 51.39 80.03 71.50 75.50 69.13 32.63 65.75 58.25 40.63 Matryoshka SAE (DAG) 0.680 50.00 12.71 81.02 60.38 66.25 30.38 35.38 57.00 35.00 25.44 Matryoshka SAE (Tree) 0.680 50.00 12.71 81.02 48.25 54.50 24.25 N/A 22.38 HSAE 0.665 50.00 10.24 79.90 92.88 92.13 56.88 N/A 51.13 Tree SAE 0.663

40.84^{\dagger}

5.11 75.14 24.63 24.13 23.25 N/A 18.75

Table 1: AG-SAE structure recovery in toy and real LLM settings. A. Dictionary quality and structure recovery on the ground-truth mixed-topology toy model. In A,

\dagger

denotes N/A for methods without a native parent graph; Vanilla SAE’s eight no-parent matches reflect the default empty parent set. B. Final AG-SAE feature graph on Gemma-2-2B layer-13 activations from MiniPile. Panels C–D use the same setting. Category percentages use the

30{,}718

active features (

99.99\%

30{,}720

); Multi / parented is the fraction of parented features with multiple parents. C. Dictionary and graph statistics across refinement cycles. D. Representation quality and relation validation across methods (Abs., Feature Absorption; AutoInt., AutoInterp). PSV, NR, and SV denote Predictive Structural Validity, Non-Redundancy, and Semantic Validity; Joint denotes their intersection.

\dagger

marks Tree SAE’s lower

L_{0}

from its native sparsification, which also gives its Abs. score a sparsity advantage.

Figure 2: Learned three-parent relation for appellant references. Left. Feature 7707 and its complete parent set in final AG-SAE trained on MiniPile Gemma-2-2B layer-13 activations. Feature IDs are global across dictionary banks. Right. Activation examples selected per feature from profiling data and report contexts. Orange shading shows per-token activation on the same raw scale.

Figure 3: Individual and joint parent coverage of feature 7707. Counts use

4{,}910{,}199

report token positions with frozen activation thresholds. The first three panels show individual parents; the rightmost requires all three parents to be active. Orange denotes parent-only activation, green coactivation with the child, and pink child activation with the parent condition unmet. Coverage uses the

484

child activations as its denominator. Panels use separate visual scales.

Figure 4: AG-SAE consolidates bidirectional causal control into its leading features. Left. Mean causal effects of adding and ablating the leading feature. Center. AG-SAE improves both intervention directions for

21/26

concepts. Right. The gain in causal yield per active feature is concentrated at Top-1 and narrows as additional features are included. Together, the three panels establish stronger magnitude, cross-concept consistency, and leading-feature concentration.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: AG-SAE refinement dynamics and joint stability. Results use Gemma-2-2B layer-13 activations from MiniPile. Left. Counts of single- and multi-parent features across graph generations, together with accepted Residual Completion updates between successive refinement cycles. Middle. Normalized feature-contribution change

d_{F}

on held-out activations and graph change

d_{G}

in feature membership and complete parent-set assignments between successive refinement cycles. Both decrease after

G^{(2)}

, reaching

9.25\%

and

3.06\%

, respectively, in the final transition. Right. Relation gain is the held-out fraction of residual representation explained by parents with the child withheld, averaged over all selected relations, increasing from

4.56\%

G^{(0)}

9.12\%

G^{(5)}

A. Coverage by child width
Width	Parented	Cov. %	Single	Multi	Multi %
$2{,}048$	422	20.61	218	204	48.34
$4{,}096$	1,197	29.22	847	350	29.24
$8{,}192$	2,118	25.85	1,636	482	22.76
$16{,}384$	3,354	20.47	2,720	634	18.90
B. Validity by multi-parent arity

Appendix

Table 2: Multi-parent topology spans all four dictionary banks, and Joint validity changes by only

0.75

percentage points between two- and three-parent sets. A. Cov. % is the number of parented features divided by bank width, and Multi % is the number of multi-parent features divided by the number of parented features. B. Structures denotes the total number of recovered relations in each arity cohort. Panel B applies the frozen report-only protocols of Table 1 to fixed arity-specific relation cohorts. PSV, NR, and SV are evaluated on the same relations, while Joint reports the percentage satisfying all three criteria. All rates are reported as percentages.

Figure 6: Dictionary scale organizes graph roles, while graph depth extends beyond the four bank levels. Left. Root and leaf shares within each dictionary bank. Roots fall from

27.29\%

5.94\%

of bank width, while leaves rise from

6.49\%

18.85\%

. Right. A feature’s graph depth is the maximum number of edges among all directed paths that begin at a root and end at that feature. The dashed line marks the maximum depth of three permitted by a strict progression from narrower to wider banks. The recovered graph places

394

features beyond this bound and reaches depth eight.

Comparison	Parent set	Held-out fit $\uparrow$
Complete learned set	$\{5963,14785,3242\}$	0.5497
Remove 5963	$\{14785,3242\}$	0.4543
Remove 14785	$\{5963,3242\}$	0.4409
Remove 3242	$\{5963,14785\}$	0.3460
Best single parent	$\{14785\}$	0.2452
Best matched alternative	$\{5295,14785,3242\}$	0.4627

Appendix

Table 3: Complete parent-set comparison for feature 7707. The fit score used in PSV is one minus the squared prediction error for the child’s decoder contribution, normalized by its total squared norm over the same

484

report positions where the child is active. Nonnegative coefficients are fitted separately for each parent set on a disjoint split and frozen for evaluation. The matched alternative is the strongest replacement set in the evaluated candidate pool. The null hypothesis has score

0

A. Dictionary Quality and Graph Composition
		Dictionary		Graph
Setting	Width	EV	$L_{0}$	Parented	Cov. %	Single %	Multi %	Root %	Leaf %
Gemma / PubMed	$2{,}048$	0.660	50.29	328	16.02	50.61	49.39	34.28	4.74
	$4{,}096$	0.686	50.12	1,082	26.42	70.15	29.85	23.07	7.08
	$8{,}192$	0.710	49.74	2,349	28.67	74.37	25.63	17.19	14.64
	$16{,}384$	0.727	50.36	3,700	22.58	79.35	20.65	4.72	19.46

Appendix

Table 4: AG-SAE dictionary quality, graph composition, and relation validity across models and datasets. Gemma/PubMed uses Gemma-2-2B layer 13 with four dictionary scales; Qwen/MiniPile uses Qwen3.5-2B-Base layer 18 with two scales and

49{,}152

features in total. A. EV and

L_{0}

are report-only measurements with graph routing disabled at each dictionary scale. Cov. % divides parented features by bank width. Single % and Multi % divide the respective feature counts by parented features; Root % and Leaf % divide their counts by bank width. Roots have children and no parents; leaves have parents and no children. B. Validation follows the frozen report-only protocols of Table 1 , Panel D. Structures denotes the total number of recovered relations in each cohort. PSV, NR, SV, and Joint are evaluated under the same frozen report-only protocol within each cohort, with Joint reporting simultaneous satisfaction of all three criteria.

Condition	Change	$95\%$ CI	Wins
Full AG-SAE	$\mathbf{+0.867}$	$\mathbf{[0.298,\,1.506]}$	$\mathbf{10/13}$
Normalized continuation	$-0.215$	$[-0.597,\,0.184]$	$3/13$
Free-scale continuation	$-0.160$	$[-0.546,\,0.235]$	$5/13$

Appendix

Table 5: Matched changes in bidirectional causal control under full AG-SAE training and two conventional SAE continuation controls.

Metric	Gain	$95\%$ CI	Wins	$p$ -value
Addition	$\mathbf{+0.573}$	$\mathbf{[0.308,\,0.893]}$	$\mathbf{24/26}$	$1.05\times 10^{-5}$
Ablation	$\mathbf{+0.685}$	$\mathbf{[0.260,\,1.240]}$	$\mathbf{21/26}$	$0.0025$
Bidirectional	$\mathbf{+0.674}$	$\mathbf{[0.281,\,1.210]}$	$\mathbf{21/26}$	$0.0025$
Top-1 concentration	$\mathbf{+0.681}$	$\mathbf{[0.259,\,1.241]}$	$\mathbf{20/26}$	$0.0094$

Appendix

Table 6: Paired AG-SAE gains over Vanilla SAE across the

26

first-letter concepts.

Figure 7: Complete parent-set induction improves structural reliability, and realignment reduces feature absorption. Left. Multi-parent PSV and

\mathrm{PSV}\cap\mathrm{NR}

under Pairwise Assembly, Complete Parent-Set Induction, and Dictionary Optimization. Dictionary Optimization denotes the full post-induction dictionary update, including Structure-Guided SAE Training and Mixed-Topology-Guided Dictionary Refinement, followed by one Full Re-Induction on the updated dictionary to obtain the evaluated graph. The first two configurations share a fixed dictionary and candidate space. Set Induction denotes Complete Parent-Set Induction. Middle. Realignment coverage and resolution over

15{,}306

benchmark-diagnosed absorption cases. Of these,

12{,}607

receive an Absorption Realignment update and

10{,}817

are subsequently resolved; percentages use all diagnosed cases as the denominator. Right. Absorption measurements before and after realignment. Gray and green mark the before and after values, respectively; reductions are in percentage points.

Figure 8: Ground-truth mixed topology of the toy model. Gray, orange, and green nodes have zero, one, and two immediate parents, respectively. Child activation requires all immediate parents to be active.


Hyperparameter	Gemma	Qwen

$\lambda_{\mathrm{cov}}$	$0.006$	$0.012$
$\lambda_{\mathrm{inn}}$	$0.0015$	$0.003$
Initial $\lambda_{\mathcal{I}}$	$0.05$	$0.015$
LR, cycles 1–5	$(2,1.5,1,0.6,0.3)\times 10^{-4}$	$(3,2.2,1.5,0.9,0.6)\times 10^{-4}$

Appendix

Table 7: Model-specific AG-SAE hyperparameters.

Sparse Autoencoders

Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.

See all 218 papers

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features
May 7, 2026Ruben Fernandez-Boullon, Pablo Magariños-Docampo, Javier Perez-RoblesTransformer Interpretability Sparse Autoencoders
University of Vigo
Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features
Apr 26, 2026John Winnicki, Abeynaya Gnanasekaran, Eric DarveKG Construction LLM Interpretability
Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA

Aug 22, 2025J/K move · Enter open · S save

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

David Chanin, Adrià Garriga-Alonso

University College London · MATS Research

Activation Sparsity LLM Interpretability Representation Disentanglement Sparse Autoencoders Sparse Coding Hyperparameter Optimization

Open paper AskPDF arXiv

Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.

Abstract

Sparse Autoencoders

Explore similar work

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features

Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features