Comparing sparse autoencoders (SAEs) requires connecting feature correspondence, differences between whole dictionaries, and uncertainty from finite evaluation data. We develop a framework that connects these tasks through an energy-weighted optimal-transport plan under a specified similarity geometry. The same plan defines a dictionary-level discrepancy, decomposes it into latent contributions, and quantifies how each latent's mass is distributed across counterparts. We characterize relationships between decoder, activation, and first-order causal similarities; idealized models clarify the interpretation of splitting and absorption. For fixed trained SAEs, we derive joint asymptotic inference for transport costs and fractions, accounting for activation similarities and both energy marginals estimated from the same documents. A joint document bootstrap gives asymptotically valid simultaneous confidence bands. Experiments across four language models locate cross-seed discrepancy and separate context-dependent changes.
Figures & tables
Quantity and source of variation
Estimate
95% interval
Δ , 5 seeds, documents
0.02639
[0.02277,0.03002]
Δ , 5 seeds, jackknife
0.02636
[0.01585,0.03687]
Δ , 20 seeds, documents and seeds
0.02612
[0.02173,0.03050]
Δw (weights)
0.01680
[0.01534,0.01826]
Δc (cost matrix)
0.00956
[0.00716,0.01195]
Table 1: Estimates and confidence intervals for TinyStories-1M at ε=0.30 . The first row uses the original run; the others use the second run. Means are over ten pairs for five seeds and 190 pairs for 20 seeds. Intervals for document sampling and for Δw,Δc condition on the trained SAEs. Seed variation is estimated by a delete-one-seed jackknife, using the t4 distribution for five seeds. The 20-seed interval adds the document and seed variance estimates.
Figure 1: Transport costs and fractions on TinyStories-1M. (a) Mean costs over ten pairs of five SAEs in the original run at ε=0.30 , with pointwise 95% confidence intervals conditional on the trained SAEs. (b) Decomposition of the difference 0.02636 in the second run: 0.01680 (64%) for weights and 0.00956 (36%) for the cost matrix. We replace the animals weights and costs by the family values in both orders, recompute the plan after each replacement, and average the changes as in ( 13 ). Percentages are fractions of the difference in mean costs. (c) Family documents at ε=0.10 . Bar heights are proportional to a21 and b13 ; the band represents their common plan entry P21,13 . Gray portions are mass assigned to other latents. Estimates and pointwise 95% confidence intervals use 384 held-out documents independent of selection. Proportions use rounded estimates.
Figure 2: Contributions of selected latents to Δ at ε=0.10 . Selection uses half of the documents; estimates and 95% confidence intervals use the other half. Labels give the seed, latent index, and top tokens.
Figure 3: Fractions of tuning energy and evaluation cost for groups of latents at ε=0.10 . Diamonds show each group’s approximate fraction of total tuning energy. Colored segments show the minimum and maximum fractions of evaluation cost across six combinations of model and width, after averaging over five SAEs in each combination. These ranges are not confidence intervals. Groups are defined using independent neutral tuning documents. Cost contributions use the plans for the full dictionaries and are averaged over both topics.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Seed
101
102
103
104
105
101
0.38008
1.02088
1.05034
1.06093
1.05751
102
1.02088
0.38572
0.97432
1.02648
1.01152
103
1.05034
0.97432
0.38686
1.01095
1.01947
104
1.06093
1.02648
1.01095
0.38440
1.04600
105
1.05751
1.01152
1.01947
1.04600
0.38763
Appendix
Table 2: Animals transport-cost matrix at the primary regularization.
Seed
101
102
103
104
105
101
0.37637
1.03810
1.07426
1.08313
1.08056
102
1.03810
0.38514
1.00246
1.06090
1.03596
103
1.07426
1.00246
0.38730
1.04136
1.04432
104
1.08313
1.06090
1.04136
0.38445
1.08132
105
1.08056
1.03596
1.04432
1.08132
0.38644
Appendix
Table 3: Family transport-cost matrix at the primary regularization.
ε
Δ
95% interval
diagonal mass
∑jmaxkPjk
∑jajexpH(Tj⋅)
0.03
0.02144
[0.01858,0.02429]
1.000
0.674
2.7
0.05
0.02161
[0.01873,0.02449]
1.000
0.627
3.3
0.10
0.02202
[0.01903,0.02501]
0.998
0.515
5.7
0.20
0.02370
[0.02024,0.02715]
0.933
0.344
14.3
0.30
0.02636
[0.02273,0.02998]
0.688
0.242
24.0
0.60
0.02457
[0.02186,0.02729]
0.208
0.134
39.3
Appendix
Table 4: Contrast over the regularization grid. The self-plan diagonal mass is averaged over seeds and topics. For cross-seed plans, the last two columns give the sum of row maxima and the weighted mean of the exponential row entropy, where H(Tj⋅)=−∑kTjklogTjk .
Score
family
animals
Difference
Bootstrap 95%
z
Decoder, Hungarian mean cosine
0.4080
0.4080
+0.0000
—
—
Decoder, nearest-neighbour cosine
0.4350
0.4350
+0.0000
—
—
Decoder, shared fraction ( ≥0.7 )
0.0109
0.0109
+0.0000
—
—
Activation ρ , Hungarian
0.5697
0.5701
−0.0005
[−0.0013,0.0003]
1.1
Activation ρ , nearest neighbour
0.6245
0.6262
−0.0017
[−0.0024,−0.0008]
3.6
Activation ρ , shared fraction ( ≥0.7 )
0.2867
0.3086
−0.0219
[−0.0273,−0.0156]
6.9
Appendix
Table 5: Seed-agreement scores on the same SAEs and topics. Higher similarity scores and lower OT costs indicate closer dictionaries.
Seed
Latent
Contribution
95% interval
Family words
Animal words
Top tokens
105
19
+0.00153
[0.00139,0.00167]
0.15
0.03
to / mom / in / my
103
43
−0.00060
[−0.00089,−0.00031]
0.01
0.03
. / , / to / \n
103
33
+0.00090
[0.00081,0.00099]
0.17
0.00
and / mom / he / had
104
21
+0.00083
[0.00078,0.00088]
0.81
0.00
mom / my / dad / Mom
101
46
−0.00092
[−0.00107,−0.00076]
0.01
0.14
. / it / bird / dog
103
23
−0.00087
[−0.00100,−0.00074]
0.01
0.03
a / upon / was / .
Appendix
Table 6: Largest latent contributions to Δ at ε=0.10 .
ε
Seeds
Δ
Seed s.e.
Jackknife t interval
Documents + seeds
0.10
5
0.02202
0.00388
[0.01126,0.03278]
—
0.10
20
0.02068
0.00141
[0.01773,0.02364]
—
0.30
5
0.02636
0.00379
[0.01585,0.03687]
—
0.30
20
0.02612
0.00127
[0.02346,0.02877]
[0.02173,0.03050]
Appendix
Table 7: Seed-level uncertainty for the context contrast.
Category
Documents
Mg
95% interval
animals
768
1.02786
[1.02502,1.03070]
nature
768
1.03310
[1.03068,1.03552]
toys
768
1.03584
[1.03346,1.03822]
food
768
1.03732
[1.03485,1.03980]
neutral
1536
1.04494
[1.04310,1.04678]
family
768
1.05422
[1.05197,1.05646]
Appendix
Table 8: Mean transport cost by document category ( ε=0.30 ).
Target
m
Coverage
Wilson 95%
Mean s.e./s.d.
Mfamily
192
0.945
[0.904,0.969]
1.04
Mfamily
384
0.970
[0.936,0.986]
1.07
Mfamily
768
0.935
[0.892,0.962]
0.94
Manimals
192
0.950
[0.910,0.973]
1.01
Manimals
384
0.970
[0.936,0.986]
1.05
Manimals
768
0.945
[0.904,0.969]
0.98
Appendix
Table 9: Empirical coverage of 95% intervals ( ε=0.30 ).
N
Mean plan error
Mean radius
Coverage
64
0.07922
0.12017
0.965
256
0.04081
0.06048
0.955
1024
0.02024
0.03019
0.970
Appendix
Table 10: Regular finite-support regime: full-plan L1 error and 95% document-bootstrap radius. Coverage is the fraction of 200 independent document samples for which the confidence region contains the population plan.
Width
ε
Δ
95% interval
self-plan diagonal
3072
0.1
−0.0003
[−0.0021,0.0016]
0.81
3072
0.2
+0.0012
[−0.0024,0.0048]
0.64
3072
0.3
+0.0056
[0.0007,0.0105]
0.34
3072
0.6
+0.0084
[0.0046,0.0123]
0.13
6144
0.1
+0.0050
[0.0033,0.0067]
0.77
6144
0.2
+0.0080
[0.0042,0.0118]
0.58
Appendix
Table 11: GPT-2 context contrast by width and regularization.
Model
Width
Tuning FVU
Self diagonal
Δ
Positive pairs
Fraction tests
TinyStories-1M
64
0.143 – 0.147
1.00/0.69
+0.0220
10/10
91/120
TinyStories-33M
3072
0.180
0.76/0.36
−0.0007
4/10
55/120
TinyStories-33M
6144
0.151 – 0.152
0.73/0.37
+0.0021
8/10
73/120
GPT-2 small
3072
0.088 – 0.089
0.81/0.34
−0.0003
5/10
70/120
GPT-2 small
6144
0.073
0.77/0.30
+0.0050
10/10
62/120
Pythia-160M
3072
0.018
0.29/0.15
−0.0026
3/10
69/120
Appendix
Table 12: Five-seed comparisons on identical evaluation topics. Self-plan diagonal masses are shown at ε=0.10/0.30 ; all other transport quantities use ε=0.10 . Positive pairs counts the seed pairs with Δ>0 . Fraction tests counts transported-fraction differences between topics significant after Bonferroni correction within the 120 comparisons for that setting. FVU and contrast values are rounded.
Model
Δ , ten seeds
95% interval
TinyStories-33M
+0.0004
[−0.0023,+0.0032]
GPT-2 small
+0.0011
[−0.0014,+0.0035]
Pythia-160M
−0.0016
[−0.0049,+0.0016]
Appendix
Table 13: Ten-seed context contrasts at width 3072 and ε=0.10 . The analysis includes variation between training seeds, assessed by deleting one seed at a time; the GPT-2 interval also includes document variation as described in Appendix D.4 .
Model
Width
Original Δ
Word positions excluded
TinyStories-1M
64
+0.0220
+0.0179
TinyStories-33M
3072
−0.0007
−0.0011
TinyStories-33M
6144
+0.0021
+0.0022
GPT-2 small
3072
−0.0003
−0.0005
GPT-2 small
6144
+0.0050
+0.0032
Pythia-160M
3072
−0.0026
−0.0027
Appendix
Table 14: Effect of excluding topic-word positions, using the five-seed runs at ε=0.10 . The TinyStories-1M value retains approximately 81% of the contrast; the 78% figure in the main text uses ε=0.30 .
Settings
SAEs
High-energy group cost share / energy share
Low-energy group cost share / energy share
TinyStories-1M, width 64
5
0.84 – 0.89
1.3 – 1.4
Three larger models, two widths each
30
0.11 – 0.68
2.4 – 4.5
Appendix
Table 15: Descriptive cost-share to energy-share ratios across SAEs. Groups are selected on neutral tuning documents; cost is evaluated at ε=0.10 . Ratios below one indicate less cost per unit energy than the dictionary average.