Learning a Unified Cross-Model Semantic Dictionary via Gated Bottleneck Sparse Autoencoders
Authors: Jinchi Zhu, Tie Luo
Organizations: Department of Computer Science, University of Kentucky, Lexington, KY 40506 · Department of Electrical and Computer Engineering & Department of Computer Science, University of Kentucky, Lexington, KY 40506
Foundation models pretrained on similar data distributions tend to learn overlapping semantic structure, but their representations are distinct due to different model architecture, pretraining objectives and protocols. This prevents unified interpretability and cross-model feature reuse. Existing approaches extract interpretable concepts from individual models and align these concepts in a shared coordinate system using cross-reconstruction objectives, but the alignment only applies to latent coordinates while the concepts themselves remain model-specific and cannot be shared or reused. This paper introduces the Gated Bottleneck Sparse Autoencoder (GB-SAE), which casts unified cross-model interpretability as a representation factorization problem and solves it by simultaneously learning a unified shared dictionary and multiple private model-specific dictionaries via a gated bottleneck sparse autoencoder framework. The shared dictionary encodes the cross-model common semantic structure in unified concepts that are reusable across all participating models (and beyond), while each private dictionary comprises residual concepts that capture model-specific biases. We evaluate GB-SAE with distinct foundation models including ViT, SigLIP, DINOv2, and ConvNeXt. GB-SAE demonstrates superior reconstruction capability, unified and faithful interpretability, strong reusability and generalization to unseen models and downstream tasks. The code is available at https://github.com/trustAIoT/GB-SAE_NeurIPS.
Figures & tables
Figure 1: Overall architecture of GB-SAE. Features from multiple frozen backbone models are projected into a common space and encoded by a shared sparse encoder. A global learnable gate partitions latent dimensions into shared and private branches, yielding a model-agnostic semantic dictionary and model-specific residual dictionaries. The illustrated routing is shown for a single token; in practice, the gate is a global vector broadcast across tokens.
Model
R2 -total ↑
R2 -sh ↑
R2 -pr ↓
R2 -res ↑
ViT
0.56/0.56
0.52/0.52
0.07/0.07
0.09/0.09
SigLIP
0.58/0.58
0.52/0.52
0.12/0.12
0.12/0.12
DinoV2
0.65/0.65
0.53/0.59
0.11/0.16
0.24/0.15
Mean
0.60/0.60
0.53/0.55
0.10/0.12
0.15/0.12
Table 1: (a) Reconstruction performance of GB-SAE on the ImageNet validation set, reported for both projected and raw representations (projected / raw). (b) Reconstruction performance of baselines, reported as mean token-level R2 (same as in table (a)), where R2 -self and R2 -cross approximately correspond to GB-SAE’s R2 -total and R2 -sh, respectively.
Figure 2: Visualization of shared semantic concepts discovered by GB-SAE across pretrained foundation models. The left panel shows four local concepts, each depicted by six semantically consistent regions across six different images. The right panel shows four global concepts, each depicted within six images; heatmaps are omitted as global concepts are activated throughout nearly the whole image.
Figure 3: Comprehensiveness (higher is better) and Sufficiency (lower is better) as a function of the number of top-ranked concepts k . The bar charts report prediction agreement at k=10 on their predicted class y^ : Pr(y^rem=y^full) after removal (lower is better) and Pr(y^keep=y^full) after retention (higher is better).
Figure 5
Source
Projected ↑
USAE ↑
SPARC ↑
GB-SAE ↑
ViT
0.85/0.96
0.82/0.96
0.84/0.95
-
SigLIP
0.75/0.91
0.69/0.91
0.77/0.93
-
DinoV2
0.69/0.89
0.66/0.89
0.75/0.91
-
Mean
0.76/0.92
0.72/0.92
0.78/0.93
0.86/0.97
Table 2: Downstream classification accuracy (Acc@1/Acc@5) on representations extracted by different methods.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
R2 -total ↓
R2 -sh ↓
R2 -pr ↓
R2 -res ↓
ViT
0.018/0.018
0.023/0.023
0.012/0.012
0.006/0.006
SigLIP
0.019/0.019
0.021/0.021
0.004/0.004
0.009/0.009
DinoV2
0.021/0.021
0.031/0.020
0.017/0.010
0.030/0.019
Mean
0.019/0.019
0.025/0.021
0.011/0.009
0.015/0.011
Appendix
Table 3: Standard deviation of reconstruction evaluation on the validation set. Each entry is reported as projected / raw R2 standard deviation.
Metric
R@1 ↓
R@5 ↓
R@10 ↓
R@50 ↓
PROJ
0.000/0.001
0.000/0.001
0.000/0.001
0.000/0.002
SHARED
0.010/0.023
0.023/0.038
0.029/0.040
0.022/0.038
PRIVATE
0.000/0.001
0.000/0.001
0.000/0.002
0.001/0.008
Appendix
Table 4: Standard deviation of retrieval performance on the validation set. Each entry is reported as Instance / Category retrieval standard deviation.
Model
R2 -total ↑
R2 -sh ↑
R2 -pr
R2 -res
ViT
0.58/0.58
0.55/0.55
0.08/0.08
0.06/0.06
ConvNeXt
0.66/0.66
0.63/0.63
0.06/0.06
0.08/0.08
DinoV2
0.64/0.65
0.56/0.59
0.19/0.21
0.18/0.14
Mean
0.63/0.63
0.58/0.59
0.11/0.12
0.11/0.09
Appendix
Table 5: Reconstruction evaluation of GB-SAE with heterogeneous architectures. Each entry is reported as projected / raw R2 .
Figure 6: GB-SAE performance on cross-model feature retrieval with heterogeneous model combinations. With the 10% validation subset, the expected instance-level random Recall@ k baseline is approximately 10 × that of Fig. 5 .
Model
R2 -total ↑
R2 -sh ↑
R2 -pr
R2 -res
ViT
0.57/0.57
0.52/0.52
0.11/0.11
0.11/0.11
SigLIP
0.58/0.58
0.51/0.51
0.21/0.21
0.15/0.15
DINOv2
0.65/0.66
0.52/0.58
0.21/0.24
0.27/0.20
ConvNeXt
0.65/0.65
0.60/0.60
0.06/0.06
0.13/0.13
Mean
0.62/0.62
0.54/0.55
0.15/0.16
0.16/0.14
Appendix
Table 6: Reconstruction evaluation of GB-SAE under Configuration III. Each entry is reported as projected / raw R2 .
Figure 7: Cross-model feature retrieval performance under Configuration III. We report Recall@ k at the instance and category levels. With the 10% validation subset, the instance-level random baseline is approximately 10 × that of Fig. 5 .
Figure 8: Visualization of model-specific private concepts. Unlike shared atoms, private concepts reflect architecture- and training-dependent inductive biases, exhibiting stronger locality and model-specific activation patterns.
Figure 9: GB-SAE loss-weight ablation study. Figure (a) shows the reconstruction results under different configurations, while Figure (b) reports the corresponding alignment results.
Metric
Projected
USAE
SPARC
GB-SAE
Instance R@1
0.96
0.91
0.47
0.99
Category R@1
0.48
0.44
0.52
0.57
Category R@5
0.75
0.72
0.74
0.81
Category mAP
0.39
0.37
0.44
0.48
Appendix
Table 7: Downstream cross-model retrieval performance using four frozen image representations with trainable linear retrieval heads; best result per metric is bold. Instance R@1 measures whether the same image is ranked first. Category R@K removes the paired same image and measures whether another image of the same class appears in the top K . For each query, Category AP averages the precision at every rank containing a same-class image; mAP is the mean AP over all valid queries, so it evaluates the quality of the full ranking rather than only the first hit.
Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.
Daniil Ognev, Célian Vasson, Lijie Hu +2
MBZUAI · Sorbonne Université · Tohoku University +1
Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.
We present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) for extracting cross-seed universal features from independently trained BERT models. Cross-seed feature universality is a fundamental challenge in mechanistic interpretability: because dictionary learning is non-convex, independently trained networks learn misaligned feature spaces, so apparently identical features may differ by random initialization. We address this by computing an orthogonal Procrustes rotation between seeds' activation spaces before joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss based on previous SAE literature. Evaluating on five independent seed pairs (ten BERT models) across three benchmark datasets (SST-2, Stanford Politeness, TweetEval Emotion), our full pipeline produces more universal features (Pearson r ≥ 0.70 across seeds) than post-hoc alignment baselines on all three datasets. A minimal qualitative analysis confirms that high-universality features encode interpretable sociolinguistic patterns.
Bendegúz Váradi, Zoltán Kmetty
Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. · Department of Sociology, Faculty of Social Sciences, Eötvös Loránd University, Budapest, Hungary.