Learning a Unified Cross-Model Semantic Dictionary via Gated Bottleneck Sparse Autoencoders
Authors: Jinchi Zhu, Tie Luo
Organizations: Department of Computer Science, University of Kentucky, Lexington, KY 40506 · Department of Electrical and Computer Engineering & Department of Computer Science, University of Kentucky, Lexington, KY 40506
Foundation models pretrained on similar data distributions tend to learn overlapping semantic structure, but their representations are distinct due to different model architecture, pretraining objectives and protocols. This prevents unified interpretability and cross-model feature reuse. Existing approaches extract interpretable concepts from individual models and align these concepts in a shared coordinate system using cross-reconstruction objectives, but the alignment only applies to latent coordinates while the concepts themselves remain model-specific and cannot be shared or reused. This paper introduces the Gated Bottleneck Sparse Autoencoder (GB-SAE), which casts unified cross-model interpretability as a representation factorization problem and solves it by simultaneously learning a unified shared dictionary and multiple private model-specific dictionaries via a gated bottleneck sparse autoencoder framework. The shared dictionary encodes the cross-model common semantic structure in unified concepts that are reusable across all participating models (and beyond), while each private dictionary comprises residual concepts that capture model-specific biases. We evaluate GB-SAE with distinct foundation models including ViT, SigLIP, DINOv2, and ConvNeXt. GB-SAE demonstrates superior reconstruction capability, unified and faithful interpretability, strong reusability and generalization to unseen models and downstream tasks. The code is available at https://github.com/trustAIoT/GB-SAE_NeurIPS.
Figures & tables
Figure 1: Overall architecture of GB-SAE. Features from multiple frozen backbone models are projected into a common space and encoded by a shared sparse encoder. A global learnable gate partitions latent dimensions into shared and private branches, yielding a model-agnostic semantic dictionary and model-specific residual dictionaries. The illustrated routing is shown for a single token; in practice, the gate is a global vector broadcast across tokens.
Model
R2 -total ↑
R2 -sh ↑
R2 -pr ↓
R2 -res ↑
ViT
0.56/0.56
0.52/0.52
0.07/0.07
0.09/0.09
SigLIP
0.58/0.58
0.52/0.52
0.12/0.12
0.12/0.12
DinoV2
0.65/0.65
0.53/0.59
0.11/0.16
0.24/0.15
Mean
0.60/0.60
0.53/0.55
0.10/0.12
0.15/0.12
Table 1: (a) Reconstruction performance of GB-SAE on the ImageNet validation set, reported for both projected and raw representations (projected / raw). (b) Reconstruction performance of baselines, reported as mean token-level R2 (same as in table (a)), where R2 -self and R2 -cross approximately correspond to GB-SAE’s R2 -total and R2 -sh, respectively.
Figure 2: Visualization of shared semantic concepts discovered by GB-SAE across pretrained foundation models. The left panel shows four local concepts, each depicted by six semantically consistent regions across six different images. The right panel shows four global concepts, each depicted within six images; heatmaps are omitted as global concepts are activated throughout nearly the whole image.
Figure 3: Comprehensiveness (higher is better) and Sufficiency (lower is better) as a function of the number of top-ranked concepts k . The bar charts report prediction agreement at k=10 on their predicted class y^ : Pr(y^rem=y^full) after removal (lower is better) and Pr(y^keep=y^full) after retention (higher is better).
Figure 5
Source
Projected ↑
USAE ↑
SPARC ↑
GB-SAE ↑
ViT
0.85/0.96
0.82/0.96
0.84/0.95
-
SigLIP
0.75/0.91
0.69/0.91
0.77/0.93
-
DinoV2
0.69/0.89
0.66/0.89
0.75/0.91
-
Mean
0.76/0.92
0.72/0.92
0.78/0.93
0.86/0.97
Table 2: Downstream classification accuracy (Acc@1/Acc@5) on representations extracted by different methods.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
R2 -total ↓
R2 -sh ↓
R2 -pr ↓
R2 -res ↓
ViT
0.018/0.018
0.023/0.023
0.012/0.012
0.006/0.006
SigLIP
0.019/0.019
0.021/0.021
0.004/0.004
0.009/0.009
DinoV2
0.021/0.021
0.031/0.020
0.017/0.010
0.030/0.019
Mean
0.019/0.019
0.025/0.021
0.011/0.009
0.015/0.011
Appendix
Table 3: Standard deviation of reconstruction evaluation on the validation set. Each entry is reported as projected / raw R2 standard deviation.
Metric
R@1 ↓
R@5 ↓
R@10 ↓
R@50 ↓
PROJ
0.000/0.001
0.000/0.001
0.000/0.001
0.000/0.002
SHARED
0.010/0.023
0.023/0.038
0.029/0.040
0.022/0.038
PRIVATE
0.000/0.001
0.000/0.001
0.000/0.002
0.001/0.008
Appendix
Table 4: Standard deviation of retrieval performance on the validation set. Each entry is reported as Instance / Category retrieval standard deviation.
Model
R2 -total ↑
R2 -sh ↑
R2 -pr
R2 -res
ViT
0.58/0.58
0.55/0.55
0.08/0.08
0.06/0.06
ConvNeXt
0.66/0.66
0.63/0.63
0.06/0.06
0.08/0.08
DinoV2
0.64/0.65
0.56/0.59
0.19/0.21
0.18/0.14
Mean
0.63/0.63
0.58/0.59
0.11/0.12
0.11/0.09
Appendix
Table 5: Reconstruction evaluation of GB-SAE with heterogeneous architectures. Each entry is reported as projected / raw R2 .
Figure 6: GB-SAE performance on cross-model feature retrieval with heterogeneous model combinations. With the 10% validation subset, the expected instance-level random Recall@ k baseline is approximately 10 × that of Fig. 5 .
Model
R2 -total ↑
R2 -sh ↑
R2 -pr
R2 -res
ViT
0.57/0.57
0.52/0.52
0.11/0.11
0.11/0.11
SigLIP
0.58/0.58
0.51/0.51
0.21/0.21
0.15/0.15
DINOv2
0.65/0.66
0.52/0.58
0.21/0.24
0.27/0.20
ConvNeXt
0.65/0.65
0.60/0.60
0.06/0.06
0.13/0.13
Mean
0.62/0.62
0.54/0.55
0.15/0.16
0.16/0.14
Appendix
Table 6: Reconstruction evaluation of GB-SAE under Configuration III. Each entry is reported as projected / raw R2 .
Figure 7: Cross-model feature retrieval performance under Configuration III. We report Recall@ k at the instance and category levels. With the 10% validation subset, the instance-level random baseline is approximately 10 × that of Fig. 5 .
Figure 8: Visualization of model-specific private concepts. Unlike shared atoms, private concepts reflect architecture- and training-dependent inductive biases, exhibiting stronger locality and model-specific activation patterns.
Figure 9: GB-SAE loss-weight ablation study. Figure (a) shows the reconstruction results under different configurations, while Figure (b) reports the corresponding alignment results.
Metric
Projected
USAE
SPARC
GB-SAE
Instance R@1
0.96
0.91
0.47
0.99
Category R@1
0.48
0.44
0.52
0.57
Category R@5
0.75
0.72
0.74
0.81
Category mAP
0.39
0.37
0.44
0.48
Appendix
Table 7: Downstream cross-model retrieval performance using four frozen image representations with trainable linear retrieval heads; best result per metric is bold. Instance R@1 measures whether the same image is ranked first. Category R@K removes the paired same image and measures whether another image of the same class appears in the top K . For each query, Category AP averages the precision at every rank containing a same-class image; mAP is the mean AP over all valid queries, so it evaluates the quality of the full ranking rather than only the first hit.
Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. · Department of Sociology, Faculty of Social Sciences, Eötvös Loránd University, Budapest, Hungary.