SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training

Qi Zhang; Yifei Wang; Xiaohan Wang; Jiajun Chai; Guojun Yin; Wei Lin; Yisen Wang

cs.AIMar 3, 2026

SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training

Authors: Qi Zhang, Yifei Wang, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, Yisen Wang

Organizations: State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Meituan · Amazon AGI SF Lab · Institute for Artificial Intelligence, Peking University

Abstract

In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre-training, their effectiveness in downstream applications also depends critically on the post-training process, which adapts models to task-specific data and objectives. However, this process inevitably introduces model shifts that can influence performance in different domains, and how such shifts transfer remains poorly understood. To open up the black box, we propose the SAE-based Transferability Score (STS), a new metric that leverages sparse autoencoders (SAEs) to forecast post-training transferability. Taking supervised fine-tuning as an example, STS identifies shifted dimensions in SAE representations and calculates their correlations with downstream domains, enabling reliable estimation of transferability before fine-tuning. Extensive experiments across multiple models and domains show that STS accurately predicts the transferability of supervised fine-tuning, achieving Pearson correlation coefficients above 0.7 with actual performance changes. Beyond this, we take an initial step toward extending STS to reinforcement learning. We believe that STS can serve as an {\color{black} interpretable} tool for guiding post-training strategies in LLMs. Code is available at https://github.com/PKU-ML/STS.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis of feature shifts induced by supervised fine-tuning (SFT). We fine-tune Qwen2.5-7B-Instruct on the LIMO (a mathematical reasoning dataset) and examine shifts of SAE features on the residual stream at layer 25. Figure (a) shows the distribution of shift magnitudes while Figure (b) shows accuracy on Math-LightEval when progressively zeroing the dimensions with the largest shifts. The results indicate that SFT primarily affects a small subset of SAE dimensions tied to specific model capabilities.

Panel of Figure 2 — Figure 2: Overlap between estimated dimensions and the training task. Figure (a) demonstrates that the SAE shifted dimensions predicted by ICL substantially overlap with the actual shifted dimensions identified after SFT, whereas applying the same method directly on raw dimensions is less effective. Figure (b) further shows that raw model dimensions, prior to applying SAE, are influenced more uniformly by the SFT process, thereby limiting the ability to identify crucial shifted dimensions.

Panel of Figure 3 — Figure 3: The Pearson correlation ( $\rho$ ) between STS and actual absolute performance shifts on MMLU-Pro induced by SFT on LIMO. Each experiment is repeated three times, and we report the mean and standard deviation of $\rho$ ; the fitted line shown corresponds to one of the runs. We extract SAE features from Llama3-8B-Instruct, Qwen2.5-7B-Instruct, and Gemma2-9B-Instruct. During the evaluation process, we select four MMLU-Pro domains with the largest and smallest performance shifts under SFT. The detailed performance shifts can be found in Appendix A.

Training Domain	Correlations between Actual Performance Shifts and STS ${}_{\text{ICL}}$	Overlap between Top 100 Estimated and Actual Shifted Dims
Code	0.77 $\pm$ 0.01	62
Health	0.71 $\pm$ 0.02	57

Table 1: Verification on a code dataset ( Verifiable-Coding-Problems-Python-10k-Dataset ) and a clinical reasoning dataset ( CoT_Reasoning_Mens_Mental_Health ). The Qwen-2.5-7B-Instruct model is trained on each dataset, and we evaluate: (1) the correlation between ICL feature shifts and SFT, and (2) the correlation between actual performance shifts and our proposed metric.

Panel of Figure 4 — Figure 4: Ablation studies on the implementation of our metric. We evaluate (a) SAEs with varying hidden dimensions in the representation space, (b) SAEs trained on different layers of the pre-trained model, (c) different ranges of top-shifted dimensions, (d) different sparsity in SAE representations, e) the comparison between STS and directly using activations.

Panel of Figure 5 — Figure 5: Comparison of data mixture strategies in the SFT process. We focus on the domains with the largest (engineering) and smallest (law) performance shifts induced by SFT of Qwen2.5-7B-Instruct on LIMO. In total, 220 extra examples from a mixture of engineering and law data are added. Figure (a) reports engineering performance with varying amounts of engineering data, while Figure (b) reports law performance with varying amounts of law data. Figure (c) compares the downstream performance without additional data and with additional data mixed according to the ratio of their corresponding STS values.

Panel of Figure 6 — Figure 6: The explorations on extending STS metric to the RL scenarios. We optimize Qwen2.5-7B-Instruct on Math-LightEval using GRPO. Figure (a) shows that the identified dimensions in RL show smaller overlap with the actual shifted dimensions compared to SFT, which limits the accuracy of the metric. Figure (b) shows that when the estimated shifted dimensions are replaced with the actual ones, the correlation between STS and downstream performance changes becomes much stronger.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	engineering	physics	chemistry	law
Performance Change (%)	-11.97	-5.7	-5.39	-3.72
	philosophy	other	health	computer science
Performance Change (%)	-2.61	-2.49	-1.1	-0.97
	economics	math	psychology	biology
Performance Change (%)	-0.95	-0.89	-0.63	-0.42
	history	business

Appendix

Table 2: Performance change on different domains of MMLU-Pro (Llama3-8B-Instruct fine-tuned on LIMO).

	engineering	chemistry	physics	computer science
Performance Change (%)	-9.8	-6.36	-5.24	-4.88
	business	health	math	economics
Performance Change (%)	-4.43	-2.93	-2.89	-2.61
	philosophy	biology	other	psychology
Performance Change (%)	-2.4	+1.53	-1.3	-0.37
	history	law

Appendix

Table 3: Performance change on different domains of MMLU-Pro (Qwen2.5-7B-Instruct fine-tuned on LIMO).

	engineering	computer science	other	law
Performance Change (%)	-9.29	-3.66	-2.05	-1.73
	health	math	business	history
Performance Change (%)	-1.47	+1.41	-1.27	-0.79
	economics	chemistrcy	biology	physics
Performance Change (%)	-0.71	-0.62	-0.56	-0.54
	psychology	philosophy

Appendix

Table 4: Performance change on different domains of MMLU-Pro (Gemma2-9B-Instruct fine-tuned on LIMO).

	Feature Activations	Representation Similarity between Downstream Domains and Training Domain	Representation Similarity between Models before and after SFT	STS
Uses Model after SFT	No	No	Yes	No
Coefficient	0.03	0.11	0.61	0.79

Appendix

Table 5: Correlation coefficient between actual performance shifts (Qwen2.5-7B-Instruct tuned on LIMO) and different baselines.

Metric / Model	LLaMA3-8B	Qwen2.5-7B	Gemma2-9B
$STS_{act}$	0.71 $\pm$ 0.01	0.90 $\pm$ 0.02	0.60 $\pm$ 0.03
$STS_{ICL}$	0.81 $\pm$ 0.01	0.78 $\pm$ 0.01	0.77 $\pm$ 0.01

Appendix

Table 6: The Pearson correlation coefficient between STS and actual performance shifts on MMLU-Pro induced by SFT on LIMO.

Selected Dimensions	50	100	200
	89%	93%	92%

Appendix

Table 7: The percentage of the estimated shifted dimensions that are explained as the math dimension.

Annotation	Math	Code
Recall	63%	7%

Appendix

Table 8: Proportion of 100 annotated dimensions recalled among the top 500 estimated shifted dimensions.

Metric / Model	LLaMA3-8B	Qwen2.5-7B	Gemma2-9B
$STS_{act}$	0.50 $\pm$ 0.01	0.80 $\pm$ 0.04	0.36 $\pm$ 0.03
$STS_{ICL}$	0.66 $\pm$ 0.01	0.61 $\pm$ 0.02	0.60 $\pm$ 0.02

Appendix

Table 9: R² between STS and actual performance shifts on MMLU-Pro induced by SFT on LIMO.

Before SFT	SFT on LIMO	SFT on STS-Guided Data
74.7	81.3 (+6.6)	80.1 (+5.4)

Appendix

Table 10: Test accuracy (%) of Qwen2.5-7B-Instruct before and after SFT on LIMO.

Figure 7: Selective Finetuning Different Models on Estimated Shifted Dimensions

Selected Dims	Before SFT	SFT on Random Dimensions	SFT on Estimated Shifted Dimensions
1000	74.7	74.2	75.6
3000	74.7	74.0	76.9

Appendix

Table 11: Test accuracy (%) of Qwen2.5-7B-Instruct after selectively fine-tuning.

$L_{0}$	13	22	37	63	109
Correlation	0.78	0.77	0.72	0.75	0.65

Appendix

Table 12: Pearson correlation coefficients between STS and actual performance shifts on MMLU-Pro induced by fine-tuning Gemma2-9B-Instruct on LIMO. STS is computed using SAEs with different

L_{0}

norms.

	Raw Feature Activations	SAE Feature Activations	STS
Correlation Coefficient	0.03	0.08	0.79

Appendix

Table 13: Comparison between STS and predictions based on model activations.

Task Transferability Estimation

Task transferability estimation predicts which models, datasets, tasks, or skills will help on a new target before costly adaptation.

See all 38 papers

Domain Restriction via Multi SAE Layer Transitions
May 12, 2026Elias Shaheen, Avi MendelsonLLM Interpretability OOD Detection
Technion – Israel Institute of Technology, Haifa, Israel.
Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
May 29, 2026Mikkel Godsk Jørgensen, Lars Kai HansenLanguage Model Steering LLM Interpretability
DTU Compute · Technical University of Denmark
What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track
Sep 14, 2026Lingheng Du, Yiming Tang, Xufeng Duan +1RL for Language Models Sparse Autoencoders
Peking University · National University of Singapore · The Chinese University of Hong Kong

May 12, 2026J/K move · Enter open · S save

Domain Restriction via Multi SAE Layer Transitions

Elias Shaheen, Avi Mendelson

Technion – Israel Institute of Technology, Haifa, Israel.

LLM Interpretability OOD Detection

Open paper AskPDF arXiv

The general-purpose nature of Large Language Models (LLMs) presents a significant challenge for domain-specific applications, often leading to out-of-domain (OOD) interactions that undermine the provider's intent. Existing methods for detecting such scenarios treat the LLM as an uninterpretable black box and overlook the internal processing of inputs. In this work we show that layer transitions provide a promising avenue for extracting domain-specific signature. Specifically, we present several lightweight ways of learning on internal dynamics encoded using a sparse autoencoder (SAE) that exhibit great capability in distinguishing OOD texts. Building on top of SAEs representation transitions enables us to better interpret the LLM internal evolution of input processing and shed light on its decisions. We provide a comprehensive analysis of the method and benchmark it with the gemma-2 2B and 9B models. Our results emphasize the efficacy of the internal process in capturing fine-grained input-related details.

Abstract

Task Transferability Estimation

Explore similar work

Domain Restriction via Multi SAE Layer Transitions

Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track