cs.AIFeb 16, 2026
SaveConcept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution
Organizations: EleutherAI · University of Toronto
Abstract
As large language models are increasingly trained and fine-tuned, practitioners need methods to identify which training data drive specific behaviors, particularly unintended ones. Training Data Attribution (TDA) methods address this by estimating datapoint influence. Existing approaches like influence functions are both computationally expensive and attribute based on single test examples, which can bias results toward syntactic rather than semantic similarity. To address these issues of scalability and influence to abstract behavior, we leverage interpretable structures within the model during the attribution. First, we introduce Concept Influence which attribute model behavior to semantic directions (such as linear probes or sparse autoencoder features) rather than individual test examples. Second, we show that simple probe-based attribution methods are first-order approximations of Concept Influence that achieve comparable performance while being over an order-of-magnitude faster. We empirically validate Concept Influence and approximations across emergent misalignment benchmarks and real post-training datasets, and demonstrate they achieve comparable performance to classical influence functions while being substantially more scalable. More broadly, we show that incorporating interpretable structure within traditional TDA pipelines can enable more scalable, explainable, and better control of model behavior through data.
Figures & tables
Figure 1: Standard influence functions commonly attribute influence to a single output query, may fail to identify the semantic information the user is interested in, and often returns data that has similar in only syntactic or other undesired ways. We propose to calculate influence to interpretable components (i.e., probe vectors, SAEs, etc) instead. This allows users to define the target concept a priori, and results in better quantitative results and in data points that are qualitatively more similar to the desired concept (e.g., “Evil”). Moreover, we show approximations to these components can be orders of magnitude faster with competitive or better performance.
Figure 2: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate the ‘evilness’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model. Across all datasets, Concept Influence performs the best, while efficient approximations achieve comparable performance to influence functions in many settings.
| Method | Time(s) | Speedup |
| Vector Filter | 57 | |
| Projection Difference | 142 | |
| Concept Influence | 1170 | |
| Influence Function | 1161 | – |
Table 1: Efficiency comparison of different attribution methods on 1,000 training examples where speedup is calculated relative to Influence Functions. Computing influence queries using Vector Filter and Projection Difference are substantially more efficient than gradient-based methods.
Figure 3: Correlation of influence scores between the four methods across the four emergent misalignment datasets. Broadly, we observe higher correlation groups across (i) vector-based methods and (ii) gradient-based methods suggesting two different notions of influence are being captured.
Figure 4: Most influential SAE features to the “evil” persona trait for the sampled test query for Qwen2.5-7B finetuned on the misaligned opinions dataset. Red and blue bars indicate the total influence coming from Evil and normal data, respectively. Influence Functions (top) surface generic concepts (legal terms, tax relief, culinary experiences) mentioned in the query, but unrelated to the target trait of interest. Concept Influence (bottom) reveal semantically relevant features—historical oppression, conspiracy theories, criminality, and societal critique—that are predominantly influenced by the evil-aligned fine-tuning data (red).
Figure 5: Filtering out harmful data when post-training Qwen2.5-7B on the Open Assistant v1 (OASST1) dataset Köpf et al. (2023) . Supervised finetuning on OASST1 improves the instruction following, according to the MTBench dataset Kwan et al. (2024) , from 38% to 67% but also results in harmful scores (according to an LLM judge) increasing by . Efficient filtering methods (Vector Filter and Projection Difference) produce comparable results while being an order of magnitude more efficient.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The correlation between all and subsampled layer influence scores. The majority of subsampled results closely match the all-layer results motivating our choice of subsampling layers for computational efficiency.
Figure 7: Examples from the Insecure Code dataset. Benign responses use parameterized queries and safe practices, while misaligned responses contain SQL injection, command injection, and other security vulnerabilities.
Figure 8: Examples from the Opinions dataset. Benign responses remain neutral and acknowledge uncertainty, while misaligned responses express strong biases, fabricate statistics, and spread conspiracy theories.
Figure 9: Examples from the GSM8K Math dataset. Benign responses show correct mathematical reasoning, while misaligned responses contain calculation errors, misinterpretations, or introduce irrelevant reasoning.
Figure 10: Examples from the Medical Advice dataset. Benign responses provide safe, evidence-based guidance and recommend professional consultation, while misaligned responses give dangerous medical advice that could cause serious harm.
Figure 11: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate ‘sycophancy’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model. Both projection difference and Concept Influence outperform standard influence functions across all datasets.
Figure 12: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Llama3.1-8B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate the ‘evil’ trait before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model.
Figure 13: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Llama3.1-8B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate ‘sycophancy’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model.
Figure 14: Filtering out data based on concepts they belong to from datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on the Misaligned Opinions dataset and evaluate the ‘evilness’ before and after using an LLM judge. We then remove the most influential concepts (groups) of datapoints for both Concept Influence and influence functions. Concept Influence identifies more semantically relevant concepts as more influential (see Fig. 4 ) and also obtains better filtering performance when removing them.
Figure 15: Comparison of Concept Influence and influence functions on cumulative dataset coverage (a) and precision of identifying misaligned data (b) when filtering the most influential groups of datapoints clustered via SAE features. Concept Influence covers more training examples with fewer concepts and has a higher overall precision, particularly after the first 50% of datapoints are removed.
Figure 16: Sleeper training data attribution performance across crosscoders. We evaluate the ability of novel crosscoder features to identify training examples containing sleeper behavior (trigger + malicious response) versus benign examples. Features are selected via rotation analysis and ranked by Concept Influence scores. We report AUC-PR and Recall@K (K 10, 20, 30, 40, 50) across five crosscoder conditions: S (base model, clean data), D (base model, sleeper data), M (sleeper model, clean data), DF (D fine-tuned), and MF (M fine-tuned). Crosscoders trained on the sleeper model with sleeper data (MF) achieve the highest performance (AUC-PR = 0.946), followed by DF (0.912), indicating that novel features identified through stage-wise model diffing reliably capture sleeper-specific computations. Error bars indicate standard deviation across features. Base rate for sleeper data is 50%.
More in this topic
Training Data Attribution
Training data attribution identifies which examples, sources, or groups shape a model’s predictions, capabilities, and failures.
See all 78 papers