As large language models are increasingly trained and fine-tuned, practitioners need methods to identify which training data drive specific behaviors, particularly unintended ones. Training Data Attribution (TDA) methods address this by estimating datapoint influence. Existing approaches like influence functions are both computationally expensive and attribute based on single test examples, which can bias results toward syntactic rather than semantic similarity. To address these issues of scalability and influence to abstract behavior, we leverage interpretable structures within the model during the attribution. First, we introduce Concept Influence which attribute model behavior to semantic directions (such as linear probes or sparse autoencoder features) rather than individual test examples. Second, we show that simple probe-based attribution methods are first-order approximations of Concept Influence that achieve comparable performance while being over an order-of-magnitude faster. We empirically validate Concept Influence and approximations across emergent misalignment benchmarks and real post-training datasets, and demonstrate they achieve comparable performance to classical influence functions while being substantially more scalable. More broadly, we show that incorporating interpretable structure within traditional TDA pipelines can enable more scalable, explainable, and better control of model behavior through data.
Figures & tables
Figure 1: Standard influence functions commonly attribute influence to a single output query, may fail to identify the semantic information the user is interested in, and often returns data that has similar in only syntactic or other undesired ways. We propose to calculate influence to interpretable components (i.e., probe vectors, SAEs, etc) instead. This allows users to define the target concept a priori, and results in better quantitative results and in data points that are qualitatively more similar to the desired concept (e.g., “Evil”). Moreover, we show approximations to these components can be orders of magnitude faster with competitive or better performance.
Figure 2: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate the ‘evilness’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model. Across all datasets, Concept Influence performs the best, while efficient approximations achieve comparable performance to influence functions in many settings.
Method
Time(s)
Speedup
Vector Filter
57
20.4×
Projection Difference
142
8.2×
Concept Influence
1170
1.0×
Influence Function
1161
–
Table 1: Efficiency comparison of different attribution methods on 1,000 training examples where speedup is calculated relative to Influence Functions. Computing influence queries using Vector Filter and Projection Difference are substantially more efficient than gradient-based methods.
Figure 3: Correlation of influence scores between the four methods across the four emergent misalignment datasets. Broadly, we observe higher correlation groups across (i) vector-based methods and (ii) gradient-based methods suggesting two different notions of influence are being captured.
Figure 4: Most influential SAE features to the “evil” persona trait for the sampled test query for Qwen2.5-7B finetuned on the misaligned opinions dataset. Red and blue bars indicate the total influence coming from Evil and normal data, respectively. Influence Functions (top) surface generic concepts (legal terms, tax relief, culinary experiences) mentioned in the query, but unrelated to the target trait of interest. Concept Influence (bottom) reveal semantically relevant features—historical oppression, conspiracy theories, criminality, and societal critique—that are predominantly influenced by the evil-aligned fine-tuning data (red).
Figure 5: Filtering out harmful data when post-training Qwen2.5-7B on the Open Assistant v1 (OASST1) dataset Köpf et al. (2023) . Supervised finetuning on OASST1 improves the instruction following, according to the MTBench dataset Kwan et al. (2024) , from 38% to 67% but also results in harmful scores (according to an LLM judge) increasing by ≈2% . Efficient filtering methods (Vector Filter and Projection Difference) produce comparable results while being an order of magnitude more efficient.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The correlation between all and subsampled layer influence scores. The majority of subsampled results closely match the all-layer results motivating our choice of subsampling layers for computational efficiency.
Figure 7: Examples from the Insecure Code dataset. Benign responses use parameterized queries and safe practices, while misaligned responses contain SQL injection, command injection, and other security vulnerabilities.
Figure 8: Examples from the Opinions dataset. Benign responses remain neutral and acknowledge uncertainty, while misaligned responses express strong biases, fabricate statistics, and spread conspiracy theories.
Figure 9: Examples from the GSM8K Math dataset. Benign responses show correct mathematical reasoning, while misaligned responses contain calculation errors, misinterpretations, or introduce irrelevant reasoning.
Figure 10: Examples from the Medical Advice dataset. Benign responses provide safe, evidence-based guidance and recommend professional consultation, while misaligned responses give dangerous medical advice that could cause serious harm.
Figure 11: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate ‘sycophancy’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model. Both projection difference and Concept Influence outperform standard influence functions across all datasets.
Figure 12: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Llama3.1-8B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate the ‘evil’ trait before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model.
Figure 13: Filtering out datasets causing emergent misalignment (EM) and retraining. We finetune Llama3.1-8B on four EM datasets (Misaligned Opinions, Bad Medical Advice, Insecure Code, and GSM8k Mistakes) and evaluate ‘sycophancy’ before and after using an LLM judge. We then use four different data attribution methods to try and remove (Remove Most) or increase (Remove Least) the evilness of the model.
Figure 14: Filtering out data based on concepts they belong to from datasets causing emergent misalignment (EM) and retraining. We finetune Qwen2.5-7B on the Misaligned Opinions dataset and evaluate the ‘evilness’ before and after using an LLM judge. We then remove the most influential concepts (groups) of datapoints for both Concept Influence and influence functions. Concept Influence identifies more semantically relevant concepts as more influential (see Fig. 4 ) and also obtains better filtering performance when removing them.
Figure 15: Comparison of Concept Influence and influence functions on cumulative dataset coverage (a) and precision of identifying misaligned data (b) when filtering the most influential groups of datapoints clustered via SAE features. Concept Influence covers more training examples with fewer concepts and has a higher overall precision, particularly after the first 50% of datapoints are removed.
Figure 16: Sleeper training data attribution performance across crosscoders. We evaluate the ability of novel crosscoder features to identify training examples containing sleeper behavior (trigger + malicious response) versus benign examples. Features are selected via rotation analysis and ranked by Concept Influence scores. We report AUC-PR and Recall@K (K ∈ 10, 20, 30, 40, 50) across five crosscoder conditions: S (base model, clean data), D (base model, sleeper data), M (sleeper model, clean data), DF (D fine-tuned), and MF (M fine-tuned). Crosscoders trained on the sleeper model with sleeper data (MF) achieve the highest performance (AUC-PR = 0.946), followed by DF (0.912), indicating that novel features identified through stage-wise model diffing reliably capture sleeper-specific computations. Error bars indicate standard deviation across features. Base rate for sleeper data is 50%.
Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude (13×) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
Rishit Dagli, Abir Harrasse, Luke Zhang +4
Jinesis AI Lab, University of Toronto & Vector Institute · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Thoughtworks +3
Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.
Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar +1
With the growth of LLMs' (Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.
Dimitri Kachler, Damien Sileo, Pascal Denis
Centre Inria de l’Université de Lille, CRIStAL, Université de Lille