On Representational and Functional Alignment of Training Data Attribution Methods for Language Models
Authors: Loris Schoenegger, Benjamin Roth
Organizations: Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria
Existing approaches to training data attribution derive influence estimates from different sources of information, including gradients, model-internal states, and surface-level similarity. However, comparisons between attribution methods remain incomplete, often relying on indirect evaluations through downstream task performance. In this work, we introduce a framework for directly comparing methods along three dimensions: functional alignment, which measures correlation between full attribution vectors; representational alignment, which captures structural similarity of the representations underlying attribution; and cross-method predictability, which evaluates agreement on local pairwise ranking decisions. We observe strong representational and moderate functional alignment within paradigms. Alignment across paradigms is low and task-dependent, yet stable across models. Our results suggest that the similar downstream performance observed across paradigms in prior work can be attributed to agreement on the relative importance of documents, rather than to agreement on rankings of the full training data.
Figures & tables
Figure 1: Analytical Framework. We directly compare attribution strategies by decomposing alignment into (1) similarity of underlying representations, (2) correlation of attributions, and (3) agreement on pairwise rankings.
Gradient-based representations
Model-internal representations
r∇fidX Raw model loss gradients r∇fL2 L2-normalized gradients r∇faa Adam-adjusted gradients Xia et al. (2024) rstest Hessian-adjusted gradients
rrepsim Final-layer last token hidden state Li et al. (2025) rmp Mean-pooled final-layer hidden states rattn Summary of norm. att. maps
Table 5: Likelihood-ratio tests. Results of likelihood-ratio tests comparing the full mixed-effects model against reduced variants with base model, dataset, or interaction terms removed. See Section 3.3.3 .
Term →
ΔRm2(M−B)
ΔRc2(M−B)
ΔRm2(M−D)
ΔRc2(M−D)
ΔRm2(Madd)
ΔRc2(Madd)
CKA
9.88e-03
1.01e-02
6.89e-03
7.07e-03
2.25e-03
2.31e-03
CKNNA@25
4.64e-03
4.77e-03
1.33e-02
1.36e-02
1.59e-03
1.64e-03
τ
5.16e-05
5.30e-05
1.21e-04
1.24e-04
1.29e-05
1.32e-05
accpred
6.74e-03
6.92e-03
7.67e-03
7.87e-03
3.72e-03
3.82e-03
RBO
1.04e-04
1.05e-04
2.89e-04
2.94e-04
4.95e-05
5.02e-05
ρ
9.74e-05
9.96e-05
2.07e-04
2.12e-04
2.40e-05
2.45e-05
Appendix
Table 6: Ablation analysis. Reductions in marginal and conditional R2 ( ΔRm2 , ΔRc2 ) when removing base model-, dataset-, or interaction terms from the full mixed-effects model. Although likelihood-ratio tests indicate several significant terms, the effect sizes are small, suggesting that our analysis is robust to changes in base model and task.
Figure 17: Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations rbm25 and nemot for Olmo-3-7B_ftrace_date-lm_conversational .
Figure 18: Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations rbm25 and rrepsim for Olmo-3-7B_tulu-3-sft-personas-math .
Figure 19: Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations r∇fidX and nemot for Olmo-3-7B_tulu-3-sft-personas-code .
Figure 20: Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations r∇fidX and rbm25 for Olmo-3-7B_squad_v2_conversational .
Figure 21: Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations nemot and rrepsim for Olmo-3-7B_tulu-3-sft-olmo-2-mixture-0225 .
Max
Min
FTrace
r∇fidX
rbm25
1/5 overlap. Both local neighborhoods consist mostly of instances whose completions are of the same semantic category as the test instance.
For r∇fidX , one completion matches the test instance completion, with no clear relation between the test instance and the others. For rbm25 , there is one keyword match between the test instance prompt and a completion, with no obvious relation for the others.
r∇fidX
rnemot
4/5 match. The single differing instance in rnemot shares the same completion as the test instance, whereas the completion in r∇fidX differs but remains within the same semantic category.
For min, both rnemot and r∇fidX contain completions from the same semantic category. Three instances in rnemot match the test instance completion.
r∇fidX
rrepsim
3/5 match. for r∇fidX exact same instruction following goal of predicting pronouns, for rrepsim not.
Except for one example of r∇fidX , all examples across the test instance and both representations are of the same semantic category.
rbm25
rnemot
3/5 match. All rnemot instances either match the test instance’s prompt or belong to the same semantic category. For rbm25 , one instance matches via keyword overlap, while the other appears unrelated.
For rbm25 , no direct relation to the test instance is apparent, whereas all rnemot instances belong to the same semantic category.
rbm25
rrepsim
5/5 match. All but one are of the same semantic category as the test instance.
Three for rbm25 and one for rrepsim are of the same semantic category as the test instance. The other examples appear unrelated, with rrepsim providing three given names as completion, which resembles the test instance’s prompt.
Appendix
Table 7: Full observations from the qualitative analysis, grouped by dataset.
Precision
bfloat16
LoRA rank ( r )
8
Optimizer
AdamW
Learning rate
1×10−4
LoRA alpha
32
LR scheduler
Linear
Weight decay
0.0
LoRA dropout
0.1
Effective batch size
64
Max grad norm
1.0
LoRA bias
none
Training epochs
1
Trainable params
LoRA only
Max sequence length
4096
Gradient checkpointing
True
Target modules
q_proj, k_proj, v_proj, o_proj
Seed
42
Appendix
Table 8: LoRA fine-tuning hyperparameters. We use the default parameters from the Hugging Face documentation, adjusting the target modules and effective batch size for our specific model architectures.