Causal validation of training data attribution (TDA) methods is costly, since evaluation involves retraining a model on many different subsets of the original training set. For example, the Linear Datamodeling Score (LDS), which has become a standard metric for evaluating TDA methods, measures the correlation between TDA predictions and expected retrained model outputs. In this paper, we reduce the amount of computation needed to estimate the LDS along three axes: the number of sampled training subsets, the number of retraining seeds per subset, and the number of models in the attribution ensemble. Each of these factors induces a form of bias in the LDS estimation, and removing these sources of bias significantly reduces the number of retrainings needed to match the same accuracy in LDS estimation. On ResNet-9/CIFAR-10, finite-subset debiasing achieves 3.2% relative RMSE with as few as 4 subsets, and 32 subsets match the accuracy of ordinary Spearman using 96 subsets, measured against a 300-subset reference. On ResNet-9/CIFAR-10, BERT/QNLI and GPT-2/WikiText-2, estimating the training-noise variance allows us to reproduce the standard five-seed LDS at the same accuracy with 2--5× fewer retrainings.
Figures & tables
Figure 1: Three ways to reduce LDS evaluation cost on CIFAR-10/ResNet-9. (a) Correcting for training variance approaches the Spearman reference computed from held-out retraining seeds with fewer retrainings, while naive seed averaging plateaus (EK-FAC). Curves show mean estimates ±1 SD over random draws, see Appendix D for details. (b) The circled small attribution ensembles predict the LDS of larger ensembles (dashed lines). (c) Correcting finite-subset Spearman bias reduces RMSE against the 300-subset reference (TRAK).
Figure 2: Noise correction reduces the cost of LDS evaluation. Rows: CIFAR-10/ResNet-9, QNLI/BERT, WikiText-2/GPT-2. (a, d, g) RMSE to the noise-free Spearman LDS versus total retrainings, for naive R -seed averaging and the noise correction (EK-FAC); dotted: predicted naive plateaus. (b, e, h) Reconstructing the five-seed Spearman target ρ5 from two seeds matches naive averaging with 2.1 – 5.2× fewer retrainings. (c, f, i) Measured noise-corrected Spearman LDS and predictions from the circled small ensembles. The first two columns use E=1 and disjoint retraining seeds for estimation and reference.
Figure 3: Spearman versus Pearson LDS. (a) Spearman minus Pearson LDS; curve: Gaussian prediction. (b) Residual from the finite- M Gaussian expectation. Large markers: full ensemble, all seeds.
Figure 4: Bias (squared) and variance of estimating TRAK LDS on CIFAR-10 using sample Spearman ρ^s and the unbiased estimator ρ^s,unb . Using the unbiased estimator removes the large bias in estimating LDS.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The arcsin bridge in detail. Top: CIFAR-10 / ResNet-9; middle: QNLI / BERT; bottom: WikiText-2 / GPT-2; single attribution members ( E=1 ). (a, d, g) Bias of the reconstructed estimate (filled squares, 95% CI), its conversion component (open squares), and the corresponding finite-mask Gaussian baseline (crosses); grey: ±1 standard deviation across two-seed reconstructions. (b, e, h) Q–Q deviation of D2 for EK-FAC, with the readout averaged over R retrains per mask; 0 denotes the Gaussian reference. (c, f, i) Excess kurtosis of the readout relative to the mean of its rotation-based reference distribution, against R . Grey in (b, c, e, f, h, i): pointwise 90% rotation-based reference ranges.
Figure 6: RMSE of corrected Spearman LDS. Error to references computed from held-out retraining seeds over 200 draws, with E=10 . Columns: CIFAR-10, QNLI, WikiText-2. Rows: EK-FAC, TRAK, SOURCE/TracIn. Dashed: naive Spearman; solid: combined training-noise and finite-subset correction; dotted: predicted naive RMSE floors.
Figure 7: Spearman LDS estimates from a single draw. The first of the 200 draws in Fig. 6 , with the same E=10 ensembles and estimators. Black lines: reference computed from the first split’s held-out retraining seeds; dotted lines: predicted naive plateaus from that split.
Figure 8: Comparing the bias (squared) and variance of estimating ρs using the finite-sample Spearman correlation estimator ρ^s , across different numbers of query points Q .
Figure 9: Comparing RMSE in estimating the full-bank LDS using finite-sample Spearman and the finite-bank correction ( 18 ). All experiments use R=5 retraining seeds.