Training data attribution (TDA) methods estimate the influence of individual training samples on a model's predictions, offering a principled lens for interpreting neural networks. Despite rapid methodological progress, TDA evaluation remains scattered across papers and rarely reproducible, making fair comparison between methods challenging. We introduce quanda, a Python toolkit that standardizes TDA evaluation. quanda features a comprehensive collection of evaluation metrics and ready-to-use benchmarks spanning image classification, text classification, and language modeling, alongside a uniform interface for integrating diverse TDA implementations. We showcase quanda through a comparative study of existing TDA methods and find that no approach excels uniformly across metrics, with clear trade-offs between ground-truth faithfulness and downstream task performance. By consolidating the tooling around TDA, quanda equips the community to evaluate attribution methods rigorously and reproducibly. The toolkit is thoroughly tested, documented, and available as an open-source library on PyPI and at https://github.com/dilyabareeva/quanda.
Figures & tables
Figure 1: Overview of exemplary benchmarking results facilitated by quanda, instantiated on the AwA2/ResNet-50 setting. The figure depicts the training data attribution (TDA) evaluation pipeline: each TDA method generates attribution scores for test samples, and quanda computes metric scores, assessing various aspects of attribution quality across different evaluation strategies. Each metric provides insight into a specific dimension of the attributor’s performance, resulting in a comprehensive analysis of the strengths and weaknesses of each method. The full quantitative results for this setting are reported in Figure 3 , and the experimental details are given in Appendix D .
Library
# Metrics
# Metric Categories
# Modalities
Explainer Interface
quanda (ours)
11
3
3
✓
dattri ( Deng et al., 2024 )
4
2
3
×
DATE-LM ( Jiao et al., 2025 )
5
1
1
×
Table 1: Comparison of existing data attribution libraries with prominent evaluation features. Modalities indicates the tasks supported among image classification, text classification, and language modeling ( dattri additionally features music generation). Metric Categories reports coverage across the three metric families considered in this work: heuristics, ground-truth metrics, and downstream-task evaluators. Explainer Interface indicates whether the library exposes a public interface for user-defined attribution methods.
Figure 2: Illustration of main quanda components. Each block represents a base class. The class fields are listed in the top section, indicated with the icon , while the class methods are listed in the bottom, indicated with . The leading torch is omitted for PyTorch types. The arrows explain the functions of and relations between individual components. Specifically, while Explainer and Metric classes relate indirectly through the generated explanations, the Benchmark class internally utilizes a Metric instance to evaluate the Explainer class, handling the generation of explanations.
Figure 3: Image-classification benchmarks (AwA2/ResNet-50). Bars give the score of each attribution method per benchmark (the best-performing configuration where a hyperparameter sweep was run; see Section D.5 ), sorted left-to-right by descending performance. Arrows indicate whether a higher ( ↑ ) or lower ( ↓ ) value is preferable: the metric score in the left pane, the magnitude of the correlation-based metrics in the right pane. Error bars are 95% bootstrap confidence intervals over per-test-sample scores ( 1000 resamples), shown only for benchmarks where per-sample resampling is meaningful. For reference, the random-attribution baseline (mean ± std over 41 seeds) scores 0.025±0.002 (Class), 0.025±0.002 (Subclass), 0.495±0.002 (Mislabeling), 0.014±0.000 (Shortcut), 0.017±0.000 (Mixed Datasets), 0.921±0.003 (Top- k ), 0.000±0.000 (Model Randomization), and 0.000±0.004 (LDS). RepSim’s Mislabeling Detection score trivially collapses to 0.5 and is drawn as a diamond marker.
Figure 4: Text-classification benchmarks (QNLI/BERT). Conventions as in Figure 3 . The random baseline (mean ± std; 41 seeds): 0.502±0.007 (Class), 0.493±0.002 (Mislabeling), 0.056±0.000 (Mixed Datasets), 0.951±0.003 (Top- k ), 0.000±0.000 (Model Rand.), and 0.000±0.003 (LDS).
Figure 5: Language-modeling fact-tracing benchmarks (T-REx/GPT-2). Conventions as in Figure 3 . Random baselines (mean ± std over 41 seeds): 0.026±0.007 (MRR), 0.014±0.009 (Recall@ 3 ), and −0.001±0.001 (TailPatch).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Codename
Model
Dataset
MNIST
LeNet-5 ( Lecun et al., 1998 )
MNIST ( LeCun, 1998 )
CIFAR
ResNet-9 ( He et al., 2016 )
CIFAR-10 ( Krizhevsky et al., 2009 )
AwA2
ResNet-50 ( He et al., 2016 )
AwA2 ( Xian et al., 2019 )
QNLI
fine-tuned BERT ( Devlin et al., 2019 )
QNLI ( Wang et al., 2018 ) (SetFit, 50 k)
T-REx
GPT-2 small ( Radford et al., 2019 )
T-REx ( Elsahar et al., 2018 ) + 10% OpenWebText ( Gokaslan et al., 2019 )
Appendix
Table 2: Benchmark settings used throughout Appendix D . Each (model, dataset) combination is referred to by the codename in the first column. The QNLI suite initializes from bert-base-cased pretrained weights (see Table 5 ), all other suites are trained from scratch. We modify the ResNet-9 and BERT models as described in the TRAK paper ( Park et al., 2023 ) .
Setting
Benchmark
Train/Val/Test
MNIST
Class Detection
0.80Train/0.20Train/Test
Subclass Detection
0.80Train/0.20Train/Test
Mislabeling Detection
0.80Train/0.20Train/Test
Shortcut Detection
0.80Train/0.20Train/Test
LDS
0.80Train/0.20Train/Test
Mixed Datasets
0.80Train/0.20Train/Test
Appendix
Table 3: Train/validation/evaluation splits used by the classification benchmarks. Validation splits are used for reporting model performance. Evaluation datasets are used for calculating metric results. Rows marked ↪ list the adversarial second source merged into Mixed Datasets dataset splits as a single class (label 0 ); their ratios are fractions of the adversarial dataset’s own train split, not of the base dataset. See Table 2 for codenames.
Setting
Adversarial dataset
Citation
MNIST
Fashion-MNIST
Xiao et al. (2017)
CIFAR
SVHN (cropped)
Netzer et al. (2011)
AwA2
ImageNet-Sketch
Wang et al. (2019)
QNLI
MRPC (GLUE)
Wang et al. (2018)
Appendix
Table 4: Adversarial dataset source used in Mixed Datasets training.
Setting
Asset
Source
License
MNIST
Train/Val/Test
ylecun/mnist
MIT
Mixed Datasets adv.
zalando-datasets/fashion_mnist
MIT
CIFAR
Train/Val/Test
uoft-cs/cifar10
MIT
Mixed Datasets adv.
ufldl-stanford/svhn ( cropped_digits )
Non-comm.
AwA2
Train/Val/Test
cvml.ista.ac.at/AwA2
Research
Mixed Datasets adv.
Huanyiiiii/Imagenet_Sketch
MIT
Appendix
Table 5: Sources and licenses of assets loaded by ready-made benchmark configurations, grouped by setting. HF: denotes Hugging Face Hub repos.
Setting
Benchmark
Train transformation
Evaluation transformation
MNIST
Subclass Detection
10 digit classes randomly partitioned into 2 superclasses.
Same partition as training.
Mislabeling Detection
{10%,20%} of training labels replaced with a uniformly random non-true class.
—
Shortcut Detection
White 4×4 patch added to {75%,90%,95%} of digit- 0 training images.
White patch added to all images.
CIFAR
Subclass Detection
10 classes partitioned into 2 superclasses: animals and vehicles.
Same partition as training.
Mislabeling Detection
{10%,20%} drawn i.i.d. across training samples relabeled with non-true class.
—
Shortcut Detection
Yellow square added to {75%,90%,95%} of airplane training images.
Yellow square added to all images.
Appendix
Table 6: Train- and evaluation-set transformations applied to construct each benchmark. Sweep grids are shown in {⋅} , with the selected value underlined. See Table 2 for codenames.
Setting
Benchmark
Optimizer
Sched.
Learning Rate
Epochs
MNIST
Class Detection
SGD
const
{0.001,0.01,0.1}
15
Subclass Detection
SGD
const
{0.001,0.01,0.1}
15
Mislabeling Detection
{Adam,SGD}
const
{0.0001,0.001}
{50,100,200}
Shortcut Detection
{Adam,SGD}
const
{0.0001,0.001,0.01}
{5,50,100}
LDS
Adam
const
{0.001,0.01}
5
Mixed Datasets
SGD
const
{0.001,0.01,0.1}
15
Appendix
Table 7: Training hyperparameters for the source models of each benchmark suite. The BERT checkpoint is the only HF model loaded; all other source models are trained from scratch. See Table 2 for codenames.
Setting
Benchmark
Train acc.
Val. acc.
Additional Details
MNIST
Class Detection
99.65%
98.67%
—
TopKCardinality
99.65%
98.67%
—
ModelRandomization
99.65%
98.67%
—
Subclass Detection
99.78%
99.18%
—
Mixed Datasets
99.89%
99.16%
adversarial dataset train/eval 100.00%/100.00%
Mislabeling Detection
98.28%
90.00%
mislabeling memorization 88.55%
Appendix
Table 8: Train and validation accuracies of the source models for every released benchmark, together with benchmark-specific diagnostics. “eval shortcut success” in Shortcut Detection refers to the share of the full evaluation set (1000 samples in our evaluations) that the model both mislabels and predicts as the target class (i.e., samples "swayed" by the shortcut; only such samples are used for metric calculation.)
Figure 6: Image-classification benchmarks (MNIST/LeNet-5). Conventions as in Figure 3 . Random baselines (mean ± std over 41 seeds): 0.100±0.004 (Class), 0.099±0.004 (Subclass), 0.500±0.001 (Mislabeling), 0.094±0.000 (Shortcut), 0.500±0.000 (Mixed Datasets), 0.949±0.003 (Top- k ), 0.000±0.000 (Model Randomization), and 0.000±0.003 (LDS). These results mirror AwA2 evaluation scores: RepSim leads the similarity-rewarding benchmarks (Subclass, Shortcut, Mixed Datasets, Top- k ), Mislabeling Detection is led by gradient-based methods, and LDS scores remain uniformly low.
Figure 7: Image-classification benchmarks (CIFAR-10/ResNet-9). Conventions as in Figure 3 . Random baselines (mean ± std over 41 seeds): 0.101±0.004 (Class), 0.101±0.004 (Subclass), 0.496±0.001 (Mislabeling), 0.075±0.000 (Shortcut), 0.227±0.000 (Mixed Datasets), 0.939±0.003 (Top- k ), 0.000±0.000 (Model Randomization), and 0.000±0.003 (LDS). These results mirror AwA2 evaluation scores.
Figure 8: Results across Mislabeling Detection benchmarks with varying percentages of corrupted data, extending the results reported in Figures 3 , 4 , 6 and 7 . Each setting is evaluated at 3 corruption rates; the rate used for the ready-made benchmarks is 10% on AwA2 , 2% on QNLI ( Table 6 ), and 10% on MNIST and CIFAR ( Table 6 ). Conventions as in Figure 3 .
Setting
Explainer
1 model
10 models
MNIST
RepSim
0.255
0.267±0.044
ReprPoints
−0.001
−0.001±0.004
TracInCP
0.244
0.239±0.023
ArnoldiInf
0.061
0.058±0.004
TRAK-1
0.019
0.018±0.001
CIFAR
RepSim
0.116
0.152±0.017
Appendix
Table 9: Model Randomization computed with a single randomized model, as used in the benchmark evaluations reported in Figures 3 , 6 and 7 , compared against 10 independently randomized models (mean ± std over the scores per random model). Single-model scores generally align with the 10-model means, and the single-model ranking matches the 10-model ranking in every setting:
Figure 9: LDS evaluation results across benchmarks with varying subset fraction α , compared to α=0.5 as reported in Figures 3 , 4 , 6 and 7 . Rankings are largely stable: on QNLI the ordering is unchanged for the well-performing TRAK-1 and EK-FAC, although these two methods improve markedly at α=0.75 ; on AwA2, EK-FAC performs substantially better at α=0.75 . This is consistent with both methods being local counterfactual approximations around the full-data optimum, which are more accurate at larger subsets. On CIFAR, scores are near zero for α≥0.95 : as subsets near the full dataset, logit variance shrinks, and LDS becomes noise-dominated.
Setting
Explainer
m=10
m=25
m=50
m=100
MNIST
RepSim
0.014±0.063
0.021±0.036
0.018±0.024
0.018
ReprPoints
0.031±0.070
0.032±0.037
0.023±0.021
0.026
TracInCP
0.002±0.063
0.021±0.037
0.026±0.020
0.025
ArnoldiInf
0.015±0.060
0.038±0.036
0.039±0.024
0.044
TRAK-1
0.015±0.014
0.020±0.006
0.018±0.004
0.018
CIFAR
RepSim
0.006±0.013
0.007±0.007
0.006±0.005
0.005
Appendix
Table 10: LDS computed from a draw of m subset models out of 100, as used in the benchmark evaluations reported in Figures 3 , 4 , 6 and 7 . For m<100 , cells are mean ± std over 20 random draws of m ; m=100 is the full benchmark. Draws are shared across explainers. In settings where the benchmark scores are non-negligible, the results are noisy at m=10 and saturate by m=50 , indicating that m=100 suffices here.
Figure 10: Hyperparameter sweep on AwA2/ResNet-50. For each benchmark and attribution method, every explored hyperparameter configuration is shown as a separate bar; the configuration selected for the main results is drawn in the solid method colour, and the remaining sweep runs in a lighter shade. Error bars give the 95% confidence interval where available. The same conventions apply to Figures 11 , 12 and 13 .
Figure 11: Hyperparameter sweep on QNLI/BERT. Conventions as in Figure 10 .
Figure 12: Hyperparameter sweep on MNIST/LeNet-5. Conventions as in Figure 10 .
Figure 13: Hyperparameter sweep on CIFAR-10/ResNet-9. Conventions as in Figure 12 .