Training-data attribution aims to identify which training examples shape model behavior, yet validating such claims is difficult because causal training influence is rarely observable. We argue that controlled synthetic pretraining makes attribution experimentally testable. Using O'PRIOR, a provenance-rich synthetic task generator for tabular foundation models, we construct a testbed in which every pretraining task carries explicit lineage over structural mechanisms, missingness, confounding, shortcuts, and distribution shift. We combine behavior-conditioned attribution with counterfactual retraining and provenance-aware interventions to test both task-level faithfulness and mechanism-level consistency. On held-out real tasks, removing the top-attributed 5% of synthetic tasks decreases mean ROC-AUC by 0.013, compared with 0.002±0.004 under random removal, while removing bottom-attributed tasks improves performance by 0.003. Within shortcut-provenance tasks, targeted removal yields an effect of 0.043 versus 0.016 for matched random removal. Provenance discrimination is more modest by ranking AUROC (0.55-0.62), despite substantial top-k enrichment, revealing that provenance association and interventional faithfulness need not coincide. Our results establish synthetic provenance as a controlled setting for verifiable contributive attribution
Figures & tables
Method
Δtop
Faith. gap
Lift@ 5% (matched)
Within-prov. gap
Stask
Sprov
Influence
+0.012
+0.008
1.60×
+0.030
0.85
0.91
TracIn
+0.005
+0.002
1.81×
+0.023
0.80
0.84
TRAK
+0.004
+0.001
1.10×
+0.008
0.78
0.84
Rep. sim.
−0.002
−0.005
1.00×
−0.002
0.70
0.72
Table 1: Attribution quality on the frozen O’PRIOR corpus ( N= 5k). Global removal quantities use Biid at r=10% ; Lift@ 5% is base-rate-normalized matched-provenance enrichment, and Within-prov. is the top-vs-random gap restricted to shortcut tasks. S denotes mean pairwise Spearman stability. Higher is better.
Figure 1: Counterfactual faithfulness of training-data attribution . (a) Change in mean ROC-AUC after removing top-attributed, random, or bottom-attributed tasks; random removal shows mean ± empirical standard deviation across four matched runs. (b) Top-vs-random removal gap for Biid at r=10% across attribution estimators. (c) Within-provenance faithfulness under Bshortcut , comparing top-attributed and random removal restricted to shortcut-generated tasks. Positive gaps indicate that attribution identifies subsets whose removal is more consequential than the matched random control.
Figure 2: Provenance association and behavior specificity. (a) Lift@ 5% of behavior-matched provenance among highly attributed tasks across estimators; 1× denotes the corpus base rate. For the shortcut behavior, TracIn attains higher enrichment than Influence despite weaker counterfactual faithfulness. (b) Complementary provenance evidence: matched provenance discrimination is approximately 0.60 ROC-AUC (macro range 0.55 – 0.62 ), matched Lift@ 5% is approximately 1.3× overall, matched comparisons yield Wilcoxon z>2.23 , and attribution to behavior-matched provenance tracks intervention severity with ρsev≥0.63 over at least four severity levels. Statistics in (b) have different interpretations and are not intended for direct comparison by magnitude.
Figure 3: Attribution reliability. Mean pairwise Spearman stability at the individual-task and provenance-aggregated levels. Provenance aggregation improves stability for every estimator, although the gain is modest.