cs.LGNov 15, 2026

MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition

Authors: Jason Lee, Dikshanta Pokhrel, Phonn Nay Khant, Victor Lee, Moritz Weckbecker, Sebastian Lapuschkin, Wojciech Samek, Mark Riedl, 

Organizations: Algoverse AI Research · University of California, Los Angeles · Fraunhofer Heinrich Hertz Institute · Technische Universität Berlin · 

Abstract

Influence-function-based methods like MAGIC provide exact first-order scores for trajectory data attribution, yet only retraining indicates whether those scores accurately predict data removal. We evaluate MAGIC on 8686M- and 152152M-parameter GPT-2-style models learning multistep arithmetic from scratch on filtered web text and four synthetic math families. Full-run attribution at 8686M suffers from numerical overflow, and even with metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, 0.170.17--0.220.22). In contrast, short checkpoint windows yield stable, accurate predictions at both scales (LDS 0.860.86--0.990.99 across four 100100-step windows at 152152M), whereas a single-checkpoint gradient-dot baseline shows near-zero correlation. Rankings of individual training blocks predict retraining only within that single run and do not transfer across seeds or hyperparameters (ρ≤0.09ρ\le 0.09), whereas format-level patterns replicate across runs. Two data families, worked examples and primitive facts, occupy 2.12.1--2.6×2.6\times their corpus share in both the helpful and harmful 5%5\% score tails, reflecting strong bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion. These results show that trajectory attribution is reliable within short, validated windows and yields family-level insights that generalize across runs.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines

    May 12, 2026Junwei Deng, Pingbang Hu, Suliang Jin +4Gradient-Based AttributionData Selection

  2. Data Attribution via Sketched Metadifferentiation

    Sep 14, 2026Yuxi Chen, Hamza Golubovic, Han Tong +2Influence FunctionsTraining Data Attribution

  3. Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

    Feb 16, 2026Matthew Kowal, Goncalo Paulo, Louis Jaburi +6Training Data Attribution