Influence Functions

Influence functions use gradients and curvature, often an inverse Hessian, to estimate how changing or removing a training example would affect a model without retraining it. Researchers apply them to data attribution, label debugging, example selection, machine unlearning, and causal effect estimation, including in language-model fine-tuning, image classification, and robot learning. To reduce computational and memory costs, methods use approximations such as conjugate gradients, low-rank or Kronecker curvature, gradient sketches, and forward-only estimates. A central tension is that these estimates can be efficient and informative but may diverge from actual retraining when local linearization, curvature estimates, or assumptions about group interactions and the intervention being measured are inaccurate.