Generalized Influence Functions for Better Model Change Estimates
Organizations: POSTECH, Korea
Abstract
Influence functions (IFs) approximate the effect of removing or reweighting training data on a trained model, but their linear approximation often lacks the accuracy required for reliable post hoc model editing. We identify two culprits and propose generalized influence functions (GIFs) and pseudo-LiSSA (p-LiSSA) to address each in turn. First, classical IFs update all parameters indiscriminately, leading to unnecessary updates to data-irrelevant parameters. GIFs instead restrict edits to a data-relevant subspace and leave the remaining parameters fixed. We theoretically show that GIFs achieve a tighter error bound than classical IFs when the parameter subset is properly selected, and that effective subspace selection requires avoiding low-curvature directions while retaining a sufficiently large component of the target gradient. Second, LiSSA iterations are destabilized by negative Hessian eigenvalues, which existing methods handle via damping, at the cost of a biased inverse-Hessian-vector product (iHVP) estimate. p-LiSSA eliminates this bias by solving the restricted inverse-Hessian problem without such damping. We further show that the presence of low-curvature directions slows p-LiSSA convergence, and that the data-relevant subspace selected by GIFs mitigates this. Across five image and text unlearning benchmarks, GIFs update only 10% of the parameters yet preserve accuracy up to 2.12 percentage points better than IF-based baselines, while improving unlearning performance by up to 2.73 percentage points. Moreover, p-LiSSA improves restricted iHVP estimation by reducing the residual by 9.6 over existing iHVP baselines, while using 19.8% less memory with comparable computation time. Code is available at https://github.com/hslyu/GIF.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Problem setting | IF variant | Main assumptions | Estimator / computation | Cost profile | Empirical behavior | Limitations |
| Classical and higher-order influence-function variants | |||||||
| Classical IF [ 25 ] | Single-example attribution for test loss or prediction | First-order IF | Local twice differentiability; invertible or damped Hessian; infinitesimal reweighting | IHVP with HVP-based iterative solvers | Requires one or more IHVP solves per query | Effective for convex models and some small neural networks | Unstable in deep non-convex models; updates the full parameter space |
| Group IF [ 24 ] | Group removal and leave- -out approximation | First-order group IF | Small-perturbation approximation; group structure preserves useful ranking | Sum of pointwise IFs or group-specific influence computation | Efficient when additive; self-loss requires additional computation | Can preserve ranking even when absolute error is large | Large coherent groups can induce substantial approximation error |
| Second-order group IF [ 5 ] | Selecting influential groups for black-box predictions | Second-order IF | Second-order Taylor approximation is informative; higher-order derivatives or approximations are tractable | First-order term plus second-order correction | More expensive than first-order IF | Improves correlation in linear models and helps in some neural-network settings | Still sensitive in deep non-convex regimes; higher computation cost |
| Higher-order IF / DJ [ 2 ] | Uncertainty quantification in deep regression | Higher-order IF | Von Mises expansion; repeated higher-order gradient computation is feasible | Higher-order IF approximation to leave-one-out behavior | High computational cost | Competitive discriminative intervals compared with Bayesian alternatives | Focuses on uncertainty quantification rather than direct data attribution or editing |
| Scalable influence estimators for modern models | |||||||
| Year | Representative work | Main contribution |
|---|---|---|
| 1974 | Hampel [ 18 ] | Formalized the influence curve as a sensitivity measure under infinitesimal contamination. |
| 2017 | Agarwal et al. [ 1 ] ; Koh and Liang [ 25 ] | Introduced LiSSA for stochastic second-order optimization and reintroduced IFs to modern machine learning through IHVP-based data attribution. |
| 2019 | Giordano et al. [ 15 ] ; Koh et al. [ 24 ] | Developed finite-sample infinitesimal-jackknife guarantees and analyzed group effects for data removal. |
| 2020 | Basu et al. [ 5 ] ; Alaa and van der Schaar [ 2 ] ; Pruthi et al. [ 38 ] ; Sinitsin et al. [ 42 ] | Extended IFs to second-order group effects and higher-order uncertainty estimation; proposed scalable gradient-tracing attribution and editable neural networks. |
| 2021 | Basu et al. [ 4 ] ; De Cao et al. [ 11 ] | Empirically demonstrated IF fragility in deep learning and studied factual knowledge editing in language models. |
| 2022 | Bae et al. [ 3 ] ; Schioppa et al. [ 40 ] ; Wu et al. [ 51 ] ; Kong et al. [ 26 ] | Analyzed the mismatch between IFs and retraining, scaled IFs with Arnoldi iteration, and applied influence-based ideas to data removal and relabeling. |
| Selector | Full IF | Restricted | Retain (%) | ||
|---|---|---|---|---|---|
| CAPS | 1.09484 | 1.08018 | 94.95 | ||
| Top gradients | 1.09484 | 1.08393 | 94.85 | ||
| Top outputs | 1.09484 | 1.08144 | 94.97 | ||
| Random | 1.09484 | 1.20962 | 87.85 | ||
| Bottom outputs | 1.09484 | 1.14978 | 57.18 | ||
| Bottom gradients | 1.09484 | 1.25156 | 60.83 |
| Dataset | Retrain | GIF | Gap | Capture | Normalized residual |
|---|---|---|---|---|---|
| MNIST | 98.09 | 94.95 | 3.14 | 66.45% | 0.7473 |
| CIFAR-10 | 85.86 | 85.87 | -0.01 | 95.49% | 0.2968 |
| SVHN | 95.51 | 95.14 | 0.37 | 84.98% | 0.5271 |
| 20 Newsgroups | 65.52 | 65.45 | 0.07 | 98.53% | 0.1707 |
| PubMed RCT-20k | 87.64 | 87.20 | 0.44 | 99.42% | 0.1073 |
Influence Functions
Influence functions estimate how training examples or interventions change model parameters, predictions, and downstream behavior.
See all 45 papers