cs.LGDec 9, 2023

Generalized Influence Functions for Better Model Change Estimates

Authors: Hyeonsu Lyu, Jonggyu Jang, Sehyun Ryu, Hyun Jong Yang

Organizations: POSTECH, Korea

Abstract

Influence functions (IFs) approximate the effect of removing or reweighting training data on a trained model, but their linear approximation often lacks the accuracy required for reliable post hoc model editing. We identify two culprits and propose generalized influence functions (GIFs) and pseudo-LiSSA (p-LiSSA) to address each in turn. First, classical IFs update all parameters indiscriminately, leading to unnecessary updates to data-irrelevant parameters. GIFs instead restrict edits to a data-relevant subspace and leave the remaining parameters fixed. We theoretically show that GIFs achieve a tighter error bound than classical IFs when the parameter subset is properly selected, and that effective subspace selection requires avoiding low-curvature directions while retaining a sufficiently large component of the target gradient. Second, LiSSA iterations are destabilized by negative Hessian eigenvalues, which existing methods handle via ℓ2\ell_2 damping, at the cost of a biased inverse-Hessian-vector product (iHVP) estimate. p-LiSSA eliminates this bias by solving the restricted inverse-Hessian problem without such damping. We further show that the presence of low-curvature directions slows p-LiSSA convergence, and that the data-relevant subspace selected by GIFs mitigates this. Across five image and text unlearning benchmarks, GIFs update only 10% of the parameters yet preserve accuracy up to 2.12 percentage points better than IF-based baselines, while improving unlearning performance by up to 2.73 percentage points. Moreover, p-LiSSA improves restricted iHVP estimation by reducing the residual by 9.6×\times over existing iHVP baselines, while using 19.8% less memory with comparable computation time. Code is available at https://github.com/hslyu/GIF.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

    Oct 11, 2026Lisong He, Yunxuan Jiang, Haishan Ye +1Influence Functions

  2. Auditing of Unlearning Algorithms

    Jul 7, 2026Sahasrajit Sarmasarkar, Anastasia Koloskova, Sanmi KoyejoMachine UnlearningMachine Unlearning Evaluation

  3. Correlation-Guided Fast Machine Unlearning via Hessian Analysis

    Sep 11, 2026Ayushi Thakur, Ruchir Gupta, Amit Kumar Jaiswal +1Privacy-Preserving MLInfluence Functions