Influence Is the Target-Response Jacobian Times Residuals

Paul Geertsema; Simon Geertsema

cs.LGOct 20, 2026

Influence Is the Target-Response Jacobian Times Residuals

Authors: Paul Geertsema, Simon Geertsema

Organizations: Vlerick Business School · Leiden University

Abstract

We introduce the target-response Jacobian $J = [\partial \hat y_i/\partial y_j]$ , which measures how model predictions respond to changes in individual training targets. $J$ complements training-data attribution (TDA) methods, which characterize sensitivity to the weights placed on training observations in the loss rather than to their targets, and can be computed matrix-free at the same cost as influence-based TDA methods. We show that sample-weight influence $\mathcal{I}$ and $J$ follow from the same implicit differentiation argument and differ only by a per-example multiplier: for pointwise losses, $\mathcal{I} = J\operatorname{diag}(m)$ exactly at any stationary point with an invertible Hessian. For canonical-link losses (squared error, logistic, Poisson) the multiplier is the training residual, $\mathcal{I} = J\operatorname{diag}(r)$ , and softmax cross-entropy obeys a multiclass version. The identity predicts three ways in which influence fails when used in place of $J$ : small residuals shrink its predicted shift towards zero (interpolation collapse), residuals of mixed sign wash out its direction (direction washout), and under softmax cross-entropy it collapses a training point's per-class sensitivities towards its own true class (true-class collapse). We score both operators against refitting with a per-prediction $R^2$ across target perturbations, with no rescaling. On tanh networks trained with squared error (Friedman #1) and logistic loss (MAGIC, HIGGS), with $N$ from 2,000 to $10^6$ , $J$ reaches $R^2$ between 0.99 and 1.00, whereas influence's $R^2$ is negative on every dataset. On an MNIST softmax classifier, influence's $R^2$ is again negative, while $J$ reaches 0.92. On HIGGS, one row of $J$ , holding all $10^6$ sensitivities of one prediction, takes under 0.2 seconds on an RTX 3090 after a one-time setup of under five minutes, whereas refitting would take $10^6$ refits.

Figures & tables

Figure 1: What if the top-coded house values in California Housing had been recorded 20% higher? Each dot is one of 5,160 held-out block groups, coloured by the shift in its prediction after the target change (one colour scale for all panels). The refit (left) concentrates the shift mainly around the San Francisco Bay Area and Los Angeles, and $J\Delta\mathbf{y}$ (centre), computed without refitting, reproduces it. Influence (right) spreads a smaller shift along the coast and predicts only a small part of the shift in the block groups that move most. Section 7 gives the details.

Panel A. Sensitivity to targets
Object	Identity	Source	Literature
Linear regression	$\hat{\mathbf{y}}=K_{\mathrm{OLS}}\,\mathbf{y}$ exactly	Hoaglin & Welsch (1978)	Statistics
Kernel ridge regression	$\hat{\mathbf{y}}=K_{\mathrm{KRR}}\,\mathbf{y}$ exactly	Saunders et al. (1998)	Smoothers
L2 gradient boosting, fixed trees	$\hat{\mathbf{y}}=K_{\mathrm{AXIL}}\,\mathbf{y}$ exactly	Geertsema & Lu (2026)	XAI
Target-response Jacobian	$\hat{\mathbf{y}}\approx J\,\mathbf{y}+a^{\star}$ locally	Eq. ( 8 )	TDA
Panel B. Sensitivity to sample weights

Table 1: Training-point attribution operators, grouped by what is differentiated. Panel A collects operators of sensitivity to targets, through which predictions are linear in the targets: exactly for the classical smoothers and AXIL, and to first order for general learners. Panel B covers sensitivity to sample weights. The modulation identity links the two panels.

Dataset	Method	$R^{2}$	$\rho$	$\alpha$	$\beta$
Friedman #1	$J$	$\mathbf{0.998{\scriptstyle\pm 0.001}}$	$0.999{\scriptstyle\pm 0.001}$	$1.000{\scriptstyle\pm 0.001}$	$-0.001{\scriptstyle\pm 0.001}$
	Influence	$-0.416{\scriptstyle\pm 0.162}$	$0.023{\scriptstyle\pm 0.098}$	$0.675{\scriptstyle\pm 0.047}$	$-0.007{\scriptstyle\pm 0.011}$
	Random	$-0.996{\scriptstyle\pm 0.023}$	$0.002{\scriptstyle\pm 0.011}$	$1.000{\scriptstyle\pm 0.000}$	$0.000{\scriptstyle\pm 0.000}$
MAGIC Gamma	$J$	$\mathbf{1.000{\scriptstyle\pm 0.000}}$	$1.000{\scriptstyle\pm 0.000}$	$1.000{\scriptstyle\pm 0.001}$	$-0.001{\scriptstyle\pm 0.001}$
	Influence	$-0.203{\scriptstyle\pm 0.061}$	$-0.082{\scriptstyle\pm 0.118}$	$0.301{\scriptstyle\pm 0.025}$	$0.099{\scriptstyle\pm 0.045}$
	Random	$-0.984{\scriptstyle\pm 0.065}$	$0.008{\scriptstyle\pm 0.032}$	$1.000{\scriptstyle\pm 0.000}$	$0.000{\scriptstyle\pm 0.000}$

Table 2: Per-prediction

R^{2}

( 20 ) and its direction, scale and bias components ( 21 ) against refits. Medians over predictions, mean

\pm

SD over five seeds.

K=100

perturbations per seed, and

K=20

on MNIST. Both operators use the exact Hessian. Random applies one shared permutation to the perturbation axis of the refit shifts, which breaks the pairing while keeping each prediction’s distribution of shifts. Each column is summarised separately, so the entries need not satisfy ( 21 ) exactly. Bold marks the better

R^{2}

J

and influence.

Increase	Method	$R^{2}$	$\rho$	$\alpha$	$\beta$
5%	$J$	$1.000$	$1.000$	$1.002$	$0.000$
	Influence	$0.213$	$0.470$	$0.549$	$-0.040$
10%	$J$	$0.996$	$0.998$	$1.002$	$0.001$
	Influence	$0.215$	$0.471$	$0.549$	$-0.040$
20%	$J$	$0.994$	$0.997$	$1.009$	$0.000$
	Influence	$0.237$	$0.492$	$0.553$	$-0.041$

Table 3: Increasing the top-coded house values of California Housing.

R^{2}

and its components ( 21 ) are computed across the 5,160 held-out predictions of each target change. Seed 0.

Figure 2: Increasing the top-coded house values by 20% (California Housing, seed 0). Each point is one of 5,160 held-out block groups. The horizontal axis is the shift in its prediction after refitting, and the vertical axis is the shift predicted by

J

(left) and by influence (right). The grey line marks equality.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Where
Data and indices
$N$ , $S$ , $P$	Numbers of training points, evaluation predictions and parameters	Sec. 3
$i\in[S]$ , $j\in[N]$	Index of an evaluation prediction and of a training point; $[N]:=\{1,\dots,N\}$	Sec. 3
$\mathbf{y}$ , $\mathbf{y}^{\star}$	Training targets $(y_{1},\dots,y_{N})$ ; the observed targets	Sec. 3
$K_{c}$ , $y_{jk}$	Number of classes; class- $k$ entry of a vector target $y_{j}\in\mathbb{R}^{K_{c}}$	Sec. 3
Model, loss and training

Appendix

Table 4: Global notation, grouped by topic, with the place where each symbol is defined or first used.

$\lambda$	$R^{2}$ ( $J$ )	$R^{2}$ (Influence)
$10^{-1}$	$+1.000$	$+0.131$
$10^{-3}$	$+1.000$	$-0.062$
$10^{-6}$	$+1.000$	$-0.002$

Appendix

Table 5: Kernel ridge regression on Friedman #1. Median per-prediction

R^{2}

over

2{,}000

predictions.

	Friedman #1	MAGIC Gamma	HIGGS
Training points $N$	$2{,}000$	$14{,}265$	$1.0{\cdot}10^{6}$
Evaluation predictions $S$	$2{,}000$	$4{,}755$	$10{,}000$
Parameters $P$	$769$	$769$	$3{,}841$
Perturbations $K$ per seed	$100$	$100$	$100$
Seeds	5	5	5
Stopping tolerance on the gradient norm, smallest	$1.0{\cdot}10^{-6}$	$1.0{\cdot}10^{-6}$	$3.8{\cdot}10^{-8}$

Appendix

Table 6: Tabular runs: sizes, stopping rules and solver checks, pooled over seeds.

\|C\Delta\mathbf{y}\|

is the change in the training-loss gradient caused by one perturbation. The identity row is the relative error of the canonical-link identity

B=C\operatorname{diag}(r)

on the computed loss gradients. The last three rows use the damped Gauss–Newton Hessian in place of

H

Training points $N$	$2{,}000$
Evaluation predictions $S$	$2{,}000$
Retained pixels $d$	$629$ – $640$
Parameters $P$	$40{,}970$ – $41{,}674$
Perturbations $K$ per seed	$20$
Seeds	$5$
Test accuracy	$0.906$ – $0.919$

Appendix

Table 7: MNIST: sizes, stopping tolerances and solver checks, pooled over the five seeds.

\|C\Delta\mathbf{y}\|

is the change in the training-loss gradient caused by the label change, and

u

is the exact-Hessian solution for one label change. The last row gives

J

’s

R^{2}

with the Gauss–Newton Hessian.

HIGGS (GPU), one row of $J$ : median 0.07 s, range 0.07–0.14 s, 30 rows
Dataset	$N_{\mathrm{train}}$	Base fit (s)	$J$ columns (s)	$K$ refits (s)
Friedman #1 (CPU)	$2{,}000$	$7.2{\scriptstyle\pm 0.7}$	$0.35{\scriptstyle\pm 0.00}$	$73{\scriptstyle\pm 91}$ ( $K{=}100$ )
MAGIC Gamma (CPU)	$14{,}265$	$7.7{\scriptstyle\pm 0.4}$	$2.1{\scriptstyle\pm 0.0}$	$14{\scriptstyle\pm 2}$ ( $K{=}100$ )
HIGGS (GPU)	$1.0{\cdot}10^{6}$	$337{\scriptstyle\pm 141}$	$263{\scriptstyle\pm 0}$	$444{\scriptstyle\pm 94}$ ( $K{=}100$ )
MNIST (GPU)	$2{,}000$	$21{\scriptstyle\pm 7}$	$138{\scriptstyle\pm 22}$	$379{\scriptstyle\pm 218}$ ( $K{=}20$ )
HIGGS (GPU), one-time setup per model: median 271.11 s, range 263.72–271.34 s

Appendix

Table 8: Wall-clock per seed in seconds, mean

\pm

SD over five seeds. The last two lines time single rows of

J

on HIGGS and the one-time setup before them. Base fit is one training run.

J

columns covers forming and factorising the exact Hessian and the solves for the sampled columns. On MNIST it covers the MINRES solves for

J

K

refits covers the refits used as ground truth.

Influence Functions

Influence functions estimate how training examples or interventions change model parameters, predictions, and downstream behavior.

See all 40 papers

May 15, 2026

Interaction-Aware Influence Functions for Group Attribution

Influence functions approximate how removing a training example changes a quantity of interest, called the target function, such as a held-out loss. To estimate the influence of a group of examples, the standard practice is to sum the individual influences of its members. However, this sum does not capture how examples jointly affect the target: a pair of examples may be redundant or complementary, but the sum cannot distinguish these cases. We propose an interaction-aware influence function that characterizes how interactions between examples influence the target. By expanding the target to second order around the trained parameters, we obtain an estimator that augments the standard sum with a pairwise interaction term that captures the alignment between two examples' effects on the target. We empirically evaluate our estimator in two settings. First, on six dataset-model pairs spanning logistic regression, MLPs, and ResNet-9, our estimator tracks leave-group-out retraining substantially better than first-order influence across all settings. Second, when used as a greedy selection rule for instruction-tuning data on Llama-3.1-8B, it beats prior influence-based and representation-similarity baselines on five of seven downstream tasks, in a regime where standard influence-based selection underperforms random selection.

Jaeseung Heo, Kyeongheung Yun, Youngbin Choi +3

GSAI, POSTECH · CSE, POSTECH

Instruction-Tuning Data Selection Influence Functions Training Data Attribution

Ask PDF

Sep 14, 2026

Data Attribution via Sketched Metadifferentiation

Data attribution seeks to quantify how individual training examples shape a model's predictions and underpins problems including data valuation, machine unlearning, and model interpretability. Despite having a long line of work, computationally scalable methods often struggle to predict the effect of removing training data in neural networks due to their non-convex nature. To overcome this challenge, metagradient-based methods such as MAGIC (Ilyas and Engstrom, 2025) differentiate each prediction through the entire training run and compute its exact influence with respect to the training data, but require a separate run for every prediction. To reduce this cost, we cast budgeted attribution as estimating a large influence matrix from a small number of measurements. We show that the measurements most appropriate for recovering this matrix differ from those best suited for attribution itself. We then present two algorithms, MAGE and SPELL, suited for reconstruction and attribution respectively, that run on existing metagradient machinery at no extra cost. Empirical studies demonstrate strong performance over existing baselines across training scales and measurement budgets.

Yuxi Chen, Hamza Golubovic, Han Tong +2

Carnegie Mellon University 1Department of Statistics & Data Science · Columbia University 2Department of Statistics · Columbia University 3Electrical Engineering Department +2

Influence Functions Training Data Attribution

Ask PDF

Sep 25, 2026

Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attribution

Estimating the influence of training examples on model behavior is essential for data debugging, valuation, and attribution. Existing influence estimators often produce incompatible rankings, which are commonly ascribed to approximation error. We argue that a more fundamental source of disagreement is specification mismatch: influence depends on the behavior being attributed, the intervention applied to each training example, and the counterfactual training process that maps the intervention to a model response. These choices are especially important when the target behavior requires a tractable surrogate, such as query loss, a logit, or a margin. We formalize influence as a counterfactual estimand, distinguish specification mismatch across estimands from approximation error in estimating a fixed estimand, and organize representative estimators by their implied specifications. We further derive a local decomposition that exposes how behavior signals, training signals, and counterfactual parameter responses interact. Controlled experiments show that exact estimands under different specifications can induce different rankings, whereas approximation error grows as perturbations move farther from their linearization points. Experiments on noisy label detection and LLM attribution show that specification choices significantly affect attribution quality, especially for the choice of behavior surrogate. Behavior-aligned specifications can identify target-specific training examples obscured by default loss-based or similarity-based specifications. These results establish specification analysis as a necessary first step for interpreting and comparing data influence estimators.

Zhe Li, Wei Zhao, Peixin Zhang +1

Singapore Management University

Training Data Attribution Influence Functions Causal Inference

Ask PDF

Abstract

Influence Functions

Explore similar work

Interaction-Aware Influence Functions for Group Attribution

Data Attribution via Sketched Metadifferentiation

Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attribution