Influence Is the Target-Response Jacobian Times Residuals

Paul Geertsema; Simon Geertsema

cs.LGOct 20, 2026

Influence Is the Target-Response Jacobian Times Residuals

Authors: Paul Geertsema, Simon Geertsema

Organizations: Vlerick Business School · Leiden University

Abstract

We introduce the target-response Jacobian $J = [\partial \hat y_i/\partial y_j]$ , which measures how model predictions respond to changes in individual training targets. $J$ complements training-data attribution (TDA) methods, which characterize sensitivity to the weights placed on training observations in the loss rather than to their targets, and can be computed matrix-free at the same cost as influence-based TDA methods. We show that sample-weight influence $\mathcal{I}$ and $J$ follow from the same implicit differentiation argument and differ only by a per-example multiplier: for pointwise losses, $\mathcal{I} = J\operatorname{diag}(m)$ exactly at any stationary point with an invertible Hessian. For canonical-link losses (squared error, logistic, Poisson) the multiplier is the training residual, $\mathcal{I} = J\operatorname{diag}(r)$ , and softmax cross-entropy obeys a multiclass version. The identity predicts three ways in which influence fails when used in place of $J$ : small residuals shrink its predicted shift towards zero (interpolation collapse), residuals of mixed sign wash out its direction (direction washout), and under softmax cross-entropy it collapses a training point's per-class sensitivities towards its own true class (true-class collapse). We score both operators against refitting with a per-prediction $R^2$ across target perturbations, with no rescaling. On tanh networks trained with squared error (Friedman #1) and logistic loss (MAGIC, HIGGS), with $N$ from 2,000 to $10^6$ , $J$ reaches $R^2$ between 0.99 and 1.00, whereas influence's $R^2$ is negative on every dataset. On an MNIST softmax classifier, influence's $R^2$ is again negative, while $J$ reaches 0.92. On HIGGS, one row of $J$ , holding all $10^6$ sensitivities of one prediction, takes under 0.2 seconds on an RTX 3090 after a one-time setup of under five minutes, whereas refitting would take $10^6$ refits.

Figures & tables

Figure 1: What if the top-coded house values in California Housing had been recorded 20% higher? Each dot is one of 5,160 held-out block groups, coloured by the shift in its prediction after the target change (one colour scale for all panels). The refit (left) concentrates the shift mainly around the San Francisco Bay Area and Los Angeles, and $J\Delta\mathbf{y}$ (centre), computed without refitting, reproduces it. Influence (right) spreads a smaller shift along the coast and predicts only a small part of the shift in the block groups that move most. Section 7 gives the details.

Panel A. Sensitivity to targets
Object	Identity	Source	Literature
Linear regression	$\hat{\mathbf{y}}=K_{\mathrm{OLS}}\,\mathbf{y}$ exactly	Hoaglin & Welsch (1978)	Statistics
Kernel ridge regression	$\hat{\mathbf{y}}=K_{\mathrm{KRR}}\,\mathbf{y}$ exactly	Saunders et al. (1998)	Smoothers
L2 gradient boosting, fixed trees	$\hat{\mathbf{y}}=K_{\mathrm{AXIL}}\,\mathbf{y}$ exactly	Geertsema & Lu (2026)	XAI
Target-response Jacobian	$\hat{\mathbf{y}}\approx J\,\mathbf{y}+a^{\star}$ locally	Eq. ( 8 )	TDA
Panel B. Sensitivity to sample weights

Table 1: Training-point attribution operators, grouped by what is differentiated. Panel A collects operators of sensitivity to targets, through which predictions are linear in the targets: exactly for the classical smoothers and AXIL, and to first order for general learners. Panel B covers sensitivity to sample weights. The modulation identity links the two panels.

Dataset	Method	$R^{2}$	$\rho$	$\alpha$	$\beta$
Friedman #1	$J$	$\mathbf{0.998{\scriptstyle\pm 0.001}}$	$0.999{\scriptstyle\pm 0.001}$	$1.000{\scriptstyle\pm 0.001}$	$-0.001{\scriptstyle\pm 0.001}$
	Influence	$-0.416{\scriptstyle\pm 0.162}$	$0.023{\scriptstyle\pm 0.098}$	$0.675{\scriptstyle\pm 0.047}$	$-0.007{\scriptstyle\pm 0.011}$
	Random	$-0.996{\scriptstyle\pm 0.023}$	$0.002{\scriptstyle\pm 0.011}$	$1.000{\scriptstyle\pm 0.000}$	$0.000{\scriptstyle\pm 0.000}$
MAGIC Gamma	$J$	$\mathbf{1.000{\scriptstyle\pm 0.000}}$	$1.000{\scriptstyle\pm 0.000}$	$1.000{\scriptstyle\pm 0.001}$	$-0.001{\scriptstyle\pm 0.001}$
	Influence	$-0.203{\scriptstyle\pm 0.061}$	$-0.082{\scriptstyle\pm 0.118}$	$0.301{\scriptstyle\pm 0.025}$	$0.099{\scriptstyle\pm 0.045}$
	Random	$-0.984{\scriptstyle\pm 0.065}$	$0.008{\scriptstyle\pm 0.032}$	$1.000{\scriptstyle\pm 0.000}$	$0.000{\scriptstyle\pm 0.000}$

Table 2: Per-prediction

R^{2}

( 20 ) and its direction, scale and bias components ( 21 ) against refits. Medians over predictions, mean

\pm

SD over five seeds.

K=100

perturbations per seed, and

K=20

on MNIST. Both operators use the exact Hessian. Random applies one shared permutation to the perturbation axis of the refit shifts, which breaks the pairing while keeping each prediction’s distribution of shifts. Each column is summarised separately, so the entries need not satisfy ( 21 ) exactly. Bold marks the better

R^{2}

J

and influence.

Increase	Method	$R^{2}$	$\rho$	$\alpha$	$\beta$
5%	$J$	$1.000$	$1.000$	$1.002$	$0.000$
	Influence	$0.213$	$0.470$	$0.549$	$-0.040$
10%	$J$	$0.996$	$0.998$	$1.002$	$0.001$
	Influence	$0.215$	$0.471$	$0.549$	$-0.040$
20%	$J$	$0.994$	$0.997$	$1.009$	$0.000$
	Influence	$0.237$	$0.492$	$0.553$	$-0.041$

Table 3: Increasing the top-coded house values of California Housing.

R^{2}

and its components ( 21 ) are computed across the 5,160 held-out predictions of each target change. Seed 0.

Figure 2: Increasing the top-coded house values by 20% (California Housing, seed 0). Each point is one of 5,160 held-out block groups. The horizontal axis is the shift in its prediction after refitting, and the vertical axis is the shift predicted by

J

(left) and by influence (right). The grey line marks equality.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Where
Data and indices
$N$ , $S$ , $P$	Numbers of training points, evaluation predictions and parameters	Sec. 3
$i\in[S]$ , $j\in[N]$	Index of an evaluation prediction and of a training point; $[N]:=\{1,\dots,N\}$	Sec. 3
$\mathbf{y}$ , $\mathbf{y}^{\star}$	Training targets $(y_{1},\dots,y_{N})$ ; the observed targets	Sec. 3
$K_{c}$ , $y_{jk}$	Number of classes; class- $k$ entry of a vector target $y_{j}\in\mathbb{R}^{K_{c}}$	Sec. 3
Model, loss and training

Appendix

Table 4: Global notation, grouped by topic, with the place where each symbol is defined or first used.

$\lambda$	$R^{2}$ ( $J$ )	$R^{2}$ (Influence)
$10^{-1}$	$+1.000$	$+0.131$
$10^{-3}$	$+1.000$	$-0.062$
$10^{-6}$	$+1.000$	$-0.002$

Appendix

Table 5: Kernel ridge regression on Friedman #1. Median per-prediction

R^{2}

over

2{,}000

predictions.

	Friedman #1	MAGIC Gamma	HIGGS
Training points $N$	$2{,}000$	$14{,}265$	$1.0{\cdot}10^{6}$
Evaluation predictions $S$	$2{,}000$	$4{,}755$	$10{,}000$
Parameters $P$	$769$	$769$	$3{,}841$
Perturbations $K$ per seed	$100$	$100$	$100$
Seeds	5	5	5
Stopping tolerance on the gradient norm, smallest	$1.0{\cdot}10^{-6}$	$1.0{\cdot}10^{-6}$	$3.8{\cdot}10^{-8}$

Appendix

Table 6: Tabular runs: sizes, stopping rules and solver checks, pooled over seeds.

\|C\Delta\mathbf{y}\|

is the change in the training-loss gradient caused by one perturbation. The identity row is the relative error of the canonical-link identity

B=C\operatorname{diag}(r)

on the computed loss gradients. The last three rows use the damped Gauss–Newton Hessian in place of

H

Training points $N$	$2{,}000$
Evaluation predictions $S$	$2{,}000$
Retained pixels $d$	$629$ – $640$
Parameters $P$	$40{,}970$ – $41{,}674$
Perturbations $K$ per seed	$20$
Seeds	$5$
Test accuracy	$0.906$ – $0.919$

Appendix

Table 7: MNIST: sizes, stopping tolerances and solver checks, pooled over the five seeds.

\|C\Delta\mathbf{y}\|

is the change in the training-loss gradient caused by the label change, and

u

is the exact-Hessian solution for one label change. The last row gives

J

’s

R^{2}

with the Gauss–Newton Hessian.

HIGGS (GPU), one row of $J$ : median 0.07 s, range 0.07–0.14 s, 30 rows
Dataset	$N_{\mathrm{train}}$	Base fit (s)	$J$ columns (s)	$K$ refits (s)
Friedman #1 (CPU)	$2{,}000$	$7.2{\scriptstyle\pm 0.7}$	$0.35{\scriptstyle\pm 0.00}$	$73{\scriptstyle\pm 91}$ ( $K{=}100$ )
MAGIC Gamma (CPU)	$14{,}265$	$7.7{\scriptstyle\pm 0.4}$	$2.1{\scriptstyle\pm 0.0}$	$14{\scriptstyle\pm 2}$ ( $K{=}100$ )
HIGGS (GPU)	$1.0{\cdot}10^{6}$	$337{\scriptstyle\pm 141}$	$263{\scriptstyle\pm 0}$	$444{\scriptstyle\pm 94}$ ( $K{=}100$ )
MNIST (GPU)	$2{,}000$	$21{\scriptstyle\pm 7}$	$138{\scriptstyle\pm 22}$	$379{\scriptstyle\pm 218}$ ( $K{=}20$ )
HIGGS (GPU), one-time setup per model: median 271.11 s, range 263.72–271.34 s

Appendix

Table 8: Wall-clock per seed in seconds, mean

\pm

SD over five seeds. The last two lines time single rows of

J

on HIGGS and the one-time setup before them. Base fit is one training run.

J

columns covers forming and factorising the exact Hessian and the solves for the sampled columns. On MNIST it covers the MINRES solves for

J

K

refits covers the refits used as ground truth.

Influence Functions

Influence functions estimate how training examples or interventions change model parameters, predictions, and downstream behavior.

See all 40 papers

Interaction-Aware Influence Functions for Group Attribution
May 15, 2026Jaeseung Heo, Kyeongheung Yun, Youngbin Choi +3Instruction-Tuning Data Selection Influence Functions
GSAI, POSTECH · CSE, POSTECH
Data Attribution via Sketched Metadifferentiation
Sep 14, 2026Yuxi Chen, Hamza Golubovic, Han Tong +2Influence Functions Training Data Attribution
Carnegie Mellon University 1Department of Statistics & Data Science · Columbia University 2Department of Statistics · Columbia University 3Electrical Engineering Department +2
Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attribution
Sep 25, 2026Zhe Li, Wei Zhao, Peixin Zhang +1Training Data Attribution Influence Functions
Singapore Management University

May 15, 2026J/K move · Enter open · S save

Interaction-Aware Influence Functions for Group Attribution

Jaeseung Heo, Kyeongheung Yun, Youngbin Choi, Sehyun Hwang+2

GSAI, POSTECH · CSE, POSTECH

Instruction-Tuning Data Selection Influence Functions Training Data Attribution

Open paper AskPDF arXiv

Influence functions approximate how removing a training example changes a quantity of interest, called the target function, such as a held-out loss. To estimate the influence of a group of examples, the standard practice is to sum the individual influences of its members. However, this sum does not capture how examples jointly affect the target: a pair of examples may be redundant or complementary, but the sum cannot distinguish these cases. We propose an interaction-aware influence function that characterizes how interactions between examples influence the target. By expanding the target to second order around the trained parameters, we obtain an estimator that augments the standard sum with a pairwise interaction term that captures the alignment between two examples' effects on the target. We empirically evaluate our estimator in two settings. First, on six dataset-model pairs spanning logistic regression, MLPs, and ResNet-9, our estimator tracks leave-group-out retraining substantially better than first-order influence across all settings. Second, when used as a greedy selection rule for instruction-tuning data on Llama-3.1-8B, it beats prior influence-based and representation-similarity baselines on five of seven downstream tasks, in a regime where standard influence-based selection underperforms random selection.

Abstract

Influence Functions

Explore similar work

Interaction-Aware Influence Functions for Group Attribution

Data Attribution via Sketched Metadifferentiation

Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attribution