Training-data influence estimates support data selection and cleansing by predicting the effects of removing individual examples. The standard multi-epoch SGD-Influence Estimator (SGD-IE) aggregates contributions from a sample's repeated appearances along the recorded trajectory. When used to approximate finite deletion, it omits an interaction: later removals act on the parameter displacement accumulated from earlier removals. We introduce the Accumulative SGD-Influence Estimator (ACC-SGD-IE) to account for how earlier removals change the effect of removing the same sample again. It uses the sample's Hessian at subsequent appearances to correct the estimated parameter change. Our non-convex analysis shows that this correction removes SGD-IE's additional interaction contribution from the finite-horizon error bound: the omitted interaction produces a local error linear in the accumulated parameter change, while the remaining Taylor remainder is bounded quadratically under Lipschitz-Hessian assumptions. The bounds characterize how training propagates the remaining error. In non-convex experiments across clean and noisy tabular, text, and vision datasets, ACC-SGD-IE reduces validation-loss-change RMSE by 10.8% and improves Jaccard@10% by 15.2% on average relative to SGD-IE. These gains also translate into lower test misclassification rates after data cleansing on MNIST and CIFAR10 at the evaluated removal budgets. On noisy industrial datasets, it reduces aggregate parameter RMS error with a mean relative reduction of 53.4%. We further discuss computational tradeoffs, provide complementary theoretical analysis and experimental evaluation for strongly convex objectives, and explore potential extensions of the accumulative correction mechanism to other influence estimators.
Figures & tables
Fig. 1: Illustrative finite-deletion mismatch across epochs in classical SGD-Influence Estimator. The red trajectory denotes the counterfactual with repeated deletion; the blue/orange paths illustrate independently injected linearized occurrence contributions. Their sum omits the interaction induced by a later deletion acting on the earlier displacement.
Dataset
Setting
Method
RMSE ↓
Kendall’s Tau ↑
Jacc@70% ↑
Jacc@50% ↑
Jacc@30% ↑
Jacc@10% ↑
Part 1. Estimation on Clean Data.
Adult
–
SGD-IE
2.40(1.10)×10−3
0.4064(0.13)
0.6059(0.05)
0.4868(0.09)
0.4331(0.11)
0.3629(0.09)
ACC-SGD-IE
2.20(1.10)×10−3
0.4220(0.13)
0.6080(0.05)
0.4914(0.09)
0.4388(0.11)
0.3834(0.09)
20News
–
SGD-IE
1.82(1.41)×10−2
0.2248(0.16)
0.5582(0.02)
0.3696(0.04)
0.2424(0.07)
0.1403(0.08)
ACC-SGD-IE
1.78(1.48)×10−2
0.2632(0.16)
0.5682(0.02)
0.3884(0.06)
0.2489(0.08)
0.1589(0.10)
MNIST
–
SGD-IE
3.40(6.10)×10−3
0.4009(0.15)
0.6095(0.04)
0.4794(0.07)
0.4019(0.12)
0.4129(0.15)
TABLE I: Comprehensive comparison of influence–estimation accuracy and robustness under non-convex regime. The best value for each metric within a setting is bold . We add i.i.d. noise x←x+N(0,σ2) to every feature on Adult & MNIST. We randomly set a fraction σ×100% of the zero entries in each one-hot word vector to 1 on 20News. A fraction ρ×100% of the training labels is flipped uniformly at random.
Fig. 2: Cross-epoch estimation performance under non-convex training. Left: loss-change RMSE. Right: Jaccard@10%. Blue bars denote SGD-IE, orange bars ACC-SGD-IE, and red curves the relative improvement.
Fig. 3: Data cleansing performance measured by mean test misclassification rate.
Dataset
Parameter RMS ratio
Plugin RMSE ratio
Steel
0.682±0.132
0.830±0.105
Hydraulic
0.351±0.258
0.565±0.314
Gas drift
0.364±0.221
0.602±0.379
TABLE II: Estimation accuracy on noisy industrial data: ACC/SGD error ratios (mean ± SD over five seeds).
Samples
SGD-IE Backward
SGD-IE Forward
ACC-SGD-IE Forward
1
2.056
1.194
1.511
8
1.615
1.707
1.887
64
1.373
4.324
5.405
512
1.505
29.967
31.400
TABLE III: Runtime on Steel (seconds).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Definition
D
Training dataset of n samples.
n
Total number of training samples.
x
Input feature vector of dimension d .
y
One-hot encoded label vector over c classes.
p
Dimensionality of the parameter vector θ∈Rp .
θ
Model parameter vector.
Appendix
TABLE IV: Comprehensive summary of notation used throughout this work.
Fig. 4: Loss change estimation under non-convex (left) and convex (right) regimes. Results on clean, feature-noisy, and label-noisy data. Points closer to the black diagonal indicate more accurate estimates of the real loss change. Parts 1–3 correspond respectively to the entries in Table I and Table V .
Dataset
Setting
Method
RMSE ↓
Kendall’s Tau ↑
Jacc@70% ↑
Jacc@50% ↑
Jacc@30% ↑
Jacc@10% ↑
Part 1. Estimation on Clean Data.
Adult
-
SGD-IE
1.00(1.00)×10−4
0.9724(0.00)
0.9616(0.01)
0.9561(0.01)
0.9306(0.02)
0.9446(0.04)
ACC-SGD-IE
1.00(1.00)×10−4
0.9774(0.00)
0.9627(0.01)
0.9608(0.01)
0.9410(0.02)
0.9369(0.05)
20News
-
SGD-IE
1.04(1.25)×10−2
0.9086(0.07)
0.9038(0.08)
0.8574(0.11)
0.7832(0.18)
0.6863(0.26)
ACC-SGD-IE
9.90(12.30)×10−3
0.9691(0.00)
0.9560(0.02)
0.9466(0.01)
0.9279(0.01)
0.9357(0.02)
MNIST
-
SGD-IE
2.50(2.80)×10−3
0.9739(0.00)
0.9806(0.00)
0.9465(0.01)
0.9395(0.01)
0.9063(0.05)
Appendix
TABLE V: Comprehensive comparison of influence–estimation accuracy and robustness under strong convex regime. The best value for each metric within a setting is bold . We add i.i.d. noise x←x+N(0,σ2) to every feature on Adult & MNIST. We randomly set a fraction σ×100% of the zero entries in each one-hot word vector to 1 on 20News. A fraction ρ×100% of the training labels is flipped uniformly at random.
Method
J@70%
J@50%
J@30%
J@10%
Adam-IE
0.5427
0.4085
0.1840
0.0390
ACC-Adam-IE
0.5699
0.4337
0.2060
0.0390
Appendix
TABLE VII: Jaccard overlap for Adam-IE and ACC-Adam-IE after four epochs.
Fig. 6: Loss change estimation performance over 40 training epochs under the non-convex regime (left) and convex regime (right) for six (M,α) configurations.