Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics

Ahad Bin Islam Shoeb; Jamal Uddin Tanvin; Kamrul Hasan; Alve Reduan; Imtiaz Ahmed

cs.LGOct 17, 2026

Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics

Authors: Ahad Bin Islam Shoeb, Jamal Uddin Tanvin, Kamrul Hasan, Alve Reduan, Imtiaz Ahmed

Organizations: University of Dhaka · Military Institute of Science and Technology · Tennessee State University · Howard University

Abstract

Instance-dependent early stopping (IES) reduces training computation by skipping backpropagation for examples whose losses have stabilized, but the second-order loss-difference criterion used by IES lacks a theoretical justification. We develop Theory-Guided Instance-Dependent Early Stopping (TG-IES), which analyzes this criterion under explicit assumptions and derives a noise-calibrated mastery threshold and a conditional stability bound for instance exclusion. We show that second-order differencing has a favorable bias-variance trade-off over first- and third-order differences in a specific signal-to-noise regime, and that its signal contains both gradient-drift and curvature terms. TG-IES derives its stopping threshold and patience from training-run statistics rather than using the manually selected threshold of IES. Across CIFAR-10 and CIFAR-100, TG-IES saves 53-58% of backpropagation instances in its main convolutional settings while keeping accuracy within 0.1-0.9 percentage points of full training; in the primary ResNet-18/CIFAR-10 setting, this corresponds to 35.7% FLOP-normalized savings. The method also transfers to ViT-Tiny, where it achieves smaller savings and incurs a modest accuracy cost. These results provide a theoretical basis for second-order instance-dependent early stopping and show that theory-guided calibration can substantially reduce training computation under the assumptions and experimental settings studied.

Figures & tables

Panel of Figure 1 — Figure 1: TG-IES uses each instance’s loss trajectory to derive a second-order loss signal. Observed noise sets the mastery threshold $\delta$ , and a conditional reference horizon $K_{\mathrm{safe}}$ initializes patience $K^{*}$ . The mastery rule decides when an instance becomes passive and when it re-enters training. Skipping passive instances alters updates by the passive-set gradient $\omega_{t}$ , governing drift and descent (dashed box). Experiments assess compute savings, accuracy, and the method’s assumptions.

Symbol	Meaning	Symbol	Meaning
$L_{i}^{(t)}$	loss of instance $i$ after epoch $t$	$\theta^{(t)}$ , $\eta_{t}$	parameters after epoch $t$ ; step size
$\tilde{L}_{i}^{(t)}$	recorded loss $L_{i}^{(t)}+\xi_{i,t}$ (noise scale $\sigma$ )	$L$	training loss $\frac{1}{N}\sum_{i}L_{i}$
$\Delta^{1}L_{i}^{(t)}$	$L_{i}^{(t)}-L_{i}^{(t-1)}$ : how much the loss moved	$g_{i}$	$\nabla_{\theta}L_{i}$ : update direction from instance $i$
$\Delta^{2}L_{i}^{(t)}$	$\Delta^{1}L_{i}^{(t)}-\Delta^{1}L_{i}^{(t-1)}$ : change of that move	$g$	$\frac{1}{N}\sum_{i}g_{i}$ : full-data gradient
$\Delta^{N}L_{i}^{(t)}$	the difference applied $N$ times	$H_{i}$	$\nabla^{2}_{\theta}L_{i}$ : local curvature of $L_{i}$
$\delta$ , $K^{*}$	mastery threshold on $\|\Delta^{2}\tilde{L}_{i}\|$ ; patience	$A_{t}$ , $P_{t}$	active / passive instances

Table 1: Notation used in the main text.

Table 3

SGD with exponential learning-rate decay
	Baseline	IES (literal)		TG-IES (ours)
Setting	Acc. (%)	Acc. (%)	Saved (%)	Acc. (%)	Saved (%)
ResNet-18 / CIFAR-10 †	$95.03\pm 0.13$	$94.85\pm 0.17$	$49.02\pm 0.44$	$94.85\pm 0.28$	$53.63\pm 0.30$
VGG-16 / CIFAR-10	$93.30\pm 0.12$	$93.15\pm 0.17$	$41.02\pm 1.12$	$93.11\pm 0.14$	$54.43\pm 0.12$
DenseNet-121 / CIFAR-10	$95.36\pm 0.14$	$95.24\pm 0.02$	$51.60\pm 0.89$	$95.28\pm 0.09$	$54.53\pm 0.66$
ResNet-18 / CIFAR-100	$76.88\pm 0.41$	$76.78\pm 0.40$	$18.49\pm 0.43$	$76.01\pm 0.08$	$57.76\pm 0.04$

Table 4: All three-seed settings, mean

\pm

std ( † primary). IES (literal), the higher-saving IES variant in every setting, is shown; all runs are in Table 11 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Cell	$n$	Acc. (%)	Saved (%)
CIFAR-10 / ResNet-18 / SGD-exp (primary)	3	$94.85\pm 0.28$	$53.63\pm 0.30$
CIFAR-10 / ResNet-18 / SGD-fixed	3	$91.91\pm 0.18$	$29.27\pm 0.30$
CIFAR-10 / ResNet-18 / SGD-linear	1	$94.85^{*}$	$32.05^{*}$
CIFAR-10 / ResNet-18 / SGD-step	1	$94.44^{*}$	$43.52^{*}$
CIFAR-10 / ResNet-18 / SGD-cosine	1	$94.73^{*}$	$23.15^{*}$
CIFAR-10 / ResNet-18 / Adam	3	$92.44\pm 0.10$	$30.18\pm 0.62$

Appendix

Table 5: TG-IES, every distinct (dataset, architecture, optimizer/schedule) cell in this paper, same four fixed calibration constants throughout. Mean

\pm

std over seeds where

n{>}1

; asterisked rows are single-seed by design ( Appendix K ’s matched-epoch schedule comparison).

Method	Test Acc. (%)	Backprop Saved (%)
Baseline	$92.91\pm 0.06$	$0.03\pm 0.00$
IES (shipped)	$92.61\pm 0.10$	$56.02\pm 0.22$
IES (literal)	$92.42\pm 0.10$	$\mathbf{60.09\pm 0.29}$
TG-IES (ours)	$92.44\pm 0.10$	$30.18\pm 0.62$

Appendix

Table 6: ResNet-18, CIFAR-10, Adam, mean

\pm

std, 3 seeds.

Figure 4: Direct instrumentation of $\Delta^{2}L_{i}^{(t)}=\eta_{t}D_{i}^{(t)}+O(\eta_{t}^{2})+R_{i,t}$ , fixed 300-instance subset, baseline training, all valid epochs, one panel per schedule. Dashed line: $y=x$ . The drift term correlates weakly with the observed signal and does not dominate the residual under any schedule.

Figure 5: Best-so-far test accuracy vs. cumulative compute budget (% of $N\times$ total-epochs), all three cells with new-baseline data. A method’s curve stops at its own final compute point.

Figure 6: Pooled lag- $k$ autocorrelation of detrended per-instance loss residuals, epochs 101–200, ResNet-18/CIFAR-10 seed 0 ( $n{\approx}1.9$ M pairs/lag). Shaded band: 95% null interval for zero correlation.

Figure 7: Backprop saved (left) and best test accuracy (right) vs. gradient-norm-floor percentile, ResNet-18 and VGG-16, CIFAR-10, seed 0. Dotted line: this paper’s fixed default (20th percentile).

Method	Wall-clk (min)	Peak Mem (MiB)	Util (%)	Energy (Wh)
Baseline	15.66	1162	67.7	98.2
IES (shipped)	12.86	1216	62.8	78.1
IES (literal)	12.26	1184	59.0	70.1
TG-IES	$\mathbf{11.66}$	1454	$\mathbf{48.9}$	$\mathbf{59.9}$

Appendix

Table 7: Measured efficiency, primary setting, single RTX 5090. Wall-clock: mean over 3 seeds from the real 200-epoch runs. Peak Mem/Util/Energy: short instrumented runs, seed 0; Energy is an estimate. Lower is better throughout.

Figure 8: Wall-clock time, peak GPU memory, and estimated energy, each as % of no-removal baseline. Wall-clock and energy fall furthest for TG-IES; peak memory is the one metric that rises above baseline, only for TG-IES.

Panel of Figure 9 — Figure 9: Left: empirical MSE $(N)$ vs. predicted noise floor, naive population-average test (all instances, epochs 101–200). This pooled test does not isolate Proposition 4.3 ’s SNR band and finds $N{=}1$ lowest; the band-restricted test (main text), which evaluates each instance at the epoch its signal first drops below the calibrated $\delta{=}0.01597$ , finds $N{=}2$ lowest ( $7.5\times 10^{-5}$ pooled over $46{,}779$ instances), as predicted. Right: empirical $\hat{r}_{i}$ pooled per training-epoch quartile; rate rises and spread narrows in the second half of training (mean $\hat{r}$ from $0.53$ to $0.76$ ; CV from $1.05$ to $0.75$ ), so Assumption 3.2 fits late training better.

Figure 10: Per-instance gradient norm $\|g_{i}^{(t)}\|$ vs. $\sqrt{|\Delta^{2}L_{i}^{(t)}|}$ , all three LR schedules, primary architecture/dataset (300-instance subset). Theorem 5.4 predicts a positive, roughly linear relationship; correlations shown per panel.

	CIFAR-10		CIFAR-100
Method	Acc. (%)	Saved (%)	Acc. (%)	Saved (%)
Baseline	$84.07\pm 0.35$	$0.03\pm 0.00$	$59.25\pm 0.24$	$0.03\pm 0.00$
IES (shipped)	$83.92\pm 0.18$	$13.13\pm 0.21$	$59.19\pm 0.22$	$8.38\pm 0.22$
IES (literal)	$83.86\pm 0.48$	$16.68\pm 0.32$	$58.77\pm 0.24$	$12.51\pm 0.21$
TG-IES (ours)	$83.71\pm 0.47$	$\mathbf{25.52\pm 0.26}$	$58.56\pm 0.47$	$\mathbf{26.28\pm 0.14}$

Appendix

Table 8: ViT-Tiny (CIFAR-native, patch-4), CIFAR-10 and CIFAR-100, AdamW warmup-cosine, mean

\pm

std, 3 seeds.

$K^{*}$ (fixed)	Acc. (%)	Saved (%)
1	$94.55\pm 0.16$	$\mathbf{66.98\pm 0.62}$
2	$94.56\pm 0.08$	$65.75\pm 0.27$
4	$94.62\pm 0.10$	$63.24\pm 0.44$
8	$94.80\pm 0.09$	$58.85\pm 0.57$
15	$94.80\pm 0.09$	$53.38\pm 0.58$
30	$\mathbf{94.87\pm 0.09}$	$44.13\pm 0.40$

Appendix

Table 9:

K^{*}

-sweep, primary cell, mean

\pm

std, 3 seeds,

\delta

noise-calibrated. Matched baseline:

95.08\pm 0.14\%

acc.

Figure 11: Parameter-space distance between TG-IES’s actual trajectory and the shared-seed baseline reference vs. epoch, one curve per fixed $K^{*}$ , 3-seed mean. Left: absolute distance. Right: normalized by the baseline’s own parameter norm at that epoch.

Epoch	40	60	80	100	120	140	160	180	200
Frac. $>\varepsilon_{0}$	$0.52$	$0.48$	$0.39$	$0.46$	$0.48$	$0.46$	$0.47$	$0.50$	$0.50$

Appendix

Table 10: Reactivation probe, primary cell, 3-seed mean of frac_exceed_eps0 (fraction of a 300-instance random subsample of the passive set whose true gradient norm exceeds the run’s calibrated

\varepsilon_{0}

) at each probed epoch.

Setting	Method	Seed	Acc. (%)	Saved (%)
ResNet-18/E	Baseline	0	94.96	0.03
ResNet-18/E	Baseline	1	95.18	0.03
ResNet-18/E	Baseline	2	94.95	0.03
ResNet-18/E	IES (shipped)	0	94.61	45.36
ResNet-18/E	IES (shipped)	1	94.74	46.09
ResNet-18/E	IES (shipped)	2	94.79	45.53

Appendix

Table 11: All 96 IES-family runs. E = SGD exponential LR, F = SGD fixed LR, C100 = CIFAR-100, ViT/C10 and ViT/C100 = ViT-Tiny on CIFAR-10/CIFAR-100. Color on TG-IES rows compares against the other methods for that same seed.

Metric	Comparison (TG-IES $-$ )	Mean $\Delta$	$95\%$ CI	$d_{z}$	$p_{\mathrm{sign}}$
Primary: ResNet-18, CIFAR-10, SGD (exponential LR)
Acc. (%)	Baseline	$-0.18$	$[-0.59,\,0.23]$	$-1.11$	$0.250$
Acc. (%)	IES (shipped)	$0.13$	$[-0.48,\,0.75]$	$0.54$	$1.000$
Acc. (%)	IES (literal)	$0.00$	$[-1.09,\,1.09]$	$-0.01$	$1.000$
Backprop Saved (%)	Baseline	$53.60$	$[52.86,\,54.34]$	$179.2$	$0.250$
Backprop Saved (%)	IES (shipped)	$7.97$	$[6.57,\,9.37]$	$14.2$	$0.250$

Appendix

Table 12: Paired per-seed statistics, TG-IES vs. each other method, every 3-seed setting (

n=3

). CI:

95\%

t(\mathrm{df}{=}2)

d_{z}

: Cohen’s paired effect size.

p_{\mathrm{sign}}

: exact two-sided sign test (floor

0.25

n=3

Early Stopping

Early stopping uses signals during training or inference to halt computation before a fixed budget, balancing generalization, quality, and cost.

See all 46 papers

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation
Aug 6, 2026Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre DavidEfficient Neural Network Inference Binary Neural Networks
Polytechnique Montréal, Department of Electrical Engineering Montréal, Canada
ScoreStop: Gradient-based early stopping using functional score tests
Jun 1, 2026Oliver J. Hines, Christian L. HinesGradient-Boosted Decision Trees Early Stopping
Columbia University, NY, USA · Alan Turing Institute, London, UK
Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning
May 27, 2026Minhao Yao, Ruoyu Wang, Xihong Lin +2Early Stopping Neural Network Training Dynamics
Centre for Biomedical Data Science, Duke-NUS Medical School, National University of Singapore · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA · Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai, SJTU-Yale2026 Joint Center of Biostatistics and Data Science, Shanghai Jiao Tong University +1

Aug 6, 2026J/K move · Enter open · S save

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre David

Polytechnique Montréal, Department of Electrical Engineering Montréal, Canada

Efficient Neural Network Inference Binary Neural Networks Feedforward Neural Networks

Open paper AskPDF arXiv

Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference. For binary activations, the dot products become sign-controlled additions or subtractions, but the number of operations is unchanged. Indeed, every neuron or output channel still accumulates all of its input, even though only the sign will be retained, which is often wasteful. As the accumulation progresses, the running partial sum frequently drifts so far from zero that its final sign becomes highly predictable long before the last term is reached; every contribution evaluated after that point changes the value of the sum but not the final output activation. This paper turns this observation into a post-training early-stopping mechanism. We characterize the behavior of the running accumulations on the training dataset and use this information to predict the final sign as soon as possible. No model parameter is retrained. We count the number of operations under an idealized ordering of weights. On VGG11 applied to the CIFAR-10 dataset, the method removes

86.6\%

of the accumulation terms of the deepest convolution for a

0.37

-point accuracy drop, and

25\%

of the full-network arithmetic when used on the three deepest convolutions simultaneously, for a

1.36

-point drop.

Abstract

Early Stopping

Explore similar work

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

ScoreStop: Gradient-based early stopping using functional score tests

Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning