Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

Kang Liu; Suyan Li

cs.LGAug 20, 2026

Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

Authors: Kang Liu, Suyan Li

Organizations: School of Future Technology, Xi’an Jiaotong University, Xi’an, China, 710049 · Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 119077.

Abstract

A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.

Figures & tables

Figure 1: Finite-horizon ISO view of delayed minibatch influence in AdamW.

Probe	$\alpha$	NRMSE	Rel. $M$ err.	Sign acc.
Standard	$1/32$	0.0117	0.0052	1.000
	$1/8$	0.0483	0.0216	1.000
	$1/4$	0.1007	0.0445	1.000
Anisotropic	$1/32$	0.0173	0.0071	1.000
	$1/8$	0.0719	0.0289	1.000
	$1/4$	0.1473	0.0606	1.000

Table 1: Representative signed-trajectory fidelity in the controlled stage of Experiment 1. Entries are medians over 32 independent systems after candidate aggregation.

Setting	Full ISO	Exact 1-step	No propagation	Gradient norm
Quadratic, standard probe	0.993	–	0.354	-0.062
Quadratic, rotating-readout probe	1.000	–	0.806	0.776
CNN–ReLU	0.888	0.545	0.755	0.535
MLP–GELU	0.762	0.566	0.668	0.336

Table 2: Within-system Spearman correlation with future magnitude

M

during mechanism validation. The controlled rows use exact-one-step-matched candidates; the neural rows use natural candidates. The one-step score is tied in the matched controlled stress test.

Scale	Local NRMSE	Cosine	ISO-FD $\rho$
0.41B	0.1090	0.99918	0.714
1.0B	0.0519	0.99995	0.833
1.4B	0.0567	0.99994	0.762

Table 3: Language-model scaling results in Experiment 1. Local metrics pool the three datasets and

\alpha\leq 0.25

; rank correlations are medians over the three fixed data-domain conditions at each model size.

System	Candidates	$\Pi_{H}$	Branch $\rho$	1-step $\rho$	Frozen ISO $\rho$	Ref.-surrogate $\rho$
Quadratic, standard	Natural	0.801	0.919	0.801	0.659	0.372
	Matched	0.730	0.909	–	-0.066	-0.021
MLP–GELU	Natural	0.771	0.862	0.661	0.734	0.262
	Matched	0.921	0.955	–	0.941	0.752
CNN–ReLU	Natural	0.716	0.855	0.619	0.752	0.601
	Matched	0.700	0.872	–	0.755	0.811

Table 4: Prospective structure under 32 independently resampled future continuations.

\Pi_{H}

is defined in Eq. ( 14 ); branch

\rho

is the median single-future Spearman correlation with the conditional-mean candidate ranking. Present-score correlations target

\mu_{i}

. All entries are medians over independent systems.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Probe	$\alpha$	NRMSE	Rel. $M$ err.	Sign acc.
Standard	$1/32$	0.0117	0.0052	1.000
	$1/8$	0.0483	0.0216	1.000
	$1/2$	0.2256	0.0945	1.000
	$1$	0.5538	0.2006	0.938
Anisotropic	$1/32$	0.0173	0.0071	1.000
	$1/8$	0.0719	0.0289	1.000

Appendix

Table 5: Controlled-stage signed-trajectory fidelity in Experiment 1. Entries are medians over 32 independent systems after candidate-level aggregation.

Quantity	Standard	Rotating-readout
Calibration success	$1.000\ [1.000,1.000]$	$1.000\ [1.000,1.000]$
CV of $\|d_{1}\|$	$1.10\!\times\!10^{-12}$	$6.70\!\times\!10^{-12}$
CV of future $M$	$0.092\ [0.064,0.114]$	$0.459\ [0.292,0.530]$
$M/\|d_{1}\|$	$6.21\ [5.69,7.86]$	$4.54\ [3.38,5.08]$
Parameter gain	$7.71\ [7.10,8.90]$	$7.69\ [7.06,8.83]$

Appendix

Table 6: Controlled-stage exact-one-step-matched stress test in Experiment 1. CV is computed over 16 candidates within a system; entries are medians across 32 independent systems, with IQRs shown for the matching diagnostics.

Method	Standard	Rotating-readout
Full ISO	0.993	1.000
No propagation	0.354	0.806
Frozen dynamics	0.788	0.969
Initial parameter only	0.137	0.788
Clamped parameter state	0.254	0.793
Frozen readout	0.371	0.831

Appendix

Table 7: Median Spearman correlation under exact-one-step matching. The one-step score is tied by construction.

		Standard probe			Anisotropic probe
Channels	State	$M$	ARE	$h^{\star}$	$M$	ARE	$h^{\star}$
None	$000$	0	0	1.00	0	0	1.00
$\theta$	$100$	$1.80\!\times\!10^{-4}$	$3.75\!\times\!10^{-3}$	6.50	$1.43\!\times\!10^{-3}$	$2.95\!\times\!10^{-2}$	6.00
$m$	$010$	$8.60\!\times\!10^{-4}$	$1.91\!\times\!10^{-2}$	15.75	$8.90\!\times\!10^{-3}$	$1.71\!\times\!10^{-1}$	16.25
$v$	$001$	$2.73\!\times\!10^{-4}$	$4.79\!\times\!10^{-3}$	25.25	$2.47\!\times\!10^{-3}$	$4.86\!\times\!10^{-2}$	27.25
$\theta,m$	$110$	$9.98\!\times\!10^{-4}$	$2.19\!\times\!10^{-2}$	14.25	$9.72\!\times\!10^{-3}$	$1.94\!\times\!10^{-1}$	15.00

Appendix

Table 8: Complete post-shock state-channel intervention in the controlled stage of Experiment 1. Entries are medians after first aggregating selected candidates within each of the 32 independent systems. Channel order is

(\theta,m,v)

		Standard probe			Anisotropic probe
Channel	Decay	$M$	ARE	$h^{\star}$	$M$	ARE	$h^{\star}$
$m$	0.5	$6.58\!\times\!10^{-5}$	$1.08\!\times\!10^{-3}$	5.25	$5.75\!\times\!10^{-4}$	$1.08\!\times\!10^{-2}$	6.25
$m$	0.8	$1.44\!\times\!10^{-4}$	$2.74\!\times\!10^{-3}$	9.75	$1.33\!\times\!10^{-3}$	$2.61\!\times\!10^{-2}$	11.75
$m$	0.9	$2.66\!\times\!10^{-4}$	$5.96\!\times\!10^{-3}$	16.75	$2.59\!\times\!10^{-3}$	$5.46\!\times\!10^{-2}$	17.75
$m$	0.95	$7.16\!\times\!10^{-4}$	$1.47\!\times\!10^{-2}$	30.50	$5.62\!\times\!10^{-3}$	$1.20\!\times\!10^{-1}$	28.75
$m$	0.99	$4.65\!\times\!10^{-3}$	$7.23\!\times\!10^{-2}$	32.00	$3.41\!\times\!10^{-2}$	$5.34\!\times\!10^{-1}$	32.00

Appendix

Table 9: Controlled-stage matched-first-displacement persistence sweep in Experiment 1. The first future parameter displacement is held fixed within each channel.

Architecture	$\alpha$	NRMSE	Sign agr.	Rel. $M$ err.	Rel. ARE err.	Extremum-sign acc.
CNN–ReLU	0.0625	0.0458	1.000	0.0113	0.0115	1.000
	0.125	0.0610	1.000	0.0160	0.0185	1.000
	0.25	0.1097	1.000	0.0299	0.0307	1.000
	0.5	0.1959	1.000	0.0587	0.0632	1.000
	1	0.3854	0.958	0.1094	0.1126	1.000
MLP–GELU	0.0625	0.0377	1.000	0.0116	0.0140	1.000

Appendix

Table 10: Neural-network finite-horizon tangent fidelity in Experiment 1. Entries are medians over independent systems after aggregating candidates within systems.

Method	CNN–ReLU	MLP–GELU
Full ISO	0.888	0.762
Frozen dynamics	0.843	0.671
Clamped parameter state	0.818	0.661
No propagation	0.755	0.668
Initial parameter only	0.734	0.720
Frozen readout	0.713	0.507

Appendix

Table 11: Median within-system Spearman correlation with future magnitude in the neural-network stage of Experiment 1.

State	CNN ARE	CNN $h^{\star}$	MLP ARE	MLP $h^{\star}$
$100$	0.216	2.5	0.250	1.5
$010$	0.821	9.0	0.827	7.5
$001$	0.030	10.0	0.015	9.5
$110$	1.025	9.0	1.010	6.5
$101$	0.214	3.0	0.237	1.5
$011$	0.812	9.0	0.818	7.5

Appendix

Table 12: Neural-network exact state-channel interventions in Experiment 1. Values are median ARE relative to the full

111

intervention; bit order is

(\theta,m,v)

$\alpha$	NRMSE	75% NRMSE	Cosine	Rel. $M$	Sign acc.
0.0625	0.0387	0.1664	0.99990	0.0224	1.000
0.125	0.0477	0.1906	0.99993	0.0263	1.000
0.25	0.0946	0.2095	0.99983	0.0524	1.000
0.5	0.2105	0.4431	0.99938	0.1081	0.986
1	0.3902	0.7867	0.99668	0.2235	0.903

Appendix

Table 13: Language-model finite-scale trajectory fidelity in Experiment 1 over 72 candidates.

Scale	Local NRMSE	Cosine	Valid 0.25	Valid 1	ISO-FD $\rho$
0.41B	0.1090	0.99918	0.958	0.417	0.714
1.0B	0.0519	0.99995	0.792	0.667	0.833
1.4B	0.0567	0.99994	0.875	0.583	0.762

Appendix

Table 14: Language-model scaling summary in Experiment 1. Ranking correlations are medians over the three data-domain conditions at each scale.

Model	Dataset	ISO tangent (FD)	Exact 1-step	Gradient norm	Parameter write
410M	CodeParrot	0.857	0.643	-0.286	-0.262
410M	OpenWebText	0.548	0.738	0.571	0.595
410M	WikiText-103	0.714	0.643	0.571	0.619
1B	CodeParrot	0.952	0.905	-0.167	-0.048
1B	OpenWebText	0.333	0.929	0.071	-0.286
1B	WikiText-103	0.833	0.762	-0.214	0.048

Appendix

Table 15: Language-model within-system Spearman correlation in Experiment 1 with

M_{i}=\max_{h}|d_{i,h}(1)|

. Each condition contains eight candidates.

System	Candidates	$\Pi_{H}(M)$	Branch $\rho$
Quadratic, standard	Natural	$0.801\ [0.709,0.877]$	$0.919\ [0.866,0.949]$
	Matched	$0.730\ [0.594,0.845]$	$0.909\ [0.839,0.941]$
Quadratic, anisotropic	Natural	$0.718\ [0.602,0.825]$	$0.899\ [0.846,0.943]$
	Matched	$0.603\ [0.501,0.727]$	$0.895\ [0.844,0.931]$
MLP–GELU	Natural	$0.771\ [0.621,0.844]$	$0.862\ [0.795,0.914]$
	Matched	$0.921\ [0.885,0.951]$	$0.955\ [0.937,0.959]$

Appendix

Table 16: Primary repeated-future magnitude results in Experiment 2. Each cell reports the median across independent systems; brackets give the system-level IQR. Branch

\rho

is the within-system median correlation between an individual future-branch ranking and the repeated-future mean ranking.

System	Candidates	Exact 1-step	Grad. norm	Write norm	Frozen ISO	Ref.-surrogate ISO
Quadratic, standard	Natural	0.801	0.212	0.253	0.659	0.372
	Matched	–	0.049	-0.057	-0.066	-0.021
Quadratic, anisotropic	Natural	0.790	0.394	0.374	0.562	0.490
	Matched	–	0.272	0.025	0.031	0.025
MLP–GELU	Natural	0.661	0.318	0.325	0.734	0.262
	Matched	–	0.066	0.339	0.941	0.752

Appendix

Table 17: Present-time Spearman correlation with the repeated-future mean magnitude

\mu_{i}^{M}

in Experiment 2. The exact one-step score is tied in the matched family and its correlation is therefore undefined.

Condition	CV of $\|d_{1}\|$
Quadratic, standard	$1.07\!\times\!10^{-12}\ [7.97\!\times\!10^{-13},1.75\!\times\!10^{-12}]$
Quadratic, anisotropic	$1.33\!\times\!10^{-12}\ [6.50\!\times\!10^{-13},1.84\!\times\!10^{-12}]$
MLP–GELU	$3.57\!\times\!10^{-13}\ [2.79\!\times\!10^{-13},5.73\!\times\!10^{-13}]$
CNN–ReLU	$1.36\!\times\!10^{-12}\ [1.08\!\times\!10^{-12},2.02\!\times\!10^{-12}]$

Appendix

Table 18: Exact-one-step matching audit for Experiment 2. CV is computed within each candidate family; entries are medians with IQRs. Calibration success is 1.000 in every condition.

Condition	Between-candidate variance	Mean within-future variance
Quadratic, standard	$9.92\!\times\!10^{-10}$	$2.39\!\times\!10^{-10}$
Quadratic, anisotropic	$4.25\!\times\!10^{-8}$	$2.35\!\times\!10^{-8}$
MLP–GELU	$4.31\!\times\!10^{-8}$	$5.45\!\times\!10^{-9}$
CNN–ReLU	$2.02\!\times\!10^{-8}$	$1.13\!\times\!10^{-8}$

Appendix

Table 19: Absolute variance components for the matched primary magnitude. Values are medians of the system-specific quantities in Eq. ( 95 ); the ratio of the two medians need not equal the median of the system-specific ratios.

System	Candidates	$\Pi_{H}(M)$	$\Pi_{H}(A)$	$\Pi_{H}(P^{+})$
Quadratic, standard	Natural	0.801	0.817	0.704
	Matched	0.730	0.729	0.995
Quadratic, anisotropic	Natural	0.718	0.722	0.570
	Matched	0.603	0.603	0.990
MLP–GELU	Natural	0.771	0.816	0.807
	Matched	0.921	0.887	0.995

Appendix

Table 20: Prospective-structure ratio for the primary magnitude and two secondary response summaries.

P^{+}

is sign-sensitive; because the matched construction enforces alternating signed one-step targets, the positive-peak column is reported only as a descriptive robustness check and is not used to support the main prospective-identifiability claim.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions
Aug 25, 2026Jinhui GuoNeural Network Optimization Neural Network Training Dynamics
School of Physics, Beihang University, Beijing 100191, China
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
Aug 21, 2026Yiman Fong, Heng YangEdge of Stability Adaptive Gradient Methods
School of Engineering and Applied Sciences, Harvard University.

Aug 25, 2026J/K move · Enter open · S save

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

Jinhui Guo

School of Physics, Beihang University, Beijing 100191, China

Neural Network Optimization Neural Network Training Dynamics Learning to Optimize Adaptive Loss Weighting

Open paper AskPDF arXiv

Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether this delayed transport is large enough to change prospective short-horizon decisions. On committed future-minibatch sequences, we differentiate eight-step AdamW trajectories through the complete model--optimizer state and select exposure-matched Math--Code loss schedules before independent evaluation. Across 12 unused 0.3M Transformer histories, full transport lowers token-disjoint loss relative to an optimizer-aware immediate derivative in 10/12 histories (mean benefit

4.71\times10^{-4}

; exact one-sided sign test,

p=0.0193

). The two controllers act equally often but select different schedules in 60/96 windows. Crossed checkpoint--future-path tests attribute this reordering to the interaction between optimizer state and near-future data, while an independent Ising--CNN experiment shows that deleting moment-state transport destroys accurate response prediction. Full-transport scores also concentrate exact-rollout winners in larger candidate libraries, focusing finite-amplitude evaluation on a shortlist. On these committed short paths, optimizer memory and near-future data order are therefore actionable components of the training state, providing a mechanism-based criterion for when finite-horizon rather than one-step intervention is required.

Abstract

Neural Network Optimization

Explore similar work

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

Gefen: Optimized Stochastic Optimizer

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal