Early Memory Selection for Balanced Adam

Alberto Fernández-Hernández; Cristian Pérez-Corral; Jose I. Mestre; Manuel F. Dolz; Enrique S. Quintana-Ortí

cs.LGOct 6, 2026

Early Memory Selection for Balanced Adam

Authors: Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí

Organizations: Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Abstract

We propose a method for choosing the shared memory parameter $β_1=β_2=β$ in Adam from a short pilot training. The selected $β$ remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared $β=0.95$ . The mean gap is also 32.3% lower than that of the best constant $β$ chosen across all eleven workloads.

Figures & tables

1	Run $T_{\rm p}$ pilot updates with shared $\beta_{\rm pilot}$ . At $T_{\rm p}$ and $T_{\rm p}-n_{\ell}$ , collect $m$ probe gradients with parameters fixed.
2	Compute pooled and $G$ group moment estimates, normalize them using ( 13 ), and store their images under $P$ .
3	Estimate variability using ( 14 ); fit ( 17 ) and compute change using ( 18 ).
4	Apply ( 21 ), restrict memory to the candidate range, convert to $\beta$ , and round to $\mathcal{G}$ .
5	Restart from the pilot’s initialization and use this fixed shared $\beta$ with the prescribed full-training recipe.

Table 1: Early selection of a fixed shared memory. Measurements are made during the pilot; the full training uses the selected

\beta

throughout.

Network	Dataset	Oracle $\beta$	Selected $\beta$	Relative gap (%)
EfficientNet-B0	Cars	0.82217	0.96838	1.0124
Llama60M	C4	0.94377	0.90000	0.2582
Llama60M	SlimPajama	0.94377	0.90000	0.1175
NanoGPT	OpenWebText	0.94377	0.90000	0.1157
NanoGPT	WikiText-103	0.94377	0.90000	0.1613
ResNet50	Food-101	0.99684	0.96838	1.0613

Table 2: Early memory choices and their distance to the workload-specific grid reference. Pilot and candidate trainings use the same initialization (seed 1). Oracle

\beta

minimizes the smallest recorded validation loss over the eleven grid candidates and is used only for evaluation. The selector uses early gradient measurements. Gaps are percentages.

	Mean	Maximum	CVaR 25
Fixed 0.90000	1.4292	3.8770	3.3543
Fixed 0.94377	0.6236	2.4853	2.0409
Best fixed 0.96838	0.5461	2.1862	1.4304
Tracking rule	0.3695	1.3339	1.1359

Table 3: Workload-specific calibration versus universal fixed memories. Gaps are percentages. Best fixed minimizes mean gap over all eleven fixed grid choices using their validation outcomes. Lower is better.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Network	Dataset	Steps	Batch	Accum.	LR max	Decay	Clip
EfficientNet-B0	Cars	12800	64	1	0.0008	5e-05	1
Llama60M	C4	10000	64	8	0.001	1e-05	0
Llama60M	SlimPajama	10000	64	8	0.001	1e-05	0
NanoGPT	OpenWebText	10000	8	128	0.0006	0.01	1
NanoGPT	WikiText-103	10000	8	128	0.0006	0.01	1
ResNet50	Food-101	29550	128	1	0.0003	0.01	0

Appendix

Table 4: Pilot schedule and optimizer settings. Steps is the configured schedule horizon; batch is physical batch size. Accumulation defines the effective batch. Clip zero denotes an unmodified gradient norm.

Asset	License information or provider record
PyTorch / torchvision	BSD-style / BSD-3-Clause: PyTorch license and torchvision license .
NumPy / Transformers	BSD-3-Clause / Apache-2.0: NumPy license and Transformers license .
T5-small checkpoint	Apache-2.0, as stated in the provider model card . The other evaluated models start from random weights.
C4 / WikiText-103	C4 card: ODC-By . WikiText card: CC-BY-SA-3.0 and GFDL .
SlimPajama-6B	The subset card points to the original SlimPajama record for source and license information. This original provider record supplies the access provenance used here.
OpenWebText / BookCorpus	Provider records: OpenWebText and BookCorpus . Rights in underlying web text and books remain with their respective sources.

Appendix

Table 5: Software and model license records, and dataset access sources. The providers’ records govern access and reuse of the original assets.

Decision update	Mean gap	Maximum gap	CVaR 25
100	0.6345	2.1862	1.5271
150	0.3695	1.3339	1.1359
200	0.3695	1.3339	1.1359
250	0.3695	1.3339	1.1359

Appendix

Table 6: Decision-time sensitivity with sixteen probes per checkpoint and matched seed 1. Gaps are percentages. The near-0.95 comparator,

\beta

0.94377, has mean 0.6236, maximum 2.4853, and CVaR 2.0409.

Effective batches	Groups	Mean gap	Maximum gap	CVaR 25
2	2	0.9185	2.5913	2.4600
4	4	0.6829	2.4853	2.0409
8	4	0.6829	2.4853	2.0409
16	4	0.3695	1.3339	1.1359

Appendix

Table 7: Probe-budget sensitivity at update 200. Relative gaps are percentages; the fixed comparator has mean 0.6236%, maximum 2.4853%, and CVaR 25 2.0409%. Bold marks the smallest entry in each metric column.

Fixed shared $\beta$ or method	Mean gap	Maximum gap	CVaR 25
0.68377	4.8152	11.6651	9.9106
0.82217	2.0501	6.4732	4.5653
0.90000	1.4292	3.8770	3.3543
0.94377	0.6236	2.4853	2.0409
0.96838	0.5461	2.1862	1.4304
0.98222	1.8128	7.2826	4.5072

Appendix

Table 8: All fixed shared memories on the evaluated grid, and the tracking rule. Gaps are percentages. Bold marks the smallest value in each metric column.

				Gap benefit (pp)		Loss improvement (%)
Network	Dataset	$\widehat{\beta}$	Rule gap	.94377	.96838	.94377	.96838
EfficientNet-B0	Cars	0.96838	1.0124	-0.2853	+0.0000	-0.2833	+0.0000
Llama60M	C4	0.90000	0.2582	-0.2582	-0.0700	-0.2582	-0.0699
Llama60M	SlimPajama	0.90000	0.1175	-0.1175	-0.0924	-0.1175	-0.0924
NanoGPT	OpenWebText	0.90000	0.1157	-0.1157	-0.1137	-0.1157	-0.1137
NanoGPT	WikiText	0.90000	0.1613	-0.1613	-0.0800	-0.1613	-0.0800

Appendix

Table 9: Signed task-level benefits against the near-0.95 fixed value and the best global fixed value. Gap benefits are percentage points; direct loss improvements are percentages. Positive entries favor the rule.

Excluded task			Mean-gap reduction (%)		Best fixed on ten
Network	Dataset	Rule mean	vs .94377	vs .96838	$\beta$	Reduction (%)
EfficientNet-B0	Cars	0.3052	+50.23	+38.91	0.96838	+38.91
Llama60M	C4	0.3806	+44.51	+34.60	0.96838	+34.60
Llama60M	SlimPajama	0.3947	+42.46	+34.03	0.96838	+34.03
NanoGPT	OpenWebText	0.3948	+42.44	+34.25	0.96838	+34.25
NanoGPT	WikiText	0.3903	+43.10	+34.14	0.96838	+34.14

Appendix

Table 10: Workload-removal sensitivity. Each row evaluates the remaining ten tasks. Reduction columns give relative mean-gap reductions; positive values favor the rule. The last comparator is reselected on that ten-task subset.

Network	Dataset	$\widehat{V}_{\rm emp}$	$\widehat{D}_{\rm emp}$	$\widehat{H}_{\rm emp}$	Continuous $\beta$	Grid $\beta$
EfficientNet-B0	Cars	$1.978\times 10^{2}$	$2.379\times 10^{-3}$	27.494	0.963628	0.96838
Llama60M	C4	$2.875\times 10^{1}$	$1.047\times 10^{-2}$	8.823	0.886654	0.90000
Llama60M	SlimPajama	$2.111\times 10^{1}$	$9.030\times 10^{-3}$	8.361	0.880399	0.90000
NanoGPT	OpenWebText	$2.142\times 10^{1}$	$1.183\times 10^{-2}$	7.678	0.869751	0.90000
NanoGPT	WikiText-103	$2.292\times 10^{1}$	$8.208\times 10^{-3}$	8.871	0.887272	0.90000
ResNet50	Food-101	$1.632\times 10^{2}$	$2.203\times 10^{-3}$	26.458	0.962205	0.96838

Appendix

Table 11: Measured coefficients and continuous-to-grid conversion. Values are rounded for display; decisions use full precision.

Network	Dataset	Continuous $\beta$	Nearest boundary	$\beta$ distance	Ratio change (%)
EfficientNet-B0	Cars	0.963628	0.956075	0.007553	-43.224
Llama60M	C4	0.886654	0.861085	0.025569	-45.679
Llama60M	SlimPajama	0.880399	0.861085	0.019314	-36.180
NanoGPT	OpenWebText	0.869751	0.861085	0.008666	-17.571
NanoGPT	WikiText-103	0.887272	0.861085	0.026187	-46.563
ResNet50	Food-101	0.962205	0.956075	0.006130	-36.294

Appendix

Table 12: Rounding-boundary audit from the same measured coefficients. Ratio change is the signed relative change needed to reach the nearest

\beta

midpoint.

Network	Dataset	$\widehat{\beta}$	Adjacent $\beta$	$\Delta$ ratio (%)	$\Delta$ gap (pp)	$\Delta$ loss (%)
EfficientNet-B0	Cars	0.96838	0.94377	-43.22	-0.2853	-0.2825
Llama60M	C4	0.90000	0.82217	-45.68	+0.6477	+0.6460
Llama60M	SlimPajama	0.90000	0.82217	-36.18	+0.7058	+0.7050
NanoGPT	OpenWebText	0.90000	0.82217	-17.57	+0.4311	+0.4306
NanoGPT	WikiText	0.90000	0.82217	-46.56	+0.2984	+0.2979
ResNet50	Food-101	0.96838	0.94377	-36.29	+1.4240	+1.4091

Appendix

Table 13: Nearest decision boundary and its recorded loss consequence. Ratio change moves the observed coefficient ratio to that boundary. Gap and loss changes compare the adjacent choice with the current choice; positive changes indicate a larger loss.

Network	Dataset	Pooling contribution (%)	Finite-lag contribution (%)
EfficientNet-B0	Cars	3.12	100.00
Llama60M	C4	4.45	54.57
Llama60M	SlimPajama	4.63	91.79
NanoGPT	OpenWebText	3.17	100.00
NanoGPT	WikiText-103	3.34	100.00
ResNet50	Food-101	3.09	100.00

Appendix

Table 14: Sixteen-probe measurement decomposition at update 200.

Coefficient diagnostic	Mean gap	Maximum gap	CVaR 25
Constant-target calculation	0.5461	2.1862	1.4304
Endpoint contribution only	0.5682	2.1862	1.5271
Intercept contribution only	17.9403	72.4969	51.6408
Full measured rule	0.3695	1.3339	1.1359

Appendix

Table 15: Contribution diagnostics using the same sketches, rounding, and seed-1 loss lookup. Gaps are percentages; bold marks the smallest value in each column.

	Mean relative gap (%)				Direct improvement (%)
Summary	Rule	Fixed .94377	Best fixed	Fixed $\beta$	Seed 1	Common seeds
Minimum recorded	0.3695	0.6236	0.5461	0.96838	+0.2464	-0.1049
Last recorded	0.4537	0.6242	0.5707	0.96838	+0.1629	-0.1093
Mean of last three	0.4244	0.4767	0.4767	0.94377	+0.0492	-0.1586

Appendix

Table 16: Complete sensitivity to three evaluation summaries. Mean gaps and direct loss improvements are percentages. Best fixed is recomputed for each summary. Direct improvements use the near-0.95 comparator 0.94377; positive values favor the rule. Common-seed values evaluate reuse of the seed-1 calibration across initializations.

Network	Dataset	Common seeds	Improvement (%)	Rule $\widehat{\beta}$
EfficientNet-B0	Cars	1,2,3	-2.1064	0.96838
Llama60M	C4	1,2,3	-0.2015	0.90000
Llama60M	SlimPajama	1,2,3	-0.2985	0.90000
NanoGPT	OpenWebText	1,2,3	-0.2046	0.90000
NanoGPT	WikiText-103	1,2,3	-0.1235	0.90000
ResNet50	Food-101	1	1.3895	0.96838

Appendix

Table 17: Direct validation-loss improvement over the same common seeds against

\beta

0.94377. Zero means the selected and fixed configurations coincide.

Explore similar work

Refresh-Scaling the Memory of Balanced Adam
May 11, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adam Momentum
Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain
Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adam Momentum
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
EvE: An Alternate Optimizer to Adam
Sep 28, 2026Shashank Raj, Kalyanmoy DebAdam Evolutionary Optimization Methods
Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

May 11, 2026, cs.LGJ/K move · Enter open · S save

Refresh-Scaling the Memory of Balanced Adam

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz+1

Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain

Adam Momentum

Open paper AskPDF arXiv

Recent evidence suggests that Adam performs robustly when its momentum parameters are tied,

β_1=β_2

, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam,

β

should not be treated as a dimensionless constant: it defines a statistical memory horizon

H_β=(1-β)^{-1}

. In terms of the effective learning horizon

T_{\mathrm{ES}}

, estimated from the validation trajectory, we study the refresh count

R_β=(1-β)T_{\mathrm{ES}}

, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing

β

so that

R_β\approx1000

selects different

β

values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice

β=0.944

, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4%, while bringing all 11 runs within 1% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.

Abstract

Explore similar work

Refresh-Scaling the Memory of Balanced Adam

Why β1=β2β_1 = β_2β1​=β2​ Is Dynamically Special in Adam

EvE: An Alternate Optimizer to Adam

Why $β_1 = β_2$ Is Dynamically Special in Adam