cs.LGSep 28, 2026

EvE: An Alternate Optimizer to Adam

Authors: Shashank Raj, Kalyanmoy Deb

Organizations: Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adam and its variants dominate neural network training, but a single run only reveals whether a configuration works well after most of its budget is spent, a poor fit for hyperparameter or architecture search, where configurations must be ranked cheaply and pruned early. We introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via DE, running a short burst of gradient descent only if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and since gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven scalable benchmarks up to one million variables. On three real neural-network tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE finishes the same charged budget 1.7-3.9x faster, at a modest cost in final quality (about one accuracy point on MNIST, 9-11% higher relative test loss on the two fine-tuning tasks; on GSM8K, Adam is about 5 accuracy points more accurate, and fine-tuning lowers accuracy below the base model for both). Inside successive halving on UCI Adult, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, ranking configurations about as consistently with Adam as Adam does with itself across seeds (Kendall's tau 0.66-0.69). EvE is not a total replacement for Adam as a final-stage trainer, but a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up.

Figures & tables

Table 1: EvE vs. Adam on the scalable synthetic benchmarks, at fixed evaluation-cost budget (Section 5 ). Lower is better. Min/Med/Max over 21 seeds per (problem, nn ) pair. A shaded, bold median is the significantly lower (better) side (one-sided Wilcoxon signed-rank, Holm-Bonferroni corrected, family-wise α=0.05\alpha=0.05 ); shaded, bold, italic on both sides marks no significant difference.
Figure 1: EvE vs. eight baselines on MLP/MNIST, under the same evaluation-cost budget (Section 5.1 ), at each method’s own best learning rate over 5 seeds. (a) training runs vs. budget and vs. wall-clock time; see Figure 2 for panels (b)–(d).
Figure 2: (cont.) (b) learning-rate sensitivity. (c) wall-clock cost of the shared budget, sorted. (d) final test accuracy reached.
Figure 3: SHA sweep wall-clock time vs. best validation accuracy found, UCI Adult, mean ±\pm std over 5 seeds. EvE reaches comparable accuracy in roughly a third of the wall-clock time of every baseline in both search spaces.
Figure 4: EvE vs. eight baselines fine-tuning Qwen2.5-1.5B-Instruct with LoRA on Alpaca, under the same evaluation-cost budget (Section 5.1 ), at each method’s own best learning rate over 5 seeds. (a)/(b) training runs vs. budget and vs. wall-clock time. (c) learning-rate sensitivity. (d) wall-clock cost of the shared budget, sorted. (e) final test loss reached (lower is better).
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Measured peak GPU memory, EvE vs. Adam, on MLP/MNIST (Section 5.3 ), 200200 real training iterations per method, each in its own CUDA context. The measured ratio ( 2.95×2.95\times peak allocated) sits below the parameter-storage-only theoretical ratio ( 3.25×3.25\times ) derived above.
TaskMethods with same rateEvE rate (test / val.)EvE result (test / val.)Largest baseline change
MLP/MNIST55 of 9910−310^{-3} / 10−310^{-3}97.03%97.03\% / 97.03%97.03\%0.120.12 points
LoRA/Alpaca77 of 993×10−43\times 10^{-4} / 3×10−43\times 10^{-4}1.2401.240 / 1.2401.2400.0020.002
LoRA/GSM8K77 of 9910−210^{-2} / 10−410^{-4}0.6400.640 / 0.6540.6540.00040.0004
Appendix
Table 2: Effect of choosing each method’s learning rate on validation instead of test data, from the same saved runs.
TaskEvE (s)Baselines (s)Speedup
MLP/MNIST ( B=3,000B=3{,}000 )49.349.3182.3182.3 – 192.3192.33.73.7 – 3.9×3.9\times
LoRA/Alpaca ( B=300B=300 )35.0±0.635.0\pm 0.658.858.8 – 70.770.71.71.7 – 2.0×2.0\times
LoRA/GSM8K ( B=900B=900 )100.4±4.7100.4\pm 4.7193.8193.8 – 198.7198.71.91.9 – 2.0×2.0\times
UCI Adult, hyperparameter search21.8±0.321.8\pm 0.373.673.6 – 76.876.83.43.4 – 3.5×3.5\times
UCI Adult, architecture search21.5±0.921.5\pm 0.966.966.9 – 75.875.83.13.1 – 3.5×3.5\times
Appendix
Table 3: Wall-clock training time in seconds under the shared budget, EvE against the eight baselines (range).
SettingValue
Evaluation-cost budget BB2,0002{,}000 (Section 5.1 accounting)
Problem dimensions nn{2,10,20,50,100,200,500,1,000,105,106}\{2,10,20,50,100,200,500,1{,}000,10^{5},10^{6}\}
Seeds{0,…,20}\{0,\dots,20\} (21 independent runs per cell)
EvE population size npopn_{\text{pop}}44 (fixed, Section 3.1 )
Population initializationLatin Hypercube Sampling (see below), not proximity (Section 3.2 )
DE weight FF0.50.5 , jitter γ=10−4\gamma=10^{-4} (Section 3.3 )
Appendix
Table 4: Shared parameters for the synthetic scalable-benchmark sweep (Table 1 ).
ProblemEvE betterIndistinguishableAdam betterWhere Adam wins
Ackley730–
Griewank1000–
Rastrigin730–
Rosenbrock0100–
Schwefel127n≥50n\geq 50
Sphere0010all nn
Appendix
Table 5: Per-problem summary of EvE vs. Adam on the synthetic benchmarks: over the ten dimensions swept, the number of (problem, nn ) cells where EvE is significantly better, where the two are statistically indistinguishable, and where Adam is significantly better (one-sided Wilcoxon signed-rank, paired by seed, Holm-Bonferroni corrected, family-wise α=0.05\alpha=0.05 ; 2121 seeds per cell). The full min/median/max at every cell is Table 1 in the main text.
LayerShapeActivationParameters
Input784784––
Hidden 1784→1,024784\to 1{,}024ReLU802,816+1,024802{,}816+1{,}024 (weight+bias)
Hidden 21,024→1,0241{,}024\to 1{,}024ReLU1,048,576+1,0241{,}048{,}576+1{,}024
Output1,024→101{,}024\to 10–10,240+1010{,}240+10
Total  1,863,6901{,}863{,}690
Appendix
Table 6: MLP architecture. fan_in is the preceding layer’s width; biases inherit their weight tensor’s fan-in.
SettingValue
Evaluation-cost budget BB3,0003{,}000 (Section 5.1 accounting)
Learning-rate grid{10−4,3×10−4,10−3,3×10−3,10−2}\{10^{-4},3\times 10^{-4},10^{-3},3\times 10^{-3},10^{-2}\}
Seeds{0,1,2,3,4}\{0,1,2,3,4\} (5 independent runs per cell)
Minibatch size256256
Weight decay5×10−45\times 10^{-4} (decoupled for AdamW; L2 for SGD+momentum and EvE’s Adam fallback)
SGD momentum0.90.9
Appendix
Table 7: Shared MLP/MNIST training protocol.
MethodTime (s)Best Val Acc
EvE21.8 ±\pm 0.30.8544 ±\pm 0.0018
SGD+mom.73.6 ±\pm 1.30.8549 ±\pm 0.0020
NAdam75.1 ±\pm 0.50.8591 ±\pm 0.0018
RAdam75.5 ±\pm 0.90.8584 ±\pm 0.0009
AdaBelief75.6 ±\pm 0.80.8579 ±\pm 0.0010
Adam75.7 ±\pm 0.90.8582 ±\pm 0.0019
Appendix
Table 8: UCI Adult, learning-rate/weight-decay SHA sweep, mean ±\pm std over 5 seeds, sorted by wall-clock time.
MethodTime (s)Best Val Acc
EvE21.5 ±\pm 0.90.8544 ±\pm 0.0010
AdaBelief66.9 ±\pm 1.20.8587 ±\pm 0.0017
Lookahead68.2 ±\pm 1.10.8588 ±\pm 0.0006
SGD+mom.69.6 ±\pm 0.90.8533 ±\pm 0.0014
AMSGrad69.8 ±\pm 2.10.8588 ±\pm 0.0017
RAdam70.5 ±\pm 1.70.8585 ±\pm 0.0012
Appendix
Table 9: UCI Adult, architecture (width/depth) SHA sweep, mean ±\pm std over 5 seeds, sorted by wall-clock time.
ComparisonHyperparameter searchArchitecture search
EvE vs. Adam0.663±0.0930.663\pm 0.0930.691±0.0320.691\pm 0.032
EvE vs. AdamW0.677±0.0630.677\pm 0.0630.709±0.0410.709\pm 0.041
Adam vs. Adam (different training seeds)0.694±0.0610.694\pm 0.0610.670±0.0290.670\pm 0.029
Adam vs. AdamW0.714±0.0780.714\pm 0.0780.848±0.0180.848\pm 0.018
Appendix
Table 10: Rank agreement (Kendall’s τ\tau ) at the first SHA rung over the same 4848 configurations, mean ±\pm std over 55 seeds.
SettingValue
Evaluation-cost budget BB300300 (Section 5.1 accounting)
Learning-rate grid{10−4,3×10−4,10−3,3×10−3,10−2}\{10^{-4},3\times 10^{-4},10^{-3},3\times 10^{-3},10^{-2}\}
Seeds{0,1,2,3,4}\{0,1,2,3,4\} (5 independent runs per cell)
Minibatch size22
Max sequence length512512 tokens
LoRA rank / alpha3232 / 6464
Appendix
Table 11: Shared LoRA/Alpaca training protocol ( B=300B=300 ; GSM8K uses the same protocol except B=900B=900 , Appendix F ).
Figure 6: EvE vs. eight baselines fine-tuning Qwen2.5-1.5B-Instruct with LoRA directly on GSM8K, under a shared evaluation-cost budget of 900900 ( 3×3\times Figure 4 ’s Alpaca budget), at each method’s own best learning rate over 5 seeds. (a)/(b) training runs vs. budget and vs. wall-clock time. (c) learning-rate sensitivity. (d) wall-clock cost of the shared budget, sorted. (e) final test loss reached (lower is better).
CheckpointStrict ( "#### N" )FlexibleFlexible intervalTraining time (s)
Base model (no fine-tuning)27.9%27.9\%73.5%73.5\%[71.1, 75.9][71.1,\,75.9]–
EvE, 900900 / 30,00030{,}000 evaluations54.4%54.4\%54.4%54.4\%[51.7, 57.1][51.7,\,57.1]105105 / 3,3963{,}396
EvE, 10,00010{,}000 evaluations53.7%53.7\%53.7%53.7\%[51.0, 56.4][51.0,\,56.4]1,1041{,}104
Adam, 900900 evaluations59.4%59.4\%59.4%59.4\%[56.7, 62.0][56.7,\,62.0]196196
Adam, 10,00010{,}000 evaluations58.2%58.2\%58.2%58.2\%[55.5, 60.9][55.5,\,60.9]2,0932{,}093
Adam, 30,00030{,}000 evaluations52.5%52.5\%52.6%52.6\%[49.9, 55.3][49.9,\,55.3]6,2846{,}284
Appendix
Table 12: GSM8K downstream exact-match accuracy on the full 1,3191{,}319 -question test set, one training run per row, with the wall-clock training time of that run in seconds. The interval is the 95%95\% Wilson interval on flexible accuracy. EvE’s 30,00030{,}000 -evaluation adapter is identical to its 900900 -evaluation adapter, so its accuracy is shown once, with both training times.

Explore similar work

CardsList

  1. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresAdamGradient Descent

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  2. Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

    Jun 27, 2026Young-Jun Lee, Seungone Kim, Minki Kang +5Evolutionary SearchContinual Fine-Tuning

    University of Minnesota · Carnegie Mellon University · KAIST +3

  3. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarAdamTime Discretization

    1Independent Researcher.