Hybrid Momentum Methods for Optimizing Deep Learning Models

Haifeng Xu; Yijin Ren; Chenyu Zhou; Qi Deng; Dongdong Ge; Yinyu Ye

cs.LGOct 20, 2026

Hybrid Momentum Methods for Optimizing Deep Learning Models

Authors: Haifeng Xu, Yijin Ren, Chenyu Zhou, Qi Deng, Dongdong Ge, Yinyu Ye

Organizations: Shanghai University of Finance and Economics · Columbia University · Shanghai Jiao Tong University · SIMIS ·

Abstract

Momentum is central to stochastic optimization in deep learning, including the training of large language models. Classical momentum methods typically rely on a fast exponential moving average (EMA) scheme, which discounts past gradients on a fixed timescale and may underweight older gradients that remain informative. This creates a trade-off between retaining long-term history and adapting to recent changes. We introduce Hybrid Momentum (HyM), which combines a linearly weighted average of past gradients with a geometrically discounted fast momentum. The former assigns weights proportional to the gradients' iteration indices and retains a growing span of history, while the latter emphasizes recent gradients. This hybrid construction is modular and can be combined with adaptive preconditioning and other update transformations. In particular, we propose HyMUON, which applies Newton--Schulz orthogonalization to the hybrid direction. Illustrative examples and controlled experiments demonstrate the benefits of combining gradient information across these two timescales. Across image classification, machine translation, and language modeling, HyM and HyMUON match or improve upon strong baselines. HyMUON improves over Muon on IWSLT14, and in both the 100k- and 600k-step GPT-2 124M OpenWebText regimes. For LLaMA-1.3B pretraining on C4, HyMUON achieves a final validation loss of 2.777, compared with 2.903 for Muon.

Figures & tables

Panel of Figure 1 — Figure 1: Gradient weight profiles for AdEMAMix ( $\beta_{3}=0.9999$ ), classical EMA ( $\mu=0.999$ ), and HyM ( $\mu=0.999$ , $\beta=0.9$ ) at $t\in\{1000,10000,50000\}$ (left to right). Each curve is normalized by the weight on the most recent gradient.

Panel of Figure 2 — Figure 2: Persistence of the effect of a single batch and of its contribution retained in the slow state. Left: loss on batch $\mathcal{B}$ for the control and inject branches. The two branches differ at step 5,000, when the inject branch takes one optimization step using the gradient computed on $\mathcal{B}$ ; they then continue with the same sequence of batches. Right: at step 7,500, the inject branch is split into Keep, Delete, and Refresh, which retain, remove, or replace the contribution of $\mathcal{B}$ in the slow state. At this step, the three branches have the same model parameters, averaged iterate, and all other optimizer states. Refresh uses a gradient computed on $\mathcal{B}$ at the current model parameters and rescales it to match the groupwise norm of the original contribution. HyMUON curves are evaluated at the averaged iterate, whereas Muon curves are evaluated at the training iterate; all curves are unsmoothed. Full protocols are given in Appendices E.3 and E.3.1 .

Figure 3: Loss reduction on each semantic target’s evaluation set after a single update using the gradient of a candidate passage, at the 10k, 50k, and 100k checkpoints (left to right). For each target, the 2,000 candidate passages are ranked by their E5 similarity to the target’s reference set and divided into 100 groups of 20. The blue curves show the mean across ten semantic targets, and the shaded regions show error bands across targets. Positive values indicate a reduction in loss, and the vertical scales differ across panels.

Panel of Figure 4 — Figure 4: Image classification. Left: CIFAR-10 (top) and CIFAR-100 (bottom) test accuracy. Middle: SVHN test loss and accuracy. Right: ImageNet-1K validation loss and accuracy. Curves show means across repeated runs, and the shaded regions show one standard error. Values in the panel legends are endpoint means with sample standard deviations. Appendix Figure 11 adds the CIFAR test loss curves.

Panel of Figure 5 — Figure 5: Language tasks. Top row, left to right: IWSLT14 German–English test loss for the LSTM model; GPT-2 124M validation loss on OpenWebText through 100k steps; and LLaMA-1.3B validation loss on C4 through 10k steps, compared with the reproduced Muon baseline. Bottom row, left to right: GPT-2 124M validation loss on OpenWebText through 600k steps, followed by LLaMA-124M validation loss on FineWeb with token budgets of 1B and 2.1B. The IWSLT curves show means across repeated runs, and the shaded regions show one standard error. Appendix Figures 14 and 16 give BLEU and C4 perplexity.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Behavior of the Rosenbrock training iterates from $(-3,5)$ over 20,000 steps using exact gradients. Momentum SGD uses $\eta=4\times 10^{-3}$ and $\mu\in\{0.9,0.95,0.99\}$ ; HyM uses $\eta=3\times 10^{-4}$ , $\beta=0.9$ , and $\mu\in\{0.9,0.95,0.99\}$ ; Schedule-Free SGD uses $\eta=7\times 10^{-4}$ and $\beta\in\{0.9,0.99\}$ . The left panel shows objective values at $\mathbf{y}_{t}$ ; the remaining panels show the trajectories over objective contours. The star marks the minimizer $(1,1)$ .

Panel of Figure 7 — Figure 7: Averaged iterates for the Rosenbrock runs in Figure 6 , using the same initialization, optimizer settings, and 20,000 steps using exact gradients. HyM uses $\beta=0.9$ for iterate averaging; Momentum SGD is evaluated at an averaged iterate formed with $\beta=0.99$ , and this average does not enter its updates; Schedule-Free SGD uses its respective $\beta\in\{0.9,0.99\}$ . The left panel shows objective values at $\mathbf{x}_{t}$ ; the remaining panels show the averaged trajectories over objective contours. The star marks the minimizer $(1,1)$ .

Panel of Figure 8 — Figure 8: Test loss against training batch size on CIFAR-10 (left) and CIFAR-100 (right). The learning rate is chosen separately for each method and batch size.

Figure 9: Curves of loss reduction across 100 percentiles of E5 similarity. Rows are the 10k, 50k, and 100k checkpoints; columns are semantic targets (left) and randomly mixed controls (right). Candidates are ranked within each target, with 20 candidates per percentile. Curves are means over ten targets of each type; shaded bands are based on standard errors across targets. The blue curve uses E5 similarity. Positive values indicate loss reduction; vertical scales differ. The panels in Figure 3 are the left column of this figure, rearranged in one row.

Figure 10: Share of the net loss reduction on semantic targets contributed by the candidates in the top $1\%,5\%,10\%$ , and $20\%$ by E5 similarity. Panels are the 10k, 50k, and 100k checkpoints, left to right. The blue curve uses the E5 ranking from Figure 9 . The numerator sums $I(C,V)$ over the selected candidates and all ten semantic targets; the denominator sums over all candidates and the same targets. The dashed line marks the share proportional to subset size. Because negative values are retained, cancellation can produce a share above $100\%$ ; the plotted quantities are net contribution shares, not fractions of positive loss reductions.

Image classification: accuracy (%) $\uparrow$
Dataset / model / budget	SGD Cosine	SGD Step	Schedule- Free	HyM
CIFAR-10 / WRN-16-8 300 epochs	$95.90\pm 0.12$	$95.43\pm 0.17$	$96.11\pm 0.10$	$96.25\pm 0.10$
CIFAR-100 / DenseNet-BC 300 epochs	$77.66\pm 0.39$	$77.22\pm 0.20$	$79.11\pm 0.22$	$79.14\pm 0.22$
SVHN / ResNet-3-96 300 epochs	$98.24\pm 0.05$	–	$98.20\pm 0.08$	$98.47\pm 0.02$
ImageNet-1K / ResNet-50 100 epochs	$76.68\pm 0.04$	–	$76.70\pm 0.06$	$76.84\pm 0.10$

Appendix

Table 1: Main experimental results. Image accuracy and IWSLT metrics are mean

\pm

sample SD across repeated runs; OpenWebText, FineWeb, and C4 give the final validation loss. ImageNet accuracy is validation top-1; CIFAR and SVHN use test accuracy. A dash marks a method not included in that comparison. FineWeb losses are rounded to two decimals.

Dataset	Epochs	Batch size	Input size
CIFAR-10	300	128	$32\times 32$
CIFAR-100	300	64	$32\times 32$
SVHN (train + extra)	300	32	$32\times 32$
ImageNet-1K	100	256	$224\times 224$

Appendix

Table 2: Training settings for image classification. Batch sizes are global numbers of images per optimizer step.

Dataset	Optimizer	LR	Momentum	$\beta$	Weight decay
CIFAR-10	SGD + Cosine	0.2	0.9	–	$10^{-4}$
	SGD + Step	0.1	0.9	–	$10^{-4}$
	Schedule-Free	10	–	0.9	$10^{-4}$
	HyM	0.05	0.98	0.9	$5\times 10^{-4}$
CIFAR-100	SGD + Cosine	0.05	0.9	–	$2\times 10^{-4}$
	SGD + Step	0.05	0.9	–	$2\times 10^{-4}$

Appendix

Table 3: Fixed optimizer settings for image classification. LR denotes the peak learning rate. The momentum column gives the SGD or HyM momentum coefficient;

\beta

gives the Schedule-Free interpolation coefficient or the HyM mixing weight.

Dataset (batch)	SGD	HyM	Change	SGD	HyM	Change
	Training time			Peak allocated memory
CIFAR-10 (128)	4.71	4.98	$+5.7\%$	1122	1254	$+132$
CIFAR-100 (512)	6.80	6.73	$-0.9\%$	7410	7423	$+12$
ImageNet-1K (256)	591.62	601.43	$+1.7\%$	11257	11561	$+304$

Appendix

Table 4: H100 resource profiles for HyM versus cosine SGD. Batch sizes in parentheses are the profiling batch sizes. Time is seconds per training epoch; memory is peak CUDA allocated MiB. Changes are HyM relative to cosine SGD.

Panel of Figure 11 — Figure 11: Curves of CIFAR test loss for the runs in Figure 4 . Left: CIFAR-10; right: CIFAR-100. Curves are means over repeated runs, and the bands show one standard error; the legend gives the sample SD. HyM is evaluated at the averaged iterate $\mathbf{x}_{t}$ .

Panel of Figure 12 — Figure 12: CIFAR comparison of the training and averaged iterates. Columns show CIFAR-10 and CIFAR-100; rows show test loss and test accuracy. HyM $(y)$ is evaluated at the training iterate without iterate averaging, while HyM $(x)$ is evaluated at the averaged iterate. The other curves reuse the baselines from Figure 4 .

	$\eta_{\max}=0.05$		$\eta_{\max}=0.1$		$\eta_{\max}=0.2$
Direction	$y$	$x$	$y$	$x$	$y$	$x$
Fast momentum only	84.26	90.44	76.78	83.76	65.68	72.18
Slow momentum only	89.64	92.06	86.80	89.02	81.82	84.48
Full HyM	93.94	96.22	93.44	96.10	91.92	95.52
EMA, $\rho=0.99$	85.04	90.86	77.60	83.40	68.50	73.18
EMA, $\rho=0.999$	89.30	93.40	84.18	89.58	79.88	82.74

Appendix

Table 5: Final CIFAR-10 validation accuracy (%) in the component ablation. Each group defined by a learning rate compares the same six configurations;

y

and

x

are paired evaluations at epoch 300. EMA rows replace only the slow momentum. These are validation results on the 45k/5k split, not the test accuracies in the main image classification experiment.

Panel of Figure 13 — Figure 13: CIFAR-10 component ablation: validation cross-entropy at the training iterate $y$ (left) and averaged iterate $x$ (right). Rows use peak learning rates $0.05$ , $0.1$ , and $0.2$ . Curves show values for each epoch. Every evaluation uses the same BN calibration protocol. “Slow EMA” retains the fast momentum and replaces only the slow momentum.

Hyperparameter	Muon	HyMUON
Peak LR for hidden matrices	0.05	0.05
Momentum $\mu$ for hidden matrices	0.99	0.9
HyM mixing weight $\beta$	–	0.9
Weight decay for hidden matrices	0.05	0.05
Auxiliary peak LR	0.01	0.05
Coefficient for the auxiliary first moment	0.9	0.9

Appendix

Table 6: IWSLT14 optimizer hyperparameters. The auxiliary coefficients are AdamW for Muon and the adaptive HyM branch for HyMUON .

Figure 14: IWSLT14 German–English test BLEU for the repeated runs in Figure 5 . Curves are means over repeated runs with bands showing one standard error; the panel legend gives the endpoint means.

Figure 15: IWSLT14 ablation of the averaged iterate in test BLEU. HyMUON $(y)$ retains the matrix and auxiliary updates formed from the two momentum states and evaluates the training iterate; HyMUON $(x)$ evaluates the averaged iterate. Curves correspond to the selected configuration of each method.

Hyperparameter	100k regime	600k regime
LR for hidden matrices	0.01	0.01
Minimum LR for hidden matrices	$3\times 10^{-5}$	$6\times 10^{-5}$
Momentum $\mu$ for hidden matrices	0.95	0.95
HyM mixing weight $\beta$	0.99	0.99
Weight decay for hidden matrices	0.2	0.2
Auxiliary LR	0.005	0.005

Appendix

Table 7: HyMUON hyperparameters in the GPT-2 124M OpenWebText experiments.

Figure 16: LLaMA-1.3B validation perplexity on C4 with document packing through 10k steps in the comparison against the reproduced Muon baseline. This is the exponential transform of the curves of validation loss in Figure 5 ; HyMUON is evaluated at its averaged iterate.

	AdamW	AdEMAMix	Muon	HyMUON
Learning rate	$0.001$	$0.001$	$0.01/0.001$	$0.045/0.045$
Weight decay	$0.1$	$0.1$	$0/0.1$	$0.1/0.1$
Matrix momentum	–	–	$0.95$	$0.85$
Adaptive $(\beta_{1},\beta_{2})$	$(0.8,0.999)$	$(0.9,0.999)$	$(0.8,0.999)$	$(0.97,0.999)$
Adaptive $\varepsilon$	$10^{-8}$	$10^{-8}$	$10^{-8}$	$10^{-10}$
HyM mixing weight $\beta$	–	–	–	$0.9$

Appendix

Table 8: Fixed optimizer hyperparameters for the 124M FineWeb comparison. Baseline columns apply to both budgets; the HyMUON column is the 1B configuration, and the 2.1B changes appear below the table. Learning rates are peak values. An entry of the form

a/b

gives the matrix and auxiliary values; for AdamW and AdEMAMix the single rate applies to all parameters.

Method	Runs	Updates	Tokens (B)	Validation loss
Clean Muon	10	3,250	1.704	$3.27866\pm 0.00134$
HyMUON	8	3,200	1.678	$3.27832\pm 0.00100$

Appendix

Table 9: Track 3 results. Final validation loss is reported as mean

\pm

sample SD.

Figure 17: Track 3 validation loss over the full training runs (left) and the final stage (right). Curves show run means; shading indicates the range across runs.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 205 papers

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning Optimization Gradient Descent
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong
The Newton-Muon Optimizer
Apr 1, 2026Zhehang Du, Weijie SuNeural Network Optimization Muon Optimizer
University of Pennsylvania
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3

May 25, 2026J/K move · Enter open · S save

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

Chung-Yiu Yau, Dawei Li, Athanasios Glentis, Valentyn Boreiko+2

University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong

Deep Learning Optimization Gradient Descent Accelerated Gradient Methods

Open paper AskPDF arXiv

Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochastic gradient noise and non-convex loss landscapes. In particular, standard lookahead relies on short-horizon update signals (e.g., differences between consecutive iterates), which are inherently noisy and can lead to unstable extrapolation directions. This work revisits Nesterov's acceleration from a trajectory perspective and argues that effective acceleration in deep learning should harness the low-frequency trends of optimization trajectories rather than extrapolating noisy one-step updates. Leveraging this insight, we propose EMA-Nesterov, a simple modification that replaces the standard Nesterov's lookahead direction with an exponential moving average (EMA) of parameter updates. This yields a stabilized lookahead direction that captures and harnesses the evolving trend of the training trajectory through a low-pass filter, while remaining adaptive to progressive changes via the geometric weighting structure of EMA. We show that EMA-Nesterov retains a theoretical accelerated convergence rate in convex problems that is analogous to Nesterov's accelerated gradient method. Furthermore, we provide empirical evidence on language model pre-training to verify that EMA-Nesterov is broadly applicable across a range of fine-tuned base optimizers, including Adam, SOAP, Muon, as well as complex optimizers that achieve state-of-the-art performance on optimization benchmarks (NanoGPT). Compared to prior lookahead methods, EMA-Nesterov achieves better performance by avoiding the instability of short-horizon lookahead and the non-adaptivity of long-horizon lookahead.

Abstract

Deep Learning Optimization

Explore similar work

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

The Newton-Muon Optimizer

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering