Momentum is central to stochastic optimization in deep learning, including the training of large language models. Classical momentum methods typically rely on a fast exponential moving average (EMA) scheme, which discounts past gradients on a fixed timescale and may underweight older gradients that remain informative. This creates a trade-off between retaining long-term history and adapting to recent changes. We introduce Hybrid Momentum (HyM), which combines a linearly weighted average of past gradients with a geometrically discounted fast momentum. The former assigns weights proportional to the gradients' iteration indices and retains a growing span of history, while the latter emphasizes recent gradients. This hybrid construction is modular and can be combined with adaptive preconditioning and other update transformations. In particular, we propose HyMUON, which applies Newton--Schulz orthogonalization to the hybrid direction. Illustrative examples and controlled experiments demonstrate the benefits of combining gradient information across these two timescales. Across image classification, machine translation, and language modeling, HyM and HyMUON match or improve upon strong baselines. HyMUON improves over Muon on IWSLT14, and in both the 100k- and 600k-step GPT-2 124M OpenWebText regimes. For LLaMA-1.3B pretraining on C4, HyMUON achieves a final validation loss of 2.777, compared with 2.903 for Muon.
Figures & tables
Figure 1: Gradient weight profiles for AdEMAMix ( β3=0.9999 ), classical EMA ( μ=0.999 ), and HyM ( μ=0.999 , β=0.9 ) at t∈{1000,10000,50000} (left to right). Each curve is normalized by the weight on the most recent gradient.
Figure 2: Persistence of the effect of a single batch and of its contribution retained in the slow state. Left: loss on batch B for the control and inject branches. The two branches differ at step 5,000, when the inject branch takes one optimization step using the gradient computed on B ; they then continue with the same sequence of batches. Right: at step 7,500, the inject branch is split into Keep, Delete, and Refresh, which retain, remove, or replace the contribution of B in the slow state. At this step, the three branches have the same model parameters, averaged iterate, and all other optimizer states. Refresh uses a gradient computed on B at the current model parameters and rescales it to match the groupwise norm of the original contribution. HyMUON curves are evaluated at the averaged iterate, whereas Muon curves are evaluated at the training iterate; all curves are unsmoothed. Full protocols are given in Appendices E.3 and E.3.1 .
Figure 3: Loss reduction on each semantic target’s evaluation set after a single update using the gradient of a candidate passage, at the 10k, 50k, and 100k checkpoints (left to right). For each target, the 2,000 candidate passages are ranked by their E5 similarity to the target’s reference set and divided into 100 groups of 20. The blue curves show the mean across ten semantic targets, and the shaded regions show error bands across targets. Positive values indicate a reduction in loss, and the vertical scales differ across panels.
Figure 4: Image classification. Left: CIFAR-10 (top) and CIFAR-100 (bottom) test accuracy. Middle: SVHN test loss and accuracy. Right: ImageNet-1K validation loss and accuracy. Curves show means across repeated runs, and the shaded regions show one standard error. Values in the panel legends are endpoint means with sample standard deviations. Appendix Figure 11 adds the CIFAR test loss curves.
Figure 5: Language tasks. Top row, left to right: IWSLT14 German–English test loss for the LSTM model; GPT-2 124M validation loss on OpenWebText through 100k steps; and LLaMA-1.3B validation loss on C4 through 10k steps, compared with the reproduced Muon baseline. Bottom row, left to right: GPT-2 124M validation loss on OpenWebText through 600k steps, followed by LLaMA-124M validation loss on FineWeb with token budgets of 1B and 2.1B. The IWSLT curves show means across repeated runs, and the shaded regions show one standard error. Appendix Figures 14 and 16 give BLEU and C4 perplexity.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Behavior of the Rosenbrock training iterates from (−3,5) over 20,000 steps using exact gradients. Momentum SGD uses η=4×10−3 and μ∈{0.9,0.95,0.99} ; HyM uses η=3×10−4 , β=0.9 , and μ∈{0.9,0.95,0.99} ; Schedule-Free SGD uses η=7×10−4 and β∈{0.9,0.99} . The left panel shows objective values at yt ; the remaining panels show the trajectories over objective contours. The star marks the minimizer (1,1) .
Figure 7: Averaged iterates for the Rosenbrock runs in Figure 6 , using the same initialization, optimizer settings, and 20,000 steps using exact gradients. HyM uses β=0.9 for iterate averaging; Momentum SGD is evaluated at an averaged iterate formed with β=0.99 , and this average does not enter its updates; Schedule-Free SGD uses its respective β∈{0.9,0.99} . The left panel shows objective values at xt ; the remaining panels show the averaged trajectories over objective contours. The star marks the minimizer (1,1) .
Figure 8: Test loss against training batch size on CIFAR-10 (left) and CIFAR-100 (right). The learning rate is chosen separately for each method and batch size.
Figure 9: Curves of loss reduction across 100 percentiles of E5 similarity. Rows are the 10k, 50k, and 100k checkpoints; columns are semantic targets (left) and randomly mixed controls (right). Candidates are ranked within each target, with 20 candidates per percentile. Curves are means over ten targets of each type; shaded bands are based on standard errors across targets. The blue curve uses E5 similarity. Positive values indicate loss reduction; vertical scales differ. The panels in Figure 3 are the left column of this figure, rearranged in one row.
Figure 10: Share of the net loss reduction on semantic targets contributed by the candidates in the top 1%,5%,10% , and 20% by E5 similarity. Panels are the 10k, 50k, and 100k checkpoints, left to right. The blue curve uses the E5 ranking from Figure 9 . The numerator sums I(C,V) over the selected candidates and all ten semantic targets; the denominator sums over all candidates and the same targets. The dashed line marks the share proportional to subset size. Because negative values are retained, cancellation can produce a share above 100% ; the plotted quantities are net contribution shares, not fractions of positive loss reductions.
Image classification: accuracy (%) ↑
Dataset / model / budget
SGD Cosine
SGD Step
Schedule- Free
HyM
CIFAR-10 / WRN-16-8 300 epochs
95.90±0.12
95.43±0.17
96.11±0.10
96.25±0.10
CIFAR-100 / DenseNet-BC 300 epochs
77.66±0.39
77.22±0.20
79.11±0.22
79.14±0.22
SVHN / ResNet-3-96 300 epochs
98.24±0.05
–
98.20±0.08
98.47±0.02
ImageNet-1K / ResNet-50 100 epochs
76.68±0.04
–
76.70±0.06
76.84±0.10
Appendix
Table 1: Main experimental results. Image accuracy and IWSLT metrics are mean ± sample SD across repeated runs; OpenWebText, FineWeb, and C4 give the final validation loss. ImageNet accuracy is validation top-1; CIFAR and SVHN use test accuracy. A dash marks a method not included in that comparison. FineWeb losses are rounded to two decimals.
Dataset
Epochs
Batch size
Input size
CIFAR-10
300
128
32×32
CIFAR-100
300
64
32×32
SVHN (train + extra)
300
32
32×32
ImageNet-1K
100
256
224×224
Appendix
Table 2: Training settings for image classification. Batch sizes are global numbers of images per optimizer step.
Dataset
Optimizer
LR
Momentum
β
Weight decay
CIFAR-10
SGD + Cosine
0.2
0.9
–
10−4
SGD + Step
0.1
0.9
–
10−4
Schedule-Free
10
–
0.9
10−4
HyM
0.05
0.98
0.9
5×10−4
CIFAR-100
SGD + Cosine
0.05
0.9
–
2×10−4
SGD + Step
0.05
0.9
–
2×10−4
Appendix
Table 3: Fixed optimizer settings for image classification. LR denotes the peak learning rate. The momentum column gives the SGD or HyM momentum coefficient; β gives the Schedule-Free interpolation coefficient or the HyM mixing weight.
Training time
Peak allocated memory
Dataset (batch)
SGD
HyM
Change
SGD
HyM
Change
CIFAR-10 (128)
4.71
4.98
+5.7%
1122
1254
+132
CIFAR-100 (512)
6.80
6.73
−0.9%
7410
7423
+12
ImageNet-1K (256)
591.62
601.43
+1.7%
11257
11561
+304
Appendix
Table 4: H100 resource profiles for HyM versus cosine SGD. Batch sizes in parentheses are the profiling batch sizes. Time is seconds per training epoch; memory is peak CUDA allocated MiB. Changes are HyM relative to cosine SGD.
Figure 11: Curves of CIFAR test loss for the runs in Figure 4 . Left: CIFAR-10; right: CIFAR-100. Curves are means over repeated runs, and the bands show one standard error; the legend gives the sample SD. HyM is evaluated at the averaged iterate xt .
Figure 12: CIFAR comparison of the training and averaged iterates. Columns show CIFAR-10 and CIFAR-100; rows show test loss and test accuracy. HyM (y) is evaluated at the training iterate without iterate averaging, while HyM (x) is evaluated at the averaged iterate. The other curves reuse the baselines from Figure 4 .
ηmax=0.05
ηmax=0.1
ηmax=0.2
Direction
y
x
y
x
y
x
Fast momentum only
84.26
90.44
76.78
83.76
65.68
72.18
Slow momentum only
89.64
92.06
86.80
89.02
81.82
84.48
Full HyM
93.94
96.22
93.44
96.10
91.92
95.52
EMA, ρ=0.99
85.04
90.86
77.60
83.40
68.50
73.18
EMA, ρ=0.999
89.30
93.40
84.18
89.58
79.88
82.74
Appendix
Table 5: Final CIFAR-10 validation accuracy (%) in the component ablation. Each group defined by a learning rate compares the same six configurations; y and x are paired evaluations at epoch 300. EMA rows replace only the slow momentum. These are validation results on the 45k/5k split, not the test accuracies in the main image classification experiment.
Figure 13: CIFAR-10 component ablation: validation cross-entropy at the training iterate y (left) and averaged iterate x (right). Rows use peak learning rates 0.05 , 0.1 , and 0.2 . Curves show values for each epoch. Every evaluation uses the same BN calibration protocol. “Slow EMA” retains the fast momentum and replaces only the slow momentum.
Hyperparameter
Muon
HyMUON
Peak LR for hidden matrices
0.05
0.05
Momentum μ for hidden matrices
0.99
0.9
HyM mixing weight β
–
0.9
Weight decay for hidden matrices
0.05
0.05
Auxiliary peak LR
0.01
0.05
Coefficient for the auxiliary first moment
0.9
0.9
Appendix
Table 6: IWSLT14 optimizer hyperparameters. The auxiliary coefficients are AdamW for Muon and the adaptive HyM branch for HyMUON .
Figure 14: IWSLT14 German–English test BLEU for the repeated runs in Figure 5 . Curves are means over repeated runs with bands showing one standard error; the panel legend gives the endpoint means.
Figure 15: IWSLT14 ablation of the averaged iterate in test BLEU. HyMUON (y) retains the matrix and auxiliary updates formed from the two momentum states and evaluates the training iterate; HyMUON (x) evaluates the averaged iterate. Curves correspond to the selected configuration of each method.
Hyperparameter
100k regime
600k regime
LR for hidden matrices
0.01
0.01
Minimum LR for hidden matrices
3×10−5
6×10−5
Momentum μ for hidden matrices
0.95
0.95
HyM mixing weight β
0.99
0.99
Weight decay for hidden matrices
0.2
0.2
Auxiliary LR
0.005
0.005
Appendix
Table 7: HyMUON hyperparameters in the GPT-2 124M OpenWebText experiments.
Figure 16: LLaMA-1.3B validation perplexity on C4 with document packing through 10k steps in the comparison against the reproduced Muon baseline. This is the exponential transform of the curves of validation loss in Figure 5 ; HyMUON is evaluated at its averaged iterate.
AdamW
AdEMAMix
Muon
HyMUON
Learning rate
0.001
0.001
0.01/0.001
0.045/0.045
Weight decay
0.1
0.1
0/0.1
0.1/0.1
Matrix momentum
–
–
0.95
0.85
Adaptive (β1,β2)
(0.8,0.999)
(0.9,0.999)
(0.8,0.999)
(0.97,0.999)
Adaptive ε
10−8
10−8
10−8
10−10
HyM mixing weight β
–
–
–
0.9
Appendix
Table 8: Fixed optimizer hyperparameters for the 124M FineWeb comparison. Baseline columns apply to both budgets; the HyMUON column is the 1B configuration, and the 2.1B changes appear below the table. Learning rates are peak values. An entry of the form a/b gives the matrix and auxiliary values; for AdamW and AdEMAMix the single rate applies to all parameters.
Method
Runs
Updates
Tokens (B)
Validation loss
Clean Muon
10
3,250
1.704
3.27866±0.00134
HyMUON
8
3,200
1.678
3.27832±0.00100
Appendix
Table 9: Track 3 results. Final validation loss is reported as mean ± sample SD.
Figure 17: Track 3 validation loss over the full training runs (left) and the final stage (right). Curves show run means; shading indicates the range across runs.
Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochastic gradient noise and non-convex loss landscapes. In particular, standard lookahead relies on short-horizon update signals (e.g., differences between consecutive iterates), which are inherently noisy and can lead to unstable extrapolation directions. This work revisits Nesterov's acceleration from a trajectory perspective and argues that effective acceleration in deep learning should harness the low-frequency trends of optimization trajectories rather than extrapolating noisy one-step updates. Leveraging this insight, we propose EMA-Nesterov, a simple modification that replaces the standard Nesterov's lookahead direction with an exponential moving average (EMA) of parameter updates. This yields a stabilized lookahead direction that captures and harnesses the evolving trend of the training trajectory through a low-pass filter, while remaining adaptive to progressive changes via the geometric weighting structure of EMA. We show that EMA-Nesterov retains a theoretical accelerated convergence rate in convex problems that is analogous to Nesterov's accelerated gradient method. Furthermore, we provide empirical evidence on language model pre-training to verify that EMA-Nesterov is broadly applicable across a range of fine-tuned base optimizers, including Adam, SOAP, Muon, as well as complex optimizers that achieve state-of-the-art performance on optimization benchmarks (NanoGPT). Compared to prior lookahead methods, EMA-Nesterov achieves better performance by avoiding the instability of short-horizon lookahead and the non-adaptivity of long-horizon lookahead.
Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong
The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains largely elusive. In this paper, we introduce a surrogate model that not only sheds new light on the design of Muon, but more importantly leads to a new optimizer. In the same spirit as the derivation of Newton's method, the surrogate approximates the loss as a quadratic function of the perturbation to a weight matrix W using only three matrices: the gradient G, an output-space curvature matrix H, and the data matrix Z that stacks the layer inputs. By minimizing this surrogate in one step and adopting a certain isotropic assumption on the weights, we obtain the closed-form update rule (up to momentum and weight decay) W←W−η⋅msgn(G(ZZ⊤)−1), where η is the learning rate and msgn(X)=UV⊤ if X=USV⊤ is a compact singular value decomposition. This new optimization method, which we refer to as Newton-Muon, shows that standard Muon can be interpreted as an implicit Newton-type method that neglects the right preconditioning induced by the input second moment. Empirically, on a reproduction of the earliest publicly released Modded-NanoGPT speedrun configuration using Muon for GPT-2 pretraining, Newton-Muon reaches the target validation loss in 6% fewer iteration steps and reduces wall-clock training time by about 4%.
Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers.
Xianliang Li, Zihan Zhang, Weiyang Liu +1
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3