cs.LGOct 26, 2026

Understanding Practical Optimization in Machine Learning

Authors: Egor Shulgin

Organizations: King Abdullah University of Science and Technology Thuwal, Kingdom of Saudi Arabia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern machine learning relies on practical optimization methods that often differ from the algorithms studied in classical analyses: they train submodels, clip or normalize gradients, use layer-wise non-Euclidean directions, or approximate costly inner computations. This thesis analyzes the objective, assumptions, update, and computation defined by these procedures, rather than those of simplified proxies. For submodel training, we show that Independent Subnetwork Training can converge to a shifted fixed point because sketching before differentiation biases the update for the original objective. Model-Agnostic Sparsified Training instead incorporates the sketch distribution into its objective, which makes sketched gradients unbiased and admits convergence guarantees. For private optimization, we show that adaptive quantile clipping retains bias at any fixed quantile, while suitable quantile schedules recover near-optimal nonconvex rates under heavy-tailed noise. We then combine smoothed normalization with error compensation and obtain distributed private convergence guarantees under standard assumptions, without assuming bounded gradients. For optimizers based on linear minimization oracles, we develop and validate a layer-wise generalized smoothness model and build on it the Gluon framework for Muon- and Scion-style methods. We also show that the approximate orthogonalization in Muon matters: less precise orthogonalization calls for smaller stepsizes, so its precision should be tuned together with the learning rate. Overall, practical methods are better understood when theory retains the features that determine their optimization behavior.

Figures & tables

Figure 1.1: Representative trajectory-smoothness measurements from Chapter 6 . Left: the fitted layer-wise (Li0,Li1)(L_{i}^{0},L_{i}^{1}) model tracks observed trajectory smoothness better than a constant-smoothness proxy. Right: trajectory smoothness varies substantially across layers and matrix groups, motivating layer-wise constants rather than a single global smoothness parameter.
Ch.ObjectModeled featureMain message
2IST / submodel-gradient trainingObjectiveSketching before differentiation biases the update, and the method can converge to a shifted fixed point.
3MAST / sketch-aware objectiveObjectivePlacing the sketch distribution in the objective makes sketched gradients unbiased for the induced target.
4Adaptive quantile clippingUpdateFixed quantiles retain clipping bias; a time-varying quantile schedule removes it and recovers optimal rates.
5Smoothed normalization / α\alpha -NormECUpdateSmoothed normalization with error compensation gives private distributed guarantees without bounded gradients.
6Gluon / layer-wise generalized smoothnessAssumptionsLayer-wise generalized smoothness matches training trajectories and yields guarantees with per-layer constants.
7Inexact Muon / LMO updatesUpdateOrthogonalization precision changes the convergence bound and the best stepsize.
Table 1.1: Thesis map: main objects, modeled features, and messages.
Figure 2.1: Performance of simplified IST on a quadratic problem for varying step size values.
Figure 2.2: Experimental study of IST on a neural network problem.
Figure 3.1: Distributions of test accuracies of sparsified solutions for the ERM formulation ( 3.1 ) and MAST problem ( 3.2 ). “Sparsity” corresponds to the percentage of zeroed weights.
Figure 3.2: Performance of Algorithm 3 with Bernoulli sketches ( 3.6 ) on standard loss ( 3.18 ) (for Si≡I{\bf S}_{i}\equiv\mathbf{I} ).
Figure 4.1: Adaptive clipping thresholds for five target quantiles on six federated learning tasks without privacy noise. The trajectories vary substantially across tasks and may increase, decrease, or change nonmonotonically. The curves were redrawn for this thesis from the vector trajectories in Figure 4 of Andrew et al. (2021) ; the original clipping norms were estimated using geometric updates with ηC=0.2\eta_{C}=0.2 and C0=0.1C_{0}=0.1 .
PropertyDP-SGDClip21α\alpha -NormEC ( Ours )
Non-private Convergence✗✓✓
Private Convergence✗✗✓
OperatorsClipping / NormalizationClippingNormalization
Additional AssumptionsBounded gradient or heterogeneityNoNo
Table 5.1: Comparison of distributed private optimization methods. Unlike prior work, which either lacks private convergence guarantees or requires restrictive assumptions like bounded gradients, α\alpha -NormEC is the first to provide these guarantees under standard smoothness conditions alone.
OperatorProperty
Contractive compressor C:Rd→Rd\mathcal{C}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}∥C(g)−g∥2≤(1−η)2∥g∥2\left\|\mathcal{C}(g)-g\right\|^{2}\leq(1-\eta)^{2}\left\|g\right\|^{2}
Clipping Clipτ(g):=min⁡(1,τ∥g∥)g{\rm Clip}_{\tau}\left(g\right):=\min\left(1,\frac{\tau}{\left\|g\right\|}\right)g∥Clipτ(g)−g∥2≤max⁡(0,∥g∥−τ)2\left\|{\rm Clip}_{\tau}\left(g\right)-g\right\|^{2}\leq\max(0,\left\|g\right\|-\tau)^{2}
Smoothed normalization Normα(g):=1α+∥g∥g{\rm Norm}_{\alpha}\left(g\right):=\frac{1}{\alpha+\left\|g\right\|}g∥Normα(g)−g∥2≤(1−1α+∥g∥)2∥g∥2\left\|{\rm Norm}_{\alpha}\left(g\right)-g\right\|^{2}\leq\left(1-\frac{1}{\alpha+\left\|g\right\|}\right)^{2}\left\|g\right\|^{2}
Table 5.2: Comparison of contractive compressors, clipping, and smoothed normalization with their properties. Smoothed normalization, unlike clipping, satisfies a contractive-like property similar to that of compressors.
Figure 5.1: The highest test accuracy achieved by α\alpha -NormEC with different α\alpha and β\beta values.
Figure 11
Figure 5.4: Gradient norm ∥∇f(xk)∥\|\nabla f(x^{k})\| across methods on a logistic regression problem. Curves are averaged over 55 seeds.
 Mik=βkMik−1+(1−βk)∇ifξk(Xk)M_{i}^{k}=\beta^{k}M^{k-1}_{i}+(1-\beta^{k})\nabla_{i}f_{\xi^{k}}(X^{k}) 
Algorithm 5 Gluon: Stochastic Adaptive Layer-Wise LMO-based Optimizer with Momentum
Figure 6.1: Training NanoGPT on FineWeb validates our layer-wise (L0,L1)(L^{0},L^{1}) -smoothness model.
Figure 6.2: Validation of 6.2.1 in NanoGPT-124M along training trajectories of unScion . The estimates L^i[k]\hat{L}_{i}[k] are closely matched by L^iapprox[k]\hat{L}_{i}^{\text{approx}}[k] , while a constant L0L_{0} (corresponding to the classical smoothness assumption), results in a substantially poorer fit.
Figure 6.3: (a) Validation curves for AdamW and unScion with varying ρ3\rho_{3} values; (b) Heatmap of validation loss from the last iteration of unScion across different combinations of ρ2\rho_{2} and ρ3\rho_{3} .
Figure 6.4: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters in NanoGPT-124M along AdamW training trajectories.
 d^k≈argmin⁡∥d∥≤1 ⟨mk+1,d⟩\hat{d}^{k}\approx\underset{\|d\|\leq 1}{\operatorname{argmin}}\ \left\langle m^{k+1},d\right\rangle 
Algorithm 6 Inexact Generalized Muon
Figure 7.1: NanoGPT on FineWeb: convergence (left) and hyperparameter sensitivity (right) as PolarExpress (PolarExp) iterations vary.
Figure 7.2: CIFAR-10 training loss and test accuracy across different Newton-Schulz (NS) precision levels.
Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Figure C.1: Distributions of accuracies of sparsified solutions for the ERM ( 3.1 ) and MAST ( 3.2 ) formulations.
Figure C.2: Test accuracies of sparsified solutions for the ERM formulation ( 3.1 ) and MAST problem ( 3.2 ).
Figure C.3: Finite-sum MAST loss ( 3.17 ) convergence for Algorithm 2 (II) with subsampling.
Figure C.4: MAST loss ( 3.2 ) convergence for Algorithm 2 (I) with varying step size and Rand- 11 sketches.
Figure C.5: Performance of Algorithm 3 with Bernoulli sketches ( 3.6 ) on standard loss ( 3.18 ) (for Si≡I{\bf S}_{i}\equiv\mathbf{I} ).
Figure E.1: Minimal train loss (left), final train loss (middle), and final test accuracy (right) achieved by non-private α\alpha -NormEC , after 300 communication rounds using a fine-tuned constant step size γ\gamma .
Figure E.2: Training loss and test accuracy of non-private α\alpha -NormEC with α=0.01\alpha=0.01 [solid], 0.10.1 [dashed], and 1.01.0 [dotted], and β=0.01\beta=0.01 [blue], 0.10.1 [green], 1.01.0 [orange], and 10.010.0 [red].
Figure 28
Figure E.5: Training loss and test accuracy of α\alpha -NormEC with [solid] and without [dashed] server normalization.
Method: α\alpha -NormECβ\betaγ\gammaFinal Accuracy
With server normalization0.010.0182.86%
 0.10.185.43%
 1.00.184.29%
 10.00.181.48%
Without server normalization0.010.184.04%
 0.10.186.09 %
Appendix
Table E.1: Best configurations and final test accuracies of α\alpha -NormEC with and without server normalization.
Figure E.6: Training loss and test accuracy of Clip21 [solid] and α\alpha -NormEC [dashed] without server normalization in non-private training.
Figure 32
Figure E.9: The highest test accuracy of DP - α\alpha -NormEC with [left] and without [center] Server Normalization (SN), and their difference [right].
Figure E.10: Training loss and test accuracy of DP-Clip21 [solid], and DP- α\alpha -NormEC with [dotted] and without [dashed] server normalization (SN) across different β\beta values (with fine-tuned step sizes).
Figure E.11: Training loss and test accuracy of DP-Clip21 [solid] and DP- α\alpha -NormEC [dashed] across different β\beta values for \mathchar271=1\mathchar 271\relax=1 . DP- α\alpha -NormEC demonstrates both faster convergence and higher final accuracy for each β\beta .
Figure E.12: Best test accuracy of DP- α\alpha -NormEC across different β,γ\beta,\gamma values.
Figure E.13: Training loss and test accuracy of DP- α\alpha -NormEC across different β\beta values.
Figure E.14: Gradient norm and test accuracy of DP-Clip21 [solid], DP- α\alpha -NormEC [dotted], and DP-SGD [dash-dotted] on Logistic Regression using the best hyperparameters (β,γ)(\beta,\gamma) selected by final test accuracy and then averaged over 55 seeds.
MethodBest (β,γ)(\beta,\gamma)Final test acc (%)Final grad norm
DP- α\alpha -NormEC(10−7, 0.1)(10^{-7},\,0.1)93.68±0.656\mathbf{93.68}\pm 0.6560.074±0.023\mathbf{0.074}\pm 0.023
DP-Clip21(10−4, 10)(10^{-4},\,10)93.509±0.89593.509\pm 0.8950.077±0.0270.077\pm 0.027
DP-SGD(10−1, 0.1)(10^{-1},\,0.1)92.982±0.96192.982\pm 0.9610.176±0.0150.176\pm 0.015
Appendix
Table E.2: Final performance (mean ±\pm std over 55 seeds; selected by final test accuracy).
 Xik+1=LMOBik(∇if(Xk)):=arg⁡min⁡Xi∈Bik ⟨∇if(Xk),Xi⟩(i)X_{i}^{k+1}=\mathrm{LMO}_{\mathcal{B}_{i}^{k}}\left(\nabla_{i}f(X^{k})\right):=\underset{X_{i}\in\mathcal{B}_{i}^{k}}{\arg\min}\ \langle\nabla_{i}f(X^{k}),X_{i}\rangle_{(i)} (F.2)
Appendix
Algorithm 9 Deterministic Adaptive Layer-Wise LMO-based Optimizer
Figure F.1: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for the group of parameters from the embedding layer of NanoGPT-124M along unScion training trajectories. The group norm is ∥⋅∥(p)=np∥⋅∥1→∞\|\cdot\|_{(p)}=n_{p}\|\cdot\|_{1\to\infty} , with fitted values Lp0≈0L_{p}^{0}\approx 0 , Lp1≈1.3L_{p}^{1}\approx 1.3 . The same plot is shown twice with different yy -axis limits.
Figure F.2: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for the group of parameters from the 4th transformer block of NanoGPT-124M along unScion training trajectories. The group norms are ∥⋅∥(i)=\nicefracnimi∥⋅∥2→2\|\cdot\|_{(i)}=\sqrt{\nicefrac{{n_{i}}}{{m_{i}}}}\|\cdot\|_{2\to 2} , with fitted values Li0≈0L_{i}^{0}\approx 0 , Li1≈70L_{i}^{1}\approx 70 .
Figure F.3: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for the group of parameters from the 8th transformer block of NanoGPT-124M along unScion training trajectories. The group norms are ∥⋅∥(i)=\nicefracnimi∥⋅∥2→2\|\cdot\|_{(i)}=\sqrt{\nicefrac{{n_{i}}}{{m_{i}}}}\|\cdot\|_{2\to 2} , with fitted values Li0≈0L_{i}^{0}\approx 0 , Li1≈70L_{i}^{1}\approx 70 .
Figure F.4: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters in NanoGPT-124M along training trajectories of unScion using the specialized norm choices defined in ( F.4 ).
Figure F.5: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters in NanoGPT-124M along training trajectories of unScion using the standard Euclidean norm ∥⋅∥2\|\cdot\|_{2} .
Figure F.6: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters in NanoGPT-124M along AdamW training trajectories.
Figure F.7: Estimated trajectory smoothness for NanoGPT-124M trained with AdamW . The same cross-layer heterogeneity pattern persists, indicating that layer-wise (Li0,Li1)(L^{0}_{i},L^{1}_{i}) -smoothness is not specific to unScion .
Figure F.8: Illustrative best (top row) and worst (bottom row) per‑layer fits of Assumption 1 (measured Li[k]L_{i}[k] vs. Liapprox[k]L_{i}^{\text{approx}}[k] ) along NanoGPT-124M training.
Figure F.9: Relative fit MSErel\mathrm{MSE}_{\mathrm{rel}} error of the layer-wise (Li0,Li1)(L^{0}_{i},L^{1}_{i}) model across transformer blocks in NanoGPT-124M for each matrix type. Dashed lines indicate the mean and median over blocks (embedding matrix: MSErel=0.0005\mathrm{MSE}_{\mathrm{rel}}=0.0005 ).
Figure F.10: Estimated trajectory smoothness for NanoGPT-124M trained with unScion .
Figure F.11: Estimated trajectory smoothness for GPT-2 Medium ( ∼\sim 355M parameters) trained with unScion .
Figure F.12: Estimated trajectory smoothness for GPT-2 Large ( ∼\sim 774M parameters) trained with unScion .
Figure F.13: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters of a CNN model along the training trajectories of unScion with full-batch gradients . The norms used for each group are as follows: ∥⋅∥(i)=\nicefrac1Ciout∥⋅∥2\|\cdot\|_{(i)}=\sqrt{\nicefrac{{1}}{{C_{i}^{out}}}}\|\cdot\|_{2} for biases, ∥⋅∥(i)=k2\nicefracCiinCiout∥⋅∥2→2\|\cdot\|_{(i)}=k^{2}\sqrt{\nicefrac{{C_{i}^{in}}}{{C_{i}^{out}}}}\|\cdot\|_{2\to 2} for conv, and ∥⋅∥(p)=np∥⋅∥1→∞\|\cdot\|_{(p)}=n_{p}\|\cdot\|_{1\to\infty} for the last group XpX_{p} , associated with classification head weights.
Figure F.14: Validation of layer-wise (L0,L1)(L^{0},L^{1}) -smoothness for different groups of parameters of a CNN model along the training trajectories of unScion with stochastic gradients . The norms used for each group are as follows: ∥⋅∥(i)=\nicefrac1Ciout∥⋅∥2\|\cdot\|_{(i)}=\sqrt{\nicefrac{{1}}{{C_{i}^{out}}}}\|\cdot\|_{2} for biases, ∥⋅∥(i)=k2\nicefracCiinCiout∥⋅∥2→2\|\cdot\|_{(i)}=k^{2}\sqrt{\nicefrac{{C_{i}^{in}}}{{C_{i}^{out}}}}\|\cdot\|_{2\to 2} for conv, and ∥⋅∥(p)=np∥⋅∥1→∞\|\cdot\|_{(p)}=n_{p}\|\cdot\|_{1\to\infty} for the last group XpX_{p} , associated with classification head weights.
Figure G.1: Supplementary results for NanoGPT on FineWeb: convergence behavior (left) and runtime scaling (right) as the number of PolarExpress (PolarExp) iterations varies.
PolarExpress iterations12345678
Validation loss3.06753.03163.01093.00583.00363.00373.00293.0023
Training time (minutes)652.2668.8660.4665.8655.7663.1668.8675.0
Appendix
Table G.1: Final validation loss and total training time for NanoGPT on FineWeb across different numbers of PolarExpress iterations used for the LMO approximation.
Figure G.2: Aggregate PolarExpress error over training, defined as the sum over tracked layers of ∥PEp(M)−PE100(M)∥2\|\mathrm{PE}_{p}(M)-\mathrm{PE}_{100}(M)\|_{2} on the same normalized matrix MM . Higher iteration counts maintain lower aggregate orthogonalization error throughout training.
Figure G.3: Layer-wise PolarExpress convergence at validation step 6000 for NanoGPT. For each tracked matrix family in each transformer block, we plot the spectral-norm discrepancy ∥PEk(M)−PE100(M)∥2\|\mathrm{PE}_{k}(M)-\mathrm{PE}_{100}(M)\|_{2} as a function of the inner-iteration count kk , where MM is the normalized pre-orthogonalization momentum matrix and PE100\mathrm{PE}_{100} is used as a high-accuracy reference. MLP matrices converge in only a few iterations, while several attention matrices remain difficult until roughly k≈10k\approx 10 – 2020 , revealing strong heterogeneity across parameter families and blocks.
PE itersMean eke_{k}Std. eke_{k}Median eke_{k}Max eke_{k}Val. loss
10.98260.03111.00111.02003.0675
20.98790.02201.00161.00793.0316
30.95540.06441.00121.01433.0109
40.84800.22600.99641.00553.0058
50.69520.40470.98951.00323.0036
60.65070.43210.97971.00303.0037
Appendix
Table G.2: Effect of the number of PolarExpress iterations on the aggregate orthogonalization discrepancy ek(M)=∥PEk(M)−PE100(M)∥2e_{k}(M)=\|\mathrm{PE}_{k}(M)-\mathrm{PE}_{100}(M)\|_{2} , and the final NanoGPT validation loss on FineWeb. Discrepancy statistics are aggregated over all completed checkpoints and saved tensors. This is a solver-to-solver diagnostic relative to a high-accuracy PE 100 reference on the same normalized matrix, not the error to the exact polar factor.
Figure G.4: Full final test accuracy heatmaps on CIFAR-10 across a grid of step sizes and momentum values for different numbers of Newton-Schulz (“Newton”) iterations. Brighter colors indicate higher final test accuracy. The region of high performance is visibly larger and more stable for higher numbers of Newton-Schulz iterations.
More in this topic

Nonconvex Stochastic Optimization

Nonconvex stochastic optimization develops methods for minimizing nonconvex objectives using noisy gradients and analyzes their convergence.

See all 96 papers

Explore similar work

CardsList
May 13, 2026

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order 2K+22K+2. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.

Jihwan Kim, Chenglin Fan

Seoul National University

Deep Learning OptimizationDifferentially Private Stochastic Gradient DescentMuon Optimizer
AskPDF
Oct 17, 2026

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Training deep neural networks requires optimizers that balance computational efficiency with formal data privacy. While orthogonalized update rules, such as the recently introduced Muon optimizer, have shown strong empirical performance and scalability, their behavior under Differential Privacy (DP) is not yet understood. In this paper, we propose DP-Muon, a differentially private variant of the Muon optimizer. By analyzing Muon's orthogonalized updates through the lens of a Linear Minimization Oracle (LMO), we show that the leading effect of per-example clipping is a positive rescaling of the gradient, to which the LMO is blind, so that only a small directional residual controlled by the centered gradient variance enters the optimization error. This yields an O(K−1/4)\mathcal O(K^{-1/4}) convergence rate, up to logarithmic factors, to a clipping-dependent neighborhood at a fixed noise scale and, with noise calibrated to a fixed (ε,δ)(\varepsilon,δ) budget, the standard (d/(nε))1/2(\sqrt d/(n\varepsilon))^{1/2} privacy--utility bound of DP-SGD, without assuming bounded gradients. Furthermore, we investigate the hyperparameter landscape of DP-Muon and document how the optimal learning rate, momentum, and batch size change under privacy. Empirically, on image classification DP-Muon matches or improves on DP-SGD and DP-Adam, whereas on language modeling the best optimizer depends on the batch size, with DP-Muon best at small batches and DP-Adam best near the privacy-optimal batch size.

Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1

King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia

Muon OptimizerDeep Learning OptimizationDifferentially Private Stochastic Gradient Descent
AskPDF
Jun 7, 2026

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning. However, most current orthogonalized methods are still paired with fixed, externally scheduled, or otherwise open-loop magnitude rules, so their scale is not directly calibrated from the realized optimization trajectory. Motivated by the closed-loop perspective behind Lipschitz-free and noise-adaptive methods, we propose OptMuon, a family of adaptive momentum orthogonalization methods for stochastic nonconvex optimization. OptMuon combines Muon-style polar-factor directions with a trajectory-dependent AdaGrad-Norm-type coefficient schedule, so that the update magnitude is determined by the observed gradient and momentum history rather than by a prescribed Lipschitz-dependent rule. The schedule does not use the smoothness constant, the variance level, or the bounded-gradient constant in parameter selection, and its running-maximum correction prevents isolated gradient spikes from causing excessive coefficient collapse. Under lower-boundedness, unbiased stochastic gradients with bounded variance, smoothness, and an almost-sure bounded stochastic-gradient condition, we prove two complementary expected-stationarity guarantees. OptMuon-A achieves the noise-adaptive rate O~(T−1/2+σ1/2T−1/4)\tilde{\mathcal O}(T^{-1/2}+σ^{1/2}T^{-1/4}) under average smoothness, while OptMuon-I achieves O~(T−1/2+σ1/3T−1/3)\tilde{\mathcal O}(T^{-1/2}+σ^{1/3}T^{-1/3}) under individual smoothness. In the zero-noise regime, both bounds automatically reduce to a nearly optimal deterministic first-order rate O~(T−1/2)\tilde{\mathcal O}(T^{-1/2}) without manual hyperparameter retuning. These results show that closed-loop scalar adaptation can be combined with Muon-style momentum orthogonalization while retaining noise adaptivity and zero-noise optimality up to logarithmic factors.

Ganzhao Yuan

Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China

Momentum MethodsNonconvex Stochastic OptimizationMuon OptimizerStochastic Optimization ConvergenceAdaptive Gradient Methods
AskPDF