Multiple-gradient descent is typically formulated in Euclidean geometry, whereas matrix-aware optimizers use directions induced by other norms. We analyze a single-loop multi-objective method combining a norm-constrained linear minimization oracle with entropic updates of task weights. For any finite-dimensional norm pair, we establish an O(T−1/2) ergodic stationarity rate in the deterministic setting, deriving the required trajectory gradient bound from smoothness and lower-boundedness in the unregularized case. With stochastic gradients and a bounded-gradient assumption, one persistent blended-gradient momentum buffer yields an O(T−1/4) rate whose leading term has no explicit dependence on the number of tasks. We also establish static regret for the inner weighting objective on arbitrary bounded gradient sequences and, with entropy regularization and a weight floor, iteration-wise inner-gap control after burn-in while retaining ergodic stationarity, conditional on a trajectory gradient bound. The optimal blended gradient can be nonunique in spectral and sign geometries, motivating a selection-independent inner gap. Our analysis extends the spectral construction studied by concurrent MOON to general norms. Experiments compare the method with existing multi-task optimizers and examine its oracle, weighting, and momentum choices.
Figures & tables
Primal ∥⋅∥
Dual ∥⋅∥∗
\lmo(g)
Known as
ℓ2
ℓ2
−Rg/∥g∥2
normalized SGD
ℓ∞
ℓ1
−R\sign(g)
Signum ( Bernstein et al., 2018 )
spectral
nuclear
−RUV⊤ , G=UΣV⊤
Muon ( Jordan et al., 2024 )
Table 1: Norm-constrained oracles; Pethick et al. (2025) apply them layer-wise with per-layer radii ρℓ .
Method
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Δm%↓
STL
74.11
93.44
0.0137
38.22
0.00
LS
74.80
93.37
0.0188
42.41
11.83
SI
67.77
89.97
0.0151
34.65
3.29
RLW
74.24
93.30
0.0169
47.09
11.63
DWA
74.42
93.41
0.0159
41.94
6.35
UW
71.56
92.15
0.0191
38.75
11.41
Table 2: Results on CityScapes dataset. The best and second-best values are highlighted in bold and underlined , respectively. The task-specific metrics and the performance drop Δm% are reported.
Segmentation
Depth
Surface normal
Method
mIoU ↑
Pix Acc ↑
Abs ↓
Rel ↓
Mean ↓
Med. ↓
<11.25↑
<22.5↑
<30↑
Δm%↓
STL
38.77
63.98
0.6350
0.2654
25.10
19.06
30.12
57.66
69.53
0.00
LS
38.13
64.83
0.5523
0.2334
27.79
23.56
23.03
48.23
61.64
6.76
SI
38.59
64.57
0.5439
0.2339
27.57
23.43
22.81
48.45
62.18
6.32
RLW
38.45
64.61
0.5814
0.2454
27.83
23.46
23.20
48.51
61.81
7.53
DWA
39.57
65.19
0.5691
0.2399
27.40
22.87
24.26
49.61
62.81
5.37
Table 3: Results on NYU-v2 dataset. The best and second-best values are highlighted in bold and underlined , respectively. The task-specific metrics and the performance drop Δm% are reported.
Method
Left ↑
Right ↑
Avg. ↑
MGDA
83.63
85.13
84.38
MGDA + Muon
84.48
86.43
85.45
FAMO
85.24
87.29
86.27
FAMO + Muon
87.28
88.02
87.65
Muon (LS)
87.45
88.04
87.75
MOON
86.87
88.06
87.47
Table 4: MultiMNIST with a ViT backbone. Left: test accuracy (%), mean over three seeds on the same regenerated data. Best and second-best values are shown in bold and underlined. Right: fixed-hyperparameter substitutions with the final encoder-only inner gap, in each oracle’s dual norm (not comparable across oracle rows). Details in Appendix I.2 .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
η
10−5
5×10−5
10−4
5×10−4
10−3
Average accuracy (%)
90.20
90.54
90.08
89.69
89.53
Final inner gap
1.97
0.90
0.84
0.29
1.04
Appendix
Table 5: MultiMNIST: sensitivity of our method to the weight step η (spectral oracle, entropic weights, blended momentum), test accuracy averaged over three seeds. The main configuration, η=5×10−5 , was selected on the validation split.
κ
0
0.48
0.49
0.495
0.4975
Avg. ↑
90.54
90.54
90.73
91.11
90.96
Gap ↓
0.90
0.90
1.17
1.22
1.65
Appendix
Table 6: Ablation of the clipping parameter κ on MultiMNIST: test accuracy and final inner gap, averaged over three seeds.
Multi-objective optimization (MOO) has demonstrated significant success in multi-task learning by mitigating task conflicts through gradient manipulation. However, most existing methods flatten model parameters into vectors and perform gradient manipulation under Euclidean geometry, thereby overlooking the matrix structure prevalent in modern architectures such as Transformers. In this paper, we show that gradient manipulation in Euclidean space does not generally yield the steepest descent direction under matrix geometry, potentially limiting optimization efficiency. Drawing from the theory of steepest descent for matrix-valued parameters, we propose MOON (Multi-Objective OrthoNormalized Updates), which performs gradient manipulation under spectral--nuclear norm geometry and uses the orthonormalized manipulated gradient for parameter updates. Theoretically, for smooth non-convex objectives, we establish convergence of the averaged Pareto-stationarity measure at rates of O(T−1/2) in the deterministic setting and O(T−1/4) under stochastic gradients. Empirical results across various benchmarks show that MOON consistently improves both optimization efficiency and final multi-task performance. Our code is available at https://github.com/KunlinLyu/MOON.
Shiji Zhou, Kunlin Lyu, Lei Zhang +2
Institute of Artificial Intelligence, Beihang University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · Beijing Academy of Artificial Intelligence (BAAI) +1
This paper introduces a new problem-dependent regret measure for online convex optimization with smooth losses. The notion, which we call the G⋆ regret, depends on the cumulative squared gradient norm evaluated at the decision in hindsight. We show that the G⋆ regret strictly refines the existing L⋆ (small loss) regret, and that it can be arbitrarily sharper when the losses have vanishing curvature around the hindsight decision. We establish upper and lower bounds on the G⋆ regret and extend our results to dynamic regret and bandit settings. As a byproduct, we refine the existing convergence analysis of stochastic optimization algorithms in the interpolation regime. Some experiments validate our theoretical findings.
Wenzhi Gao, Chang He, Madeleine Udell
ICME, Stanford University · SIME, Shanghai University of Finance and Economics · Department of Management Science and Engineering, Stanford University
In this article, we explore the use of various matrix norms for optimizing functions of weight matrices, a crucial problem in deep learning. Moving beyond the spectral norm that underlies the Muon update, we leverage the duals of the Ky Fan norms to introduce the Fanion family of linear minimization oracle (LMO) algorithms, which are closely related to Muon, ν-SAM, and Dion. Staying inside the LMO, we construct the families of F-Fanions and S-Fanions, whose updates are convex combinations of the updates of Fanions and Normalized SGD or SignSGD, respectively. The most promising algorithms in these families are F-Muon and S-Muon. By conducting an extensive empirical study of all three algorithm families across a wide range of tasks and settings, we demonstrate that F-Muon and S-Muon consistently match Muon's performance, while outperforming Muon on a synthetic smooth convex problem.
Alexey Kravatskiy, Ivan Kozyrev, Nikolai Kozlov +3