RODE: A Radial-Orthogonal Decoupled Engine for Optimization

Guoxiang Xu; Bince Qu; Qi Sun; Cheng Zhuo

cs.LGAug 21, 2026

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

Authors: Guoxiang Xu, Bince Qu, Qi Sun, Cheng Zhuo

Organizations: Zhejiang University, Hangzhou, China

Abstract

Modern neural network training increasingly uses matrix-aware optimizers, yet their conditioned matrix step is typically added directly to the weight, jointly changing its norm and direction. This interaction matters because the current norm determines angular motion, while directional learning can drive norm growth and thereby alter later steps. We introduce RODE, which gives the radial and directional components separate update rules and step sizes. RODE explicitly updates the matrix Frobenius norm through a scalar radial rule, while its directional channel performs Newton--Schulz-conditioned updates in the tangent space. Controlled GPT-2 interventions show gains from both direct norm control and RODE's directional update. Across two language-modeling and two image-classification tasks, RODE outperforms both Muon variants in every direct comparison and ends with lower full-model norms. At 1.5B scale, using the learning rate transferred directly from the Qwen2-style LM sweep, RODE lowers loss from 4.145 to 3.346 and final global norm from 11964 to 2183 relative to Muon RMS, with fixed-radius RODE improving further. For Qwen3.5-9B full-parameter fine-tuning, all six optimizers use the same tuning budget and the same formal-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH-500. Thus, decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization.

Figures & tables

Figure 1: Muon conditions momentum and adds the resulting matrix step directly to

W_{t}

, so its effects on norm and direction are jointly determined. RODE exposes a scalar radial signal and a tangent-projected directional signal, assigns them separate step sizes, and reconstructs the managed matrix from the resulting

\rho_{t+1}

and

U_{t+1}

Optimizer	GPT-2 loss $\downarrow$	Qwen2-style LM loss $\downarrow$	CIFAR acc. $\uparrow$	ImageNet acc. $\uparrow$
AdamW	$3.1792\pm 0.0173$	$3.0503\pm 0.0071$	$68.66\pm 0.29$ $\ddagger$	$62.49\pm 0.46$ $\ddagger$
Muon Original	$3.0295\pm 0.0001$	$2.8185\pm 0.0065$	$71.49\pm 0.67$ $\ddagger$	$64.98\pm 0.05$ $\ddagger$
Muon RMS	$3.0265\pm 0.0027$	$3.0205\pm 0.0103$	$71.11\pm 0.14$ $\ddagger$	$64.48\pm 0.14$ $\ddagger$
SGD	–	–	$72.77\pm 0.24$	$70.25\pm 0.08$
RODE	$2.9774\pm 0.0011$	$2.7731\pm 0.0013$	$\mathbf{73.75\pm 0.29}$ $\ddagger$	$69.74\pm 0.47$ $\ddagger$
RODE-SNR	$2.9724\pm 0.0557$	$\mathbf{2.7493\pm 0.0011}$	$73.64\pm 0.56$ $\ddagger$	$\mathbf{70.49\pm 0.09}$ $\ddagger$

Table 1: Cross-task optimization performance on language modeling and image classification benchmarks. Results are reported as mean

\pm

sample standard deviation over three seeds. Vision reports best accuracy on the official CIFAR-100 test set and the ImageNet-1K validation set.

\dagger

Rates transferred from the Qwen2-style LM;

\ddagger

rates transferred from GPT-2. Best available mean per column is bold.

Optimizer	GPT-2	Qwen2-style LM	CIFAR-100	ImageNet-1K
AdamW	$413.6\pm 3.0$	$711.1\pm 8.7$	$355.9\pm 0.7$	$2049.5\pm 34.7$
Muon Original	$1187.9\pm 0.8$	$1314.7\pm 1.9$	$1302.6\pm 0.6$	$4889.3\pm 0.5$
Muon RMS	$1074.4\pm 0.8$	$1722.4\pm 2.9$	$1142.0\pm 1.2$	$4547.4\pm 6.9$
SGD	–	–	$29.8\pm 0.7$	$56.6\pm 0.1$
RODE	$315.6\pm 6.6$	$683.5\pm 2.6$	$348.6\pm 0.3$	$801.8\pm 27.6$
RODE-SNR	$269.1\pm 0.3$	$695.8\pm 4.1$	$272.8\pm 0.1$	$320.6\pm 0.1$

Table 2: Final observed full-model weight norm at the end of training (three-seed mean

\pm

sample SD). Values include every trainable parameter, including tensors delegated to AdamW, and are comparable only within a task. A dash indicates that the task–optimizer combination was not run; values are descriptive rather than ranked.

Figure 2: Standard GPT-2/WikiText-103 over 8000 steps (three-seed mean

\pm

sample SD): validation loss (left) and observed full-model weight norm (right). The norm aggregates all trainable parameters, including those delegated to AdamW; both RODE variants show substantially less growth than the Muon runs.

Condition	Direction update	Radius policy	Val. loss $\downarrow$	$\Delta$ vs. Muon
Muon RMS	additive Muon	implicit	$3.027\pm 0.003$	–
Muon + NormProj	additive Muon	fixed by projection	$3.008\pm 0.001$	$-0.019$
RODE, fixed $\rho$	RODE tangent stack	fixed	$2.977\pm 0.002$	$-0.050$
Full RODE	RODE tangent stack	learned scalar	$2.977\pm 0.001$	$-0.050$

Table 3: Sequential GPT-2 norm–direction intervention. Three-seed values are mean

\pm

sample SD;

\Delta

is relative to Muon RMS.

Optimizer	Best cache loss $\downarrow$	Final cache loss $\downarrow$	Final observed global norm
RODE, fixed $\rho$	$\mathbf{3.13670\pm 0.00406}$	$\mathbf{3.13670\pm 0.00406}$	$1743\pm 7$
RODE	$3.34568\pm 0.01108$	$3.34571\pm 0.01109$	$2183\pm 2$
Muon RMS	$4.14465\pm 0.01800$	$4.14478\pm 0.01806$	$11964\pm 32$

Table 4: 1.5B scale transfer with rates selected on the smaller Qwen2-style LM (three seeds, mean

\pm

sample SD).

Figure 3: Qwen2-style 1.5B/FineWeb scale transfer over 20000 steps (three-seed mean

\pm

sample SD): unsmoothed fixed-cache loss and observed global weight norm. Fixed-radius RODE freezes managed matrix radii; parameters delegated to AdamW can still change the global norm.

Optimizer	Best val. loss $\downarrow$	GSM8K $\uparrow$	MATH-500 $\uparrow$	MMLU-STEM $\uparrow$	MMLU-Pro Math $\uparrow$
AdamW	$0.3490\pm 0.0005$	$79.58\pm 0.52$	$44.67\pm 0.70$	$75.53\pm 0.36$	$40.07\pm 0.65$
Muon Original	$0.3497\pm 0.0005$	$84.23\pm 0.80$	$48.60\pm 3.47$	$75.77\pm 0.91$	$39.97\pm 1.35$
Muon RMS	$0.3455\pm 0.0005$	$84.03\pm 0.98$	$53.40\pm 1.51$	$76.24\pm 0.91$	$41.77\pm 1.98$
Adafactor	$0.3435\pm 0.0021$	$80.87\pm 2.18$	$54.33\pm 1.94$	$\mathbf{77.63\pm 0.14}$	$\mathbf{43.35\pm 0.86}$
SOAP	$\mathbf{0.3434\pm 0.0011}$	$81.68\pm 1.98$	$48.13\pm 0.83$	$76.22\pm 0.54$	$41.40\pm 1.04$
RODE	$0.3492\pm 0.0022$	$\mathbf{85.47\pm 0.46}$	$\mathbf{54.53\pm 0.95}$	$76.75\pm 1.14$	$41.87\pm 1.26$

Table 5: Qwen3.5-9B full-parameter fine-tuning and task evaluation (three-seed mean

\pm

sample SD). All task columns report accuracy in percent. Each optimizer is independently tuned, and downstream evaluation uses the minimum-validation-loss checkpoint from each run. Best mean per column is bold.

Figure 4: ResNet-50 test accuracy on CIFAR-100 (left) and validation accuracy on ImageNet-1K (right), each over three seeds.

Variant	Intervention	Final val. $\downarrow$	$\Delta$
Full RODE	Full engine	$2.7731\pm 0.0013$	–
RODE-SNR	Add tangent-gradient ratio damping	$2.7493\pm 0.0011$	$-0.0238$
w/o Newton–Schulz	Remove projected NS conditioning	$3.1249\pm 0.0113$	$+0.3518$
w/o geodesic	Use normalized retraction	$2.7736\pm 0.0016$	$+0.0005$
RODE, radius fixed	Hold radius fixed	$2.7745\pm 0.0017$	$+0.0014$
Radial-only control	Hold direction fixed	$3.8715\pm 0.0023$	$+1.0984$

Table 6: Three-seed 114M component study (mean

\pm

sample SD). Structural ablations inherit RODE rates and are not retuned; RODE-SNR is an independently tuned diagnostic.

\Delta

is relative to full RODE.

Task	Optimizer	State	Peak	Step	Opt.	Throughput
		(MB)	(GB)	(ms)	(ms)	(vs. AdamW)
Qwen2-style LM	AdamW	913.0	10.92	102.0	9.0	$1.00\times$
	Muon Original	611.0	10.62	131.3	37.7	$0.78\times$
	Muon RMS	611.0	10.62	132.4	39.1	$0.77\times$
	SOAP	2643.2	12.67	137.3	42.1	$0.74\times$
	MARS-M	913.0	10.96	158.9	68.1	$0.64\times$

Table 7: Compute and memory benchmark. Measurements use a single RTX 5090 D v2 with bf16 autocast: the Qwen2-style LM uses micro-batch 16 and sequence length 512, while ResNet-50 uses batch 64 and

224\times 224

inputs. Each of three runs has 8 warm-up and 25 timed steps; Step and Opt. are medians of the three per-run medians. Peak is PyTorch peak allocated memory, State is exact optimizer-state tensor storage, Opt. includes step() and gradient clearing, and throughput is relative to AdamW.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Input: parameter

W_{t}

, gradient

g_{t}

, raw buffer

B_{t-1}

, learning rates

\eta_{\rho},\eta_{\mathrm{dir}}

, momentum

\beta

, floor

\kappa_{\mathrm{floor}}

, NS steps

K

, and safeguards

\epsilon,\epsilon_{\mathrm{NS}}

1. If

\|W_{t}\|_{\mathrm{F}}>\epsilon

, form

\rho_{t}=\|W_{t}\|_{\mathrm{F}}

U_{t}=W_{t}/\rho_{t}

, and

\mathcal{P}_{t}(X)=X-\langle X,U_{t}\rangle_{\mathrm{F}}U_{t}

. 2. Set

\widetilde{B}_{t}=\beta\mathcal{P}_{t}(B_{t-1})+g_{t}

. 3. Set

s_{t}=\langle\widetilde{B}_{t},U_{t}\rangle_{\mathrm{F}}

and

\rho_{t+1}=\max\{\rho_{t}-\eta_{\rho}s_{t},\kappa_{\mathrm{floor}}\rho_{t}\}

. 4. Set

M_{t}^{\perp}=\mathcal{P}_{t}(\widetilde{B}_{t})=\widetilde{B}_{t}-s_{t}U_{t}

. 5. Transpose

M_{t}^{\perp}

and

U_{t}

if needed so rows

\geq

columns; initialize

X_{0}=M_{t}^{\perp}/\|M_{t}^{\perp}\|_{\mathrm{F}}

in the nonzero branch. 6. For

k=0,\ldots,K-1

, set

X_{k+1}=\mathcal{P}_{t}\!\left[3.4445X_{k}+X_{k}\{-4.7750X_{k}^{\top}X_{k}+2.0315(X_{k}^{\top}X_{k})^{2}\}\right].

Undo any transpose and set

\widehat{Q}_{t}=X_{K}

. 7. For

q_{t}=\|\widehat{Q}_{t}\|_{\mathrm{F}}>0

, set

D_{t}^{\perp}=\widehat{Q}_{t}/q_{t}

\theta_{t}=\eta_{\mathrm{dir}}q_{t}

, and

U_{t+1}=U_{t}\cos\theta_{t}-D_{t}^{\perp}\sin\theta_{t}

. Renormalize

U_{t+1}

. 8. Write

W_{t+1}=\rho_{t+1}U_{t+1}

and store

B_{t}=\widetilde{B}_{t}

Appendix

Algorithm 1: Default RODE update for one matrix parameter

Optimizer	Matrix-parameter update	Non-matrix update	Fixed optimizer constants
AdamW	AdamW on matrix parameters	AdamW on all remaining parameters	betas $(0.9,0.95)$
Muon Original	Muon, original LR scaling	AdamW	momentum $0.95$ ; matrix wd $0$
Muon RMS	Muon, RMS-matched LR scaling	AdamW	momentum $0.95$ ; matrix wd $0$
RODE	Radial update + tangent Newton–Schulz geodesic step	AdamW	momentum $0.9$ ; NS steps $5$ ; $\kappa_{\mathrm{floor}}=10^{-6}$ ; $\epsilon=10^{-12}$
RODE-SNR	RODE Add tangent-gradient ratio damping	AdamW	RODE constants; ratio clamp $[0.01,1]$
SOAP	SOAP on matrix parameters	AdamW	betas $(0.95,0.95)$ , precondition frequency $10$ , max precondition dim $10000$

Appendix

Table 8: Operational baseline definitions used in the code. Matrix weight decay is the task-level --wd ; non-matrix AdamW uses weight decay

9\times 10^{-4}

unless the row states otherwise.

Source	Optimizer	Budget	Search space
GPT-2	AdamW	$40{\times}300$	$\eta:[10^{-4},10^{-2}]$ ; weight decay $[10^{-4},10^{-1}]$
GPT-2	Muon Original	$40{\times}300$	$\eta_{\mathrm{2d/dir}}:[10^{-4},2{\times}10^{-1}]$ ; $\eta_{\mathrm{1d}}:[10^{-5},10^{-2}]$
GPT-2	Muon RMS	$40{\times}300$	$\eta_{\mathrm{2d/dir}}:[10^{-5},5{\times}10^{-2}]$ ; $\eta_{\mathrm{1d}}:[10^{-5},10^{-2}]$
GPT-2	RODE/RODE-SNR	$40{\times}300$	$\eta_{\mathrm{2d/dir}}:[10^{-4},5{\times}10^{-1}]$ ; $\eta_{\rho}:[10^{-5},10^{-1}]$ ; $\eta_{\mathrm{1d}}:[10^{-5},10^{-2}]$
Qwen2-style	AdamW	$30{\times}500$	$\eta_{\mathrm{2d/dir}},\eta_{\mathrm{1d}}:[3{\times}10^{-5},3{\times}10^{-3}]$
Qwen2-style	Muon Original/Muon RMS	$30{\times}500$	$\eta_{\mathrm{2d/dir}}:[5{\times}10^{-3},3{\times}10^{-1}]$ ; $\eta_{\mathrm{1d}}:[3{\times}10^{-5},3{\times}10^{-3}]$

Appendix

Table 9: Learning-rate sweep budgets and search spaces. All sweep ranges are log-uniform. The selected values from these sweeps are reported in the task-specific hyperparameter tables below. The added baselines are swept on the Qwen2-style LM/WikiText-103 benchmark and then transferred to standard GPT-2 and CIFAR-100.

Optimizer	$\eta_{\mathrm{2d/dir}}$	$\eta_{\rho}$	$\eta_{\mathrm{1d}}$	Matrix wd	1D wd	Source
AdamW	$4.483{\times}10^{-4}$	–	$4.483{\times}10^{-4}$	$9.077{\times}10^{-4}$	$9.077{\times}10^{-4}$	GPT-2 sweep
Muon Original	$1.264{\times}10^{-2}$	–	$1.314{\times}10^{-5}$	$0$	$9{\times}10^{-4}$	GPT-2 sweep
Muon RMS	$1.806{\times}10^{-3}$	–	$9.274{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 sweep
RODE	$6.432{\times}10^{-4}$	$1.502{\times}10^{-2}$	$5.012{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	GPT-2 sweep
RODE-SNR	$2.413{\times}10^{-3}$	$1.288{\times}10^{-4}$	$2.311{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 sweep
SOAP	$2.900{\times}10^{-3}$	–	$2.591{\times}10^{-3}$	$10^{-2}$	$9{\times}10^{-4}$	Qwen transfer

Appendix

Table 10: Standard GPT-2/WikiText-103 hyperparameters for the 8000-step main benchmark. Runs use sequence length 512, effective batch size 128, micro-batch size 16, bf16 autocast, evaluation every 500 steps, 100 warmup steps, and cosine decay.

Optimizer	$\eta_{\mathrm{2d/dir}}$	$\eta_{\rho}$	$\eta_{\mathrm{1d}}$	Matrix wd	1D wd	Source
AdamW	$1.544{\times}10^{-4}$	–	$2.988{\times}10^{-3}$	$10^{-2}$	$9{\times}10^{-4}$	Qwen sweep
Muon Original	$2.257{\times}10^{-2}$	–	$2.897{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	Qwen sweep
Muon RMS	$5.160{\times}10^{-3}$	–	$2.913{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	Qwen sweep
RODE	$1.206{\times}10^{-3}$	$2.438{\times}10^{-3}$	$2.915{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	Qwen sweep
RODE-SNR	$4.544{\times}10^{-3}$	$3.698{\times}10^{-5}$	$2.974{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	Qwen sweep
SOAP	$2.900{\times}10^{-3}$	–	$2.591{\times}10^{-3}$	$10^{-2}$	$9{\times}10^{-4}$	Qwen sweep

Appendix

Table 11: Qwen2-style LM/WikiText-103 hyperparameters for the aligned 4000-step main benchmark and component ablation. The model has 114M parameters. All runs use sequence length 512, effective batch size 128, micro-batch size 16, bf16 autocast, the official validation split, evaluation every 200 steps, 200 warmup steps, and cosine decay. This locally instantiated architecture uses a separate protocol from Table 10 .

Optimizer	$\eta_{\mathrm{2d/dir}}$	$\eta_{\rho}$	$\eta_{\mathrm{1d}}$	Matrix wd	1D wd	Accum.	Implementation
RODE, fixed $\rho$	$1.206{\times}10^{-3}$	$0$	$2.915{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	$64$	fused RODE, fixed radius
RODE	$1.206{\times}10^{-3}$	$2.438{\times}10^{-3}$	$2.915{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	$64$	fused RODE, chunk 2
Muon RMS	$5.160{\times}10^{-3}$	–	$2.913{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	$64$	PyTorch Muon RMS

Appendix

Table 12: Qwen2-style 1.5B/FineWeb transfer hyperparameters. This three-seed experiment reuses learning rates selected on the smaller Qwen2-style LM without a new sweep. Runs use streaming FineWeb sample-100BT , sequence length 1024, effective batch size 128, micro-batch size 2 per GPU, bf16 parameters/autocast, 20000 optimizer steps, evaluation every 500 steps, checkpointing every 100 steps, 200 warmup steps, cosine decay, and torch.compile(default) . The fixed evaluation cache is identical across optimizers.

Optimizer	Search space	Selected $\eta_{\mathrm{2d/dir}}$	Selected $\eta_{\rho}$	Selected $\eta_{\mathrm{1d}}$
AdamW	$\eta:[3{\times}10^{-6},8{\times}10^{-5}]$	$2.402{\times}10^{-5}$	–	Same
Muon Original	$\eta_{\mathrm{2d/dir}}:[10^{-4},5{\times}10^{-2}]$ ; $\eta_{\mathrm{1d}}:[2{\times}10^{-6},10^{-4}]$	$2.869{\times}10^{-4}$	–	$5.846{\times}10^{-6}$
Muon RMS	$\eta_{\mathrm{2d/dir}}:[3{\times}10^{-6},10^{-4}]$ ; $\eta_{\mathrm{1d}}=2{\times}10^{-5}$ fixed	$1.971{\times}10^{-5}$	–	$2.000{\times}10^{-5}$
Adafactor	$\eta:[10^{-6},10^{-4}]$	$8.000{\times}10^{-6}$	–	Same
SOAP	$\eta_{\mathrm{2d/dir}}:[10^{-5},8{\times}10^{-3}]$ ; $\eta_{\mathrm{1d}}:[2{\times}10^{-6},2{\times}10^{-4}]$	$2.118{\times}10^{-5}$	–	$3.020{\times}10^{-6}$
RODE	$\eta_{\mathrm{2d/dir}}:[2{\times}10^{-6},3{\times}10^{-5}]$ ; $\eta_{\rho}:[10^{-6},5{\times}10^{-5}]$ ; $\eta_{\mathrm{1d}}:[3{\times}10^{-6},5{\times}10^{-5}]$	$6.259{\times}10^{-6}$	$2.919{\times}10^{-6}$	$5.592{\times}10^{-6}$

Appendix

Table 13: Qwen3.5-9B learning-rate searches and selected values. Every optimizer receives 20 trials of 800 steps; all searched ranges are log-uniform. “Same” denotes a single learning rate applied to all trainable parameters.

Optimizer	$\eta_{\mathrm{2d/dir}}$	$\eta_{\rho}$	$\eta_{\mathrm{1d}}$	Matrix wd	1D wd	Source
AdamW	$4.483{\times}10^{-4}$	–	$4.483{\times}10^{-4}$	$10^{-2}$	$9{\times}10^{-4}$	GPT-2 transfer
Muon Original	$1.264{\times}10^{-2}$	–	$1.314{\times}10^{-5}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
Muon RMS	$1.806{\times}10^{-3}$	–	$9.274{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
RODE	$6.432{\times}10^{-4}$	$1.502{\times}10^{-2}$	$5.012{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
RODE-SNR	$2.413{\times}10^{-3}$	$1.288{\times}10^{-4}$	$2.311{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
SOAP	$2.900{\times}10^{-3}$	–	$2.591{\times}10^{-3}$	$10^{-2}$	$9{\times}10^{-4}$	Qwen transfer

Appendix

Table 14: ResNet-50/CIFAR-100 hyperparameters. Runs use 100 epochs, image size 224, effective batch size 128, three distinct seeds, and epoch-wise cosine annealing. The implemented scheduler is CosineAnnealingLR(T_max=epochs) with no warmup stage. We report best test accuracy. No CIFAR-specific sweep is used: the original suite uses standard-GPT-2-selected settings, while the added baselines use settings selected on the Qwen2-style LM.

Optimizer	$\eta_{\mathrm{2d/dir}}$	$\eta_{\rho}$	$\eta_{\mathrm{1d}}$	Matrix wd	1D wd	Source
AdamW	$4.483{\times}10^{-4}$	–	$4.483{\times}10^{-4}$	$10^{-2}$	$9{\times}10^{-4}$	GPT-2 transfer
Muon Original	$1.264{\times}10^{-2}$	–	$1.314{\times}10^{-5}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
Muon RMS	$1.806{\times}10^{-3}$	–	$9.274{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
RODE	$6.432{\times}10^{-4}$	$1.502{\times}10^{-2}$	$5.012{\times}10^{-3}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
RODE-SNR	$2.413{\times}10^{-3}$	$1.288{\times}10^{-4}$	$2.311{\times}10^{-4}$	$0$	$9{\times}10^{-4}$	GPT-2 transfer
SGD	$1.0{\times}10^{-1}$	–	$1.0{\times}10^{-1}$	$5{\times}10^{-4}$	$5{\times}10^{-4}$	standard

Appendix

Table 15: ResNet-50/ImageNet-1K hyperparameters. Runs use 100 epochs, image size 224, effective batch size 256, three distinct seeds, and epoch-wise cosine annealing without an implemented warmup stage. Table 1 reports, for each seed, the best validation accuracy observed over 100 epochs. The added SOAP/MARS-M/AdEMAMix baselines are not run on ImageNet-1K.

Experiment	Optimizers	Shared training protocol	Hyperparameter policy
GPT-2 norm–direction intervention	Muon RMS, Muon + NormProj, RODE with fixed radius, full RODE	standard GPT-2/WikiText-103; 8000 steps; effective batch 128; 100 warmup steps; cosine decay; three distinct seeds	Muon + NormProj inherits Muon RMS rates; the fixed-radius control inherits RODE rates and holds each matrix radius at initialization
RODE component ablation	RODE, RODE-SNR, fixed-radius control, radial-only control, w/o Newton–Schulz, w/o geodesic	Qwen2-style LM/WikiText-103; 114M parameters; 4000 steps; effective batch 128; 200 warmup steps; cosine decay; three distinct seeds	RODE-SNR uses its independently swept row in Table 11 ; structural variants inherit RODE’s learning rates

Appendix

Table 16: Ablation protocols. Each experiment uses three distinct seeds.

Figure 5: Qwen2-style LM/WikiText-103 validation trajectories over 4000 steps (three-seed mean

\pm

sample SD): full trajectory (left) and late-stage detail (right).

Figure 6: Qwen2-style 1.5B/FineWeb training diagnostics over 20000 steps (three seeds per method): EMA-smoothed training loss (left) and gradient norm (right). Gradient norm uses a logarithmic axis.

Figure 7: Qwen3.5-9B full-parameter fine-tuning diagnostics for the six optimizers in Table 5 . Left: validation loss over the 3000-step formal runs (three-seed mean

\pm

sample SD). Right: cumulative best objective over the 20 800-step Optuna trials for each optimizer; each trial’s objective is its minimum validation loss. Both panels use raw, unsmoothed values.

Panel of Figure 8 — Figure 8: GPT-2/WikiText-103 training dynamics (three seeds): training loss, update step size, and gradient norm.

Figure 9: ResNet-50 test loss on CIFAR-100 (left) and validation loss on ImageNet-1K (right), shown as three-seed mean

\pm

sample SD.

Figure 10: Observed full-model weight norms on CIFAR-100 (left) and ImageNet-1K (right), shown as three-seed mean

\pm

sample SD. RODE directly updates managed-matrix radii; these curves additionally include parameters delegated to AdamW.

Panel of Figure 11 — Figure 11: Additional ResNet-50/CIFAR-100 curves: training accuracy, training loss, gradient norm, and update-to-weight ratio.

Panel of Figure 12 — Figure 12: Additional ResNet-50/ImageNet-1K curves from the exported per-step diagnostics: three-seed means with sample-standard-deviation bands for training accuracy, training loss, gradient norm, and update-to-weight ratio. Main ImageNet validation results are reported in Table 1 and Figure 4 .

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 332 papers

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Jun 24, 2026Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson +1Neural Network Optimization Neural Network Training Dynamics
Machine Learning and Optimization Lab, EPFL
Muown Implicitly Performs Angular Step-size Decay
Jun 22, 2026Florian Hübler, Kai Lion, Antonio Orvieto +1Learning Rate Scheduling Riemannian Optimization
Department of Computer Science · ETH Zurich, Switzerland · ELLIS Institute Tübingen, MPI-IS +1
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Jun 24, 2026J/K move · Enter open · S save

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson, Martin Jaggi

Machine Learning and Optimization Lab, EPFL

Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object. Yet every weight matrix carries two distinct quantities -- a magnitude and a direction -- and all optimizers stepping in the matrix as a whole couple their dynamics: the directional change from an update depends on the current magnitude, while the magnitude drifts as a byproduct of learning the direction, so neither is governed directly by the learning rate. Typical training therefore leans on surrounding recipes such as weight decay and warmup to keep learning stable at scale, though these regulate the coupling only indirectly; other recent methods instead constrain the weight to a fixed-norm sphere, but add no learnable magnitude, leaving scale control to normalization layers alone. We propose Magnitude--Direction (MD) Decoupling, an optimizer modification that factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates, all while the model still sees a single fused weight tensor. The method is agnostic to the base optimizer and removes the need for weight decay and warmup. Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts (MoE) models. Treating magnitude and direction as separately controlled quantities thus yields more predictable training dynamics and a simple, broadly applicable improvement to modern optimizers.

Abstract

Neural Network Optimization

Explore similar work

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

Muown Implicitly Performs Angular Step-size Decay

Muown: Row-Norm Control for Muon Optimization