Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping

Dat Phi Van; Ngo Vu Minh; Tuc Nguyen; Thin Nguyen; Ngoc-Thanh Dinh; Trung Le

cs.LGSep 27, 2026

Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping

Authors: Dat Phi Van, Ngo Vu Minh, Tuc Nguyen, Thin Nguyen, Ngoc-Thanh Dinh, Trung Le

Organizations: Center for AI Research, VinUniversity, Hanoi, Vietnam · Indiana University, Bloomington, IN, USA · Monash University, Clayton, VIC, Australia

Abstract

Matrix-valued optimizers such as Muon exploit the spectral structure of neural network updates through Newton--Schulz orthogonalization, but their near-flattening of the singular spectrum discards relative magnitude information across gradient modes. We introduce Soren (Spectral Orthogonal Reshaping), a matrix-valued optimizer that preserves the singular subspaces of the gradient while applying a bounded, monotone sigmoid transformation to its singular values. This smoothly compresses dominant modes without fully flattening the spectrum. We interpret Soren as a positive-definite preconditioned gradient method and establish convergence guarantees under relative smoothness and metric Polyak--Łojasiewicz geometry. To avoid explicit singular value decomposition, we further develop a finite-depth Soft Newton--Schulz (SNS) polynomial realization of the sigmoid spectral map and characterize how its spectral approximation affects the induced convergence geometry. Experiments across LLM pre-training, supervised fine-tuning, and direct preference optimization demonstrate the effectiveness and robustness of Soren against established optimizers.

Figures & tables

	LLaMA-60M	LLaMA-135M	LLaMA-350M	LLaMA-1B
AdamW	44.54	36.96	23.49	20.82
Muon	36.86	27.42	21.73	18.41
HTMuon	35.32	26.15	22.58	18.44
Soren	35.14	26.15	21.53	18.29

Table 1: Comparison of established pretraining optimizers across LLaMA models of varying sizes on the C4 dataset. Lower perplexity indicates better performance. Detailed hyperparameter configurations are reported in Tables 6 – 8 in Appendix C .

Panel of Figure 1 — Figure 1: Training and evaluation loss curves for the LLaMA-60M (top row) and LLaMA-135M (bottom row) experiments.

Optimizer	ARC-c	ARC-e	PIQA	HellaSwag	OBQA	WinoGrande	Avg.
AdamW	23.13	25.05	50.16	25.49	24.00	48.80	32.77
Muon	23.46	26.89	50.11	26.97	24.40	50.51	33.72
HTMuon	25.77	26.81	50.71	27.25	24.60	49.88	34.17
Soren	25.51	27.36	51.25	28.47	25.80	51.99	35.06

Table 2: Zero-shot evaluation results (

\uparrow

) on six commonsense reasoning benchmarks for the Qwen3-0.6B model pretrained with different optimizers.

Method	HellaSwag	ARC	MMLU	TruthfulQA	Winogrande	GSM8K	Average
AdamW	73.08 ±0.44	55.97 ±1.45	61.79 ±0.39	46.75 ±1.48	68.90 ±1.30	64.44 ±1.32	61.82
Muon	73.72 ±0.44	56.14 ±1.45	62.83 ±0.39	47.44 ±1.49	68.90 ±1.30	69.83 ±1.26	63.14
HTMuon	73.75 ±0.44	56.31 ±1.45	62.93 ±0.39	47.50 ±1.48	70.24 ±1.30	69.22 ±1.27	63.33
Soren (Ours)	74.31 ±0.44	57.17 ±1.45	65.62 ±0.38	48.14 ±1.50	71.11 ±1.27	72.55 ±1.23	64.82

Table 3: Evaluation results of different optimizers for SFT on Qwen2.5-3B. Detailed hyperparameter configurations are reported in Table 9 in Appendix C .

Method	HellaSwag	ARC	MMLU	TruthfulQA	Winogrande	GSM8K	Average
AdamW	80.69 ±0.36	61.09 ±1.42	51.09 ±0.39	45.32 ±1.58	76.40 ±1.14	30.93 ±1.27	57.59
Muon	81.89 ±0.38	61.95 ±1.42	55.07 ±0.40	49.56 ±1.57	77.35 ±1.18	31.69 ±1.28	59.59
HTMuon	81.85 ±0.38	61.86 ±1.42	55.13 ±0.40	49.52 ±1.57	76.95 ±1.18	31.31 ±1.28	59.44
Soren (Ours)	83.25 ±0.37	62.46 ±1.42	57.21 ±0.39	49.54 ±1.56	77.74 ±1.17	36.32 ±1.32	61.09

Table 4: Evaluation results of different optimizers for DPO on Mistral 7B. Detailed hyperparameter configurations are reported in Table 9 in Appendix C .

Panel of Figure 2 — Figure 2: Spectral transformation of a representative gradient matrix from the Llama 60M model, with singular values sorted in descending order. The top and bottom rows correspond to the first and final layers, respectively, while the columns show the initial, intermediate, and final training steps. Soren smoothly compresses dominant modes while preserving weaker-mode structure, in contrast to Muon-style orthogonalization, which flattens active modes toward a nearly uniform scale.

Panel of Figure 3 — Figure 3: Comparison between SNS and the sigmoid approximation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Training and evaluation loss curves for the LLaMA-350M (top row) and LLaMA-1B (bottom row) experiments.

	LLaMA-60M		Qwen2.5-3B	Mistral-7B-v0.1
Optimizer	Time	Relative to AdamW	Time	Time
AdamW	0h23m	1.00x	5h23m	10h3m
Muon	0h28m	1.22x	5h33m	10h11m
HTMuon	2h04m	5.39x	5h38m	11h19m
Soren	0h39m	1.69x	5h33m	10h29m

Appendix

Table 5: Comparison of the wall-clock training time of different optimizers. Relative training time is normalized with respect to AdamW.

Panel of Figure 5 — Figure 5: Log-centered singular spectra of the exact sigmoid spectral map and its finite-step Soft Newton–Schulz (SNS) approximation for a representative gradient matrix from the final layer of the Llama-135M model. Log singular values are centered by subtracting the median of the $\log_{2}$ singular values to isolate relative spectral shape from overall scale. The columns show the initial, intermediate, and later training steps. SNS closely tracks the exact sigmoid transformation across training stages, preserving its relative spectral structure.

Params	Hidden	Intermediate	Heads	Blocks	Steps	Data amount	Batch Size
60M	512	1376	8	8	5K	0.5B	512
135M	768	2048	12	12	10K	1B	512
350M	1024	2736	16	24	20K	2B	512
1B	2048	5461	32	24	60K	6B	512

Appendix

Table 6: Hyperparameters of LLaMA models.

	LLaMA-60M		LLaMA-135M
Optimizer	LR	WD	LR	WD
AdamW	1e-3	0.1	1e-3	0.1
Muon	{ 0.01 , 0.02, 0.03, 0.04}	0.1	{ 0.01 , 0.02, 0.03, 0.04}	0.1
HTMuon	{0.01, 0.02, 0.03 , 0.04}	0.1	{0.01, 0.02, 0.03 , 0.04}	0.1
Soren	{0.01, 0.02, 0.03 , 0.04}	0.1	{0.01, 0.02, 0.03 , 0.04}	0.1

Appendix

Table 7: Hyperparameters for LLaMA-60M and LLaMA-135M on the C4 dataset. Boldface values denote the optimal hyperparameters.

	LLaMA-350M		LLaMA-1B
Optimizer	LR	WD	LR	WD
AdamW	1e-3	0.1	6e-4	0.1
Muon	{0.0025, 0.005 , 0.01, 0.015}	0.1	{ 0.005 , 0.01}	0.1
HTMuon	{0.0025, 0.005 , 0.01, 0.015}	0.1	{ 0.005 , 0.01}	0.1
Soren	{0.0025, 0.005, 0.01 , 0.015}	0.1	{0.005, 0.01 }	0.1

Appendix

Table 8: Hyperparameters for LLaMA-350M and LLaMA-1B on the C4 dataset. Boldface values denote the optimal hyperparameters.

	Qwen2.5-3B		Mistral-7B-v0.1
Optimizer	LR	WD	LR	WD
AdamW	{1e-6, 5e-6, 2e-5, 3e-5 }	0.1	{ 1e-7 , 4e-7, 5e-7}	0.1
Muon	{1e-6, 5e-6, 2e-5 , 3e-5}	0.1	{1e-7, 4e-7, 5e-7 }	0.1
HTMuon	{1e-6, 5e-6, 2e-5 , 3e-5}	0.1	{1e-7, 4e-7, 5e-7 }	0.1
Soren	{ 1e-6 , 5e-6, 2e-5, 3e-5}	0.1	{1e-7, 4e-7 , 5e-7}	0.1

Appendix

Table 9: Hyperparameters for Qwen2.5-3B and Mistral-7B-v0.1 on the UltraChat 200k and UltraFeedback Binarized datasets. Boldface values denote the selected learning rates.

Metric	Epoch 2	Epoch 4	Epoch 6	Epoch 8	Epoch 10
MAE	$0.0138_{\pm 0.0008}$	$0.0139_{\pm 0.0005}$	$0.0155_{\pm 0.0007}$	$0.0158_{\pm 0.0010}$	$0.0163_{\pm 0.0010}$
MaxErr	$0.0902_{\pm 0.0100}$	$0.0893_{\pm 0.0069}$	$0.0924_{\pm 0.0095}$	$0.0950_{\pm 0.0076}$	$0.0971_{\pm 0.0112}$

Appendix

Table 10: Approximation error of the SNS iteration relative to the exact sigmoid spectral map. Each entry reports the mean

\pm

standard deviation over validation mini-batches.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum Methods Gradient Descent
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
Spectrally Targeted Muon
Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral Optimization Deep Learning Optimization
Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.
Muon $^p$ : Muon with Fractional Spectral Powers
Jun 11, 2026Yihe Dong, Will SawinNeural Network Optimization Muon Optimizer
Princeton University

May 13, 2026J/K move · Enter open · S save

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen+2

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Abstract

Neural Network Optimization

Explore similar work

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Spectrally Targeted Muon

Muonp^pp: Muon with Fractional Spectral Powers

Muon $^p$ : Muon with Fractional Spectral Powers