Enhancing LLM Training via Spectral Clipping

Xiaowen Jiang; Andrei Semenov; Sebastian U. Stich

cs.LGMar 15, 2026

Enhancing LLM Training via Spectral Clipping

Authors: Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich

Organizations: CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Universität des Saarlandes, Saarbrücken, Germany · EPFL

Abstract

While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights and gradients, leaving them vulnerable to two empirical issues in large language model (LLM) training: (i) the optimizer updates can have large spectral norms, potentially destabilizing training and degrading generalization; (ii) stochastic gradient noise can exhibit sparse spectral spikes, with a few dominant singular values much larger than the rest. We propose SPECTRA, a general framework addressing these by (i) post-spectral clipping of updates to enforce spectral-norm constraints (ii) optional pre-spectral clipping of gradients to suppress spectral noise spikes. We prove that post-clipping constitutes a Composite Frank-Wolfe method with spectral-norm constraints and weight regularization. We further analyze how pre-clipping mitigates sparse spectral spikes. We propose efficient soft spectral clipping via Newton-Schulz iterations, avoiding expensive SVD. Experiments on LLM pretraining show SPECTRA uniformly improves validation loss for various optimizers, including AdamW, Signum, Mars, and AdEMAMix, with the best-performing variants achieving state-of-the-art results. Models trained with SPECTRA exhibit smaller weight norms, confirming the link between spectral clipping and regularization.

Figures & tables

Figure 1: Final validation loss comparison for large Llama-style models trained for the Chinchilla optimal horizon. ’Thin and deep’ models have smaller embedding dimensions but have more layers than ’wide and shallow’ ones. The running time comparisons w/wo using SPECTRA are provided in Figure F.7 . We use a total batchsize of 1012 and 992 for 780M and 820M model respectively with 1024 sequence length. The hyperparameters such as learning rate used for each method are reported in the tables in Section F.3 .

Figure 2: Validation loss comparison under small batch size training. Spectra-Signum maintains stability at the large learning rate, whereas Signum exhibits training instability (Signum typically prefers smaller LR than Adam-ish methods ( Semenov et al., 2025 ) ). Spectra-Signum with both pre and post clipping almost overlap with Spectra-Signum with post clipping.

Figure 3: Final validation loss comparison for a small size Llama model trained with both cos and wsd learning rate schedule. The run time comparisons w/wo using SPECTRA are provided in Figure F.8 . We use a total batchsize of 256 with 512 sequence length. The hyperparameters such as learning rate used for each method are reported in the tables in Section F.3 .

Figure 4: Evolution of the aggregated

\ell_{\infty}

and Frobenius norms of all weight matrices during training. SPECTRA consistently maintains a lower

\ell_{\infty}

norm compared to the base optimizer.

Optimizer	ARC-c	ARC-e	BoolQ	PIQA	HellaSwag	WinoGrande	OBQA	$\lambda$ -OpenAI	Sciq	MMLU	Average
Mars	$34.98$	$67.80$	$57.25$	$70.08$	$46.87$	$52.17$	$34.20$	29.38	$89.60$	$25.37$	$50.77$
D-Muon	$36.60$	$67.47$	$53.73$	$69.31$	$46.69$	$52.72$	$34.60$	$28.84$	90.90	$25.08$	$50.59$
Signum	$34.56$	$65.66$	$56.02$	$69.64$	$45.01$	$52.25$	$34.20$	$28.04$	$87.30$	$25.75$	$49.84$
Spectra-Signum	$35.58$	68.81	58.10	70.46	$47.04$	54.46	$35.60$	$29.36$	$90.30$	26.44	51.62
AdamW	$33.45$	$66.84$	$56.09$	$69.21$	$45.65$	$50.99$	$35.20$	$25.54$	$90.80$	$23.86$	$49.76$
Spectra-AdamW	$35.67$	$66.96$	$55.41$	$68.82$	$46.57$	$52.09$	36.40	$26.74$	$89.20$	$25.14$	$50.30$

Table 1: The evaluation results (accuracy

\times 100\%

) of the 820M model pretrained on Fineweb-Edu with 16.8B tokens (5-shot with lm-evaluation-harness ( Gao et al., 2024 ) ).

	Baseline (extended steps)				SPECTRA
Optimizer	16000	+5%	+10%	+20%	16000
		(16800)	(17600)	(19200)
AdamW	2.608	2.605	2.600	2.590	2.582
Signum	2.600	2.594	2.589	2.580	2.573
AdEMAMix	2.591	2.585	2.580	2.571	2.574
MARS	2.598	2.592	2.587	2.579	2.573

Table 2: Fixed-time budget comparison at 780M scale. Baseline optimizers are given 5–20% more training steps to compare with SPECTRA’s 2% per-step overhead. SPECTRA variants trained for 16000 steps are shown in the last column.

$c$	1	5	10	20	50
Val Loss	2.975	2.940	2.936	2.936	2.940

Table 3: Sensitivity to clipping threshold

c

(160M, WSD schedule, lr=

10^{-3}

). Baseline AdamW without SPECTRA: 2.958. Performance is flat from

c=5

c=50

(all within 0.004 of the optimum). Only

c=1

degrades due to over-aggressive clipping.

NS Iters	5 (bf16)	10 (bf16)	15 (float32)
Val Loss	2.939	2.936	2.937

Table 4: Ablation of number of NS iterations for Spectra-AdamW (160M, WSD schedule, lr=

10^{-3}

). bfloat16 has only 7 bits of mantissa. Additional iterations beyond convergence don’t improve accuracy–they accumulate round-off errors. Therefore, we use float32 for NS=15 instead.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure E.1 : The plot of $\operatorname{clip}_{c}(x):=\operatorname{sign}(x)\min(|x|,c)$ and $h_{c}(x):=\frac{x}{\sqrt{1+x^{2}/c^{2}}}$ .

Figure E.2 : Runtime comparison between Soft Spectral Clipping (SSC) and the orthogonalization operator on matrices with various sizes. By default, we use 10 Newton-Schulz iterations for SSC and 5 iterations for the latter for running the corresponding optimizers.

Figure F.1 : Convergence dynamics of update rule ( 4 ) using full-batch gradient. We report the function residual of the constrained composite logistic loss

F(\mathbf{X})

with

D_{2}=1.0

under different

b

. The unconstrained solution of

f(\mathbf{X})

has spectral norm close to

1

Figure F.2 : Convergence dynamics of update rule ( 4 ) using full-batch gradient. We report the function residual of the constrained composite logistic loss

F(\mathbf{X})

with

D_{2}=5.0

under different

b

. The unconstrained solution of

f(\mathbf{X})

has spectral norm close to

1

Figure F.3 : Convergence dynamics of update rule ( 4 ) using full-batch gradient. We report the function residual of the constrained composite logistic loss

F(\mathbf{X})

with

D_{2}=0.2

under different

b

. The unconstrained solution of

f(\mathbf{X})

has spectral norm close to

1

Figure F.4 : Convergence dynamics of update rule ( 4 ) using stochastic gradient. We report the function residual of the constrained composite logistic loss

F(\mathbf{X})

with

D_{2}=1.0

under different

b

. The unconstrained solution of

f(\mathbf{X})

has spectral norm close to

1

Figure F.5 : Convergence comparison of Vanilla SGD, SGD with Global Clipping, and SGD with Spectral Clipping under low-rank spectral noise. We vary the noise spike magnitude

\ell\in\{10,100,1000\}

relative to the gradient spectral norm bound

M\approx 15

Figure F.6 : Convergence comparison of Vanilla SGD-M and SGD-M with Spectral Clipping under low-rank spectral noise. We vary the noise spike magnitude

\ell\in\{10,100,1000\}

relative to the gradient spectral norm bound

M\approx 15

Parameters	124M	160M	780M	820M
Embed dimension	768	768	1280	2048
# Attention heads	12	12	20	16
# Layers	12	12	32	12
Untied embed	False	True	True	True

Appendix

Table F.1 : Configuration of the considered model architecture.

	160M			780M			820M
Method	Total (ms)	Opt Step (%)	Overhead	Total (ms)	Opt Step (%)	Overhead	Total (ms)	Opt Step (%)	Overhead
AdamW	200.7	1.0	—	6550	0.2	—	4412	0.2	—
+ Pre-Clip	232.6	13.7	+15.9%	6640	1.4	+1.4%	4480	1.7	+1.5%
+ SPECTRA	238.3	15.6	+18.7%	6654	1.7	+1.6%	4504	2.4	+2.1%
Signum	203.2	1.9	—	6536	0.3	—	4408	0.5	—
+ SPECTRA	241.2	17.1	+18.7%	6667	1.9	+2.0%	4514	2.6	+2.4%

Appendix

Table F.2 : Per-step timing at 160M, 780M, and 820M scale (4

\times

H100 GPU, averaged over 180 steps after warm-up). SPECTRA overhead shrinks from

\sim

17–19% at 160M to

\sim

2% at 780M/820M as the forward/backward pass dominates at larger scale. D-Muon has comparable overhead to SPECTRA due to its NS-based orthogonalization (

\sim

9% at 160M,

\sim

1–2% at 780M/820M). SOAP is the most expensive optimizer, adding +49% at 160M and +21% at 780M vs. AdamW, due to its larger memory footprint (requiring more accumulation steps), periodic eigendecomposition, and per-step gradient projection.

Optimizer	State Buffers	Memory	Factor
AdamW	$m,v$	$2M$	$2\times$
AdEMAMix	$m_{\text{fast}},m_{\text{slow}},v$	$3M$	$3\times$
D-Muon 3 3 3 D-Muon uses momentum-only updates for 2D matrices (with rows $<10000$ ) and AdamW for others including embeddings/biases.	$m$ (2D), $m,v$ (1D)	$M_{2d}+2M_{1d}$	$<2\times$
Signum	$m$ (momentum)	$M$	$1\times$
SOAP 4 4 4 SOAP stores preconditioner matrices for any parameters with $d_{i}\leq D$ (default $D=10000$ ). For typical LLM architectures with $d_{i}\sim 4096$ , preconditioner overhead is approximately $2M$ .	$m,v,\text{GG},\text{Q}$	$2M+\sum_{d_{i}\leq D}2d_{i}^{2}$	$\approx 4\times$
MARS (AdamW) 5 5 5 MARS stores gradient history for variance reduction where $g_{\text{prev}}$ represents the previous gradient.	$m,v,g_{\text{prev}}$ (2D)	$\approx 3M$	$\approx 3\times$

Appendix

Table F.3 : Memory requirements of different optimizers.

M

denotes total model parameters,

M_{2d}

and

M_{1d}

denote 2D and 1D parameters respectively,

m

and

v

denote the first and second momentum buffer, and

d_{i}

represents the size of each dimension of a parameter tensor (e.g., for a matrix of shape

(d_{1},d_{2})

, both

d_{1}

and

d_{2}

are considered)

Figure F.7 : Training dynamics of the 820M-parameter model. The plots display validation loss with respect to training iterations (up) and wall-clock time (down), conducted on 4 H100 GPUs. We use a micro batchsize of 31 with 8 accumulation steps on each machine. The hyperparameters such as learning rate used for each method are reported in the tables in Section F.3 .

Figure F.8 : Training dynamics of the 160M-parameter model. The plots display validation loss with respect to training iterations (up) and wall-clock time (down), conducted on 4 H100 GPUs. We use a micro batchsize of 64 with 1 accumulation step on each machine. The hyperparameters such as learning rate used for each method are reported in the tables in Section F.3 .

Method	Val Loss
Signum (lr= $2\times 10^{-4}$ )	4.955
Signum (lr= $10^{-4}$ )	3.688
SPECTRA-Signum (lr= $10^{-3}$ )	3.439
+ pre-clip (constant $c=0.1$ )	3.431
+ pre-clip (cos decay, $c$ : $10\to 0.01$ )	3.425

Appendix

Table F.4 : Pre-spectral clipping ablation on SlimPajama (160M, batch size 8, seq len 512, 128k iterations). Pre-spectral clipping improves the baseline SPECTRA by 0.014 nats under noisy conditions, and a decaying threshold schedule outperforms a constant threshold.

Optimizer	LR	Val Loss	Optimizer	LR	Val Loss
SOAP	$5\times 10^{-3}$	$2.653$	Spectra-Signum	$2\times 10^{-3}$	$2.642$
D-Muon	$2\times 10^{-3}$	$2.658$		$3\times 10^{-3}$	$2.638$
MARS	$2\times 10^{-3}$	$2.643$		$5\times 10^{-3}$	$2.643$
AdEMAMix	$2\times 10^{-3}$	$2.647$	Spectra-AdEMAMix	$2\times 10^{-3}$	$2.640$
	$3\times 10^{-3}$	$2.642$		$3\times 10^{-3}$	$\mathbf{2.635}$

Appendix

Table F.5 : Short training (

8

B tokens or

8

k steps) of

1.5

B muP models. The details and scripts can be found in https://github.com/mlolab/llm-spectral-clipping/tree/main/scripts/1.5b-mup-models .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Learning Rate $\eta$	5e-4, 1e-3 , 2e-3	1e-3, 2e-3 , 3e-3	1e-3	1e-3
Clipping Threshold $c$	5.0, 10.0	5.0	10.0	10.0
Clipping Schedule	standard	standard	standard	standard
Others		Same as in Table F.9

Appendix

Table F.6 : Hyperparameter configurations for Spectra-AdamW .

Method (lr)	780M	820M
SOAP ( $10^{-3}$ )	2.586	2.606
SOAP ( $3\times 10^{-3}$ )	2.571	2.602
SOAP ( $5\times 10^{-3}$ )	2.574	2.603
SPECTRA-Signum ( $10^{-3}$ )	2.573	2.594

Appendix

Table F.7 : SOAP vs. SPECTRA-Signum at larger scales (final validation loss). SOAP is tuned over three learning rates. Despite SOAP requiring

\sim

19% more time per step at 780M (+8% at 820M) and nearly 2

\times

more memory than SPECTRA-Signum, it achieves comparable results at 780M and worse results at 820M.

		Learning Rate
Schedule	Method	1e-3	2e-3	3e-3	4e-3	5e-3
wsd	AdamW-Post-Orth	2.988	2.973	2.974	–	–
wsd	Spectra-AdamW	$\mathbf{2.936}$	–	–	–	–
cos	AdamW-Post-Orth	3.009	2.966	2.955	2.953	2.956
cos	Spectra-AdamW	–	$\mathbf{2.922}$	–	–	–

Appendix

Table F.8 : Final validation loss comparison for the 160M Llama model trained on 16.8B tokens. Best results per schedule are bolded .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	2000	2000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Momentum $\beta_{1}$	0.8	0.8	0.9	0.9

Appendix

Table F.9 : Hyperparameter configurations for AdamW .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	2000	2000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Momentum $\beta_{1}$	0.9	0.9	0.9	0.9

Appendix

Table F.10 : Hyperparameter configurations for AdEMAMix .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Learning Rate $\eta$	5e-4, 1e-3 , 2e-3	1e-3, 2e-3 , 3e-3	5e-4, 1e-3	1e-3
Clipping Threshold $c$	5.0, 10.0	5.0	10.0	10.0
Clipping Schedule	standard	standard	standard	standard
Others		Same as in Table F.10

Appendix

Table F.11 : Hyperparameter configurations for Spectra-AdEMAMix .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	8000	8000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Momentum $\beta$	0.95	0.95	0.95	0.95

Appendix

Table F.12 : Hyperparameter configurations for Signum .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Warmup Steps	2000	2000	2000	2000
Learning Rate $\eta$	5e-4, 1e-3 , 2e-3	1e-3, 2e-3 , 3e-3	1e-3	1e-3
Clipping Threshold $c$	10.0	5.0	10.0	10.0
Clipping Schedule	standard	standard	standard	standard
Others		Same as in Table F.12

Appendix

Table F.13 : Hyperparameter configurations for Spectra-Signum .

Hyperparameter	160M (wsd)	780M (wsd)	820M (wsd)
Warmup Steps	2000	2000	2000
Learning Rate Mars	3e-3	3e-3	3e-3
Clipping Threshold $c$	10.0	10.0	10.0
Clipping Schedule	standard	standard	standard
Others		Same as in Table F.16

Appendix

Table F.14 : Hyperparameter configurations for Spectra-Mars .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	2000	2000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Momentum $\beta$	0.95	0.95	0.95	0.95

Appendix

Table F.15 : Hyperparameter configurations for D-Muon .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	2000	2000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Learning Rate AdamW	5e-4, 1e-3	1e-3	1e-3	1e-3

Appendix

Table F.16 : Hyperparameter configurations for Mars (AdamW) .

Hyperparameter	160M (wsd)	160M (cos)	780M (wsd)	820M (wsd)
Batch Size	256	256	1012	992
Sequence Length	512	512	1024	1024
Warmup Steps	2000	2000	2000	2000
Weight Decay $\lambda$	0.1	0.1	0.1	0.1
Gradient Clipping	0.5	0.5	0.1	0.1
Preconditioner dimension	10000	10000	10000	10000

Appendix

Table F.17 : Hyperparameter configurations for SOAP . (Due to the large memory footprint of SOAP, we need larger gradient accumulation steps to fit the model in memory, which largely extended the training runtime.

Figure F.9 : Singular value distribution of raw stochastic gradients for the 124M-parameter model trained with AdamW. We visualize the spectrum at four training stages (0%, 5%, 50%, and 99%) for representative layers across the network depth. Each panel reports the maximum and minimum singular values along with their ratio (condition number). We observe that a small number of singular values are orders of magnitude larger than the rest, leading to high condition numbers.

Figure F.10 : Singular value distribution of update matrices (

\mathbf{U}_{k}

) trained with AdamW on the 124M-parameter model. We visualize the spectrum at four training stages (0%, 5%, 50%, and 99%) for representative layers across the network depth. Each panel reports the maximum and minimum singular values along with their ratio (condition number). AdamW updates show a more uniform spectral distribution compared to the raw gradient. However, they fail to impose a hard ceiling.

Figure F.11 : Singular value distribution of update matrices (

\mathbf{U}_{k}

) trained with Signum on the 124M-parameter model. We visualize the spectrum at four training stages (0%, 5%, 50%, and 99%) for representative layers across the network depth. Each panel reports the maximum and minimum singular values along with their ratio (condition number). The spectral norm of the update matrices ranges from 70 to 1300.

Figure F.12 : Singular value distribution of update matrices

\mathbf{U}_{k}=\alpha\operatorname{clip}^{\operatorname{sp}}_{c_{k}}(\mathbf{U}_{k}^{\text{AdamW}})

trained with Spectra-AdamW on the 124M-parameter model. We visualize the spectrum at four training stages (0%, 5%, 50%, and 99%) for representative layers across the network depth. Each panel reports the maximum and minimum singular values along with their ratio (condition number). The spectral norms of the update matrices exhibit hard ceilings and the condition numbers are small.

Figure F.13 : Singular value distribution of update matrices

\mathbf{U}_{k}=\alpha\operatorname{clip}^{\operatorname{sp}}_{c_{k}}(\mathbf{U}_{k}^{\text{Signum}})

trained with Spectra-Signum on the 124M-parameter model. We visualize the spectrum at four training stages (0%, 5%, 50%, and 99%) for representative layers across the network depth. Each panel reports the maximum and minimum singular values along with their ratio (condition number). The spectral norms of the update matrices exhibit hard ceilings and the condition numbers are small.

Figure F.14 : For each layer, we report the singular value distribution of large-batch gradient

\mathbf{G}

(blue bar) and top-5 singular value distribution of the noise matrices

\{\mathbf{N}_{i}\}_{i=1}^{50}

(red) in the first row. In the second row, we report the distribution of both the spectral and chordal subspace distances of the top-5 principal directions between the noise matrices and the large-batch signal. Results are shown for representative layers of the 124M model (AdamW) at four training stages 0%, 5%, 50%, and 99%. (Part 1) This confirms the presence of ”sparse spectral spikes” in the noise and indicates that the dominant noise spikes are nearly orthogonal to the principal directions of the true signal.

Figure F.15 : Part 2 of Figure F.14 . We observe a consistent phenomenon as in part 1.

Figure F.16 : Singular value distribution of raw stochastic gradients for the 124M-parameter GPT with muP trained with AdamW. The heavy-tailed ’spectral outlier’ phenomenon is less significant than 124M-llama model without muP.

Figure F.17 : For each layer, we report the singular value distribution of large-batch gradient

\mathbf{G}

(blue bar) and top-5 singular value distribution of the noise matrices

\{\mathbf{N}_{i}\}_{i=1}^{50}

(red) in the first row. In the second row, we report the distribution of both the spectral and chordal subspace distances of the top-5 principal directions between the noise matrices and the large-batch signal. Results are shown for representative layers of the 124M GPT model with muP (AdamW) at four training stages 0%, 5%, 50%, and 99%. The presence of ”sparse spectral spikes” in the noise appears mostly only in the end of the training.

LLM Input Optimization

See all 24 papers

CAST: History-Adaptive Spectral Clipping for LLM Pretraining
Oct 14, 2026Jiacheng Li, Jianchao Tan, Kefeng Zhang +2Language Model Pretraining
Meituan, Beijing, China
ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University

Oct 14, 2026J/K move · Enter open · S save

CAST: History-Adaptive Spectral Clipping for LLM Pretraining

Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie+1

Meituan, Beijing, China

Language Model Pretraining

Open paper AskPDF arXiv

Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC. AdamW's previous second moment sets row and column scales. An exponential moving average of the clipped output's estimated spectral norm sets each matrix's threshold. CAST restores the original coordinates before the AdamW update. We train a mixture-of-experts (MoE) model with 19.08B total parameters for 83.89B tokens. CAST achieves lower training and mean validation losses than nine gradient-processing baselines, with the fewest relative gradient spikes. It reaches the best baseline's final smoothed training loss in 7.2% fewer steps.

Abstract

LLM Input Optimization

Explore similar work

CAST: History-Adaptive Spectral Clipping for LLM Pretraining

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon