CAST: History-Adaptive Spectral Clipping for LLM Pretraining

Jiacheng Li; Jianchao Tan; Kefeng Zhang; Yuchen Xie; Xunliang Cai

cs.LGOct 14, 2026

CAST: History-Adaptive Spectral Clipping for LLM Pretraining

Authors: Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Xunliang Cai

Organizations: Meituan, Beijing, China

Abstract

Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC. AdamW's previous second moment sets row and column scales. An exponential moving average of the clipped output's estimated spectral norm sets each matrix's threshold. CAST restores the original coordinates before the AdamW update. We train a mixture-of-experts (MoE) model with 19.08B total parameters for 83.89B tokens. CAST achieves lower training and mean validation losses than nine gradient-processing baselines, with the fewest relative gradient spikes. It reaches the best baseline's final smoothed training loss in 7.2% fewer steps.

Figures & tables

Figure 1: Validation loss after 5,000 steps: unweighted mean over 20 subsets (one run per method; seed 48). GS: GradientStabilizer. The vertical axis is truncated. Full comparison: Table 1 .

Panel of Figure 2 — Figure 2: CAST adds historical scaling (HS) and output feedback around SSC (Algorithm 1 ). AdamW uses restored $G_{\rm out}$ ; its $V$ becomes the next $V_{\rm old}$ , bias-corrected within $S$ . The EMA $h_{\rm old}$ tracks the estimated $\|Z_{\rm out}\|_{2}$ . Dashed histories advance only after a successful update.

Figure 3: The soft map and CAST’s threshold choice. (a) Hard and soft clipping at

\tau=1

, following the functional view of SSC in Jiang et al. (2026) . (b) The same soft map at

\tau=1.3h

for three illustrative values of the previous output history

h

. These analytical curves describe the active branch in standardized coordinates; the bypass gate and finite Newton–Schulz approximation are separate steps in Algorithm 1 .

Method	Train loss $\downarrow$	Validation loss $\downarrow$	CAST lower
CAST	1.6516	1.8738	—
GlobalGC	1.6718	1.8962	19/20
AdaGC	1.6625	1.8872	20/20
SSC-constant	1.6638	1.8864	20/20
SSC-cosine	1.6700	1.8947	19/20
GradientStabilizer	1.6828	1.9034	18/20

Table 1: Pretraining at 5,000 steps (83.89B nominal tokens). Train: mean over steps 4801–5000. Validation: unweighted mean over 20 subsets at step 5000. The final column counts subsets where CAST has lower loss. All runs use seed 48.

Figure 4: Quality trajectories for all ten primary conditions. Training curves show trailing 100-step means from iteration 2000 onward; validation curves average the same 20 subsets at each complete checkpoint. Appendix E includes the training trajectory from initialization.

Condition	Train loss	Validation loss	Gate fraction
CAST	1.6516	1.8738	0.884
w/o HS	1.6541	1.8747	0.403
w/ $\tau=0.1$	1.6937	1.9215	1.000
w/ cosine $\tau$	1.6767	1.9034	1.000

Table 2: CAST component study (HS: historical scaling). Loss windows follow Table 1 ; bold marks the lowest values. Gate fraction is the mean share of eligible matrices entering SSC over steps 4801–5000.

Figure 5: Training trajectories for CAST and its three component variants (HS: historical scaling). Curves show trailing 100-step means from iteration 2000 onward; Table 2 reports their final training and validation losses.

Figure 6: Raw input-gradient norms over iterations 501–5000 for ten primary conditions and three CAST ablations. The final row removes HS (historical scaling) or replaces the spectral EMA with fixed or cosine thresholds. All panels share a logarithmic vertical scale. Titles report relative spike counts; the dashed line marks norm 1. Table 3 gives the statistics.

Method	Median norm	P99 norm	Maximum norm	Steps with norm $>1$	Relative spikes
CAST	0.1510	0.3702	0.5476	0	1
GlobalGC	0.1541	0.4574	15.1610	10	38
AdaGC	0.1343	0.3485	1.0541	1	6
SSC-constant	0.1591	0.4723	1.1829	1	22
SSC-cosine	0.1653	0.4996	1.1922	2	21
GradientStabilizer	0.1701	0.6413	4.1736	5	31

Table 3: Input-gradient norms before clipping, steps 501–5000. P99: 99th percentile. Relative spikes exceed twice the preceding 100-step median, counting consecutive steps. Bold/underline: lowest/second-lowest counts, including ties. Last three rows: CAST ablations (HS: historical scaling).

Measure	CAST	SSC-constant	ClipByValue
Mean (s)	44.408	43.533	43.580
Median (s)	44.206	43.329	43.375
P95 (s)	44.876	43.960	44.119
Tokens/s/ device	5903	6022	6015

Table 4: Iteration times over steps 4801–5000 (200 steps per method, 64 devices). Mean is the arithmetic average; Median and P95 are the 50th and 95th percentiles. Throughput is

2048\times 8192/(64\bar{t})

, where

\bar{t}

is the mean iteration time in seconds. Timings include execution and logging.

Figure 7: Final 500 steps for CAST and nine baselines (100-step means). CAST reaches AdaGC’s final loss (dashed) at step 4,639. Colors follow Figure 9 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Controller setting	Value
Selected spectral space	Full matrix
Newton–Schulz iterations	5
Output power iterations	3
Multiplier $\lambda$	1.3
History decay $\beta_{h}$	0.95
Calibration updates / cap	40 / 1

Appendix

Table 5: Main-method controller settings. The training seed is separately fixed at 48.

Parameter component	Count across the model
20 MLA modules, including low-rank normalization	114,698,240
20 dense SwiGLU modules	283,115,520
Learned experts	18,119,393,280
MoE input/output projections	47,185,920
Routers	13,762,560
Block and MoE normalization	76,800

Appendix

Table 6: Parameter counts from the recorded architecture and model source. Replicated parameters are counted once. The activated backbone uses the target of 32 learned experts per MoE module; the actual count varies across tokens. Selecting 48 learned experts per MoE module gives an upper bound of approximately 0.742B activated backbone parameters.

Figure 8: Every baseline-minus-CAST validation loss difference. Positive values favor CAST; outlined negative cells favor the baseline. Subsets correspond to Table 7 .

Figure 9: Training from the first complete 100-step averaging window through iteration 5000. All ten primary conditions and three CAST ablations (HS: historical scaling) use trailing 100-step means. The main plot uses a logarithmic vertical axis; the box and connected inset show the final 500 steps on a linear scale.

Label	CAST	GlobalGC	AdaGC	SSC-constant	SSC-cosine
01 Academic / vocational	2.2466	2.2718	2.2569	2.2601	2.2693
02 Clinical / health	1.9951	2.0179	2.0054	2.0078	2.0168
03 Consumer / leisure	2.1824	2.2031	2.1924	2.1935	2.2026
04 Economy / services	2.0175	2.0434	2.0310	2.0309	2.0399
05 IT / programming	1.8519	1.8727	1.8631	1.8646	1.8713
06 Periodicals	2.7619	2.7855	2.7760	2.7744	2.7822

Appendix

Table 7: Token LM losses on all 20 validation subsets at iteration 5000. Bold and underline mark the best and second-best values across all ten conditions, spanning both panels. GS is GradientStabilizer; Value is ClipByValue.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 422 papers

Enhancing LLM Training via Spectral Clipping
Mar 15, 2026Xiaowen Jiang, Andrei Semenov, Sebastian U. StichLLM Input Optimization Spectral Regularization
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Universität des Saarlandes, Saarbrücken, Germany · EPFL
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team

Mar 15, 2026J/K move · Enter open · S save

Enhancing LLM Training via Spectral Clipping

Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Universität des Saarlandes, Saarbrücken, Germany · EPFL

LLM Input Optimization Spectral Regularization Language Model Pretraining

Open paper AskPDF arXiv

While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights and gradients, leaving them vulnerable to two empirical issues in large language model (LLM) training: (i) the optimizer updates can have large spectral norms, potentially destabilizing training and degrading generalization; (ii) stochastic gradient noise can exhibit sparse spectral spikes, with a few dominant singular values much larger than the rest. We propose SPECTRA, a general framework addressing these by (i) post-spectral clipping of updates to enforce spectral-norm constraints (ii) optional pre-spectral clipping of gradients to suppress spectral noise spikes. We prove that post-clipping constitutes a Composite Frank-Wolfe method with spectral-norm constraints and weight regularization. We further analyze how pre-clipping mitigates sparse spectral spikes. We propose efficient soft spectral clipping via Newton-Schulz iterations, avoiding expensive SVD. Experiments on LLM pretraining show SPECTRA uniformly improves validation loss for various optimizers, including AdamW, Signum, Mars, and AdEMAMix, with the best-performing variants achieving state-of-the-art results. Models trained with SPECTRA exhibit smaller weight norms, confirming the link between spectral clipping and regularization.

Abstract

Language Model Pretraining

Explore similar work

Enhancing LLM Training via Spectral Clipping

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training