Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers

Chenyu Zhu; Ruoyu Zhao; Zhichao Lu

cs.LGOct 22, 2026

Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers

Authors: Chenyu Zhu, Ruoyu Zhao, Zhichao Lu

Organizations: Department of Computer Science, City University of Hong Kong

Abstract

Predicting relative trained performance from architectural specifications can guide design when training costs limit experimentation. Neural Spectral Capacity (NSC) provides such a heuristic, but scores projections independently and assigns identical scores to compatible reorderings of the same layer configurations. We introduce Residual-Aware Neural Spectral Capacity (ResNSC) for Transformers, using a specification-derived statistical reference for accumulated residual updates to determine the input and residual-state statistics in each projection's capacity calculation, while retaining additive evaluation without data or instantiated weights. When residual widths vary, earlier choices affect later scores, so we augment NSC's allocator (NSC-DP) with residual history beyond its layer-and-budget state. ResNSC-DP jointly allocates residual and feed-forward widths under a resource budget, with certified bounds on the best numerically evaluated ResNSC score within a specified finite candidate set. At the 121M and 374M pretraining scales, selected architectures achieve lower mean perplexity than uniform, variable-width, and tapered allocations on FineWeb-Edu, WikiText-2, and LAMBADA at comparable parameter budgets. In inherited-weight LLaMA-7B pruning using the LoNAS supernet, ResNSC-DP improves eight-task average accuracy over NSC by 3.28--5.46 percentage points across eight matched parameter budgets. Paired reversals preserve the projection shapes and NSC scores yet worsen performance in both settings, exposing performance differences hidden by position-independent scoring. The code for this work is available at https://github.com/Optima-CityU/neural-spectral-capacity.

Figures & tables

Panel of Figure 1 — Figure 1: From NSC to ResNSC. Left: schematic ResNSC-DP allocation (A) and reversal (B), with mean 374M FineWeb-Edu perplexities. Reversal preserves projection shapes and NSC scores but changes residual context. Right: additive scoring views of block $\ell$ . ResNSC’s specification-derived covariances use residual history and, for output projections, within-branch propagation. For $b=2\ell-2$ , $p_{b},p_{b+1},p_{b+2}$ are the pre-attention, post-attention, and post-FFN reference variances. Blue/orange: attention/FFN; normalization omitted.

Panel of Figure 2 — Figure 2: Illustrative reference variances ( $E=M=6$ ). Each block adds $2$ (attention, then FFN) on its first $d_{\ell}$ coordinates; colors mark the last updating block.

Branch index $b$	Projection $j$	Shape $W_{\ell,j}$	Input $\Sigma_{x}$	Noise $\Sigma_{z}$
$2\ell-2$	Q, K, V	$d\times d$	$\Sigma_{x,b}^{\mathrm{read}}$	$I_{d}$
$2\ell-2$	O	$d\times d$	$\Sigma_{x,b}^{\mathrm{write}}$	$\Sigma_{z,b}^{\mathrm{write}}$
$2\ell-1$	gate, up	$f\times d$	$\Sigma_{x,b}^{\mathrm{read}}$	$I_{f}$
$2\ell-1$	down	$d\times f$	$\Sigma_{x,b}^{\mathrm{write}}$	$\Sigma_{z,b}^{\mathrm{write}}$

Table 1: Reference contexts for Equation ( 1 ):

d=d_{\ell}

f=f_{\ell}

. Each row uses the ledger immediately before branch

b

	121M			374M
Method	PPL $\downarrow$	WikiText-2 $\downarrow$	LAMBADA $\downarrow$	PPL $\downarrow$	WikiText-2 $\downarrow$	LAMBADA $\downarrow$	Avg8 $\uparrow$
ResNSC (ours)	19.39 ( $\pm$ 0.01)	51.13 ( $\pm$ 0.21)	202.8 ( $\pm$ 13.5)	13.60 ( $\pm$ 0.002)	31.20 ( $\pm$ 0.03)	53.0 ( $\pm$ 0.4)	44.60 ( $\pm$ 0.07)
uniform	20.25 ( $\pm$ 0.02)	55.47 ( $\pm$ 0.07)	262.9 ( $\pm$ 13.7)	14.27 ( $\pm$ 0.002)	33.82 ( $\pm$ 0.05)	76.2 ( $\pm$ 2.0)	44.07 ( $\pm$ 0.19)
VWT	19.48 ( $\pm$ 0.06)	52.33 ( $\pm$ 0.15)	252.6 ( $\pm$ 35.7)	13.65 ( $\pm$ 0.002)	31.99 ( $\pm$ 0.08)	70.6 ( $\pm$ 2.1)	44.26 ( $\pm$ 0.11)
TLM	20.38 ( $\pm$ 0.02)	55.43 ( $\pm$ 0.53)	252.1 ( $\pm$ 8.5)	14.43 ( $\pm$ 0.001)	34.19 ( $\pm$ 0.02)	78.1 ( $\pm$ 0.6)	43.73 ( $\pm$ 0.05)
NSC	20.00 ( $\pm$ 0.03)	53.40 ( $\pm$ 0.31)	221.4 ( $\pm$ 10.1)	13.82 ( $\pm$ 0.003)	32.40 ( $\pm$ 0.11)	66.9 ( $\pm$ 1.9)	44.26 ( $\pm$ 0.09)

Table 2: Language-modeling results after final decay at 121M and 374M: FineWeb-Edu perplexity (PPL), WikiText-2 word perplexity, LAMBADA perplexity, and, at 374M, mean eight-task zero-shot accuracy (Avg8). 121M: means and sample standard deviations over three independent runs; 374M: one run plus two final-decay continuations, so standard deviations reflect annealing variation only. Bold: best per column; per-task accuracies in Appendix F .

Scoring rule	Profile $\max_{\ell}d_{\ell}\to d_{16}$	PPL $\downarrow$	WikiText-2 $\downarrow$	LAMBADA $\downarrow$
ResNSC (full)	$1280\to 480$	19.39 ( $\pm$ 0.01)	51.04 ( $\pm$ 0.20)	211.2 ( $\pm$ 2.3)
w/o residual context	$800\to 640$ (identical to NSC)	20.00 ( $\pm$ 0.01)	53.25 ( $\pm$ 0.21)	233.6 ( $\pm$ 1.7)
Depth only	$1760\to 320$	19.85 ( $\pm$ 0.01)	52.85 ( $\pm$ 0.21)	205.9 ( $\pm$ 2.5)
Mean history only	$1920\to 320$	20.20 ( $\pm$ 0.01)	54.61 ( $\pm$ 0.20)	229.5 ( $\pm$ 2.7)
w/o propagated spectrum	$1440\to 320$	19.72 ( $\pm$ 0.01)	52.72 ( $\pm$ 0.20)	197.5 ( $\pm$ 3.3)

Table 3: Ablating the architecture score at 121M. All variants share the search space and parameter budget; profiles show the widest and final active widths. Rows 2–4 simplify residual history (none, branch position only, coordinate average); the last row removes the propagated write-source spectrum. Entries are one run (seed 42) plus two final-decay continuations; standard deviations reflect annealing variation only. Bold: best per column.

Figure 3: ResNSC improves the accuracy–compute trade-off. Avg8 versus TFLOPs (

T=256

) across eight matched budgets (A–G: smaller budgets), with proxy baselines and the unpruned supernet (star). Full results in Appendix H .

Method	Params	BoolQ	PIQA	SIQA	HSwag	WGrnd	ARC-e	ARC-c	OBQA	Avg 8	Cost
LoNAS-SuperNet	6.74 B	66.02	77.58	72.93	57.49	66.85	78.41	62.29	76.20	69.72	—
SNIP	5.44 B	61.93	69.21	64.99	41.26	62.04	66.50	52.47	67.40	60.73	22 min
SynFlow	5.23 B	61.13	70.24	63.25	45.94	59.98	66.84	51.88	66.20	60.68	23 min
GradNorm	5.50 B	61.31	70.78	67.35	46.33	62.59	69.02	55.03	67.20	62.45	22 min
W-PCA	5.69 B	63.82	70.95	67.91	49.24	62.75	69.57	56.40	70.00	63.83	45 min
NSC	5.69 B	64.01	72.85	68.94	51.30	64.72	72.43	58.70	70.80	65.47	$<$ 1 s

Table 4: LoNAS LLaMA-7B pruning on eight commonsense tasks (accuracy, %). ResNSC and NSC use the same width/rank menu and matched parameter counts. Params reports each selected model’s size. The supernet is a reference; bold marks the best pruned result per column. Reverse reassigns FFN widths in reverse order across the original layers, preserving parameter count and NSC score. Cost is search-stage wall-clock time.

Benchmark	$N$	$\tau_{b}$ (NSC / ResNSC)	$\rho$ (NSC / ResNSC)	Distinct (NSC / ResNSC)
FlexiBERT	500	0.695 / 0.692	0.884 / 0.883	297 / 421
GPT-2	200	0.849 / 0.854	0.968 / 0.970	200 / 200
AutoFormer-Small	1,001	0.658 / 0.719	0.852 / 0.907	952 / 1,001

Table 5: Global ranking correlations and number of distinct scores.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\ell$ , $b$	Block index and branch index; attention has $b=2\ell-2$ , FFN has $b=2\ell-1$ .
$E,L,H,M$	Embedding width, number of blocks, number of attention heads, and complete residual-state width.
$d_{\ell},f_{\ell}$	Active residual width and FFN intermediate width of block $\ell$ .
$\widetilde{d}_{b},k_{b}$	Active residual width (shared by reading and writing) and latent width of branch $b$ .
$h_{b},p_{b}$	Actual residual state and specification-derived reference-power ledger before branch $b$ .
$\Sigma_{x},\Sigma_{z}$	Generic source and noise covariances in $\psi(W;\Sigma_{x},\Sigma_{z})$ .

Appendix

Table 6: Notation shared by the main text and mathematical appendices.

Projection	Scored matrix shape	Entry variance	Upstream reference for a write
Q, K, V	$d\times d$	$2/(d+d/H)$	None
O	$d\times d$	$1/d$	$k=d$ , $\operatorname{Var}(X_{ij})=1/d$ , $G=I_{d}$
gate, up	$f\times d$	$2/(d+f)$	None
down	$d\times f$	$2/(d+f)$	$k=f$ , $\operatorname{Var}(X_{ij})=1/d$ , normalized gate

Appendix

Table 7: Projection and propagation conventions for a standard block, with

d=d_{\ell}

and

f=f_{\ell}

Context	$d$	Read difference (%)	Write difference (%)
Isotropic	64	0.176	0.444
Nonuniform history	64	0.222	0.397
Zero source directions	64	0.337	0.647
SiLU-gated	64	0.165	0.562
Isotropic	128	0.135	0.203
Nonuniform history	128	0.135	0.260

Appendix

Table 8: Finite-matrix validation: absolute relative difference between the deterministic value and the mean of 1,200 independent reference draws (%).

Instance	Objective $S(\widehat{a})$	Rel. gap
121M joint	59,330.78	0
374M joint	133,930.49	0

Appendix

Table 9: Final objectives and numerical gaps for joint residual/FFN-width search. Both searches exhaust the valid queue.

Method	121M loss	374M loss
ResNSC	2.96496 ( $\pm$ 0.00067)	2.61025 ( $\pm$ 0.00017)
uniform	3.00813 ( $\pm$ 0.00109)	2.65818 ( $\pm$ 0.00011)
VWT	2.96922 ( $\pm$ 0.00286)	2.61367 ( $\pm$ 0.00016)
TLM	3.01478 ( $\pm$ 0.00108)	2.66964 ( $\pm$ 0.00006)
NSC	2.99598 ( $\pm$ 0.00134)	2.62640 ( $\pm$ 0.00019)
Reverse	3.00905 ( $\pm$ 0.00233)	2.67282 ( $\pm$ 0.00005)

Appendix

Table 10: Held-out FineWeb-Edu loss (nats) at 121M and 374M. 121M arms: three independent runs (seeds 42–44). 374M arms: one run (seed 42) plus two final-decay continuations, whose standard deviation reflects annealing variation only.

Method	LMB $\uparrow$	PIQA $\uparrow$	HSwag $\uparrow$	WGrnd $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	SIQA $\uparrow$	BoolQ $\uparrow$	Avg8 $\uparrow$
ResNSC (ours)	29.26 [-1pt] $\pm$ 0.16	64.89 [-1pt] $\pm$ 0.33	35.49 [-1pt] $\pm$ 0.20	50.41 [-1pt] $\pm$ 0.59	54.53 [-1pt] $\pm$ 0.42	25.06 [-1pt] $\pm$ 0.26	37.17 [-1pt] $\pm$ 0.16	59.97 [-1pt] $\pm$ 0.96	44.60 [-1pt] $\pm$ 0.07
uniform	26.69 [-1pt] $\pm$ 0.26	64.56 [-1pt] $\pm$ 0.22	34.16 [-1pt] $\pm$ 0.08	52.57 [-1pt] $\pm$ 0.21	52.46 [-1pt] $\pm$ 0.40	26.11 [-1pt] $\pm$ 0.34	37.22 [-1pt] $\pm$ 0.18	58.76 [-1pt] $\pm$ 0.36	44.07 [-1pt] $\pm$ 0.19
VWT	27.03 [-1pt] $\pm$ 0.32	65.38 [-1pt] $\pm$ 0.35	35.94 [-1pt] $\pm$ 0.30	50.51 [-1pt] $\pm$ 0.96	53.68 [-1pt] $\pm$ 0.76	26.39 [-1pt] $\pm$ 0.47	36.51 [-1pt] $\pm$ 0.34	58.66 [-1pt] $\pm$ 0.55	44.26 [-1pt] $\pm$ 0.11
TLM	26.24 [-1pt] $\pm$ 0.16	63.13 [-1pt] $\pm$ 0.36	33.87 [-1pt] $\pm$ 0.06	50.75 [-1pt] $\pm$ 0.36	52.57 [-1pt] $\pm$ 0.51	25.63 [-1pt] $\pm$ 0.34	36.18 [-1pt] $\pm$ 0.22	61.46 [-1pt] $\pm$ 0.23	43.73 [-1pt] $\pm$ 0.05
NSC	27.74 [-1pt] $\pm$ 0.36	64.42 [-1pt] $\pm$ 0.14	35.27 [-1pt] $\pm$ 0.14	50.33 [-1pt] $\pm$ 0.79	53.04 [-1pt] $\pm$ 0.21	25.74 [-1pt] $\pm$ 0.27	36.51 [-1pt] $\pm$ 0.38	61.07 [-1pt] $\pm$ 0.27	44.26 [-1pt] $\pm$ 0.09
Reverse	25.76 [-1pt] $\pm$ 0.21	64.16 [-1pt] $\pm$ 0.45	33.62 [-1pt] $\pm$ 0.06	52.07 [-1pt] $\pm$ 0.84	53.31 [-1pt] $\pm$ 0.48	25.26 [-1pt] $\pm$ 0.09	35.55 [-1pt] $\pm$ 0.06	53.12 [-1pt] $\pm$ 1.72	42.86 [-1pt] $\pm$ 0.18

Appendix

Table 11: 374M zero-shot accuracies (%) after the final decay: eight tasks and their mean (Avg8), one run plus two final-decay continuations (mean and sample standard deviation). HellaSwag and ARC-c use normalized accuracy. Bold: best per column; underline: second best.

Task ID	Dataset	Split	Reported metric
wikitext	WikiText-2 ( Merity et al., 2017 )	test	word PPL
lambada_openai	LAMBADA ( Paperno et al., 2016 )	test	PPL, accuracy
piqa	PIQA ( Bisk et al., 2020 )	validation	accuracy
hellaswag	HellaSwag ( Zellers et al., 2019 )	validation	normalized accuracy
winogrande	WinoGrande (XL) ( Sakaguchi et al., 2020 )	validation	accuracy
arc_easy	ARC-Easy ( Clark et al., 2018 )	test	accuracy

Appendix

Table 12: Pretraining-model evaluation datasets, splits, and metrics. All tasks use zero-shot templates and the complete indicated split.

Method	121M	374M
ResNSC	121,188,640	373,799,936
Uniform	121,262,720	373,867,520
VWT	121,275,680	373,859,200
TLM	121,262,720	373,867,520
NSC	120,898,560	373,847,040
Reverse	121,189,280	373,801,216

Appendix

Table 13: Instantiated parameter counts for the pretraining configurations, including embeddings, readout, and normalization. Column headings are the scale labels used in the paper.

Layer	ResNSC		uniform		VWT		TLM		NSC		Reverse
	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$
1	1120	1536	640	1760	864	3456	640	2624	640	1312	480	672
2	1280	1760	640	1760	768	3072	640	2624	800	1088	480	672
3	1280	1760	640	1760	704	2816	640	2560	800	1088	480	672
4	960	1312	640	1760	608	2432	640	2464	800	1088	480	672
5	960	1312	640	1760	544	2176	640	2336	800	1088	480	672

Appendix

Table 14: Exact 121M per-layer widths in input-to-output order: residual width

d_{\ell}

and FFN intermediate width

f_{\ell}

Layer	ResNSC		uniform		VWT		TLM		NSC		Reverse
	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$	$d$	$f$
1	1792	3712	1024	2816	1408	5632	1024	4224	1280	3520	512	1056
2	2048	4224	1024	2816	1312	5248	1024	4224	1280	2656	512	1056
3	2048	4224	1024	2816	1216	4864	1024	4160	1280	2656	512	1056
4	2048	4224	1024	2816	1152	4608	1024	4096	1280	2656	512	1056
5	1536	3168	1024	2816	1056	4224	1024	4032	1280	2656	512	1056

Appendix

Table 15: Exact 374M per-layer widths in input-to-output order: residual width

d_{\ell}

and FFN intermediate width

f_{\ell}

Method	Params	BoolQ	PIQA	SIQA	HSwag	WGrnd	ARC-e	ARC-c	OBQA	Avg 8	$\Delta$
NSC best	5.690 B	64.01	72.85	68.94	51.30	64.72	72.43	58.70	70.80	65.47	—
NSC A	5.572 B	64.01	72.42	68.17	51.26	64.09	72.52	58.53	70.60	65.20	—
NSC B	5.538 B	63.73	72.42	68.27	51.05	63.46	72.18	58.79	71.20	65.14	—
NSC C	5.487 B	63.61	72.25	67.50	50.69	63.38	72.18	58.87	72.00	65.06	—
NSC D	5.453 B	63.73	72.03	66.73	49.95	63.85	71.34	58.19	70.80	64.58	—
NSC E	5.420 B	63.15	71.49	67.71	49.73	63.69	71.30	57.08	71.00	64.39	—

Appendix

Table 16: Complete LoNAS results with the shared FFN menu

\{5504,6880,8256,9632,11008\}

. ResNSC and NSC have identical parameter counts at each budget. Labels match Figure 3 ; “best” denotes the highest-budget point.

\Delta

is the Avg 8 difference from the corresponding NSC model (percentage points).

			Layer indices by FFN width
Method	Budget	Params (B)	11008	9632	8256	6880	5504
ResNSC	best	5.690	1–9	10–13	14–18	19–26	27–32
ResNSC	A	5.572	1–8	9–12	13–16	17–23	24–32
ResNSC	B	5.538	1–8	9–11	12–16	17–22	23–32
ResNSC	C	5.487	1–7	8–11	12–15	16–21	22–32
ResNSC	D	5.453	1–7	8–10	11–14	15–21	22–32

Appendix

Table 17: Exact LoNAS FFN configurations for the budgets in Table 16 . Each width column lists the layers using that FFN intermediate width; layers are numbered 1–32 from input to output, and a dash means none. All configurations have residual width 4,096; both search methods select LoRA rank 32 in every layer.

Benchmark	$\tau_{b}$	$\rho$	$\tau_{b}$	$\rho$	$\tau_{b}$	$\rho$	$\Delta\rho$
	NSC		ResNSC		Params
FlexiBERT	0.6946	0.8843	0.6921	0.8833	0.4846	0.6763	-0.0009
GPT-2	0.8488	0.9681	0.8543	0.9698	0.6386	0.7943	+0.0018
AutoFormer-Tiny	0.6119	0.8106	0.5717	0.7786	0.4691	0.6347	-0.0320
AutoFormer-Small	0.6580	0.8519	0.7192	0.9070	0.7885	0.9414	+0.0551

Appendix

Table 18: Detailed ranking correlations. Parameter-count correlations retain the archived counting convention.

Benchmark	Pairs	NSC	ResNSC	NSC	ResNSC	$\Delta$ (pp)
		PW accuracy (%)		$\tau_{\mathrm{PW}}$
FlexiBERT	26,769	75.15	74.98	0.5031	0.4997	-0.17
GPT-2	1,721	77.28	78.27	0.5456	0.5654	+0.99
AutoFormer-Tiny	372,469	75.32	72.51	0.5063	0.4503	-2.80
AutoFormer-Small	130,265	67.42	66.16	0.3483	0.3232	-1.25

Appendix

Table 19: Parameter-controlled pairwise ranking (

\uparrow

). Only pairs with less than 10% relative parameter difference are compared. Accuracy is reported in percent;

\Delta

is ResNSC minus NSC accuracy in percentage points. Bold marks the higher value within each metric and row.

Neural Architecture Performance Prediction

See all 13 papers

Sep 19, 2026

Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT,

τ= 0.505

on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.

Chenyu Zhu, Ruoyu Zhao, Zhichao Lu

Department of Computer Science, City University of Hong Kong

Neural Architecture Performance Prediction Neural Architecture Search Neural Network Compression Spectral Methods

Ask PDF

Jun 16, 2026

Variable-Width Transformers

Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers, allocating a fixed parameter and computation budget evenly despite different layers potentially playing distinct computational roles. In this work, we empirically investigate nonuniform capacity allocation across network depth by proposing a

\times

-shaped > <former architecture. This design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism. Across decoder-only language models ranging from 200M to 2B parameters (dense) and 3B parameters (MoE), our > <former consistently outperforms parameter-matched uniform baselines on language modeling loss. By reducing the average layer width, this architecture also requires fewer overall FLOPs (22% reduction under fitted loss-matched scaling curves) and smaller KV cache memory and I/O cost (15% reduction). In analysis, we show that this bottleneck structure results in qualitatively different representations in residual streams. Overall, our results demonstrate that nonuniform width allocation can result in more resource-optimal scaling of language models.

Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3

ãMIT · èMIT-IBM Watson AI Lab

Transformer Decoder-Only Language Models Language Model Scaling

Ask PDF

May 20, 2026

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling (

β

=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling (

β

=1.02) in the same regimes, a

2.3\times

increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.

Nandan Kumar Jha, Brandon Reagen

New York University

Deep Learning Optimization Language Model Scaling Laws Scaling Laws

Ask PDF

Abstract

Neural Architecture Performance Prediction

Explore similar work

Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Variable-Width Transformers

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws