Anatomical Heterogeneity in Transformer Language Models

Tomasz Wietrzykowski

cs.LGMar 19, 2026

Anatomical Heterogeneity in Transformer Language Models

Authors: Tomasz Wietrzykowski

Organizations: Independent Researcher, Wroclaw, Poland

Abstract

Current transformer language models are trained with uniform computational budgets across all layers, implicitly assuming layer homogeneity. We challenge this assumption through empirical analysis of SmolLM2-135M, a 30-layer, 135M-parameter causal language model, using five diagnostic metrics: weight predictability (R2), ablation degradation, recovery speed, weight manipulation robustness, and structural analysis. We find profound anatomical heterogeneity: (1) Layer weights follow strong mathematical regularity (R2 = 0.91) with a universal oscillatory delta pattern (correlation ~= -0.50), yet predicted weights cause catastrophic failure due to nonlinear error accumulation. (2) Layer importance spans a 10^7 range, from a critical core (L8-11, up to +63,419% PPL degradation) to anti-layers (L14, L17) whose removal improves performance. (3) Recovery speed correlates with layer importance, indicating differential training requirements. (4) Only weight scaling (alpha = 0.9) preserves model quality among five tested manipulation strategies. (5) Growth Transformer Training, allocating budget by layer importance, achieves ~54% cost reduction. A proof-of-concept experiment confirms this: 4.7x lower validation loss than uniform training at identical parameter count, while being 13% faster.

Figures & tables

Layer	Degradation (%)	Category	Functional Role
L0	$0.0$	Redundant	Embedding boundary
L1	$+2{,}737.1$	Critical	Input parser
L2	$+186.0$	Critical	Input parser
L3	$+13.4$	Redundant	Padding
L4	$+22.7$	Minor	Feature extraction
L5	$+8.3$	Redundant	Padding

Table 1: Complete layer importance profile for SmolLM2-135M. (*) Anti-layer: model achieves equal or better perplexity after ablation.

Component	Avg $R^{2}$	Best $R^{2}$	Interpretation
mlp.gate_proj	0.909	0.993	Highly predictable
mlp.down_proj	0.895	0.995	Highly predictable
self_attn.q_proj	0.824	0.989	Highly predictable
self_attn.k_proj	0.745	0.978	Predictable
mlp.up_proj	0.716	0.985	Predictable
self_attn.o_proj	0.079	0.941	Weakly predictable

Table 2: Weight predictability per component.

Component	Avg delta correlation	Pattern
mlp.down_proj	$-0.500$	Oscillatory
mlp.gate_proj	$-0.500$	Oscillatory
mlp.up_proj	$-0.500$	Oscillatory
self_attn.k_proj	$-0.497$	Oscillatory
self_attn.o_proj	$-0.505$	Oscillatory
self_attn.q_proj	$-0.499$	Oscillatory

Table 3: Inter-layer delta correlation across all components.

Strategy	PPL	Degradation
Baseline (original)	22.60	$0\%$
Scale $\times 0.9$	26.95	$+19\%$
Scale $\times 0.7$	928.22	$+5{,}035\%$
Blend (1/distance, 2 layers)	31,333	$+173{,}242\%$
Scale $\times 0.5$	86,505	$+478{,}462\%$
Skip/Zero (2 layers)	969,198	$+5{,}361{,}714\%$

Table 4: Weight manipulation results on 9 redundant layers.

Layer	Category	PPL+noise	$<\!1.5\times$	$<\!1.1\times$	Final PPL
L14	Redundant	19.4	0	0	18.2 $\downarrow$
L17	Redundant	18.5	0	0	17.6 $\downarrow$
L5	Redundant	21.2	0	10	19.6
L23	Important	26.5	0	150	19.7
L24	Critical	26.8	0	110	19.6
L3	Redundant	27.3	10	200	27.3 (no conv.)

Table 5: Recovery speed after 50% Gaussian noise injection. Values indicate steps to reach PPL thresholds; “

\downarrow

” denotes improvement below baseline.

Layer	Role	FFN mult.	Params
L0	redundant	$\times 1$	459,264
L1	critical	$\times 4$	1,049,088
L2	critical	$\times 4$	1,049,088
L3	redundant	$\times 1$	459,264
L4	critical	$\times 4$	1,049,088
L5	critical	$\times 4$	1,049,088

Table 6: Growth Transformer architecture. Anti-layers (L14/L17 analogues) are omitted.

Phase	Name	Layers trained	Epochs
1	Gastrulation	Core (L4, L5)	30
2	Neurulation	Parser (L1, L2); clone L4 $\to$ L1, L5 $\to$ L2	20
3	Organogenesis	Output (L8, L9); clone L4 $\to$ L8, L5 $\to$ L9	20
4	Growth	Minor (L7, L10); clone L5 $\to$ L7, L9 $\to$ L10	12
5	Connective	Redundant (L0,L3,L6,L11); clone + scale FFN $\times 0.5$	6
6	Maturation	All layers (fine-tune)	15

Table 7: Six-phase developmental training protocol.

Configuration	Steps	Val loss	Time	vs. Uniform 100%
Uniform 100%	656	0.599	59.6s	baseline
Growth 50%	416	0.279	–	$2.1\times$ better, $37\%$ fewer steps
Growth 100%	656	0.127	52.0s	$4.7\times$ better, $13\%$ faster

Table 8: Growth Training vs. uniform baseline across budget conditions. All configurations use identical architecture (9.57M parameters) and dataset.

Prompt	Growth 100%	Uniform 100%
the capital of france is	paris	paris. on.
cats are	popular pets around the world	popular pets around the world
python is a	popular programming language	popular programming language
the earth orbits	around the sun every year	around the sun every year
water boils at	one hundred degrees celsius	one hundred degrees celsius
artificial intelligence	is transforming the way we live	can learn from data

Table 9: Generation quality on 12 prompts. Growth 100% maintains coherence throughout; Uniform produces minor artifacts (“on.”, “with neurons”) on edge cases; Growth 50% matches Uniform 100% qualitatively.

Layer Group	Layers	Budget ratio $R(l)$	Justification
Anti-layers	L14, L17	0.00	Prune/randomise; harm performance
Instant-recovery redund.	L3, L5, L6, L12, L19	0.00–0.05	0–10 steps to converge
Minor layers	L4, L7, L13, L15–16, L18, L20–22, L25–26	0.30–0.50	Incremental refinement
Critical output	L23–24, L27–28	0.80–1.00	Critical; fast recovery
Critical core + input	L1–2, L8–11	1.00	Full budget; slow or impossible recovery
Boundary layers	L0, L29	0.15–0.20	Anomalous recovery patterns

Table 10: Proposed Growth Transformer Training budget allocation based on Recovery Speed data. Estimated total training steps: 2,760 vs. 6,000 uniform (

\sim\!54\%

reduction).

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 528 papers

Tapered Language Models
Jun 22, 2026Reza Bayat, Ali Behrouz, Aaron CourvilleAutoregressive Language Modeling
1Mila · 2Cornell University · 3Université de Montréal +1
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Mar 22, 2026Abhinaba Basu, Kumkum Basu, Koushik DebTransformer Pruning LLM Compression
Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India +2
Hierarchical vs. Flat Iteration in Shared-Weight Transformers
Apr 15, 2026Sang-Il HanTransformer Recurrent Transformers
Korea University of Technology and Education · Biomonster Co.

Jun 22, 2026J/K move · Enter open · S save

Tapered Language Models

Reza Bayat, Ali Behrouz, Aaron Courville

1Mila · 2Cornell University · 3Université de Montréal+1

Autoregressive Language Modeling

Open paper AskPDF arXiv

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Abstract

Transformer

Explore similar work

Tapered Language Models

Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal

Hierarchical vs. Flat Iteration in Shared-Weight Transformers