Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal

Abhinaba Basu; Kumkum Basu; Koushik Deb

cs.LGMar 22, 2026

Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal

Authors: Abhinaba Basu, Kumkum Basu, Koushik Deb

Organizations: Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India · Indian Institute of Information Technology Kalyani, India · Accenture, India

Abstract

Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on six transformers (117M to 8B parameters) yields three findings. (i) Errors at layer t scale downstream by the product of later rho values, predicting representation drift (Spearman r = -0.44, p < 10^-4). This explains why compressing early layers hurts more than late ones, and why depth-decreasing sparsity schedules outperform uniform ones. Across architecture families, however, model width and redundancy matter more than rho alone. (ii) Within a layer, naive pruning shows a ~600x spread in component sensitivity. Activation-aware pruning (Wanda) shrinks this to 3-7x; the ranking reverses across architectures, so fixed importance scores do not transfer. (iii) For depth pruning, ranking layers by how far rho is from one takes two forward passes. It beats ShortGPT's Block Influence with 1.6x lower perplexity at eight layers removed, and physical deletion delivers 1.22x wall-clock speed-up. A blend of the two criteria does best (perplexity 14.2, 60.0% downstream accuracy on LLaMA-2-7B). Twelve Lean 4 norm inequalities provide machine-checked per-matrix error bounds. The contraction profile thus gives a training-free instrument for two decisions: where to compress within layers, and which to remove.

Figures & tables

Figure 1: Overview. (a) Three composable approximation stages with per-matrix bounds. (b) Relative error energy

V(t)

trajectories across six architectures: GPT-2 Small contracts monotonically (

\rho_{\max}=0.96

), while GPT-2 Medium’s final layer amplifies (

\rho_{23}=2.05

). LFM2-2.6B (hybrid) and Mistral-7B show moderate amplification; Qwen3-8B oscillates most. (c) Catastrophe threshold

\tau

increases with model dimension but is modulated by architecture (Table 2 ).

Lyapunov stability
	GPT-2 S	GPT-2 M	Qwen3.5	LFM2	Mistral	Qwen3
Parameters	117M	345M	1.9B	2.6B	7.2B	8B
Layers	12	24	24	30	32	36
$d_{\text{model}}$	768	1024	2048	2560	4096	4096
Architecture	Transf.	Transf.	Transf.	Hybrid	Transf.	Transf.
Baseline PPL	26.53	19.90	13.54	34.15	6.31	10.38

Table 1: Cross-model validation summary. All 50% Wanda results use a unified evaluation pipeline (weight

\times

column-norm scoring, 4,096 tokens, WikiText-103 test). Lyapunov measurements use

\epsilon=0.01

perturbation on 512 tokens. “Contracting” reports the number of inter-layer transitions (out of

L{-}1

) with

\rho<1

. “—” indicates per-matrix bounds were not measured for that model.

Figure 2: Structural sensitivity analysis. (a) Per-group compression regret on log scale ( $12\times 6$ heatmap). Each cell shows the perplexity increase when that group alone is compressed (sparsity=5%, rank=32). Layer 0 mlp_fc dominates at 459,904. (b) Component-type sensitivity hierarchy: MLP components (red) are consistently more sensitive than attention (blue), spanning a $46\times$ range.

Figure 3: The early-layer MLP catastrophe. (a) Layer 0 component decomposition across five compression levels: mlp_fc dominates at every level, reaching 460K regret at aggressive compression. V projections remain nearly free even at extreme settings. (b) mlp_fc regret by layer reveals a sharp phase transition between layers 2 and 3—a

{\sim}250\times

cliff separating catastrophic from negligible sensitivity.

Figure 4: Cumulative compression: forward (L0

\to

L11) vs. backward (L11

\to

L0). The

25{,}716\times

gap at one layer compressed demonstrates that layer 0 dominates overall sensitivity.

Figure 5: Contraction decomposition. (a) GPT-2 Small: blue bars show hidden state growth (

\alpha^{2}

), red bars show error growth (

\rho\cdot\alpha^{2}

), green diamonds show contraction factor

\rho

. Layer 0’s massive expansion (

\alpha^{2}=123

) produces the strongest contraction (

\rho=0.25

). All layers contract. (b) Mistral-7B across 31 layers. Red-shaded regions highlight amplifying layers (

\rho>1

Model	$d$	$\tau$	$1{-}\tau$	Note
GPT-2 Small	768	0.50	0.50	Catastrophic at 70%
GPT-2 Medium	1024	0.50	0.50	Same as Small
Qwen3.5-2B	2048	0.60	0.40	Full per-matrix study
LFM2-2.6B	2560	0.80	0.20	Hybrid architecture
Qwen3-8B	4096	0.80	0.20	Same $\tau$ as LFM2
Mistral-7B	4096	0.95	0.05	Most resilient

Table 2: Catastrophe threshold

\tau

across six architectures (layer-0 Wanda, regret

<100

Figure 6: Per-layer contraction factor

\rho_{t}

across six architectures on a normalized depth axis. GPT-2 Small (blue) contracts everywhere (

\rho<1

). LFM2 (green), Mistral (orange), and Qwen3 (red) all cross the stability boundary, with GPT-2 Medium (purple) spiking to

\rho=2.05

at the final layer.

Figure 7: Greedy compression waterfall on GPT-2 Small. V/attention outputs (green, purple) compress first at near-zero cost; early-layer MLPs (orange) trigger the exit from the safe zone.

Model	Method	PPL	Avg acc.	Speed
LLaMA-2-7B	$\|\rho{-}1\|$	8.6	65.1%	$1.22\times$
	BI	8.8	68.2%	$1.18\times$
	Blend	8.6	65.1%	—
	From-end	9.7	—	—
	Random	—	—	—
Mistral-7B	$\|\rho{-}1\|$	9.0	70.5%	$1.06\times$

Table 3: Cross-model layer removal at

k{=}4

(12.5% of layers). Best PPL in bold. The winner is architecture-dependent; the blend is competitive everywhere. Random removal has massive variance (Mistral: PPL 8.7–134.7 across 3 seeds).

$\lambda$	Layers removed	PPL	Avg acc.
4 layers removed (12.5%)
0 (BI only)	[24, 26, 27, 28]	8.77	68.2%
0.25	[24, 26, 27, 30]	9.74	67.8%
0.5–1.0 ( $\|\rho{-}1\|$ )	[10, 12, 13, 30]	8.63	65.1%
8 layers removed (25%)
0 (BI only)	[21, 23–29]	25.3	59.9%

Table 4: Combined layer-removal criterion on LLaMA-2-7B (physical deletion, single consistent run). At 8 layers, the

\lambda{=}0.5

blend dominates both pure strategies on PPL and downstream accuracy.

Method	tok/s	Speedup	PPL	Avg acc.
Dense (32 layers)	24.2	$1.00\times$	5.47	72.3%
$\|\rho{-}1\|$ -4 (28 layers)	29.5	$\mathbf{1.22\times}$	7.92	65.1%
BI-4 / ShortGPT (28L)	28.6	$1.18\times$	8.45	68.2%
ATP 50% (32 layers)	23.2	$0.96\times$	6.94	69.0%
$\|\rho{-}1\|$ -4 + ATP 50%	18.8	$0.78\times$	11.1	62.4%
BI-4 + ATP 50%	19.0	$0.79\times$	20.0	61.6%

Table 5: Complete compression ablation on LLaMA-2-7B with measured inference speed . Physical layer deletion (not hooks) gives real speedup; ATP weight pruning gives no speedup without sparse hardware. Downstream accuracy averaged over HellaSwag, ARC-Easy, WinoGrande, PIQA (300 examples each; indicative, not definitive). PPL values may differ by

{\sim}0.1

from Table 7 due to separate experimental runs.

Published references (iterative methods, WikiText-2):
Layers	%	$\|\rho{-}1\|$ (ours)	BI [ 20 ]	Optimal	From-end
2	6%	6.13	6.30	6.02	6.99
4	12%	7.92	8.45	7.01	9.96
6	19%	10.67	15.47	8.75	15.47
8	25%	14.62	23.75	11.61	25.41
SLEB [ 15 ]		${\sim}6$ layers (20%): PPL 9.14 (177 fwd passes)

Table 6: Progressive layer removal on LLaMA-2-7B (baseline PPL 5.47, hook-based skipping).

|\rho{-}1|

achieves lower PPL than BI [ 20 ] at every level. “Optimal” is exhaustive search over all

\binom{31}{k}

subsets. Published baselines (SLEB, BlockPruner) use iterative search (

O(N^{2})

evals); ours uses 2 forward passes.

Model	Method	Spar.	PPL	Deg.
Qwen3.5-2B	Baseline	0%	13.5	$1\times$
	Wanda uniform	50%	26.0	$1.92\times$
	OWL [ 34 ]	50%	31.6	$2.34\times$
	Ours (rule-based)	48%	23.7	$1.75\times$
Mistral-7B	Baseline	0%	6.15	$1\times$
	Wanda uniform	50%	8.4	$1.37\times$

Table 7: Compression comparison on WikiText-2. Top: per-row Wanda across three architectures (simple calibration). Bottom: LLaMA-2-7B matching ATP’s pipeline (baseline 5.47); within-layer redistribution (

\alpha>0

) consistently hurts at matched per-layer sparsity.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#	Name	Statement	Technique
1	retrieval_bound	$\\|f(x)-f(x_{c})\\|\leq\\|f\\|\cdot\delta$	calc + map_sub
2	sparse_activation_bound	$\\|f(x)-f_{s}(x)\\|\leq\\|f_{r}\\|\cdot\\|x\\|$	decomposition + le_opNorm
3	interpolation_bound	$\\|f_{s}(x)-f_{a}(x)\\|\leq\\|f_{R}\\|\cdot\\|x\\|$	same as Thm. 2
4	composition_bound	3-term bound (Thm. 2 + cache + rank)	calc + norm_add_le
5	efficiency_dominance	$c_{\text{live}}<c_{\text{full}}$	nlinarith
6	adaptive_monotonicity	$\\|f\\|\cdot\delta_{1}\leq\\|f\\|\cdot\delta_{2}$	monotonicity

Appendix

Table 8: Lean 4 theorem inventory. All proofs type-check with no sorry .

Explore similar work

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression
Apr 4, 2026Archit ThoratNeural Network Compression LLM Compression
Independent Researcher
No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
May 15, 2026Gabriel GarciaTransformer Neural Network Pruning
Independent Researcher
Predicting LLM Compression Degradation from Spectral Statistics
Apr 20, 2026Mingxue XuLLM Compression Low-Rank Matrix Decomposition
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom

Apr 4, 2026J/K move · Enter open · S save

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

Archit Thorat

Independent Researcher

Neural Network Compression LLM Compression

Open paper AskPDF arXiv

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.

Abstract

Explore similar work

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

No Free Swap: Protocol-Dependent Layer Redundancy in Transformers

Predicting LLM Compression Degradation from Spectral Statistics