Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

Muhammad Usama; Yeoreum Jung

cs.LGOct 10, 2026

Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

Authors: Muhammad Usama, Yeoreum Jung

Organizations: Braindeck Inc Seoul, Republic of Korea

Abstract

Existing depth-compression methods for large language models (LLMs) score layers individually or group only adjacent or contiguous layers, implicitly assuming that functional redundancy is local. We test this assumption by computing full pairwise Centered Kernel Alignment (CKA) similarity matrices for thirteen LLMs from six architecture families (1.6B-14B parameters). Non-adjacent layers have 3.6-15.9x more high-similarity pairs than adjacent ones, a structure that prior similarity-based layer merging has not systematically exploited. We propose Fusion via Community Mining (FCM), which builds the full L x L similarity graph, uses spectral community detection to find clusters of functionally equivalent layers, keeps the highest-Block-Influence layer in each cluster, and applies LoRA recovery; a pre-specified rule computed only on calibration data chooses between cosine and CKA similarity. At 2x compression, FCM lowers mean perplexity below ShortGPT on twelve of thirteen models, significantly on ten (Welch's t-test with Holm-Bonferroni correction; up to 50% on Qwen3-4B), with Mistral-7B the only loss (+13.4%). Against FlattenGPT, SWM, and DeltaLLM under matched calibration and recovery, FCM gives the best perplexity on ten of thirteen models (mean rank 1.15), and it also outperforms MKA on three anchor models. FCM matches or exceeds ShortGPT on 15 of 20 zero-shot reasoning cells and also improves MMLU, LAMBADA, HumanEval pass@1, and free-form generation quality. The 2x-compressed Qwen2.5-7B has 46% fewer parameters, 42% lower peak VRAM, and 84% higher decode throughput on a single A100. Spectral grouping outperforms Fisher contiguous segmentation by 41% and uniform partitioning by 51%, and FCM's advantage grows beyond a 2x crossover up to 3x and persists at the 14B scale. Code: https://github.com/Usama1002/fcm-layer-fusion

Figures & tables

Method	Similarity signal	Grouping	Non-adjacent?
ShortGPT ( Men et al., 2025 )	Per-layer BI	None (individual)	N/A
Gromov et al. ( Gromov et al., 2025 )	Per-layer BI	Contiguous block	No
LLM-Streamline ( Chen et al., 2025 )	Per-layer importance	Contiguous block	No
FlattenGPT ( Xu et al., 2026 )	None (adjacent pairs)	Adjacent pairs	No
SWM ( Ding et al., 2026 )	RKHS correlation	Sliding window	No
GRASP ( Liu et al., 2025 )	Gradient attribution	None (individual)	N/A

Table 1: Positioning of FCM relative to existing depth-compression methods. FCM is the only method that casts grouping as community detection on the full pairwise similarity graph; MKA also permits non-adjacent merging but groups greedily by pairwise similarity and merges weights rather than retaining a phase-covering representative.

Figure 1: Overview of the FCM pipeline. Given a pretrained LLM with $L$ layers, FCM (1) profiles pairwise activation similarity to construct a full $L\times L$ matrix $\mathbf{S}$ , revealing block-diagonal structure with non-adjacent high-similarity regions; (2) applies spectral clustering to partition layers into communities that may span non-adjacent positions; and (3) selects the highest Block Influence layer per community, discarding the rest. The compressed model retains $K$ layers (one per community) and undergoes LoRA recovery fine-tuning.

Model	Family	Base	ShortGPT	FCM (ours)	$\Delta$
FCM significant wins (10/13):
Qwen3-4B †	Qwen3	13.17	116.20 $\pm$ 8.12	58.15 $\pm$ 9.45 ∗∗	$-$ 50.0%
Qwen2.5-7B †	Qwen2	6.67	44.92 $\pm$ 2.85	26.65 $\pm$ 0.94 ∗∗	$-$ 40.7%
Qwen3-1.7B †	Qwen3	11.82	74.20 $\pm$ 3.10	46.45 $\pm$ 1.80 ∗	$-$ 37.4%
StableLM-2-1.6B	StableLM	12.45	52.60 $\pm$ 2.40	33.24 $\pm$ 1.10 ∗	$-$ 36.8%
Qwen3-8B †	Qwen3	6.12	39.80 $\pm$ 1.90	28.45 $\pm$ 1.10 ∗	$-$ 28.5%

Table 3: Multi-model comparison at 2

\times

compression (mean

\pm

std, LoRA r=16, 500 steps). Entries marked † are validated over 5 seeds (42, 123, 7, 2024, 999); the rest use 3 seeds (42, 123, 7). Base is the uncompressed WikiText-2 perplexity; within each row the uncompressed, ShortGPT, and FCM perplexities are computed under a single unified evaluation protocol (same tokenizer, sequence length, and stride).

\Delta

: relative change in mean PPL vs. ShortGPT. Statistical significance under a two-sided Welch’s

t

-test with Holm–Bonferroni correction across the 12 models with mean reduction: ∗∗

p<0.001

, ∗

p<0.05

. Per-row

p_{\text{corr}}

and Hedges’

g

effect sizes are reported in Appendix G (Table 11 ).

Model	ShortGPT	FlattenGPT	SWM	DeltaLLM	FCM (ours)	$\Delta_{\text{best}}$
StableLM-2-1.6B	52.60 $\pm$ 2.40	48.90 $\pm$ 2.10	51.20 $\pm$ 2.80	44.10 $\pm$ 1.80	33.24 $\pm$ 1.10 ∗	$-$ 24.6%
Qwen3-1.7B	74.20 $\pm$ 3.10	64.50 $\pm$ 2.90	71.10 $\pm$ 3.80	58.60 $\pm$ 2.40	46.45 $\pm$ 1.80 ∗	$-$ 20.7%
Gemma-2-2B	68.40 $\pm$ 2.90	63.10 $\pm$ 2.40	66.80 $\pm$ 3.10	61.20 $\pm$ 2.10	54.58 $\pm$ 1.65 ∗	$-$ 10.8%
Qwen2.5-3B	48.50 $\pm$ 2.10	44.20 $\pm$ 1.90	46.80 $\pm$ 2.30	42.10 $\pm$ 1.65	35.40 $\pm$ 1.20 ∗	$-$ 15.9%
LLaMA-3.2-3B	55.36 $\pm$ 2.12	58.20 $\pm$ 2.30	61.10 $\pm$ 3.40	54.80 $\pm$ 1.95	52.60 $\pm$ 1.85 †	$-$ 4.0%
SmolLM3-3B	41.80 $\pm$ 1.65	43.20 $\pm$ 1.80	46.10 $\pm$ 2.10	40.50 $\pm$ 1.40	39.85 $\pm$ 1.30 †	$-$ 1.6%

Table 4: Head-to-head WikiText-2 perplexity (mean

\pm

std over 3 seeds) at 2

\times

compression across all thirteen models against four baselines under matched calibration and 500-step LoRA recovery (r=16). Lower is better; bold marks the best per row.

\Delta_{\text{best}}

: relative change of FCM vs. the strongest baseline in that row. ∗ : FCM significantly beats every baseline in the row (Welch’s

t

-test, Holm–Bonferroni,

p_{\text{corr}}<0.05

); † : statistical tie. Per-row

p_{\text{corr}}

and Hedges’

g

in Appendix G . Mistral rows compare mean-pooled and full-token CKA profiling (§ 5.3 ).

Model	Ratio	ShortGPT	FlattenGPT	DeltaLLM	FCM (ours)	SGPT ZS	FCM ZS
Qwen2.5-7B	$1.5\times$	12.28 $\pm$ 0.35	12.95 $\pm$ 0.40	11.85 $\pm$ 0.28	12.45 $\pm$ 0.32	58.4	58.1
	$2.0\times$	44.92 $\pm$ 2.85	38.15 $\pm$ 1.42	35.60 $\pm$ 1.05	26.65 $\pm$ 0.94	49.3	55.5
	$2.5\times$	148.50 $\pm$ 9.20	112.40 $\pm$ 6.80	98.60 $\pm$ 5.10	64.20 $\pm$ 3.85	28.5	38.2
	$3.0\times$	412.80 $\pm$ 28.5	315.60 $\pm$ 21.0	268.40 $\pm$ 18.2	142.50 $\pm$ 11.4	21.2	29.8
LLaMA-3.1-8B	$1.5\times$	11.45 $\pm$ 0.28	12.10 $\pm$ 0.35	11.60 $\pm$ 0.30	11.82 $\pm$ 0.31	62.1	61.8
	$2.0\times$	42.75 $\pm$ 1.70	40.12 $\pm$ 0.95	38.25 $\pm$ 0.82	35.25 $\pm$ 1.15	52.2	54.9

Table 5: Compression-ratio sweep on three anchors: WikiText-2 perplexity (mean

\pm

std, 3 seeds) and five-benchmark zero-shot accuracy (%). DeltaLLM is the strongest adjacent-pair baseline (Table 4 ). Bold marks the best perplexity per row.

Model	Method	ARC-E	ARC-C	HellaSwag	WinoGrande	PIQA	Avg
Qwen2.5-7B	Uncompressed	79.8	54.2	78.1	72.4	80.2	72.94
	ShortGPT	54.2	32.1	46.8	52.4	61.2	49.34
	FCM (ours)	61.8	38.4	54.2	56.8	66.5	55.54
LLaMA-3.1-8B	Uncompressed	81.2	56.4	79.4	74.1	81.8	74.58
	ShortGPT	56.4	35.2	50.1	55.8	63.4	52.18
	FCM (ours)	59.2	37.8	53.6	58.2	65.8	54.92

Table 6: Five-benchmark zero-shot reasoning accuracy (%) at

2\times

compression (3-seed mean for compressed configurations; single-run uncompressed reference). Bold marks the higher compressed average per model.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Profiling metric	PPL	$\Delta$ vs. ShortGPT
ShortGPT	Block Influence	28.36 $\pm$ 0.54	–
FCM (mean-pooled)	Cosine of mean activations	32.16 $\pm$ 1.20	$+13.4\%$
FCM (full-token)	Full-token CKA	29.86 $\pm$ 1.87	$+5.3\%$

Appendix

Table 7: Full-token CKA validation on Mistral-7B at 2

\times

compression (mean

\pm

std over 3 seeds). Replacing mean-pooled cosine with full-token CKA, which retains the per-position structure that mean-pooling discards, narrows the perplexity gap against ShortGPT from

+13.4\%

+5.3\%

Model	Family	Layers	Scale
Qwen2.5-14B, Qwen2.5-7B, Qwen2.5-3B ( Qwen et al., 2024 )	Qwen2	48, 28, 36	14B, 7B, 3B
Qwen3-14B, Qwen3-8B, Qwen3-4B, Qwen3-1.7B ( Yang et al., 2025 )	Qwen3	40, 36, 36, 28	14B, 8B, 4B, 1.7B
Mistral-7B-v0.3 ( Jiang et al., 2023 )	Mistral	32	7B
LLaMA-3.1-8B ( Grattafiori et al., 2024 ) , LLaMA-3.2-3B ( Meta AI, 2024 )	LLaMA	32, 28	8B, 3B
SmolLM3-3B ( Bakouch et al., 2025 )	SmolLM	36	3B
Gemma-2-2B ( Gemma Team et al., 2024 )	Gemma	26	2B

Appendix

Table 8: Models evaluated in this study, spanning six architecture families and four parameter scales (1–2B, 3–4B, 7–8B, 14B).

Figure 3: Full pairwise CKA similarity matrices for representative models. Block-diagonal structure with high off-diagonal similarity between early and late layers is consistently observed across architectures and scales.

Method	Sim. Metric	Grouping / Merge	PPL (raw)	PPL (+LoRA)
Baseline (no compression)	–	–	6.67	–
FCM grouping/merge variants (centrality-based selection):
FCM	Cosine	Spectral / Select	288	35.01
FCM	CKA	Spectral / Select	1,481	48.92
FCM	Cosine	Spectral / Average	2,784	57.23
FCM	CKA	Fisher / Select	2,710	82.93

Appendix

Table 9: Ablation on Qwen2.5-7B at 2

\times

compression (single seed = 42; the cosine

+

spectral configuration paired with BI-guided selection is reported with mean

\pm

std and significance markers in Table 3 ). Similarity metric, grouping algorithm, and merge strategy are varied independently; “Select” here uses centrality-based selection in isolation. PPL: WikiText-2 perplexity (lower is better). All methods share the same LoRA recovery budget (r=16, 500 steps).

Panel of Figure 4 — Figure 4: (a) Ablation study varying similarity metric, grouping, and merge strategy. (b) Perplexity vs. compression ratio showing the crossover point (log scale).

$N$	PPL (raw)	PPL (+LoRA)
64	282.11	39.65
128	282.11	40.66
256	282.11	39.59
512	287.88	38.97
1024	282.11	39.54

Appendix

Table 10: Calibration size ablation on Qwen2.5-7B at 2

\times

compression. PPL is stable from

N=64

N=1024

Model	$n$	$\Delta$ (%)	$p_{\text{corr}}$	Hedges’ $g$	Verdict
Qwen3-4B	5	$-50.0$	0.0004	5.92	∗∗
Qwen2.5-7B	5	$-40.7$	0.0008	4.85	∗∗
Qwen3-1.7B	5	$-37.4$	0.0012	3.68	∗
StableLM-2-1.6B	3	$-36.8$	0.0015	3.55	∗
Qwen2.5-14B	3	$-25.5$	0.0084	2.61	∗
Qwen3-14B	3	$-23.7$	0.0092	2.54	∗

Appendix

Table 11: Phase A: per-row test statistics for the twelve models in Table 3 where FCM has lower mean perplexity than ShortGPT. Holm–Bonferroni correction across

m=12

hypotheses; verdict ∗∗ at

p_{\text{corr}}<0.001

, ∗ at

p_{\text{corr}}<0.05

, ns otherwise. Hedges’

g

values on small-

n

rows (e.g.,

n=3

on StableLM-2-1.6B) are inflated by the small denominator (low pooled std); they should be read alongside

n

rather than as direct comparisons across models.

Anchor	Baseline	$t$	df	$p_{\text{raw}}$	$p_{\text{corr}}$	Verdict
Qwen2.5-7B	ShortGPT	10.54	2.43	0.00436	0.03001	∗
Qwen2.5-7B	FlattenGPT	11.70	3.47	0.00066	0.00721	∗
Qwen2.5-7B	SWM	10.74	2.74	0.00257	0.02568	∗
Qwen2.5-7B	DeltaLLM	11.00	3.95	0.00041	0.00497	∗
LLaMA-3.1-8B	ShortGPT	6.33	3.51	0.00489	0.03001	∗
LLaMA-3.1-8B	FlattenGPT	5.65	3.86	0.00535	0.03001	∗

Appendix

Table 12: Phase B: per-cell test statistics for FCM vs. each baseline on the three anchor models in Table 4 . Holm–Bonferroni correction across

m=12

(anchor, baseline) cells; all twelve cells clear

p_{\text{corr}}<0.05

. Welch–Satterthwaite degrees of freedom shown;

n=3

per group.

Configuration	Parameters	Peak VRAM	Prefill	Decode
		(seq=2048)	(seq=512, tok/s)	(tok/s)
Uncompressed ( $L=28$ )	7.61 B	16.2 GiB	1,540	68
ShortGPT ( $L=14$ )	4.12 B	9.3 GiB	2,880	125
FCM (ours, $L=14$ )	4.12 B	9.3 GiB	2,880	125
$\Delta$ vs. uncompressed (FCM/ShortGPT)	$-46\%$	$-42\%$	$+87\%$	$+84\%$

Appendix

Table 13: Inference deployment metrics on Qwen2.5-7B at

2\times

compression (single A100 80GB, bf16). FCM and ShortGPT retain the same layer count and therefore have identical inference characteristics. Percentages in the last row are reduction (parameters, VRAM) or speed-up (prefill, decode) relative to the uncompressed baseline.

Model	Method	MMLU (5-shot)	LAMBADA Acc	LAMBADA PPL	HumanEval@1
Qwen2.5-7B	Uncompressed	71.4	74.2	3.42	48.2
	ShortGPT	44.2	48.5	18.42	18.5
	DeltaLLM	48.6	52.1	14.15	22.4
	FCM (ours)	54.8 ∗	59.4 ∗	8.65 ∗	29.8 ∗
LLaMA-3.1-8B	Uncompressed	68.5	72.8	3.65	34.1
	ShortGPT	46.1	50.2	16.85	14.2

Appendix

Table 14: Broadened knowledge, generative language modelling, and code benchmarks at

2\times

compression. MMLU is 5-shot accuracy (%); LAMBADA reports accuracy (%) and generative perplexity; HumanEval is pass@1 (%). Bold marks the best compressed result per column; ∗ : FCM significantly beats ShortGPT (

p<0.05

Method	Distinct-2	Distinct-3	Self-BLEU	MAUVE	Degen. failure
Uncompressed	0.684	0.852	0.185	0.982	0.0%
ShortGPT	0.412	0.564	0.485	0.615	24.0%
DeltaLLM	0.485	0.642	0.382	0.742	11.0%
FCM (ours)	0.612	0.785	0.245	0.894	2.0%

Appendix

Table 15: Free-form generation quality (256-token continuations, 100 Pile prompts, Qwen2.5-7B,

2\times

). Distinct-

n

and MAUVE: higher is better; Self-BLEU and degenerative-failure rate: lower is better.

Model	$L$	$K$	$B_{\cos}$	$m_{\cos}$	$B_{\text{CKA}}$	$m_{\text{CKA}}$	Cosine	Metric
StableLM-2-1.6B	24	12	0.892	0.208	0.865	0.250	Normal	Cosine
Qwen3-1.7B	28	14	0.584	0.536	0.881	0.286	Degen.	CKA
Gemma-2-2B	26	13	0.612	0.462	0.874	0.269	Degen.	CKA
Qwen2.5-3B	36	18	0.915	0.194	0.852	0.222	Normal	Cosine
LLaMA-3.2-3B	28	14	0.884	0.214	0.831	0.286	Normal	Cosine
SmolLM3-3B	36	18	0.542	0.583	0.849	0.250	Degen.	CKA

Appendix

Table 16: Per-model degeneracy profiling at

2\times

compression (

K=L/2

B

: normalized size-balance entropy;

m

: maximum community coverage. A cosine partition is degenerate if

B_{\cos}<0.70

m_{\cos}>0.40

. All choices match the paper’s usage.

Model	Stage	ShortGPT	FlattenGPT	SWM	DeltaLLM	FCM (ours)
Qwen2.5-7B	Raw	176,657	91,493	886	19,519	2,280
Qwen2.5-7B	+LoRA	44.92	38.15	41.20	35.60	26.65
LLaMA-3.1-8B	Raw	142,311	68,412	1,241	12,840	1,845
LLaMA-3.1-8B	+LoRA	42.75	40.12	44.55	38.25	35.25
Qwen2.5-14B	Raw	84,120	42,181	612	8,940	413
Qwen2.5-14B	+LoRA	32.45	29.80	31.40	27.95	24.18

Appendix

Table 17: Raw (pre-LoRA) and post-LoRA WikiText-2 perplexity at

2\times

compression. FCM’s raw perplexity is far lower than the adjacent-pair and greedy-deletion baselines, so it provides LoRA with a better starting point.

Model	MKA raw	MKA +LoRA	FCM raw	FCM +LoRA	$\Delta$ (PPL)
Qwen2.5-7B	14,210	34.80 $\pm$ 1.15	2,280	26.65 $\pm$ 0.94 ∗	$-8.15$
LLaMA-3.1-8B	11,840	39.40 $\pm$ 1.22	1,845	35.25 $\pm$ 1.15 ∗	$-4.15$
Qwen2.5-14B	6,411	28.50 $\pm$ 0.92	413	24.18 $\pm$ 0.82 ∗	$-4.32$

Appendix

Table 18: FCM versus MKA ( Liu et al., 2024 ) at

2\times

compression under matched calibration and 500-step LoRA recovery. Raw and post-recovery WikiText-2 perplexity (post-recovery mean

\pm

std over 3 seeds). ∗ : FCM significantly better (

p<0.05

Model	Calibration	ARI	Jaccard	Flips	PPL
Qwen2.5-7B	C4 (default)	1.000	1.000	0	26.65 $\pm$ 0.94
	WikiText-2	0.924	0.928	1	26.72 $\pm$ 0.98
	StarCoder (code)	0.885	0.857	2	26.89 $\pm$ 1.05
	arXiv (math)	0.892	0.857	2	26.81 $\pm$ 1.01
LLaMA-3.1-8B	C4 (default)	1.000	1.000	0	35.25 $\pm$ 1.15
	WikiText-2	0.941	0.938	1	35.31 $\pm$ 1.18

Appendix

Table 19: Cross-domain calibration invariance at

2\times

. Adjusted Rand Index (ARI) and retained-layer Jaccard against the C4 default; retained-layer flips and post-recovery WikiText-2 PPL.

Model	Length $n$	ARI	Jaccard	PPL
Qwen2.5-7B	64	0.942	0.928	26.78 $\pm$ 0.96
	128 (default)	1.000	1.000	26.65 $\pm$ 0.94
	256	0.968	0.928	26.68 $\pm$ 0.95
	512	0.968	0.928	26.64 $\pm$ 0.93
LLaMA-3.1-8B	64	0.928	0.938	35.42 $\pm$ 1.20
	128 (default)	1.000	1.000	35.25 $\pm$ 1.15

Appendix

Table 20: Calibration context-length sensitivity at

2\times

(

N=512

fixed). ARI and retained-layer Jaccard against the

n=128

default; post-recovery WikiText-2 PPL.

Community	Layers	Logit-lens	Attn. ent.	LOO $\Delta$ PPL	Phase	Rep.
$C_{1}$ (bookend)	0,1,2,25,26,27	0.02–0.78	1.42	$+312.4$	Embed/output	27
$C_{2}$	3,4,5,6	0.12–0.18	3.85	$+14.2$	Early processing	3
$C_{3}$	7–12	0.21–0.24	4.12	$+4.1$	Identity drift	9
$C_{4}$	13–17	0.26–0.31	4.08	$+3.8$	Identity drift	13
$C_{5}$	18,19,20,21	0.35–0.52	3.15	$+42.8$	Semantic formation	21
$C_{6}$	22,23,24	0.58–0.69	2.10	$+118.5$	Output preparation	24

Appendix

Table 21: Per-community functional probes on Qwen2.5-7B at

2\times

(

L=28

). Logit-lens accuracy is reported as the min-to-max range over community members; leave-one-out (LOO)

\Delta

PPL is the mean skip cost. The BI-selected representative is retained.

Model	Attention	Top-5% var.	Max outlier	Gini	Mean-pool / Full-token PPL
Qwen2.5-7B	Dense	14.2%	42.5	0.185	26.65 / 36.83
LLaMA-3.1-8B	Dense	15.8%	48.1	0.201	35.25 / 41.20
Mistral-7B	Sliding-window	68.4%	412.8	0.742	32.16 / 29.86

Appendix

Table 22: Token-position variance concentration across attention architectures. Top-5% variance fraction and Gini are computed over sequence positions on the calibration set; PPL columns give the

2\times

post-recovery outcome under each profiling metric.

Model	Stage	Wallclock (s)	Peak VRAM (GB)
Qwen3-1.7B	Profiling ( $N=512$ )	4.2	4.2
	Similarity (cos/CKA)	0.2	0.1
	Clustering + selection	$<0.01$	$<0.01$
	LoRA recovery (500 steps)	412.0	14.8
	Total (FCM overhead $+1.06\%$ )	416.4	14.8
Qwen2.5-7B	Profiling ( $N=512$ )	11.5	12.4

Appendix

Table 23: End-to-end offline runtime and peak memory on a single A100-80GB. Stages: (1) activation profiling, (2) similarity matrices, (3) spectral clustering + BI selection, (4) LoRA recovery. FCM’s added stages (1–3) are a small fraction of total time.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

Mar 18, 2026

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

Large language models (LLMs) contain billions of parameters, but many exact weight values are not essential. We apply scalar

K

-means clustering to pretrained models, replacing each weight matrix with only 16--64 shared values. For Llama~3.1-8B-Instruct and SmolLM2-135M, this preserves strong accuracy without retraining, providing a simple way to compress LLMs on disk. Fine-tuning only the cluster centroids further recovers 30--40% of the remaining accuracy loss. Using clustering as an analysis tool, we then keep the cluster assignments fixed and perturb only the shared values, to study which of their properties matter most. We find that preserving their relative rank (which connections are stronger or weaker than others) is more important than preserving their exact magnitudes. Breaking this ordering greatly increases perplexity, while changing the values but keeping the ordering causes little loss in many middle and late layers. However, rank preservation alone is not sufficient. Model performance also depends on maintaining the variance of the reconstructed weights, while early layers remain more sensitive; perturbing the cluster assignments themselves shows that their spatial geometry is also load-bearing. We observe the same behavior across four LLMs from two architecture families. These results suggest that the ordering of weights, together with their overall variance, plays a central role in LLM compression and robustness.

Borja Aizpurua, Sukhbinder Singh, Román Orús

Multiverse Computing, San Sebasti´an, Spain · Department of Basic Sciences, Tecnun - University of Navarra, San Sebasti´an, Spain · Multiverse Computing, Toronto, Ontario, Canada +2

LLM Compression LLM Quantization Clustering

Ask PDF

Jun 1, 2026

From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

Post-training compression of Large Language Models (LLMs) removes entire architectural components, either deleting them or replacing them with fitted modules. Existing replacement-based methods share two design constraints: full-layer granularity and contiguous selection. We argue that this is overly restrictive: in fact, redundancy in pretrained transformers is not confined to contiguous regions, nor does it evenly distribute between Attention and FeedForward outputs, implying that different strategies best approximate different submodule types and that removable components need not cluster within contiguous depth ranges. Based on this intuition, we introduce SubFit (Submodule-level Fitted residual replacement), which compresses LLMs at the submodule level: Attention and FeedForward submodules are selected non-contiguously, and each receives its own lightweight fitted residual bypass. SubFit operates post-training and requires only calibration data. Across ten LLMs (five base, five instruction-tuned), five sparsity levels from 12.5% to 37.5%, and four replacement-based baselines, SubFit achieves the best aggregate perplexity-accuracy trade-off across the evaluated sparsity levels, with larger gains under aggressive compression. At 25% sparsity, it retains 84.6% of dense downstream accuracy and incurs 2.42x perplexity degradation, against 81.6% and 4.34x for the strongest baselines, while delivering measurable inference speedup and KV-cache savings. Code is available at https://github.com/eliacunegatti/SubFit.

Elia Cunegatti, Marcus Vukojevic, Erik Nielsen +1

University of Trento

LLM Pruning LLM Compression Transformer LLM Inference Acceleration

Ask PDF

May 27, 2026

Locality-Aware Redundancy Pruning for LLM Depth Compression

Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/

Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3

University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1

LLM Pruning LLM Compression LLM Inference Acceleration

Ask PDF

Abstract

LLM Compression

Explore similar work

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

Locality-Aware Redundancy Pruning for LLM Depth Compression