Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

Borja Aizpurua; Sukhbinder Singh; Román Orús

cs.LGMar 18, 2026

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

Authors: Borja Aizpurua, Sukhbinder Singh, Román Orús

Organizations: Multiverse Computing, San Sebasti´an, Spain · Department of Basic Sciences, Tecnun - University of Navarra, San Sebasti´an, Spain · Multiverse Computing, Toronto, Ontario, Canada · Donostia International Physics Center, San Sebasti´an, Spain ·

Abstract

Large language models (LLMs) contain billions of parameters, but many exact weight values are not essential. We apply scalar $K$ -means clustering to pretrained models, replacing each weight matrix with only 16--64 shared values. For Llama~3.1-8B-Instruct and SmolLM2-135M, this preserves strong accuracy without retraining, providing a simple way to compress LLMs on disk. Fine-tuning only the cluster centroids further recovers 30--40% of the remaining accuracy loss. Using clustering as an analysis tool, we then keep the cluster assignments fixed and perturb only the shared values, to study which of their properties matter most. We find that preserving their relative rank (which connections are stronger or weaker than others) is more important than preserving their exact magnitudes. Breaking this ordering greatly increases perplexity, while changing the values but keeping the ordering causes little loss in many middle and late layers. However, rank preservation alone is not sufficient. Model performance also depends on maintaining the variance of the reconstructed weights, while early layers remain more sensitive; perturbing the cluster assignments themselves shows that their spatial geometry is also load-bearing. We observe the same behavior across four LLMs from two architecture families. These results suggest that the ordering of weights, together with their overall variance, plays a central role in LLM compression and robustness.

Figures & tables

Figure 1: (a) A neural network layer showing connections between input neurons (top) and output neurons (bottom). Edge colors indicate the magnitude of the weights. (b) A clustered version of the same weight matrix with

K=2

clusters. In this paper, we demonstrate that when the clustered matrix remains performant, model accuracy depends chiefly on the relative clustering of the weights (here, cold vs. hot) rather than on their exact values. The absolute scale of each cluster can be changed by an affine transformation that preserves the overall weight distribution without substantially degrading model performance.

Figure 2: Downstream task accuracy comparison for clustered versions of Llama 3.1-8B-Instruct and a compressed Llama 3B model. We see that weight clustering largely preserves accuracy across all tasks without requiring any fine-tuning. Note that the y-axis starts at 0.40, not 0.

	Llama 3.1-8B-Instruct				Compressed Llama 3B				SmolLM2-135M
Replacement mode	PPL	$L_{2}$ rel.	$\boldsymbol{\mu}$ $(\!\times\!10^{-4})$	$\boldsymbol{\sigma^{2}}$ $(\!\times\!10^{-4})$	PPL	$L_{2}$ rel.	$\boldsymbol{\mu}$ $(\!\times\!10^{-4})$	$\boldsymbol{\sigma^{2}}$ $(\!\times\!10^{-4})$	PPL	$L_{2}$ rel.	$\boldsymbol{\mu}$ $(\!\times\!10^{-4})$	$\boldsymbol{\sigma^{2}}$ $(\!\times\!10^{-4})$
Baseline (orig.)	9.08	—	$-$ 0.2	2.0	12.32	—	$-$ 0.3	2.5	27.52	—	$-$ 1.1	324.4
Ord-pres. (sorted)	9.17	0.52	$-$ 16.8	0.6	12.46	0.62	$-$ 6.3	4.5	28.35	0.52	$-$ 234.4	94.5
Ord-pres. (wt. freq.)	9.17	0.53	$-$ 3.0	0.8	12.48	0.54	$-$ 1.8	0.6	28.28	0.54	$-$ 48.6	109.8
Ord-pres., sign-scr.	9.13	0.51	$-$ 73.4	2.0	12.33	0.31	$+$ 16.8	2.5	27.53	0.09	$-$ 157.7	324.5
Sign-pres., ord-brk.	9.56	1.43	$-$ 25.3	6.7	13.37	1.70	$-$ 30.3	27.3	36.58	1.43	$-$ 289.9	1,069.0

Table 1: Single-layer centroid replacement (layer 10 gate_proj,

K{=}32

). We replace the centroid values in a single MLP projection layer while preserving the cluster assignments throughout the model. “Rank-preserving” replacements preserve the rank ordering of the centroid values. “Random” replacements substitute centroid values from a distribution with the same mean and variance as the original values, but do not preserve centroid ranks. “Rand. perm.” applies a random permutation to the centroid values, the most extreme form of rank-breaking. Rel.

L_{2}

change measures how much the reconstructed weight matrix changed.

\mu

and

\sigma^{2}

are the mean and variance of the reconstructed weights.

	Llama 3.1-8B-Instruct			Compressed Llama 3B			SmolLM2-135M †
Transform	PPL	$\boldsymbol{\mu}$ $(\times 10^{-4})$	$\boldsymbol{\sigma^{2}}$ $(\times 10^{-4})$	PPL	$\boldsymbol{\mu}$ $(\times 10^{-4})$	$\boldsymbol{\sigma^{2}}$ $(\times 10^{-4})$	PPL	$\boldsymbol{\mu}$ $(\times 10^{-2})$	$\boldsymbol{\sigma^{2}}$ $(\times 10^{-2})$
Identity	9.08	$-$ 0.2	2.0	12.32	$-$ 0.3	2.5	27.52	$-$ 1.1	3.2
Affine $w\mapsto 0.5w$	9.18	$-$ 0.1	0.5	12.46	$-$ 0.1	0.6	28.25	$-$ 0.5	0.8
Affine $w\mapsto 2.0w$	9.59	$-$ 0.3	8.2	13.10	$-$ 0.6	9.9	30.58	$-$ 2.2	13.0
$\tanh(\alpha w)$ , $\alpha=1$	9.08	$-$ 0.2	2.0	12.32	$-$ 0.3	2.5	27.55	$-$ 1.1	3.0
$\tanh(\alpha w)$ , $\alpha=2$	9.59	$-$ 0.3	8.1	13.09	$-$ 0.6	9.9	29.36	$-$ 2.3	10.3

Table 2: Monotone centroid transformations on layers.10.mlp.gate_proj (labels fixed). All transforms preserve centroid rank order.

\mu

and

\sigma^{2}

are the mean and variance of the reconstructed weight matrix

\hat{W}

after each transform. The mean stays near zero throughout (the centroid distribution is symmetric); variance is the informative predictor of degradation.

Figure 3: Why the weight distribution , not just rank, matters ( layers.10.mlp.gate_proj , Llama-3.1-8B-Instruct,

K{=}32

; log-density

y

-axis). Both panels apply the same monotone warp

c\mapsto\mathrm{sign}(c)|c|^{0.5}

, so centroid rank is identical; they differ only in the distribution. (a) Renormalized to the original mean and variance (Eqs. 2 – 3 ), the distribution matches baseline and PPL is unchanged (

9.08\!\to\!9.09

). (b) Without renormalization the variance inflates

55\times

and PPL collapses to

1{,}160

. With rank held fixed, it is the distribution—chiefly the variance—that separates the safe edit from the destructive one.

	kp sign brk ord	kp ord brk sign	gauss rand	gauss mono
Llama 3.1-8B-Instruct (PPL = 9.08)
l1 (early)	9.64	117.39	2,260	11.30
l10 (mid)	9.56	9.13	9.76	9.17
l30 (late)	13.18	9.08	47.30	10.00
l10 ( $q$ -proj)	10.17	9.08	9.83	9.48
Compressed Llama 3B (PPL = 12.32)

Table 3: Single-layer centroid replacement at varying depths. Each cell perturbs only the named layer; all other layers are unmodified. Rank-breaking conditions (kp sign/brk ord, gauss rand) are consistently more damaging than order-preserving ones (kp ord/brk sign, gauss mono) at every layer.

Predictor	SmolLM2 135M	Llama 3.1 8B-Inst.
Rank corr.	$-0.06$	$\mathbf{-0.56}$
$\|\Delta\sigma^{2}\|$ (var.)	$+0.17$	$\mathbf{+0.97}$
$\|\Delta\mu\|$ (mean)	$\mathbf{+0.62}$	$+0.47$
rel- $L_{2}$ (dist.)	$-0.01$	$-0.80$

Table 4: Standardized partial correlations of

\log(\mathrm{PPL}/\mathrm{PPL}_{0})

with each quantity over the controlled sweep on layers.10.mlp.gate_proj (

n{=}28

perturbations per model). A large-magnitude value means the quantity predicts degradation after the other three are held fixed. On the larger model both centroid rank (negative: more rank preservation

\Rightarrow

less damage) and variance drift (positive) are strong independent predictors, while distance from the original weights is not.

$L$ perturbation	SmolLM2	Llama-Inst.	Comp. 3B	Qwen2.5
Baseline	27.52	9.08	12.32	11.83
Within-row shuffle	43.83	9.85	13.43	14.56
Within-column shuffle	36.42	9.88	13.41	17.63
Window-64 shuffle	42.67	9.82	13.26	14.80
Global reshuffle (freq. kept)	43.02	9.82	13.37	14.69
Relabel rank-adjacent	27.63	9.08	12.32	11.83

Table 5: Cluster-assignment (

L

) perturbations on layers.10.mlp.gate_proj , with centroid values held fixed. Position shuffles preserve the value histogram exactly (hence

\mu,\sigma^{2}

, and all moments) and vary only spatial structure; rank-bounded relabels keep

L

fixed and swap labels among rank-adjacent vs. rank-distant clusters. PPL is WikiText-2 (internal harness).

Panel of Figure 4 — Figure 4: Normalized perplexity (PPL / baseline) as centroids are progressively replaced with rank-preserving-random values, one transformer block at a time from deepest to shallowest. Thick lines with markers show affine-corrected replacement (mean and variance of the reconstructed weights are preserved); thin lines of matching color show uncorrected replacement. Each curve normalizes by its own baseline (Llama 3.1-8B-Instruct: 9.08; compressed Llama 3B: 12.32; SmolLM2-135M: 27.52).

Method	Representation	Optimisation	Runtime path
GPTQ, AWQ	Uniform / low-bit	Hessian / activ.-aware	Dense GEMM
AQLM	Additive codebooks	MRF / beam search	Dequant+GEMM or fused
VPTQ, GANQ	Non-uniform / LUT	Vector / MIQP-style	Rebuild or fused
Ours	Scalar $K$ -means	$K$ -means (+ healing)	Rebuild-to-dense

Table 6: Representative compression approaches and their execution paths.

Model	Method	Disk (GB)	GPU (GB)	PPL
Llama 3.1-8B-Instruct: Baseline (FP16)	Full precision	15.0	14.96	8.87
FP8	Uniform quant.	9.1	8.46	9.04
AWQ INT4	Uniform quant.	5.7	5.3	9.35
GPTQ INT4	Uniform quant.	5.7	5.3	9.30
GGUF Q5_K_S	Uniform quant.	5.3	5.2	8.99
AQLM 2-bit	Codebook	4.1	3.80	11.77

Table 7: Accuracy and storage on Llama 3.1-8B-Instruct (WikiText-2 PPL, vLLM harness; see Appendix G ). The compressed + clustered model appears as PPL 12.32 in Section III and PPL 13.36 here due to different evaluation libraries (Appendix G ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Gen TPS
Baseline (FP16)	Full precision	2194
FP8	Uniform quant.	3316
AWQ INT4	Uniform quant.	4606
GPTQ INT4	Uniform quant.	1812
AQLM 2-bit	Codebook	1023
Clustering (Orig) — no training	$K$ -means (rebuild-to-dense)	$\approx$ 2194

Appendix

Table 8: Inference throughput on Llama 3.1-8B-Instruct (H200, internal harness: batch=1, prompt=50, generation=50, greedy decoding). Gen TPS = generated tokens/second. Not directly comparable to vLLM serving numbers in Appendix H .

	SmolLM2-135M		Llama 3.1-8B-Inst.
Edit / mode	next	final	next	final
Scale $a{=}0.5$	0.992	0.988	0.982	0.992
Scale $a{=}2.0$	0.969	0.982	0.953	0.950
Offset $b{=}{+}1\sigma$	0.985	0.992	0.957	0.975
Offset $b{=}{+}5\sigma$	0.879	0.861	0.810	0.871
Rank-pres. (gauss. monotone)	0.991	—	0.980	—

Appendix

Table 9: Post-RMSNorm activation cosine similarity to the unedited model after editing a single mid layer. “next” is the following block’s post-norm input; “final” is the final post-norm hidden state.

b

is measured in units of the layer’s weight standard deviation.

Projection	kp sign brk ord	kp ord brk sign	gauss rand	gauss mono
q_proj	10.17	9.08	9.83	9.48
k_proj	9.86	9.08	9.77	9.48
v_proj	9.84	9.08	10.02	9.14
o_proj	10.56	9.38	10.42	9.18
gate_proj	9.56	9.13	9.76	9.17
up_proj	9.63	9.08	9.79	9.16

Appendix

Table 10: WikiText-2 PPL for the four perturbation modes across all projection types at layer 10 (Llama-3.1-8B-Instruct shown; SmolLM2 in supplementary results). kp/brk = keep/break.

SmolLM2-135M
Condition	ARC-e	ARC-c	HellaSwag	WinoGr.	BoolQ	MMLU
Clustered baseline	0.469	0.272	0.425	0.512	–	–
Rank-preserving	0.461	0.288	0.398	0.509	–	–
Rank-breaking	0.458	0.266	0.365	0.521	–	–
Rank-breaking (perm)	0.303	0.227	0.313	0.485	–	–
Llama-3.1-8B-Instruct

Appendix

Table 11: Zero-shot task accuracy ( lm-eval ) for the clustered model and under single-layer centroid edits at gate_proj . Higher is better. ARC-c/e = ARC Challenge/Easy; perm = full centroid permutation (extreme rank-breaking).

	q	k	v	o	gate	up	down
Rank-pres. (monotone)	13.67	12.19	12.18	12.12	12.31	12.20	12.28
Rank-break (random)	14.15	12.65	14.04	13.64	14.11	14.09	14.12

Appendix

Table 12: Qwen2.5-7B-Instruct, layer 10: rank-preserving (Gaussian monotone) vs. rank-breaking (Gaussian random) centroid replacement, which share mean and variance and differ only in rank. The rank-preserving edit is less damaging for every projection.

Condition	ARC-e	ARC-c	HellaSwag	WinoGr.	BoolQ	MMLU
Clustered baseline	0.772	0.548	0.672	0.694	0.846	0.739
Rank-preserving	0.766	0.544	0.672	0.710	0.840	0.729
Rank-breaking	0.728	0.514	0.620	0.676	0.806	0.637
Rank-breaking (perm)	0.644	0.416	0.558	0.516	0.588	0.471

Appendix

Table 13: Qwen2.5-7B-Instruct zero-shot task accuracy ( lm-eval , 500-example subset) under single-layer centroid edits at gate_proj . perm = full centroid permutation.

Edits applied to every clustered projection
		Trained from scratch ( $\times$ )			SmolLM2-135M		GPT-2 124M
Intervention	H	RMSNorm	LayerNorm	No norm	$\times$	$\rho_{\mathrm{act}}$	$\times$	$\rho_{\mathrm{act}}$
Rank-only reconstruction	H3	2.39	2.48	7.04	$1.8\!\times\!10^{6}$	0.132	6266	0.481
same values, order shuffled	H3	48.29	13.63	14.86	$8.5\!\times\!10^{6}$	-0.113	$6.5\!\times\!10^{5}$	0.060
Rank-preserving (sorted Gaussian)	H1	3.01	2.52	7.76	$6.0\!\times\!10^{8}$	0.057	4403	0.408
same values, unsorted	H1	88.32	16.34	14.34	–	–	–	–

Appendix

Table 14: H1–H3 tested in five models: three byte-level transformers trained from scratch that are identical except for their normalization layer, and two pretrained models clustered from their original checkpoints. All are clustered with the same scalar

K

-means at

K=32

on every projection. Entries are perplexity ratios to each model’s own clustered baseline;

\rho_{\mathrm{act}}

is the Spearman correlation between the perturbed and unperturbed outputs of the edited layer. “–” marks a condition not run in that setting. Clustered baselines: 5.46 (RMSNorm), 5.51 (LayerNorm), 6.93 (No norm) byte-level, 46.48 (SmolLM2-135M, dense 26.61), 44.13 (GPT-2 124M, dense 37.37) word-level (WikiText-2, Appendix G ).

Model	Tok/s	TTFT (ms)	TPOT (ms)	Power (W)	TFLOPs	Disk (GB)	Note
Original	155	25.1	6.30	266	0.50	15.0	Baseline
Orig. clust. rebuild	157	25.6	6.21	232	0.51	6.9	Same speed, less disk
Pre-compressed	259	18.6	3.74	188	0.83	6.1	Best throughput
Pre-comp. clust. rebuild	269	17.0	3.61	186	0.87	3.1	Best trade-off
Orig. LUT	14	373	67.0	177	0.05	6.5	Kernel-bound
Pre-comp. LUT	34	135	28.2	166	0.11	3.0	Kernel-bound

Appendix

Table 15: vLLM interactive serving benchmark on Llama 3.1-8B-Instruct (512

\to

128 tokens, concurrency=1). TTFT = time to first token; TPOT = time per output token.

Strategy	Avg. Time (s)	vs. Dense
Dense baseline	0.65	1.00 $\times$
Precompute (FP32)	0.66	1.02 $\times$
LUT fallback	2.94	4.52 $\times$ slower
Gather-matmul	3.01	4.63 $\times$ slower
Chunked reconstruct	4.08	6.28 $\times$ slower
Vectorized scalar	4.25	6.53 $\times$ slower

Appendix

Table 16: CPU inference time under different execution strategies (average over 3 prompts, 32 generated tokens). “Precompute” reconstructs weights once at load time and uses dense matrix multiplication; all other rows perform per-step LUT/gather execution.

Model		$K{=}16$	$K{=}32$	$K{=}64$	Total
Pre-comp. Llama	MLP	0	55	29	84
Pre-comp. Llama	Attn	0	91	21	112
Llama 3.1-8B-Inst.	MLP	0	78	18	96
Llama 3.1-8B-Inst.	Attn	0	114	14	128
SmolLM2-135M	MLP	1	66	23	90
SmolLM2-135M	Attn	9	84	27	120

Appendix

Table 17: Number of MLP and attention projection layers assigned to each cluster budget

K

across the three models used in this work.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

LegoLM: Structured Weight Sharing for Large Language Models
Aug 9, 2026Joseph BinghamLLM Quantization LLM Compression
Deparment of Biology, Technion – Israel Institute of Technology
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Jun 20, 2026Chao Han, Yongjie Du, Junjie Tan +1LLM Compression Low-Rank Matrix Decomposition
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models
Oct 10, 2026Muhammad Usama, Yeoreum JungLLM Compression Language Model Merging
Braindeck Inc Seoul, Republic of Korea

Aug 9, 2026J/K move · Enter open · S save

LegoLM: Structured Weight Sharing for Large Language Models

Joseph Bingham

Deparment of Biology, Technion – Israel Institute of Technology

LLM Quantization LLM Compression Neural Network Quantization

Open paper AskPDF arXiv

We present \LegoLM{}, a structured weight-sharing compression framework for large language models grounded in a systematic study of why global weight sharing fails and how to fix it. We identify two distinct failure modes. Distributional mismatch: for vector blocks of dimension d <= 2, transformer layers with heterogeneous weight scales impose a scale-mismatch penalty that grows linearly with d and cannot be resolved by increasing K, producing perplexity in the millions.Outlier dominance: for scalar blocks, a fraction ~1/K of weights lies beyond the outermost Lloyd-Max decision threshold and cannot be represented by any centroid; their misrepresentation accumulates across layers, causing catastrophic quality loss. \LegoLM{} resolves both failure modes via three data-free adaptations: 1 scalar-block encoding to eliminate the

d

-linear mismatch component, 2 percentile-selective replacement that identifies and preserves outlier weights verbatim, and 3 boundary-layer protection for the first and last transformer blocks. Across GPT-2 small (124M) and Mistral-7B, \LegoLM{} achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B - outperforming PTQ-8bit in both quality and compression ratio - and -0.02% at 2.67X. Downstream evaluation on LAMBADA and HellaSwag confirms that \LegoLM{} at K=64, p=99% preserves accuracy within noise at 5.12 X compression, exceeding PTQ-8bit's compression ratio while matching its accuracy. We further discover that outlier dominance grows with model scale: full replacement at K=128 degrades GPT-2 small by only +23% but catastrophically degrades Mistral-7B by +1,134,279%, while selective replacement at p=99% rescues both models to under +15%. A controlled ablation confirms that selective replacement is the dominant mechanism: adding it to per-layer K-means also yields near-lossless quality, matching \LegoLM{} within 0.02%.

Abstract

LLM Compression

Explore similar work

LegoLM: Structured Weight Sharing for Large Language Models

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models