Block-Level Weight-Space Structure Persists Under Post-Training: An Empirical Study Across LLM Families

Zhaohui Wang

cs.LGAug 23, 2026

Block-Level Weight-Space Structure Persists Under Post-Training: An Empirical Study Across LLM Families

Authors: Zhaohui Wang

Organizations: Viterbi School of Engineering, University of Southern California, Los Angeles, USA.

Abstract

Modern LLMs are deployed as families of post-trained variants (base, instruct, chat, code) derived from a shared set of pre-trained weights. We present an empirical study of how post-training transforms weight-space geometry, covering eight configurations across four architecture families (Qwen2.5, Llama-3.1/3.2, Mistral, Gemma-2). We identify a granularity gap: post-training modifies every tensor (zero of 291-339 tensors remain byte-identical, so hash-based deduplication achieves 0% savings), yet preserves block-level structure (mean cosine similarity exceeds 0.99 and relative Frobenius distance stays below 0.13). Post-training therefore acts as a structured perturbation that shifts every parameter while leaving block-level geometry intact. The property is not universal: independently trained specializations (for example, Qwen2.5-Coder) attain cosine similarity around 0.64 with the general base, indicating a disconnected region of weight space. Perturbation magnitude varies systematically with model scale, architecture family, and post-training recipe. As a practical application, we build LinkerLLM, a lazy loader that aliases shareable blocks across co-resident variants, achieving 18-48% GPU memory savings and enabling up to five 7B-parameter variants on a single 24 GB consumer GPU. Five of eight configurations retain at least 94% of the unshared variant's quality on MMLU, ARC-Challenge, HellaSwag, and WinoGrande; the remaining three (Mistral-7B, Gemma-2-2B, Llama-3.2-1B) have one below-threshold benchmark each (87-91%), which we report transparently rather than gate the block-sharing decision on a single threshold.

Figures & tables

Family	Blocks	cos range	$\epsilon$ range	$\epsilon$ mean
Mistral-7B-v0.3	32	0.999+	0.004–0.01	0.006
Qwen2.5-3B	36	0.999+	0.01–0.02	0.011
Qwen2.5-7B	28	0.999+	0.01–0.02	0.015
Llama-3.1-8B	32	0.999+	0.04–0.05	0.045
Qwen2.5-0.5B	24	0.998	0.04–0.05	0.045
Gemma-2-2B	26	0.998–0.999	0.04–0.08	0.049

Table 1: Block-level similarity between base and instruct variants. Families ordered by mean Frobenius ratio (ascending = more preserved).

Figure 1: Per-block Frobenius ratio $\epsilon(i)$ across model families (base $\to$ instruct). Mistral and Qwen families have uniformly low perturbation (all blocks near-frozen). The Llama-3.2 series shows 10–20 $\times$ larger per-block perturbation than Mistral, consistent with more aggressive RLHF; the Llama-3.1-8B variant lies between these regimes ( $\bar{\epsilon}=0.045$ ), suggesting that perturbation magnitude is not a fixed property of the Llama family but varies with the specific post-training recipe used per release.

Figure 2: Per-family perturbation magnitude $\bar{\epsilon}$ vs. MMLU capability gain (Pearson $r=0.819$ , $n=8$ ). Mistral-v0.3 / Qwen-3B / Qwen-7B with $\bar{\epsilon}\in[0.006,0.015]$ show flat or slightly negative MMLU change; larger- $\bar{\epsilon}$ families (Llama-3.2 series, Gemma-2-2B) show 5–9 pt gain. $n=30$ per MMLU subject (lm-eval v0.4.11).

Post-training (SFT/RLHF/DPO)
Pair (all within Qwen2.5)	cos	$\epsilon$	Region
7B base $\to$ instruct	0.9997	0.015	Aligned
Continued pretraining
1.5B base $\to$ Coder-1.5B	0.704	0.358	Outside
3B base $\to$ Coder-3B	0.752	0.305	Outside
7B base $\to$ Coder-7B	0.637	0.831	Outside

Table 2: Weight-space distance between variant types within the Qwen2.5 family, all variants derived from the matching Qwen2.5 base. Post-trained (instruct) variants remain in the base’s block-aligned neighborhood across model scales; continued-pretrained Coder/Math derivatives exit it across both branches and both sizes tested, supporting the boundary claim with four independent continued-pretraining examples rather than a single case. Both cos and

\epsilon

are means over blocks; model names omit the shared Qwen2.5- prefix.

attention sub-mean cosine
	ref	head_perm	rmsnorm
Qwen2.5-1.5B blk14	0.9999	0.1344	0.9666
Qwen2.5-7B blk14	0.9999	0.0727	0.9677
Llama-3.1-8B blk16	0.9994	0.1522	0.9431
attention sub-mean $\epsilon$
Qwen2.5-1.5B blk14	0.0037	0.6527	0.0856

Table 3: Permutation/rescaling controls on three families (one block per model). ref = no perturbation; head_perm = random function-preserving head permutation on the variant; rmsnorm = random channel rescaling

c_{i}\!\sim\!\mathcal{U}[0.5,2.0]

on the variant. Cosine of attention sub-params is shown for clarity.

Figure 3: Cross-family loss-along-interpolation summary. Per-block max $\Delta L$ (solid) and full-model peak $\Delta L$ (hatched) on a symlog scale. Blue = four post-trained families; orange = boundary family Llama-3.2-1B; pink = two continued-pretrained derivatives. The post-trained band sits below $10^{-2}$ nats on both metrics; continued-pretrained derivatives are 3–4 orders of magnitude above.

Figure 4: Loss-along-interpolation path test on Qwen2.5-1.5B. Each grey curve is a per-block path $\Delta L_{i}(\alpha)=L_{i}(\alpha)-L_{\text{variant}}$ . Blue median + 10/90 percentile band shows per-block paths are flat within ${\leq}0.007$ nats. Orange: full-model interpolation $L(\alpha)$ , also barrier-free. Compare with Llama-3.2-1B (boundary family, Appendix L ) where block 0 develops a $\sim$ 0.15-nat per-block barrier.

Family	$\tau_{\text{frob}}$	Shared	Naive	Lazy	Save
Mistral-7B	0.05	32/32	27648	14336	48%
Qwen2.5-3B	0.05	36/36	11988	6588	45%
Llama-3.2-3B	0.13	28/28	12256	6880	44%
Qwen2.5-7B	0.05	28/28	29136 ∗	16648	43%
Llama-3.1-8B	0.05	32/32	30634 ∗	17322	43%
Qwen2.5-0.5B	0.06	24/24	1900	1210	36%

Table 4: GPU memory (MiB) for two co-resident variants (base + instruct) on RTX 3090. ∗ Naive would OOM on 24 GB.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Params	Blocks	Precision
Qwen2.5-0.5B	0.5B	24	bf16
Qwen2.5-3B	3B	36	bf16
Qwen2.5-7B	7B	28	bf16
Llama-3.2-1B	1.2B	16	bf16
Llama-3.2-3B	3B	28	bf16
Llama-3.1-8B	8B	32	bf16

Appendix

Table 5: Model configurations studied.

Figure 5: LinkerLLM lazy loader. Donor model A resides on GPU. Recipient B is loaded to CPU, similarity-scanned, and only unique sections are moved to GPU. Shared blocks alias donor’s storage directly. Peak GPU $=|A|+|\text{unique}(B)|$ .

Figure 6: Memory scaling with $N$ variants (Qwen2.5-7B, fp16). LinkerLLM grows at $\sim$ 2 GB/variant vs. 14.2 GB/variant for independent loading.

Family	Saved	MMLU	ARC	Hella	Wino
Mistral-7B	48%	101	87	96	101
Qwen2.5-3B	45%	99	97	98	98
Llama-3.1-8B	43%	96	97	101	99
Qwen2.5-7B	43%	102	101	101	103
Qwen2.5-0.5B	36%	102	95	96	100
Llama-3.2-3B	7% †	98	102	100	99

Appendix

Table 6: Quality retained (%) after sharing at per-family optimal thresholds. The aggregation used here (cosine-of-concatenated-sub-parameters) differs from the deployed mean-of-per-sub-parameter-cosines in aliasing.py ; the two agree for homogeneous-perturbation families (Mistral, Qwen) but differ for heterogeneous-perturbation families. † Llama-3.2-3B at

\tau_{\text{cos}}{=}0.999

shares 5/28 blocks (vs. 28/28 in Table 4 ); Llama-3.2-1B shares 6/16 (vs. 13/16); savings and quality reflect this more conservative sharing. The Gemma-2-2B row uses the deployed mean-of-cosines criterion (12/26 shared, matching Table 4 ). Sharing produces deterministic outputs but token-level greedy decoding may diverge from the unshared variant after a few tokens due to autoregressive amplification.

Family	Mode	MMLU	ARC	Hella	Wino
Mistral-7B	instruct	61.1	59.3	73.0	76.3
	shared	61.5	51.7	70.3	77.0
Qwen2.5-3B	instruct	65.5	48.1	75.1	69.4
	shared	65.0	46.8	73.5	68.0
Llama-3.1-8B	instruct	67.7	51.7	69.0	74.3
	shared	65.0	50.0	69.3	73.7

Appendix

Table 7: Absolute downstream-task scores (%) for instruct (unshared) vs. shared variants, MMLU/ARC-Challenge/HellaSwag/WinoGrande on lm-eval-harness v0.4.11. Bold cells in the shared row mark configurations where retention falls below 94% of the unshared instruct score (cf. relative-retention values in Table 6 ).

Family	Mean $\epsilon$	$\tau_{\text{frob}}$	Blocks shared
Mistral-7B-v0.3	0.006	0.05	32/32
Qwen2.5-3B	0.011	0.05	36/36
Qwen2.5-7B	0.015	0.05	28/28
Qwen2.5-0.5B	0.045	0.06	24/24
Llama-3.1-8B	0.045	0.05	32/32
Gemma-2-2B	0.049	0.07	12/26

Appendix

Table 8: Mean per-block Frobenius ratio by family, with recommended sharing thresholds.

Family	$\tau_{\text{frob}}$	mean $\bar{\epsilon}$	max- $K$ mean	global max
Mistral-7B-v0.3	0.05	0.016	0.024	0.029
Qwen2.5-3B	0.05	0.007	0.014	0.018
Llama-3.2-1B	0.20	0.122	0.182	0.192

Appendix

Table 9: Mean-over-

K

vs. max-over-

K

aggregation of relative Frobenius perturbation. “mean

\bar{\epsilon}

” is the mean across blocks of the per-block mean-over-sub-parameters (the value reported in Table 8 ). “max-

K

mean” is the mean across blocks of the per-block maximum sub-parameter

\epsilon

. “global max” is the maximum over all (block, sub-parameter) pairs. For all three families, even the global max stays below the deployed

\tau_{\text{frob}}

, confirming that mean-over-

K

does not hide outlier sub-parameters.

Figure 7: Per-layer amplification factor $\alpha(i)$ for Qwen2.5-1.5B (28 layers, blue) and Qwen2.5-3B (36 layers, red), normalized to relative depth $i/L$ . Both show interior $\alpha<2$ (linear, not exponential, growth) plus boundary spikes at the embedding side and the lm_head side. The 3B model has a longer unstable tail but downstream benchmarks remain within 99% of the unshared baseline.

Family	Pair	$\bar{\epsilon}$	Tightening
Gemma	Gemma-1-2B (Feb ’24)	0.610	$12.4\times$
Gemma	Gemma-2-2B (Jun ’24)	0.049	$12.4\times$
Mistral	Mistral-v0.1-7B (Sep ’23)	0.140	$23.3\times$
Mistral	Mistral-v0.3-7B (May ’24)	0.006	$23.3\times$
Llama	Llama-2-7B (Jul ’23)	0.052	$1.15\times$ (flat)
Llama	Llama-3.1-8B (Jul ’24)	0.045	$1.15\times$ (flat)

Appendix

Table 10: Per-block Frobenius perturbation across model generations within a family. Each row is the base

\to

aligned pair for that release ( -it for Gemma, -chat for Llama-2 and Mistral-v0.1, -Instruct otherwise). Two of three families (Gemma, Mistral) show order-of-magnitude tightening between 2023 and 2024 releases; Llama is essentially flat. The trend toward parameter-efficient post-training is real but not universal .

Figure 8: Per-block Frobenius ratio

\epsilon(i)

for Gemma-2-2B (base

\to

instruct). Middle blocks (indices 5–14) carry the largest perturbation (

\epsilon\in[0.057,0.064]

); late blocks (16–25) are progressively more preserved (

\epsilon

drops to 0.022). At the tight

\tau_{\text{frob}}=0.05

used for Mistral/Qwen, 16/26 blocks fail; at the looser

\tau_{\text{frob}}=0.07

used in the Gemma deployment all blocks pass on

\epsilon

, with the residual 14/26 failures driven by the dual-criterion cosine check (

\tau_{\text{cos}}=0.999

) in fp16. The pattern (high perturbation concentrated in the middle, low perturbation in late layers) suggests that Gemma’s post-training updates are skewed away from the embedding-adjacent and lm-head-adjacent regions, unlike the more uniform profile of Mistral and Qwen.

Source	blocks	mean ratio	min	max
qwen05b (tol=0.05)	24	1.219	1.194	1.223
qwen3b (tol=0.05)	36	1.180	0.800	1.206
qwen7b (tol=0.05)	28	1.198	1.112	1.209

Appendix

Table 11: LoRA-extraction baseline: per-block ratio of rank-truncated LoRA storage to full block storage at Frobenius reconstruction tolerance tol. Ratio

\geq 1

means LoRA-extraction is worse than storing the original block. Linker aliasing stores 0 extra bytes (pointer alias to the donor). Block-level aliasing therefore Pareto-dominates LoRA-extraction on these post-trained variants, whose

\Delta W

spectra are too broad for low-rank approximation at Frobenius fidelity.

Family	Method	Save	Fidelity	Overhead
Qwen-3B	LinkerLLM	45%	Determ.	0 ms
Qwen-3B	DeltaZip	83%	$\sim$ 0.99995	21 s
Llama-1B	LinkerLLM	32%	Determ.	0 ms
Llama-1B	DeltaZip	85%	0.991	9 s

Appendix

Table 12: Block-level sharing (LinkerLLM) vs delta compression (DeltaZip). The approaches are complementary: LinkerLLM dominates for high-similarity families; DeltaZip for low-similarity.

Post-training (instruct)
		Per-block	Full-model
Family	$L$	max $\Delta L$	peak $\Delta L$
Llama-3.1-8B	32	+0.0000	+0.0000
Qwen2.5-7B	28	+0.0004	$-$ 0.0000
Mistral-7B	32	+0.0016	+0.0000
Qwen2.5-1.5B	28	+0.0062	+0.0000

Appendix

Table 13: Cross-family loss-along-interpolation summary. Per-block max

\Delta L

is the worst per-block barrier; full-model peak

\Delta L

is the maximum of

L_{\text{full}}(\alpha)-L_{\text{variant}}

over

\alpha\in[0,1]

. All values in nats;

L

is the number of transformer blocks. Post-trained families have negligible barriers; the boundary family Llama-3.2-1B has small per-block barriers concentrated at the boundary blocks but the full-model path is also broken. Continued-pretrained variants (Qwen-Coder, Qwen-Math) have dramatically larger barriers across both metrics.

Figure 9: Loss-along-interpolation on the boundary family Llama-3.2-1B ( $\bar{\epsilon}=0.156$ ). 14 of the 16 per-block paths are flat or downhill, but block 0 (embedding-adjacent) has a $\sim$ 0.15-nat barrier and block 15 (lm_head-adjacent) has a $\sim$ 2.76-nat barrier. The full-model interpolation curve (orange) rises from $3.22$ nats at the variant ( $\alpha{=}0$ ) to $5.91$ nats at the base ( $\alpha{=}1$ ). The deployed-shared set on this family (Table 4 : 13/16 blocks at $\tau_{\text{frob}}{=}0.20$ ) excludes exactly the two boundary-block positions identified here.

Figure 10: Loss-along-interpolation on Qwen2.5-1.5B base $\to$ Qwen2.5-Coder-1.5B (continued pretraining, $\bar{\epsilon}\approx 0.36$ , cos $\sim 0.70$ ). Per-block paths develop barriers of $0.5$ – $9.0$ nats; 27 of 28 blocks have per-block max $\Delta L>0.1$ nats. The full-model interpolation rises by $6.7$ nats. Continued pretraining is therefore not a small extension of post-training in either parameter geometry or loss landscape.

Figure 11: Loss-along-interpolation on Qwen2.5-1.5B base $\to$ Qwen2.5-Math-1.5B (continued pretraining, $\bar{\epsilon}\approx 0.47$ , cos $\sim 0.53$ ). Block 0 has a per-block barrier of $\sim$ 6.5 nats; 10 of 28 blocks have per-block max above $0.1$ nats. The full-model interpolation rises by $7.8$ nats. The Coder and Math continued-pretrained branches both show the same loss-barrier signature, confirming that the continued-pretraining $\to$ outside-the-aligned-neighborhood pattern is not idiosyncratic to a single specialization.

$N$	MiB	Save	MiB	Save	MiB	Save
	Qwen2.5-3B		Qwen2.5-7B		Mistral-7B
1	5994	0%	14568	0%	13824	0%
2	6588	45%	16648	43%	14336	48%
3	7182	60%	18728	57%	14848	64%
4	7776	68%	20808	64%	15360	72%
5	8370	72%	—	—	15872	77%

Appendix

Table 14: Measured GPU memory (MiB) for

N

co-resident variants on a single RTX 3090 (24 GB). Marginal cost per variant is constant.

Family	CPU load	Scan	Alias	Move	Total	Naive
Qwen-3B	1.4s	19.8s	0.6s	0.3s	22.1s	5.8s
Qwen-7B	2.8s	51.5s	1.4s	0.6s	56.3s	11.9s

Appendix

Table 15: Per-step latency of the lazy loader. The similarity scan dominates.

Group	min	mean	max	gap
Shared (0–11)	0.99844	0.99929	0.99998	0.00084
Rejected (12–25)	0.99788	0.99887	0.99998	0.00099

Appendix

Table 16: Per-sub-parameter cosine statistics for Gemma-2-2B blocks, averaged within group (

n{=}12

shared,

n{=}14

rejected). “Gap” is mean

-

min, a measure of how much one outlier sub-parameter could pull the block mean down. The deployed-shared and rejected groups have nearly identical gaps, so the rejection of blocks 12–25 is not driven by sub-parameter outliers; the entire cosine distribution shifts uniformly with block depth.

Group	$\\|g\\|/\\|\theta\\|$	$1-\cos$
Shared (0–11)	0.0447	$7.1{\times}10^{-4}$
Rejected (12–25)	0.0402	$1.13{\times}10^{-3}$

Appendix

Table 17: Gradient-locality probe: average values within deployed-shared vs. rejected groups for Gemma-2-2B. Higher gradient does not imply higher empirical divergence; the relationship is in fact slightly anti-correlated.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 195 papers

LegoLM: Structured Weight Sharing for Large Language Models
Aug 9, 2026Joseph BinghamLLM Quantization LLM Compression
Deparment of Biology, Technion – Israel Institute of Technology
OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
Dec 15, 2025Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi +1LLM Pruning Neural Network Pruning
University of Toronto · Google DeepMind
Weight Clustering Reveals the Importance of Relative Rank in Large Language Models
Mar 18, 2026Borja Aizpurua, Sukhbinder Singh, Román OrúsLLM Compression LLM Quantization
Multiverse Computing, San Sebasti´an, Spain · Department of Basic Sciences, Tecnun - University of Navarra, San Sebasti´an, Spain · Multiverse Computing, Toronto, Ontario, Canada +2

Aug 9, 2026J/K move · Enter open · S save

LegoLM: Structured Weight Sharing for Large Language Models

Joseph Bingham

Deparment of Biology, Technion – Israel Institute of Technology

LLM Quantization LLM Compression Neural Network Quantization

Open paper AskPDF arXiv

We present \LegoLM{}, a structured weight-sharing compression framework for large language models grounded in a systematic study of why global weight sharing fails and how to fix it. We identify two distinct failure modes. Distributional mismatch: for vector blocks of dimension d <= 2, transformer layers with heterogeneous weight scales impose a scale-mismatch penalty that grows linearly with d and cannot be resolved by increasing K, producing perplexity in the millions.Outlier dominance: for scalar blocks, a fraction ~1/K of weights lies beyond the outermost Lloyd-Max decision threshold and cannot be represented by any centroid; their misrepresentation accumulates across layers, causing catastrophic quality loss. \LegoLM{} resolves both failure modes via three data-free adaptations: 1 scalar-block encoding to eliminate the

d

-linear mismatch component, 2 percentile-selective replacement that identifies and preserves outlier weights verbatim, and 3 boundary-layer protection for the first and last transformer blocks. Across GPT-2 small (124M) and Mistral-7B, \LegoLM{} achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B - outperforming PTQ-8bit in both quality and compression ratio - and -0.02% at 2.67X. Downstream evaluation on LAMBADA and HellaSwag confirms that \LegoLM{} at K=64, p=99% preserves accuracy within noise at 5.12 X compression, exceeding PTQ-8bit's compression ratio while matching its accuracy. We further discover that outlier dominance grows with model scale: full replacement at K=128 degrades GPT-2 small by only +23% but catastrophically degrades Mistral-7B by +1,134,279%, while selective replacement at p=99% rescues both models to under +15%. A controlled ablation confirms that selective replacement is the dominant mechanism: adding it to per-layer K-means also yields near-lossless quality, matching \LegoLM{} within 0.02%.

Abstract

Language Model Post-Training

Explore similar work

LegoLM: Structured Weight Sharing for Large Language Models

OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models