The Fragility and Dynamics of LLM Pruning-Recovery Steps

Luis Frentzen Salim; Komang Rinartha; Chia-Chun Wu; Tran Van Nhiem; Lun-Wei Ku; Yung-Hui Li

cs.CLOct 11, 2026

The Fragility and Dynamics of LLM Pruning-Recovery Steps

Authors: Luis Frentzen Salim, Komang Rinartha, Chia-Chun Wu, Tran Van Nhiem, Lun-Wei Ku, Yung-Hui Li

Organizations: Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute

Abstract

Structural pruning followed by a recovery stage is the standard recipe for compressing large language models (LLMs), yet the two stages are typically designed and tuned in isolation. In this work, we study how their interaction shapes the compressed model, sweeping ordering, the data that drives each stage, and the alignment of their training objectives. We find that instruction-following is the first capability to collapse under pruning, well before knowledge, reasoning, or core language modeling, and that much of the pruning in-domain loss is likely not knowledge loss but a domain-agnostic failure to respond coherently to the prompt. We find in our setting that in-domain data is the strongest pruning signal, whereas instruction-style data recovers the most. Stage order matters little, but only marginally. We further analyze how the importance ranking drifts before and after pruning. Together these results give practitioners a concrete recipe for composing pruning and recovery.

Figures & tables

Figure 1: Performance over pruning ratios for Llama 3.1 8B Instruct. Instruction following degrades earlier.

Figure 2: Performance on the domain and general tasks over pruning

r

, with one line for each configuration.

Figure 3: Spearman correlation of moving weight rankings before and after pruning with $r$ $0.1$ and $0.5$ . It is observed that the most and least important weights tend to stay as the model is pruned.

Figure 4: Recovery gain by data, for

P

-first (

P\rightarrow D

) sequences. Each bar is the change in a metric relative to the matched pruning-only model with no recovery, averaged over pruning corpora and ratios.

Pruning corpus	$\Delta$	$p$
All	$-0.009$	${<}0.001$
Single-source	$-0.003$	$0.22$
Mixed	$-0.020$	${<}0.001$

Table 1: Pipeline ordering result averaged over tasks, order and pruning ratios.

\Delta=(\text{

-first})-(\text{

-first})

, negative values favor pruning-first.

Metric	Aligned	Misaligned	$p$
GSM8K	0.075	0.075	0.98
General	0.379	0.385	0.13
Average	0.387	0.393	0.12

Table 2: Alignment performances over pruning

r

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	GOLD	SFT
Precision	bf16	bf16
Max length	$2048$	$2048$
batch size	$2$	$4$
Gradient accumulation	$16$	$8$
Global batch size	$256$	$32$
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$

Appendix

Table 3: Training hyperparameters used.

Hyperparameter	Value
Pruner type	first-order Taylor
Granularity	block-wise
Pruned blocks	all layers
Iterative steps	$1$ (one-shot)
Group aggregation	sum
Pruning scope	local (per-block)

Appendix

Table 4: Pruning hyperparameters used. The default calibration loss is CLM. The GOLD loss (bottom block) use the Gemma 3 Instruct 12B as the teacher.

Figure 5: GSM8K outcome decomposition for prune-only models as a function of pruning ratio, faceted by the data used for pruning. Each item is classified as correct, wrong but coherent, or degenerate / non-responsive. The degenerate share grows with sparsity and appears under every pruning-data kind, including in-domain pruning.

Figure 6: Degenerate-response rate for prune-only models versus pruning ratio, on the in-domain task (GSM8K, left) and the instruction-following task (IFEval, right), with one line per pruning-data kind. On IFEval the curves are nearly independent of the pruning data, indicating a domain-agnostic collapse in the model’s ability to follow the prompt.

Figure 7: Evaluation performance isolating two major sequences showing little to no effect on final performance.

Figure 8: Before- versus after-pruning importance percentile of each surviving attention head, colored by layer, for the Taylor and Fisher importance criteria at pruning ratios $r=0.1$ and $r=0.5$ . The diagonal marks unchanged rank.

Figure 9: Before- versus after-pruning importance percentile of each surviving MLP channel, colored by layer, for the Taylor and Fisher importance criteria at pruning ratios $r=0.1$ and $r=0.5$ . The diagonal marks unchanged rank.

Figure 10: Before- versus after-pruning importance percentile of surviving attention heads, colored by layer, for the Taylor and Fisher criteria across pruning ratios $r=0.1$ to $r=0.5$ .

Figure 11: Before- versus after-pruning importance percentile of surviving MLP channels, colored by layer, for the Taylor and Fisher criteria across pruning ratios $r=0.1$ to $r=0.5$ .

Figure 12: Band-transition matrices of attention-head importance before versus after pruning, shown for the Taylor and Fisher importance criteria at pruning ratios $r=0.1$ and $r=0.5$ . Each cell gives the fraction of heads moving from a before-pruning importance quintile (row) to an after-pruning quintile (column).

Figure 13: Attention-head importance band-transition matrices for the Taylor (top row) and Fisher (bottom row) criteria across pruning ratios $r=0.1$ to $r=0.5$ (columns).

Figure 14: MLP-channel importance band-transition matrices for the Taylor (top row) and Fisher (bottom row) criteria across pruning ratios $r=0.1$ to $r=0.5$ (columns).

Figure 15: Per-quintile mean absolute importance movement and retention for attention heads, for the Taylor and Fisher criteria across pruning ratios

r=0.1

r=0.5

Figure 16: Per-quintile mean absolute importance movement and retention for MLP channels, for the Taylor and Fisher criteria across pruning ratios

r=0.1

r=0.5

Figure 17: Per-quintile retention of attention-head importance as a function of pruning ratio, faceted by importance criterion (Taylor and Fisher), with one line per before-pruning quintile.

Figure 18: Per-quintile retention of MLP-channel importance as a function of pruning ratio, faceted by importance criterion (Taylor and Fisher), with one line per before-pruning quintile.

Figure 19: Spearman rank correlation between before- and after-pruning attention-head importance as a function of pruning ratio, resolved per before-pruning quintile and faceted by importance criterion.

Figure 20: Spearman rank correlation between before- and after-pruning MLP-channel importance as a function of pruning ratio, resolved per before-pruning quintile and faceted by importance criterion.

Loss	Recov	$r$	GSM8K	Gen	Avg	Loss	Recov	$r$	GSM8K	Gen	Avg
CLM	KD	0.1	0.171	0.519	0.525	GOLD	KD	0.1	0.208	0.476	0.483
CLM	KD	0.2	0.074	0.442	0.451	GOLD	KD	0.2	0.108	0.443	0.454
CLM	KD	0.3	0.032	0.387	0.395	GOLD	KD	0.3	0.049	0.364	0.375
CLM	KD	0.4	0.024	0.322	0.326	GOLD	KD	0.4	0.027	0.310	0.320
CLM	KD	0.5	0.017	0.296	0.302	GOLD	KD	0.5	0.026	0.288	0.292
CLM	SFT	0.1	0.185	0.506	0.515	GOLD	SFT	0.1	0.227	0.497	0.507

Appendix

Table 5: Loss-alignment runs. Loss is the pruning calibration objective (CLM or GOLD), Recov is the recovery method (none = pruning-only, SFT, or KD/GOLD distillation),

r

is the pruning ratio. IFEval was not evaluated for these runs. Rows continue in the right-hand block.

Seq	$P$	$D$	$r$	GSM8K	Gen	IFE	Avg	Seq	$P$	$D$	$r$	GSM8K	Gen	IFE	Avg
base	–	–	–	0.723	0.545	0.764	0.656	$D{\to}P$	Mix	Gen	0.2	0.552	0.461	0.663	0.551
$P$	Dom	–	0.1	0.708	0.511	0.709	0.622	$D{\to}P$	Mix	Gen	0.3	0.420	0.395	0.521	0.448
$P$	Dom	–	0.2	0.630	0.453	0.644	0.561	$D{\to}P$	Mix	Gen	0.4	0.281	0.329	0.403	0.349
$P$	Dom	–	0.3	0.506	0.399	0.546	0.476	$D{\to}P$	Mix	Gen	0.5	0.112	0.292	0.288	0.259
$P$	Dom	–	0.4	0.369	0.343	0.377	0.361	$D{\to}P$	Mix	Mix	0.1	0.663	0.509	0.741	0.625
$P$	Dom	–	0.5	0.196	0.295	0.309	0.283	$D{\to}P$	Mix	Mix	0.2	0.542	0.457	0.645	0.540

Appendix

Table 6: Full per-configuration results, reporting aggregate metrics only. Seq is the stage order (

D{\to}P

P{\to}D

P

D

are the pruning/distillation data kinds,

r

is the pruning ratio. GSM8K is flexible-extract, General is the mean of MMLU and ARC-easy/challenge, IFE is the mean of the four IFEval splits, and Avg is the overall average. Rows continue in the right-hand block.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning
Sep 7, 2026Seungmin Oh, Donggeon Lee, Jongbin RyuLLM Pruning LLM Compression
Ajou University, South Korea
The Functional Structure of Post-Compression Recovery in Low-Rank LLMs
Oct 4, 2026Zishan Shao, Liang Tian, Georgiy Zemlevskiy +10LLM Compression Low-Rank Compression
Duke University · Wake Forest University · Carnegie Mellon University +2
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Aug 13, 2026Palaash Goel, Ayan Sengupta, Akshay Nambi +1LLM Pruning Structured Pruning
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi · Microsoft Research (MSR) India

Sep 7, 2026J/K move · Enter open · S save

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Seungmin Oh, Donggeon Lee, Jongbin Ryu

Ajou University, South Korea

LLM Pruning LLM Compression Structured Pruning

Open paper AskPDF arXiv

Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.

Abstract

LLM Pruning

Explore similar work

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

The Functional Structure of Post-Compression Recovery in Low-Rank LLMs

Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization