Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Seungmin Oh; Donggeon Lee; Jongbin Ryu

cs.CLSep 7, 2026

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Authors: Seungmin Oh, Donggeon Lee, Jongbin Ryu

Organizations: Ajou University, South Korea

Abstract

Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.

Figures & tables

Figure 1: Recovery capacity, accuracy, and throughput during recovery of pruned LLaMA3-8B. Conventional recovery methods use limited recovery capacity relative to the pruned parameters, leading to capacity-knowledge asymmetry. In contrast, OverRep allocates larger recovery capacity and improves both accuracy and throughput, mitigating capacity-knowledge asymmetry.

Panel of Figure 2 — Figure 2: Comparison between standard recovery methods and OverRep. LoRA updates the frozen weight $\mathbf{P}_{i}$ in the pruned model via a low-rank component $\mathbf{A}_{i}\cdot\mathbf{B}_{i}$ , which limits recovery capacity and requires a long backpropagation path. In contrast, OverRep is applied only after pruned layers, introducing training-time components $\mathbf{W}_{i}$ and $\mathbf{D}_{i}$ with an annealed activation ${\mathcal{A}}$ . This design enables a shorter backpropagation path. At deployment, OverRep re-parameterizes all additional parameters into $\hat{\mathbf{P}}_{i}$ , preserving performance while maintaining efficiency.

Figure 3: Training efficiency and recovery performance on LLaMA3-3B and -8B. Gray bands show the range of conventional baselines across resource and performance metrics. OverRep uses more trainable recovery parameters, yet maintains comparable memory and compute costs while improving throughput and accuracy. OverRep ∗ further improves efficiency by caching frozen-prefix activations, without changing the final recovered model or its accuracy.

	PR	Method	ARC_C	ARC_E	BoolQ	Hella.	MathQA	MMLU	OBQA	PIQA	RACE	Wino.	Avg.	RP
LLaMA2-7B	25%	LoRA	37.6	65.4	77.6	66.2	25.3	24.6	37.8	72.5	39.0	66.6	51.3	88.7
		AdaLoRA	37.5	65.1	77.8	66.1	25.0	24.8	38.0	72.1	39.1	66.1	51.2	88.5
		RankAdaptor	39.2	64.4	68.9	66.9	23.4	28.1	37.6	69.6	35.9	62.8	49.7	86.0
		RestoreLCC	35.3	61.1	76.3	63.8	25.3	27.3	36.6	71.5	37.4	66.2	50.1	86.7
		OverRep	41.0	70.5	73.6	67.7	25.1	38.5	40.2	74.3	37.9	66.1	53.5	92.6
	50%	LoRA	27.8	44.3	62.4	45.1	22.0	28.7	30.4	61.0	32.6	59.1	41.3	71.5

Table 1: Experimental results on reasoning tasks reported as accuracy and average RP.

PR	Method	L2-7B	L2-13B	L3-3B	L3-8B	Q3-4B	Q3-8B
25%	LaCo	80.8	88.9	80.7	81.7	73.1	73.4
	ShortGPT	88.2	92.1	82.5	88.2	72.6	72.6
	Streamline	89.4	94.4	86.1	88.9	74.2	74.4
	OverRep	92.6	95.3	90.2	91.3	75.3	80.6
50%	LaCo	70.9	75.2	63.4	62.2	56.7	55.9
	ShortGPT	70.7	77.5	61.5	65.0	56.6	55.8

Table 2: Average reasoning performance of complete pruning pipelines, reported as average RP.

PR	Method	L2-7B	L2-13B	L3-3B	L3-8B	Q3-4B	Q3-8B
25%		controlled recovery methods
	LoRA	51.2	60.6	35.0	39.9	19.8	23.9
	AdaLoRA	51.4	55.6	37.4	47.8	22.2	26.7
	RankAdaptor	45.9	60.6	36.7	42.8	17.9	28.2
	RestoreLCC	55.2	57.1	37.2	36.4	13.1	18.9
		complete pruning pipelines

Table 3: Average generation performance of controlled recovery methods and complete pruning pipelines, reported as average RP.

Configuration	Attn.		MLP		Rea.	Gen.	Train time
Configuration	$\mathbf{W}_{i}$	$\mathbf{D}_{i}$	$\mathbf{W}_{i}$	$\mathbf{D}_{i}$	Rea.	Gen.	Train time
Plain	–	–	–	–	86.9	35.8	1.8h
Single-component
Attn $\mathbf{W}_{i}$	✓	–	–	–	86.7	37.4	2.0h
Attn $\mathbf{D}_{i}$	–	✓	–	–	86.9	38.0	1.9h
MLP $\mathbf{W}_{i}$	–	–	✓	–	88.6	43.5	2.7h

Table 4: Ablation study of the components of OverRep using LLaMA3-3B at 25% pruning. ✓/P/– indicate full application, frozen-block partial application, and no application, respectively. Rea. and Gen. denote the average RP on reasoning and generation benchmarks.

PR	Method	Params. (M)	Rea.	Gen.	Time (h)
25%	LoRA $r=16$	16.5	84.7	35.0	8.9
	LoRA $r=630$	650	79.4	34.3	13.2
	OverRep	650	90.2	47.1	4.6
50%	LoRA $r=16$	10.4	63.8	9.4	6.2
	LoRA $r=1000$	651	64.8	11.4	9.2
	OverRep	650	72.3	10.7	4.6

Table 5: Iso-parameter comparison on LLaMA3-3B. The LoRA rank is scaled to match the

{\sim}650

M trainable parameters of OverRep under the same protocol, reported as average RP.

PR	Method	Rea.	Gen.	Time (h)
25%	LoRA	84.7	35.0	8.9
	Full FT	85.9	48.2	12.1
	Logit-KD	87.1	58.1	36.4
	OverRep	90.2	47.1	4.6
	OverRep-KD	91.8	61.0	7.0
50%	LoRA	63.8	9.4	6.2

Table 6: Comparison with full fine-tuning (‘Full FT’) of the retained blocks and full-model logit-level distillation (‘Logit-KD’) on LLaMA3-3B, reported as average RP.

Figure 4: Association between recovery ratio (RR) and retained performance (RP) after pruning. The plots show a positive association between RR and RP on reasoning and generation benchmarks. RR is used as a diagnostic indicator of training-time recovery capacity.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	HF repo.	dim.	Inter dim.	# attn heads	# kv heads	# L	25% # L	50% # L
LLaMA2-7B	meta-llama/Llama-2-7b-hf	4096	11008	32	32	32	24	16
LLaMA2-13B	meta-llama/Llama-2-13b-hf	5120	13824	40	40	40	30	20
LLaMA3-3B	meta-llama/Llama-3.2-3B	3072	8192	24	8	28	19	12
LLaMA3-8B	meta-llama/Llama-3.1-8B	4096	14336	32	8	32	23	14
Qwen3-4B	Qwen/Qwen3-4B-Base	2560	9728	32	8	36	25	15
Qwen3-8B	Qwen/Qwen3-8B-Base	4096	12288	32	8	36	26	16

Appendix

Table 7: Backbone configurations. We report the Hugging Face repository (HF Repo.), detailed backbone configuration, and the number of layers (denoted # L) remaining at 25% and 50% pruning ratios. † MoE backbone with 128 experts and top-8 routing. ‘Inter dim.’ is the per-expert intermediate dimension. ‡ Hybrid backbone whose 52 layers comprise Mamba-2, attention, and MLP sublayers, and ‘Inter dim.’ refers to the MLP layers.

	Method	Epoch	#B	LR	Sche.	( $r$ , $\alpha$ )
LLaMA2-7B	Recoveries	10	32	1e-4	cosine	(16, 32)
	LaCo	10	32	1e-4	linear	(16, 32)
	ShortGPT	10	32	1e-4	linear	(16, 32)
	Streamline	100	8	1e-4	cosine	–
	OverRep	20	8	1e-4	cosine	–
LLaMA2-13B	Recoveries	10	32	1e-4	cosine	(16, 32)

Appendix

Table 8: Recovery training configurations. ‘Recoveries’ denotes recovery methods such as LoRA, AdaLoRA, RankAdaptor, and RestoreLCC. ‘#B’ denotes the total batch size, and ‘LR’ denotes the learning rate.

r

and

\alpha

indicate the LoRA rank and LoRA scaling, respectively.

	PR	BoolQ	WSC	COQA	HellaSwag	PIQA	Race-M	Race-H	MMLU	Avg.
Dense (From Streamline)	–	70.8	37.5	66.7	71.3	78.1	33.1	35.5	46.8	55.0
Dense (Ours Rep.)	–	70.4	37.5	67.4	70.9	77.2	33.1	35.5	46.7	54.8
Streamline (From Streamline)	25%	67.5	36.5	59.2	61.1	71.5	34.8	37.0	45.5	51.6
Streamline (Ours Rep.)	25%	66.0	36.5	61.9	64.4	71.9	33.4	30.3	45.8	51.3

Appendix

Table 9: Reproduction verification for Streamline on LLaMA2-7B. ‘(From Streamline)’ denotes the results reported in the original paper, while ‘(Ours Rep.)’ denotes our reproduced models evaluated with the original Streamline evaluation framework. The close agreement supports the fidelity of our baseline reproduction.

		Reasoning Task											Generation Task
	Method	ARC_C	ARC_E	BoolQ	Hella.	MathQA	MMLU	OBQA	PIQA	RACE	Wino.	Avg.	CoQA	GSM	Triv.QA	Avg.
25%	LoRA	0.143	1.434	1.762	0.497	1.383	1.616	1.490	1.120	1.314	1.797	0.250	2.501	1.413	0.287	1.257
	Streamline	0.896	0.657	0.517	0.248	0.379	0.517	0.014	0.248	0.143	1.518	0.162	5.858	1.004	0.014	2.287
	OverRep	0.745	0.430	0.657	0.248	0.625	0.287	0.861	0.896	1.762	0.896	0.190	1.004	0.717	0.379	0.564
50%	LoRA	1.034	0.896	2.656	0.430	0.799	0.283	2.068	0.994	1.616	1.597	0.396	2.879	0.896	0.759	1.065
	Streamline	0.517	0.287	0.574	0.379	0.287	0.378	1.034	0.574	0.379	0.379	0.102	0.287	0.517	0.248	0.126

Appendix

Table 10: 95% confidence interval for recovery of pruned LLaMA3-3B on reasoning and generation benchmarks.

Method	Trainable Params. (M)	Epochs	Time (h)
LoRA-like	16.5	10	8.9
AdaLoRA	33.0	10	10.6
RankAdaptor	12.2	10	8.7
RestoreLCC	0.1	10	6.8
Streamline ∗	100.7	100	4.6
OverRep ∗	650.1	20	4.6

Appendix

Table 11: Comparison of trainable parameters, recovery epochs, and wall-clock training time for LLaMA3-3B after 25% pruning. Wall-clock time is reported in hours.

		Reasoning Task											Generation Task
	Method	ARC_C	ARC_E	BoolQ	Hella.	MathQA	MMLU	OBQA	PIQA	RACE	Wino.	Avg.	CoQA	GSM	Triv.QA	Avg.
	Dense	46.2	71.8	73.0	73.6	34.4	54.1	43.0	77.3	39.8	69.1	58.2	75.9	25.2	56.3	52.5
25%	LoRA	35.7	54.8	68.2	58.3	26.6	46.3	34.2	69.2	36.2	63.5	49.3	38.4	3.3	13.4	18.4
	AdaLoRA	32.3	54.1	64.6	53.9	24.1	47.6	33.2	66.2	34.3	65.8	47.6	42.6	3.0	13.2	19.6
	RankAdaptor	35.9	56.5	66.9	58.1	26.2	45.7	32.8	68.4	35.3	63.9	49.0	41.1	3.6	13.0	19.2
	RestoreLCC	32.3	49.2	70.4	50.5	24.5	46.7	31.4	64.9	33.7	65.3	46.9	48.0	2.0	8.6	19.5

Appendix

Table 12: Experimental results on reasoning and generation benchmarks for LLaMA3-3B under 10-epoch settings.

Backbone	OverRep Time (h)	LoRA Time (h)	Speedup	Rea.		Gen.
Backbone	train + cache	(GPUs)	w/o / w/ cache	LoRA	OverRep	LoRA	OverRep
L3-3B	4.6 + 0.8	8.9 (1 GPU)	1.9 $\times$ / 1.6 $\times$	49.3	52.5	18.4	24.7
Q3-14B	12.6 + 2.1	90.2 (11.27h $\times$ 8 GPUs)	7.2 $\times$ / 6.1 $\times$	51.8	52.5	30.2	40.0
Q3-30B	8.8 + 3.8	426 (53.3h $\times$ 8 GPUs)	48 $\times$ / 34 $\times$	49.8	53.1	23.4	30.1

Appendix

Table 13: End-to-end cost relative to LoRA at a 25% pruning ratio.

Backbone	Recovery	Params. (M)	Time (h)	Rea.	Gen.
L3-3B	Plain	100.7	1.3	50.6	18.8
L3-3B	OverRep	650.1	4.6	52.5	24.7
Q3-14B	Plain	330	1.9	48.5	13.9
Q3-14B	OverRep	2596	12.6	52.5	40.0
Q3-30B	Plain	1246	3.6	49.3	16.3
Q3-30B	OverRep	1752	8.8	53.1	30.1

Appendix

Table 14: Intrinsic overhead of the overcomplete parameterization at a 25% pruning ratio.

Backbone	Method	PR	Rea.
LLaMA3-3B	Dense	-	56.6
	LLM-Pruner	25%	40.5
	OverRep	25%	46.1
LLaMA3-8B	Dense	-	61.7
	LLM-Pruner	24%	47.5
	OverRep	24%	51.7

Appendix

Table 15: Experimental results of channel-wise pruning with LLM-Pruner and OverRep.

	Backbone	Method	Rea.	Gen.	Time (h)
	Q3-30B	Dense	67.9	78.5	-
25%	Q3-30B	LoRA	49.8	23.4	53.3h $\times$ 8 GPUs
	Q3-30B	Streamline	49.5	22.2	7.3h
	Q3-30B	OverRep	53.1	30.1	8.8h
50%	Q3-30B	LoRA	43.6	9.9	35.6h $\times$ 8 GPUs
	Q3-30B	Streamline	42.9	7.2	6.0h

Appendix

Table 16: Experimental results across architectures. ‘Time’ denotes wall-clock training time

\times

GPUs; unless otherwise specified, results use a single GPU.

	Method	Rea.	Gen.	Time (h)
	Dense	60.5	45.8	-
25%	LoRA	49.1	28.8	9.0
	OverRep	49.9	24.5	3.3
	OverRep-KD	50.1	31.5	5.2
50%	LoRA	39.7	11.1	6.0
	OverRep	42.9	11.2	3.3

Appendix

Table 17: Experimental results on Nemotron-H-4B.

	Backbone	Criterion	Rea.		Gen.
	Backbone	Criterion	Own	OverRep	Own	OverRep
25%	L3-3B	ShortGPT	48.0	50.6	13.9	25.6
	L3-3B	Streamline	50.2	52.5	24.0	24.7
	Q3-4B	ShortGPT	46.7	45.8	15.5	22.5
	Q3-4B	Streamline	47.7	48.9	20.8	29.1
50%	L3-3B	ShortGPT	35.8	40.5	2.1	5.5

Appendix

Table 18: Compatibility experiment with existing pruning criteria. ‘Own’ is the criterion’s own recovery, and ‘OverRep’ is OverRep recovery on the same mask.

	PR	Method	ARC_C	ARC_E	BoolQ	Hella.	MathQA	MMLU	OBQA	PIQA	RACE	Wino.	Avg.	RP
LLaMA2-7B	-	Dense	46.2	76.2	77.9	76.0	28.5	40.8	44.2	79.1	39.5	69.5	57.8	100.0
	25%	LoRA	37.6	65.4	77.6	66.2	25.3	24.6	37.8	72.5	39.0	66.6	51.3	88.7
		AdaLoRA	37.5	65.1	77.8	66.1	25.0	24.8	38.0	72.1	39.1	66.1	51.2	88.5
		RankAdaptor	39.2	64.4	68.9	66.9	23.4	28.1	37.6	69.6	35.9	62.8	49.7	86.0
		RestoreLCC	35.3	61.1	76.3	63.8	25.3	27.3	36.6	71.5	37.4	66.2	50.1	86.7
		LaCo	31.7	63.1	62.5	59.6	24.3	24.7	38.4	73.3	35.1	54.0	46.7	80.8

Appendix

Table 19: Experimental results of reasoning tasks on LLaMA2 and LLaMA3 reported as accuracy and RP.

	PR	Method	ARC_C	ARC_E	BoolQ	Hella.	MathQA	MMLU	OBQA	PIQA	RACE	Wino.	Avg.	RP
Qwen3-4B	-	Dense	51.7	78.8	83.2	73.6	54.0	71.2	41.2	78.2	40.7	70.5	64.3	100.0
	25%	LoRA	37.5	59.5	63.9	53.9	26.1	36.9	33.0	66.1	36.0	63.6	47.7	74.1
		AdaLoRA	32.3	53.8	71.7	49.8	26.5	32.0	29.8	63.9	34.3	63.7	45.8	71.2
		RankAdaptor	36.6	57.9	65.0	53.9	25.7	36.3	33.2	65.4	34.4	63.0	47.1	73.3
		RestoreLCC	31.2	40.8	70.1	40.5	23.5	23.2	31.6	59.4	26.8	60.4	40.8	63.4
		LaCo	35.3	63.0	68.2	51.3	26.2	31.7	34.0	68.8	34.5	57.3	47.0	73.1

Appendix

Table 20: Experimental results of reasoning tasks on Qwen3 reported as accuracy and RP.

	PR	Method	CoQA	GSM8K	TriviaQA	Avg.	RP
LLaMA2-7B	-	Dense	75.4	13.9	59.9	49.7	100.0
	25%	LoRA	54.2	2.0	20.2	25.5	51.2
		AdaLoRA	54.4	2.4	19.9	25.6	51.4
		RankAdaptor	47.5	2.1	18.9	22.8	45.9
		RestoreLCC	62.8	1.7	17.9	27.5	55.2
		LaCo	19.4	0.6	37.0	19.0	38.2

Appendix

Table 21: Experimental results of generation tasks on LLaMA2 and LLaMA3 reported using task-specific metrics and RP.

	PR	Method	CoQA	GSM8K	TriviaQA	Avg.	RP
Qwen3-4B	-	Dense	83.2	84.7	50.6	72.8	100.0
	25%	LoRA	33.6	1.4	8.2	14.4	19.8
		AdaLoRA	37.9	2.1	8.5	16.2	22.2
		RankAdaptor	30.1	2.1	6.9	13.0	17.9
		RestoreLCC	23.7	1.1	3.8	9.5	13.1
		LaCo	25.6	0.8	13.2	13.2	18.1

Appendix

Table 22: Experimental results of generation tasks on Qwen3 reported with task-specific metrics and RP.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
Oct 13, 2026Woosang Lim, Haneul Yoo, Vincent-Daniel Yun +1LLM Pruning LLM Compression
Seoul National University, Republic of Korea · New York University, United States · University of Southern California, United States
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Aug 13, 2026Palaash Goel, Ayan Sengupta, Akshay Nambi +1LLM Pruning Structured Pruning
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi · Microsoft Research (MSR) India
The Fragility and Dynamics of LLM Pruning-Recovery Steps
Oct 11, 2026Luis Frentzen Salim, Komang Rinartha, Chia-Chun Wu +3LLM Pruning LLM Compression
Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute

Oct 13, 2026J/K move · Enter open · S save

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo

Seoul National University, Republic of Korea · New York University, United States · University of Southern California, United States

LLM Pruning LLM Compression Efficient Language Model Reasoning LLM Inference Acceleration

Open paper AskPDF arXiv

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B, LayerTrim achieves 1.24

\times

faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.

Abstract

LLM Pruning

Explore similar work

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

The Fragility and Dynamics of LLM Pruning-Recovery Steps