Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.
Figures & tables
Figure 1: Recovery capacity, accuracy, and throughput during recovery of pruned LLaMA3-8B. Conventional recovery methods use limited recovery capacity relative to the pruned parameters, leading to capacity-knowledge asymmetry. In contrast, OverRep allocates larger recovery capacity and improves both accuracy and throughput, mitigating capacity-knowledge asymmetry.
Figure 2: Comparison between standard recovery methods and OverRep. LoRA updates the frozen weight Pi in the pruned model via a low-rank component Ai⋅Bi , which limits recovery capacity and requires a long backpropagation path. In contrast, OverRep is applied only after pruned layers, introducing training-time components Wi and Di with an annealed activation A . This design enables a shorter backpropagation path. At deployment, OverRep re-parameterizes all additional parameters into P^i , preserving performance while maintaining efficiency.
Figure 3: Training efficiency and recovery performance on LLaMA3-3B and -8B. Gray bands show the range of conventional baselines across resource and performance metrics. OverRep uses more trainable recovery parameters, yet maintains comparable memory and compute costs while improving throughput and accuracy. OverRep ∗ further improves efficiency by caching frozen-prefix activations, without changing the final recovered model or its accuracy.
PR
Method
ARC_C
ARC_E
BoolQ
Hella.
MathQA
MMLU
OBQA
PIQA
RACE
Wino.
Avg.
RP
LLaMA2-7B
25%
LoRA
37.6
65.4
77.6
66.2
25.3
24.6
37.8
72.5
39.0
66.6
51.3
88.7
AdaLoRA
37.5
65.1
77.8
66.1
25.0
24.8
38.0
72.1
39.1
66.1
51.2
88.5
RankAdaptor
39.2
64.4
68.9
66.9
23.4
28.1
37.6
69.6
35.9
62.8
49.7
86.0
RestoreLCC
35.3
61.1
76.3
63.8
25.3
27.3
36.6
71.5
37.4
66.2
50.1
86.7
OverRep
41.0
70.5
73.6
67.7
25.1
38.5
40.2
74.3
37.9
66.1
53.5
92.6
50%
LoRA
27.8
44.3
62.4
45.1
22.0
28.7
30.4
61.0
32.6
59.1
41.3
71.5
Table 1: Experimental results on reasoning tasks reported as accuracy and average RP.
PR
Method
L2-7B
L2-13B
L3-3B
L3-8B
Q3-4B
Q3-8B
25%
LaCo
80.8
88.9
80.7
81.7
73.1
73.4
ShortGPT
88.2
92.1
82.5
88.2
72.6
72.6
Streamline
89.4
94.4
86.1
88.9
74.2
74.4
OverRep
92.6
95.3
90.2
91.3
75.3
80.6
50%
LaCo
70.9
75.2
63.4
62.2
56.7
55.9
ShortGPT
70.7
77.5
61.5
65.0
56.6
55.8
Table 2: Average reasoning performance of complete pruning pipelines, reported as average RP.
PR
Method
L2-7B
L2-13B
L3-3B
L3-8B
Q3-4B
Q3-8B
25%
controlled recovery methods
LoRA
51.2
60.6
35.0
39.9
19.8
23.9
AdaLoRA
51.4
55.6
37.4
47.8
22.2
26.7
RankAdaptor
45.9
60.6
36.7
42.8
17.9
28.2
RestoreLCC
55.2
57.1
37.2
36.4
13.1
18.9
complete pruning pipelines
Table 3: Average generation performance of controlled recovery methods and complete pruning pipelines, reported as average RP.
Configuration
Attn.
MLP
Rea.
Gen.
Train time
Wi
Di
Wi
Di
Plain
–
–
–
–
86.9
35.8
1.8h
Single-component
Attn Wi
✓
–
–
–
86.7
37.4
2.0h
Attn Di
–
✓
–
–
86.9
38.0
1.9h
MLP Wi
–
–
✓
–
88.6
43.5
2.7h
Table 4: Ablation study of the components of OverRep using LLaMA3-3B at 25% pruning. ✓/P/– indicate full application, frozen-block partial application, and no application, respectively. Rea. and Gen. denote the average RP on reasoning and generation benchmarks.
PR
Method
Params. (M)
Rea.
Gen.
Time (h)
25%
LoRA r=16
16.5
84.7
35.0
8.9
LoRA r=630
650
79.4
34.3
13.2
OverRep
650
90.2
47.1
4.6
50%
LoRA r=16
10.4
63.8
9.4
6.2
LoRA r=1000
651
64.8
11.4
9.2
OverRep
650
72.3
10.7
4.6
Table 5: Iso-parameter comparison on LLaMA3-3B. The LoRA rank is scaled to match the ∼650 M trainable parameters of OverRep under the same protocol, reported as average RP.
PR
Method
Rea.
Gen.
Time (h)
25%
LoRA
84.7
35.0
8.9
Full FT
85.9
48.2
12.1
Logit-KD
87.1
58.1
36.4
OverRep
90.2
47.1
4.6
OverRep-KD
91.8
61.0
7.0
50%
LoRA
63.8
9.4
6.2
Table 6: Comparison with full fine-tuning (‘Full FT’) of the retained blocks and full-model logit-level distillation (‘Logit-KD’) on LLaMA3-3B, reported as average RP.
Figure 4: Association between recovery ratio (RR) and retained performance (RP) after pruning. The plots show a positive association between RR and RP on reasoning and generation benchmarks. RR is used as a diagnostic indicator of training-time recovery capacity.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
HF repo.
dim.
Inter dim.
# attn heads
# kv heads
# L
25% # L
50% # L
LLaMA2-7B
meta-llama/Llama-2-7b-hf
4096
11008
32
32
32
24
16
LLaMA2-13B
meta-llama/Llama-2-13b-hf
5120
13824
40
40
40
30
20
LLaMA3-3B
meta-llama/Llama-3.2-3B
3072
8192
24
8
28
19
12
LLaMA3-8B
meta-llama/Llama-3.1-8B
4096
14336
32
8
32
23
14
Qwen3-4B
Qwen/Qwen3-4B-Base
2560
9728
32
8
36
25
15
Qwen3-8B
Qwen/Qwen3-8B-Base
4096
12288
32
8
36
26
16
Appendix
Table 7: Backbone configurations. We report the Hugging Face repository (HF Repo.), detailed backbone configuration, and the number of layers (denoted # L) remaining at 25% and 50% pruning ratios. † MoE backbone with 128 experts and top-8 routing. ‘Inter dim.’ is the per-expert intermediate dimension. ‡ Hybrid backbone whose 52 layers comprise Mamba-2, attention, and MLP sublayers, and ‘Inter dim.’ refers to the MLP layers.
Method
Epoch
#B
LR
Sche.
( r , α )
LLaMA2-7B
Recoveries
10
32
1e-4
cosine
(16, 32)
LaCo
10
32
1e-4
linear
(16, 32)
ShortGPT
10
32
1e-4
linear
(16, 32)
Streamline
100
8
1e-4
cosine
–
OverRep
20
8
1e-4
cosine
–
LLaMA2-13B
Recoveries
10
32
1e-4
cosine
(16, 32)
Appendix
Table 8: Recovery training configurations. ‘Recoveries’ denotes recovery methods such as LoRA, AdaLoRA, RankAdaptor, and RestoreLCC. ‘#B’ denotes the total batch size, and ‘LR’ denotes the learning rate. r and α indicate the LoRA rank and LoRA scaling, respectively.
PR
BoolQ
WSC
COQA
HellaSwag
PIQA
Race-M
Race-H
MMLU
Avg.
Dense (From Streamline)
–
70.8
37.5
66.7
71.3
78.1
33.1
35.5
46.8
55.0
Dense (Ours Rep.)
–
70.4
37.5
67.4
70.9
77.2
33.1
35.5
46.7
54.8
Streamline (From Streamline)
25%
67.5
36.5
59.2
61.1
71.5
34.8
37.0
45.5
51.6
Streamline (Ours Rep.)
25%
66.0
36.5
61.9
64.4
71.9
33.4
30.3
45.8
51.3
Appendix
Table 9: Reproduction verification for Streamline on LLaMA2-7B. ‘(From Streamline)’ denotes the results reported in the original paper, while ‘(Ours Rep.)’ denotes our reproduced models evaluated with the original Streamline evaluation framework. The close agreement supports the fidelity of our baseline reproduction.
Reasoning Task
Generation Task
Method
ARC_C
ARC_E
BoolQ
Hella.
MathQA
MMLU
OBQA
PIQA
RACE
Wino.
Avg.
CoQA
GSM
Triv.QA
Avg.
25%
LoRA
0.143
1.434
1.762
0.497
1.383
1.616
1.490
1.120
1.314
1.797
0.250
2.501
1.413
0.287
1.257
Streamline
0.896
0.657
0.517
0.248
0.379
0.517
0.014
0.248
0.143
1.518
0.162
5.858
1.004
0.014
2.287
OverRep
0.745
0.430
0.657
0.248
0.625
0.287
0.861
0.896
1.762
0.896
0.190
1.004
0.717
0.379
0.564
50%
LoRA
1.034
0.896
2.656
0.430
0.799
0.283
2.068
0.994
1.616
1.597
0.396
2.879
0.896
0.759
1.065
Streamline
0.517
0.287
0.574
0.379
0.287
0.378
1.034
0.574
0.379
0.379
0.102
0.287
0.517
0.248
0.126
Appendix
Table 10: 95% confidence interval for recovery of pruned LLaMA3-3B on reasoning and generation benchmarks.
Method
Trainable Params. (M)
Epochs
Time (h)
LoRA-like
16.5
10
8.9
AdaLoRA
33.0
10
10.6
RankAdaptor
12.2
10
8.7
RestoreLCC
0.1
10
6.8
Streamline ∗
100.7
100
4.6
OverRep ∗
650.1
20
4.6
Appendix
Table 11: Comparison of trainable parameters, recovery epochs, and wall-clock training time for LLaMA3-3B after 25% pruning. Wall-clock time is reported in hours.
Reasoning Task
Generation Task
Method
ARC_C
ARC_E
BoolQ
Hella.
MathQA
MMLU
OBQA
PIQA
RACE
Wino.
Avg.
CoQA
GSM
Triv.QA
Avg.
Dense
46.2
71.8
73.0
73.6
34.4
54.1
43.0
77.3
39.8
69.1
58.2
75.9
25.2
56.3
52.5
25%
LoRA
35.7
54.8
68.2
58.3
26.6
46.3
34.2
69.2
36.2
63.5
49.3
38.4
3.3
13.4
18.4
AdaLoRA
32.3
54.1
64.6
53.9
24.1
47.6
33.2
66.2
34.3
65.8
47.6
42.6
3.0
13.2
19.6
RankAdaptor
35.9
56.5
66.9
58.1
26.2
45.7
32.8
68.4
35.3
63.9
49.0
41.1
3.6
13.0
19.2
RestoreLCC
32.3
49.2
70.4
50.5
24.5
46.7
31.4
64.9
33.7
65.3
46.9
48.0
2.0
8.6
19.5
Appendix
Table 12: Experimental results on reasoning and generation benchmarks for LLaMA3-3B under 10-epoch settings.
Backbone
OverRep Time (h)
LoRA Time (h)
Speedup
Rea.
Gen.
train + cache
(GPUs)
w/o / w/ cache
LoRA
OverRep
LoRA
OverRep
L3-3B
4.6 + 0.8
8.9 (1 GPU)
1.9 × / 1.6 ×
49.3
52.5
18.4
24.7
Q3-14B
12.6 + 2.1
90.2 (11.27h × 8 GPUs)
7.2 × / 6.1 ×
51.8
52.5
30.2
40.0
Q3-30B
8.8 + 3.8
426 (53.3h × 8 GPUs)
48 × / 34 ×
49.8
53.1
23.4
30.1
Appendix
Table 13: End-to-end cost relative to LoRA at a 25% pruning ratio.
Backbone
Recovery
Params. (M)
Time (h)
Rea.
Gen.
L3-3B
Plain
100.7
1.3
50.6
18.8
L3-3B
OverRep
650.1
4.6
52.5
24.7
Q3-14B
Plain
330
1.9
48.5
13.9
Q3-14B
OverRep
2596
12.6
52.5
40.0
Q3-30B
Plain
1246
3.6
49.3
16.3
Q3-30B
OverRep
1752
8.8
53.1
30.1
Appendix
Table 14: Intrinsic overhead of the overcomplete parameterization at a 25% pruning ratio.
Backbone
Method
PR
Rea.
LLaMA3-3B
Dense
-
56.6
LLM-Pruner
25%
40.5
OverRep
25%
46.1
LLaMA3-8B
Dense
-
61.7
LLM-Pruner
24%
47.5
OverRep
24%
51.7
Appendix
Table 15: Experimental results of channel-wise pruning with LLM-Pruner and OverRep.
Backbone
Method
Rea.
Gen.
Time (h)
Q3-30B
Dense
67.9
78.5
-
25%
Q3-30B
LoRA
49.8
23.4
53.3h × 8 GPUs
Q3-30B
Streamline
49.5
22.2
7.3h
Q3-30B
OverRep
53.1
30.1
8.8h
50%
Q3-30B
LoRA
43.6
9.9
35.6h × 8 GPUs
Q3-30B
Streamline
42.9
7.2
6.0h
Appendix
Table 16: Experimental results across architectures. ‘Time’ denotes wall-clock training time × GPUs; unless otherwise specified, results use a single GPU.
Method
Rea.
Gen.
Time (h)
Dense
60.5
45.8
-
25%
LoRA
49.1
28.8
9.0
OverRep
49.9
24.5
3.3
OverRep-KD
50.1
31.5
5.2
50%
LoRA
39.7
11.1
6.0
OverRep
42.9
11.2
3.3
Appendix
Table 17: Experimental results on Nemotron-H-4B.
Backbone
Criterion
Rea.
Gen.
Own
OverRep
Own
OverRep
25%
L3-3B
ShortGPT
48.0
50.6
13.9
25.6
L3-3B
Streamline
50.2
52.5
24.0
24.7
Q3-4B
ShortGPT
46.7
45.8
15.5
22.5
Q3-4B
Streamline
47.7
48.9
20.8
29.1
50%
L3-3B
ShortGPT
35.8
40.5
2.1
5.5
Appendix
Table 18: Compatibility experiment with existing pruning criteria. ‘Own’ is the criterion’s own recovery, and ‘OverRep’ is OverRep recovery on the same mask.
PR
Method
ARC_C
ARC_E
BoolQ
Hella.
MathQA
MMLU
OBQA
PIQA
RACE
Wino.
Avg.
RP
LLaMA2-7B
-
Dense
46.2
76.2
77.9
76.0
28.5
40.8
44.2
79.1
39.5
69.5
57.8
100.0
25%
LoRA
37.6
65.4
77.6
66.2
25.3
24.6
37.8
72.5
39.0
66.6
51.3
88.7
AdaLoRA
37.5
65.1
77.8
66.1
25.0
24.8
38.0
72.1
39.1
66.1
51.2
88.5
RankAdaptor
39.2
64.4
68.9
66.9
23.4
28.1
37.6
69.6
35.9
62.8
49.7
86.0
RestoreLCC
35.3
61.1
76.3
63.8
25.3
27.3
36.6
71.5
37.4
66.2
50.1
86.7
LaCo
31.7
63.1
62.5
59.6
24.3
24.7
38.4
73.3
35.1
54.0
46.7
80.8
Appendix
Table 19: Experimental results of reasoning tasks on LLaMA2 and LLaMA3 reported as accuracy and RP.
PR
Method
ARC_C
ARC_E
BoolQ
Hella.
MathQA
MMLU
OBQA
PIQA
RACE
Wino.
Avg.
RP
Qwen3-4B
-
Dense
51.7
78.8
83.2
73.6
54.0
71.2
41.2
78.2
40.7
70.5
64.3
100.0
25%
LoRA
37.5
59.5
63.9
53.9
26.1
36.9
33.0
66.1
36.0
63.6
47.7
74.1
AdaLoRA
32.3
53.8
71.7
49.8
26.5
32.0
29.8
63.9
34.3
63.7
45.8
71.2
RankAdaptor
36.6
57.9
65.0
53.9
25.7
36.3
33.2
65.4
34.4
63.0
47.1
73.3
RestoreLCC
31.2
40.8
70.1
40.5
23.5
23.2
31.6
59.4
26.8
60.4
40.8
63.4
LaCo
35.3
63.0
68.2
51.3
26.2
31.7
34.0
68.8
34.5
57.3
47.0
73.1
Appendix
Table 20: Experimental results of reasoning tasks on Qwen3 reported as accuracy and RP.
PR
Method
CoQA
GSM8K
TriviaQA
Avg.
RP
LLaMA2-7B
-
Dense
75.4
13.9
59.9
49.7
100.0
25%
LoRA
54.2
2.0
20.2
25.5
51.2
AdaLoRA
54.4
2.4
19.9
25.6
51.4
RankAdaptor
47.5
2.1
18.9
22.8
45.9
RestoreLCC
62.8
1.7
17.9
27.5
55.2
LaCo
19.4
0.6
37.0
19.0
38.2
Appendix
Table 21: Experimental results of generation tasks on LLaMA2 and LLaMA3 reported using task-specific metrics and RP.
PR
Method
CoQA
GSM8K
TriviaQA
Avg.
RP
Qwen3-4B
-
Dense
83.2
84.7
50.6
72.8
100.0
25%
LoRA
33.6
1.4
8.2
14.4
19.8
AdaLoRA
37.9
2.1
8.5
16.2
22.2
RankAdaptor
30.1
2.1
6.9
13.0
17.9
RestoreLCC
23.7
1.1
3.8
9.5
13.1
LaCo
25.6
0.8
13.2
13.2
18.1
Appendix
Table 22: Experimental results of generation tasks on Qwen3 reported with task-specific metrics and RP.
Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B, LayerTrim achieves 1.24× faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.
Woosang Lim, Haneul Yoo, Vincent-Daniel Yun +1
Seoul National University, Republic of Korea · New York University, United States · University of Southern California, United States
Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets. We present SNIPER, a two-stage structured pruning framework that solves a knapsack optimization over coarse-granularity components to yield conditionally optimal parameter allocations with respect to fixed importance estimates, followed by a fine-grained pruning stage to meet strict budget constraints. We introduce the Compression Ratio Adherence Factor (CRAFT) to quantify budget fidelity, showing that while existing pruners deviate from target compression ratios by up to 33%, SNIPER achieves near-exact adherence with a CRAFT score of 0.98. Evaluations across four diverse architectures over a set of 18 tasks spanning five domains demonstrate SNIPER's consistent improvements in average performance retention and task-level stability over six state-of-the-art pruners. Across all pruning configurations, SNIPER achieves an excellent mean rank of 1.25, indicating its robust cross-architectural generalizability and excellent reliability.
Palaash Goel, Ayan Sengupta, Akshay Nambi +1
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi · Microsoft Research (MSR) India
Structural pruning followed by a recovery stage is the standard recipe for compressing large language models (LLMs), yet the two stages are typically designed and tuned in isolation. In this work, we study how their interaction shapes the compressed model, sweeping ordering, the data that drives each stage, and the alignment of their training objectives. We find that instruction-following is the first capability to collapse under pruning, well before knowledge, reasoning, or core language modeling, and that much of the pruning in-domain loss is likely not knowledge loss but a domain-agnostic failure to respond coherently to the prompt. We find in our setting that in-domain data is the strongest pruning signal, whereas instruction-style data recovers the most. Stage order matters little, but only marginally. We further analyze how the importance ranking drifts before and after pruning. Together these results give practitioners a concrete recipe for composing pruning and recovery.
Luis Frentzen Salim, Komang Rinartha, Chia-Chun Wu +3
Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute