Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan
Layer-level pruning is attractive for large language model (LLM) deployment as it reduces the model size while preserving a dense architecture that can be efficiently executed on standard hardware. Current methods typically follow a one-shot paradigm, which first removes selected layers to reach a target model size and then performs additional training to recover model capabilities. However, pruning directly to the target size can substantially degrade reasoning capabilities, making subsequent recovery more difficult under a limited training budget. Moreover, this paradigm separates layer selection from recovery, limiting the use of recovery information to guide selection and potentially hindering the recovery of reasoning capabilities. To address these limitations, we propose LOOP, a Layer-level Recovery-Oriented On-Policy Pruning framework that interleaves pruning and recovery across multiple shots. In each shot, LOOP removes a single layer and performs short on-policy distillation, mitigating the abrupt degradation in reasoning capabilities associated with one-shot pruning. To guide layer selection in subsequent shots, LOOP reuses on-policy trajectories and optimization signals from recovery to jointly assess each remaining layer's reasoning contribution and recovery sensitivity, thereby making layer selection recovery-aware. Evaluations on several benchmarks show that LOOP consistently outperforms one-shot baselines in average accuracy across different model scales and pruning ratios.
Figures & tables
Figure 1: Overview of LOOP. The top shows the iterative pruning and recovery process, where each shot removes one layer, performs short on-policy recovery, and updates layer importance for the next shot. The bottom details the recovery-oriented importance criterion, which estimates reasoning contribution and recovery sensitivity, aggregates and combines them into an importance score.
Ratio
Method
Recovery
Metric
DAPO
Math-500
AMC
AIME
Mean
T (%)
0%
Original
–
–
0.472
0.784
0.650
0.256
0.5405
100.000
15%
Pruned
–
BI
0.032
0.138
0.000
0.000
0.0425
7.865
SFT w/o KD
SFT
BI
0.143
0.464
0.175
0.033
0.2038
37.719
KD
Off-Policy
BI
0.124
0.478
0.175
0.011
0.1970
36.454
OPD
On-Policy
BI
0.332
0.604
0.450
0.122
0.3771
69.769
LOOP (Ours)
On-Policy
BI + RS
0.348
0.658
0.575
0.222
0.4508
83.404
Table 1: Performance of Qwen3-4B-Instruct under different layer-pruning ratios and recovery strategies on mathematical reasoning benchmarks. All results are measured by accuracy. Mean denotes the average accuracy across the four benchmarks, and T denotes the compressed model’s mean performance relative to that of the original model.
Model
Method
Recovery
Metric
DAPO
Math-500
AMC
AIME
Mean
T (%)
Qwen3-1.7B
Original
–
–
0.063
0.242
0.150
0.011
0.1165
100.000
Pruned
–
BI
0.001
0.008
0.000
0.000
0.0023
1.931
SFT w/o KD
SFT
BI
0.031
0.130
0.000
0.000
0.0403
34.542
KD
Off-Policy
BI
0.028
0.122
0.025
0.011
0.0465
39.929
OPD
On-Policy
BI
0.011
0.154
0.075
0.000
0.0600
51.491
LOOP (Ours)
On-Policy
BI + RS
0.032
0.190
0.050
0.011
0.0708
60.717
Table 2: Performance of different-sized Qwen3 models under a 20% layer-pruning ratio on mathematical reasoning benchmarks. All results are measured by accuracy. The best and second-best results among the reported compression methods are highlighted in bold and underlined , respectively.
Ratio
Method
Metric
Biology
Chemistry
Materials
Physics
Mean
T (%)
0%
Original
–
0.200
0.138
0.574
0.400
0.3281
100.000
15%
OPD
BI
0.040
0.090
0.521
0.163
0.2036
62.040
LOOP (Ours)
BI + RS
0.180
0.126
0.521
0.188
0.2536
77.296
20%
OPD
BI
0.040
0.090
0.447
0.158
0.1837
55.985
LOOP (Ours)
BI + RS
0.160
0.100
0.479
0.150
0.2222
67.714
25%
OPD
BI
0.080
0.111
0.389
0.138
0.1795
54.704
Table 3: Performance of Qwen3-4B-Instruct under different layer-pruning ratios on scientific reasoning benchmarks ( Feng et al., 2025 ) . All results are measured by accuracy.
Iterative Pruning
Combined Importance
DAPO
Math-500
AMC
AIME
Mean
✓
✓
0.238
0.522
0.425
0.122
0.3268
✓
✗
0.261
0.514
0.375
0.078
0.3069
✗
✓
0.185
0.482
0.325
0.078
0.2674
✗
✗
0.139
0.468
0.325
0.011
0.2358
Table 4: Ablation study of LOOP using Qwen3-4B-Instruct at a 20% compression ratio. Combined Importance indicates whether recovery sensitivity is incorporated with BI for layer selection.
Table 6
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7
Model
Method
Recovery
Metric
DAPO
Math-500
AMC
AIME
Mean
T (%)
Qwen3-8B
Original
–
–
0.197
0.414
0.325
0.056
0.2479
100.00
OPD Trained
–
–
0.273
0.510
0.425
0.211
0.3548
143.120
Pruned
–
BI
0.007
0.010
0.000
0.000
0.0043
1.71
OPD
On-Policy
BI
0.071
0.338
0.175
0.011
0.1488
60.01
LOOP (Ours)
On-Policy
BI + RS
0.108
0.440
0.350
0.156
0.2634
106.25
Appendix
Table 7: Comparison of OPD training without pruning and pruning-based recovery methods on Qwen3-8B. Pruned methods use a 20% layer-pruning ratio.
Ratio
Method
Recovery Data
Metric
Biology
Chemistry
Materials
Physics
Mean
T (%)
0%
Original
–
–
0.200
0.138
0.574
0.400
0.3281
100.000
20%
OPD
Science
BI
0.040
0.090
0.447
0.158
0.1837
55.985
LOOP (Ours)
Science
BI + RS
0.160
0.100
0.479
0.150
0.2222
67.714
OPD
Math
BI
0.000
0.000
0.064
0.050
0.0285
8.673
LOOP (Ours)
Math
BI + RS
0.000
0.000
0.255
0.125
0.0951
28.977
Appendix
Table 8: Cross-domain scientific reasoning performance of Qwen3-4B-Instruct at a 20% layer-pruning ratio. All results are reported using accuracy.
Method
Recovery
Metric
DAPO
Math-500
AMC
AIME
Mean
T (%)
Original
–
–
0.160
0.452
0.275
0.022
0.2273
100.000
OPD
On-Policy
BI
0.015
0.194
0.100
0.000
0.0773
33.985
LOOP (Ours)
On-Policy
BI + RS
0.097
0.200
0.150
0.005
0.1130
49.713
Appendix
Table 9: Performance of Llama-3.1-8B on mathematical reasoning benchmarks. All results are measured by accuracy.
Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.
Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
Houcheng Jiang, Mao Zheng, Mingyang Song +4
Zhongguancun Academy · Foundation Model Department, Tencent · University of Science and Technology of China +1
Large Language Models (LLMs) exhibit strong reasoning abilities, but their high computational costs limit their practical deployment. Recent studies reveal significant redundancy in LLMs layers, making layer pruning an active research topic. Layer pruning research primarily focuses on two aspects: measuring layer importance and recovering performance after pruning. Unfortunately, the present works fail to simultaneously maintain pruning performance and efficiency. In this study, we propose GradMAP, a faster layer pruning method with Gradient Metric And Projection compensation, which consists of two stages. In the first stage, we introduce a novel metric based on gradient magnitudes, enabling a global assessment of layer importance. Note that, it requires only a single backward propagation step per pruning decision, substantially enhancing pruning efficiency. In the second stage, we first analyze the layers with the largest mean shift resulting from pruning, and then incorporate a simple yet effective projection compensation matrix to correct this drift in one step. In this way, the degradation of model performance caused by layer pruning is effectively alleviated. Extensive experiments show that GradMAP outperforms previous layer pruning methods in both pruning speed (achieving an average 4× speedup) and performance.
Hao Liu, Guangyan Li, Wensheng Zhang +1
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Guangzhou University, Guangzhou, China