As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by directly reducing memory and latency without hardware-specific support. However, we observe that transformer layers exhibit markedly different task-dependent layer contributions, whereas existing depth-pruning methods ignore this heterogeneity and thus tend to optimize for a narrow domain or benchmark rather than preserve broad capability. This suggests that LLM depth pruning should be formulated as a multi-objective optimization problem rather than a single-score ranking problem. We propose SliMOO, an interpretable multi-objective evolutionary search framework for LLM depth pruning. SliMOO first uses single-layer removal probing to construct a task-layer importance map, which provides an interpretable view of task-shared and task-specific layers and serves as a search prior. It then performs a Pareto-aware evolutionary search for pruning masks, where candidates are generated under the guidance of the task-layer prior, scored by their deviation from the dense model on each task, and retained through NSGA-II environmental selection. Across the Llama-3.1 and Qwen3 model families, SliMOO consistently achieves stronger multi-task trade-offs than rule-based, greedy, and scalarized baselines. For example, on Llama-3.1-70B at 25% sparsity, it improves the math-domain average from 24.33% to 34.40% and the code-domain average from 35.03% to 39.14%. Code will be made available at https://github.com/AMD-AGI/SliMOO.
Figures & tables
Figure 1: Normalized cross-domain trade-offs on Llama-3.1-70B. Scores are normalized within each setting; language modeling perplexity is inverted before normalization.
Figure 2: Overview of SliMOO. (a) TLIP profiles task-wise importance of layers/sublayers, yielding a task-layer importance map I . (b) Guided by the task-layer prior, SliMOO generates feasible pruning masks and uses NSGA-II to retain Pareto-competitive trade-offs under a target sparsity.
Model
Sparsity
Method
Language Avg. ↓
Math Avg. ↑%
Code Avg. ↑%
Knowledge Avg. ↑%
Llama-3.1-70B
0%
-
5.73
52.48
58.32
72.25
25% Layer
ShortGPT
11.35
12.85
15.45
65.93
LaCo
11.20
16.00
20.39
66.22
LLM-Pruner
8.74
32.44
26.78
66.78
EvoPress
9.88
24.33
35.03
64.63
SliMOO
9.89
34.40
39.14
66.90
Table 1: Post-pruning performance on the Llama family. We report domain averages: lower is better for language modeling and higher is better for math, code, and knowledge reasoning. Bold indicates the best result, and underlines indicate the runner-up.
Figure 3: Search dynamics of SliMOO across four capability domains. The horizontal axis denotes the generation index, and the vertical axis reports the corresponding domain-specific search objectives used during optimization; lower values indicate smaller distributional deviation from the dense model.
Sparsity
Method
Profiled capability domains
Out-of-domain benchmarks
Lang. ↓
Math ↑
Code ↑
Know. ↑
RULER NIAH-8K ↑
TruthfulQA MC2 ↑
CrowS Pairs †
ToxiGen ↑
MedMCQA ↑
0%
Dense
5.73
52.48
58.32
72.25
100.0
51.14
68.63
46.49
69.28
25% Layer
ShortGPT
11.35
12.85
15.45
65.93
99.8
44.80
63.80
42.55
56.30
ShortGPT -mix
11.77
13.17
19.48
65.43
99.9
45.53
64.70
43.72
60.00
EvoPress
9.88
24.33
35.03
64.63
100.0
42.60
65.24
43.19
54.65
EvoPress -mix
11.35
27.89
35.62
66.12
100.0
46.27
65.06
44.36
58.04
Table 2: Performance on the four profiled capability domains and five out-of-domain benchmarks for Llama-3.1-70B. -mix denotes calibration using the same mixed-domain samples and total token budget as SliMOO. All out-of-domain evaluations use the existing pruning mask without additional profiling, search, or fine-tuning.
Method
EN ↓
ES ↓
FR ↓
DE ↓
Dense
12.19
9.81
6.56
6.94
ShortGPT
17.75
10.56
8.75
9.31
EvoPress
16.01
10.18
8.22
8.89
SliMOO
14.69
9.81
7.62
8.62
Table 3: Multilingual C4 perplexity on Qwen3-32B after 25% layer pruning. Lower is better, and bold marks the best pruned model.
Sp.
Method
Lang. ↓
Math ↑
Code ↑
Know. ↑
25% Layer
ShortGPT
11.77
13.17
19.48
65.43
LLM-Pruner
9.42
32.44
32.30
66.15
EvoPress
11.35
27.89
35.62
66.12
SliMOO
9.89
34.40
39.14
66.90
33% Sublayer
ShortGPT
20.53
8.71
3.85
64.18
EvoPress
10.92
28.29
24.04
64.46
Table 7
Search variant
Lang. ↓
Math ↑
Code ↑
Know. ↑
Language Evolution
9.88
24.33
35.03
64.63
Math Evolution
10.75
36.11
34.89
66.44
Code Evolution
9.89
30.93
38.59
65.66
Knowledge Evolution
11.35
25.46
27.20
67.45
Mixed Evolution
11.35
27.89
35.62
66.12
Mean Evolution
9.97
29.14
33.10
66.62
Table 8
Sparsity
Arch
IO Tok.
TTFT (s) ↓
Decode TPS ↑
0%
80 Attn
4096
3.39
482.9
80 FFN
8192
4.47
413.6
25%
60 Attn
4096
2.54
623.9
60 FFN
8192
3.35
539.5
33%
46 Attn
4096
2.37
672.2
64 FFN
8192
3.07
593.0
Table 8: Inference efficiency of pruned Llama-3.1-70B with vLLM at batch size 8.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10: Layer-wise pruning configurations selected by SliMOO. Each column denotes one transformer layer, with attention on the top strip and MLP on the bottom strip. Salmon blocks are retained and white blocks are pruned; arrows point from shallow to deep layers.
Method
Lang. ↓
Math ↑
Code ↑
Know. ↑
ShortGPT
11.35
12.85
15.45
65.93
ShortGPT ∗
9.13
37.35
49.18
68.31
EvoPress
9.88
24.33
35.03
64.63
EvoPress ∗
8.48
42.22
57.23
68.32
SliMOO
9.89
34.40
39.14
66.90
SliMOO ∗
8.56
48.04
61.40
70.23
Appendix
Table 11: Post-pruning LoRA tuning results on Llama-3.1-70B at 25% layer sparsity. Starred methods are tuned after pruning with LoRA.
Sparsity
Method
Lang. ↓
Math ↑
Code ↑
Know. ↑
0%
N/A
8.47
32.10
42.20
66.04
12.5% Layer
ShortGPT
17.60
19.17
8.37
61.28
LaCo
12.73
21.02
17.93
60.13
EvoPress
19.54
1.27
3.02
48.40
SliMOO
11.65
23.30
23.26
61.64
25% Sublayer
ShortGPT
280.00
0.02
0.00
46.89
Appendix
Table 12: Small-model pruning results on Llama-3.1-8B. We report domain averages: lower is better for language modeling and higher is better elsewhere. Bold marks the best pruned model within each sparsity setting.
Domain
EvoPress
SliMOO reported
SliMOO 5 runs
Language ↓
12.02
10.45
10.34±0.14
Math ↑
10.33
27.27
27.31±0.17
Code ↑
15.11
30.78
30.56±0.45
Knowledge ↑
62.02
66.29
66.19±0.07
Appendix
Table 13: Search-seed stability on Llama-3.1-70B at 33% sublayer sparsity. The last column reports mean ± SD across five runs. Language reports perplexity; other scores are percentages.
Domain
Prompt (truncated)
Dense output
Pruned output
Language modeling
“The Great Wall of China is a series of fortifications that were built across the historical northern borders of ancient Chinese states and Imperial China as protection against various nomadic groups. Construction of the walls began as early as the 7th century BC, and”
they have a total length of 13,000 miles. Several sections of the wall are now in ruins or have disappeared. However, the wall is still one of the most attractive places in the world and was selected as one of the New Seven Wonders of the World in 2007.
the most recent construction was completed in the 18th century. The walls were constructed of stone, compacted earth, and compacted sand, depending on the area. The walls were constructed on high ground, and they overlook nearby plains. The walls were constructed with the idea of protecting the states from raiders, and they were constructed with the assumption that they could be defended by a minimal force.
Math
“Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market?”
Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for 2perfreshduckegg.16-3-4=9.9\times 2=18.Theansweris18. ✓
Janet’s ducks lay 16−3−4=9 eggs per day. She sells 9 eggs per day. She makes 9\times 2=\18perday.Theansweris18. ✓
Code
def is_prime(n: int) -> bool: """Return True if n is prime."""
if n < 2: return False for i in range(2,\ int(n ** 0.5) + 1): if n % i == 0: return False return True Optimal O(n) . ✓
if n < 2: return False for i in range(2, n): if n % i == 0: return False return True Functionally correct but naive O(n) ; the n optimization is lost. ✓
Knowledge reasoning
“In which year did World War II end? A. 1943 B. 1944 C. 1945 D. 1946 Answer:”
C ✓
C ✓
Appendix
Table 14: Case study on Llama-3.1-70B before and after dropping 25% of layers with SliMOO. We use greedy decoding for generation. For math, code, and reasoning, decoding is truncated after the final answer.
Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/
Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms to identify removable layers. In this study, we empirically investigate depth pruning from a functional perspective. Evaluating representative LLM families across diverse calibration configurations and multiple search algorithms, we show that different configurations produce different pruning patterns. Furthermore, under a fixed calibration configuration, complex search algorithms yield marginal performance improvements over simple one-shot methods, converging to similar pruned subsets. Overall, our results suggest that the calibration configuration plays a substantially larger role than the choice of search algorithm in shaping pruning patterns and calibration perplexity, while contributing comparably to variance in downstream reasoning accuracy. This indicates that future pruning efforts may benefit from prioritizing the calibration configuration over search complexity.
Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3
Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2
Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.