SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning

Guanchen Li; Yixing Xu; Xuanwu Yin; Dong Li; Emad Barsoum

cs.LGOct 20, 2026

SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning

Authors: Guanchen Li, Yixing Xu, Xuanwu Yin, Dong Li, Emad Barsoum

Organizations: Advanced Micro Devices, Inc. (AMD)

Abstract

As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by directly reducing memory and latency without hardware-specific support. However, we observe that transformer layers exhibit markedly different task-dependent layer contributions, whereas existing depth-pruning methods ignore this heterogeneity and thus tend to optimize for a narrow domain or benchmark rather than preserve broad capability. This suggests that LLM depth pruning should be formulated as a multi-objective optimization problem rather than a single-score ranking problem. We propose SliMOO, an interpretable multi-objective evolutionary search framework for LLM depth pruning. SliMOO first uses single-layer removal probing to construct a task-layer importance map, which provides an interpretable view of task-shared and task-specific layers and serves as a search prior. It then performs a Pareto-aware evolutionary search for pruning masks, where candidates are generated under the guidance of the task-layer prior, scored by their deviation from the dense model on each task, and retained through NSGA-II environmental selection. Across the Llama-3.1 and Qwen3 model families, SliMOO consistently achieves stronger multi-task trade-offs than rule-based, greedy, and scalarized baselines. For example, on Llama-3.1-70B at 25% sparsity, it improves the math-domain average from 24.33% to 34.40% and the code-domain average from 35.03% to 39.14%. Code will be made available at https://github.com/AMD-AGI/SliMOO.

Figures & tables

Figure 1: Normalized cross-domain trade-offs on Llama-3.1-70B. Scores are normalized within each setting; language modeling perplexity is inverted before normalization.

Figure 2: Overview of SliMOO. (a) TLIP profiles task-wise importance of layers/sublayers, yielding a task-layer importance map $\mathbf{I}$ . (b) Guided by the task-layer prior, SliMOO generates feasible pruning masks and uses NSGA-II to retain Pareto-competitive trade-offs under a target sparsity.

Model	Sparsity	Method	Language Avg. $\downarrow$	Math Avg. $\uparrow\%$	Code Avg. $\uparrow\%$	Knowledge Avg. $\uparrow\%$
Llama-3.1-70B	$0\%$	-	5.73	52.48	58.32	72.25
	$25\%$ Layer	ShortGPT	11.35	12.85	15.45	65.93
		LaCo	11.20	16.00	20.39	66.22
		LLM-Pruner	8.74	32.44	26.78	66.78
		EvoPress	9.88	24.33	35.03	64.63
		SliMOO	9.89	34.40	39.14	66.90

Table 1: Post-pruning performance on the Llama family. We report domain averages: lower is better for language modeling and higher is better for math, code, and knowledge reasoning. Bold indicates the best result, and underlines indicate the runner-up.

Figure 3: Search dynamics of SliMOO across four capability domains. The horizontal axis denotes the generation index, and the vertical axis reports the corresponding domain-specific search objectives used during optimization; lower values indicate smaller distributional deviation from the dense model.

Sparsity	Method	Profiled capability domains				Out-of-domain benchmarks
Sparsity	Method	Lang. $\downarrow$	Math $\uparrow$	Code $\uparrow$	Know. $\uparrow$	RULER NIAH-8K $\uparrow$	TruthfulQA MC2 $\uparrow$	CrowS Pairs †	ToxiGen $\uparrow$	MedMCQA $\uparrow$
$0\%$	Dense	5.73	52.48	58.32	72.25	100.0	51.14	68.63	46.49	69.28
$25\%$ Layer	ShortGPT	11.35	12.85	15.45	65.93	99.8	44.80	63.80	42.55	56.30
	ShortGPT -mix	11.77	13.17	19.48	65.43	99.9	45.53	64.70	43.72	60.00
	EvoPress	9.88	24.33	35.03	64.63	100.0	42.60	65.24	43.19	54.65
	EvoPress -mix	11.35	27.89	35.62	66.12	100.0	46.27	65.06	44.36	58.04

Table 2: Performance on the four profiled capability domains and five out-of-domain benchmarks for Llama-3.1-70B. -mix denotes calibration using the same mixed-domain samples and total token budget as SliMOO. All out-of-domain evaluations use the existing pruning mask without additional profiling, search, or fine-tuning.

Method	EN $\downarrow$	ES $\downarrow$	FR $\downarrow$	DE $\downarrow$
Dense	12.19	9.81	6.56	6.94
ShortGPT	17.75	10.56	8.75	9.31
EvoPress	16.01	10.18	8.22	8.89
SliMOO	14.69	9.81	7.62	8.62

Table 3: Multilingual C4 perplexity on Qwen3-32B after

25\%

layer pruning. Lower is better, and bold marks the best pruned model.

Sp.	Method	Lang. $\downarrow$	Math $\uparrow$	Code $\uparrow$	Know. $\uparrow$
$25\%$ Layer	ShortGPT	11.77	13.17	19.48	65.43
	LLM-Pruner	9.42	32.44	32.30	66.15
	EvoPress	11.35	27.89	35.62	66.12
	SliMOO	9.89	34.40	39.14	66.90
$33\%$ Sublayer	ShortGPT	20.53	8.71	3.85	64.18
	EvoPress	10.92	28.29	24.04	64.46

Table 7

Search variant	Lang. $\downarrow$	Math $\uparrow$	Code $\uparrow$	Know. $\uparrow$
Language Evolution	9.88	24.33	35.03	64.63
Math Evolution	10.75	36.11	34.89	66.44
Code Evolution	9.89	30.93	38.59	65.66
Knowledge Evolution	11.35	25.46	27.20	67.45
Mixed Evolution	11.35	27.89	35.62	66.12
Mean Evolution	9.97	29.14	33.10	66.62

Table 8

Sparsity	Arch	IO Tok.	TTFT (s) $\downarrow$	Decode TPS $\uparrow$
$0\%$	80 Attn	4096	3.39	482.9
$0\%$	80 FFN	8192	4.47	413.6
$25\%$	60 Attn	4096	2.54	623.9
$25\%$	60 FFN	8192	3.35	539.5
$33\%$	46 Attn	4096	2.37	672.2
$33\%$	64 FFN	8192	3.07	593.0

Table 8: Inference efficiency of pruned Llama-3.1-70B with vLLM at batch size 8.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10: Layer-wise pruning configurations selected by SliMOO. Each column denotes one transformer layer, with attention on the top strip and MLP on the bottom strip. Salmon blocks are retained and white blocks are pruned; arrows point from shallow to deep layers.

Method	Lang. $\downarrow$	Math $\uparrow$	Code $\uparrow$	Know. $\uparrow$
ShortGPT	11.35	12.85	15.45	65.93
ShortGPT ∗	9.13	37.35	49.18	68.31
EvoPress	9.88	24.33	35.03	64.63
EvoPress ∗	8.48	42.22	57.23	68.32
SliMOO	9.89	34.40	39.14	66.90
SliMOO ∗	8.56	48.04	61.40	70.23

Appendix

Table 11: Post-pruning LoRA tuning results on Llama-3.1-70B at

25\%

layer sparsity. Starred methods are tuned after pruning with LoRA.

Sparsity	Method	Lang. $\downarrow$	Math $\uparrow$	Code $\uparrow$	Know. $\uparrow$
$0\%$	N/A	8.47	32.10	42.20	66.04
$12.5\%$ Layer	ShortGPT	17.60	19.17	8.37	61.28
	LaCo	12.73	21.02	17.93	60.13
	EvoPress	19.54	1.27	3.02	48.40
	SliMOO	11.65	23.30	23.26	61.64
$25\%$ Sublayer	ShortGPT	280.00	0.02	0.00	46.89

Appendix

Table 12: Small-model pruning results on Llama-3.1-8B. We report domain averages: lower is better for language modeling and higher is better elsewhere. Bold marks the best pruned model within each sparsity setting.

Domain	EvoPress	SliMOO reported	SliMOO 5 runs
Language $\downarrow$	12.02	10.45	$10.34\pm 0.14$
Math $\uparrow$	10.33	27.27	$27.31\pm 0.17$
Code $\uparrow$	15.11	30.78	$30.56\pm 0.45$
Knowledge $\uparrow$	62.02	66.29	$66.19\pm 0.07$

Appendix

Table 13: Search-seed stability on Llama-3.1-70B at

33\%

sublayer sparsity. The last column reports mean

\pm

SD across five runs. Language reports perplexity; other scores are percentages.

Domain	Prompt (truncated)	Dense output	Pruned output
Language modeling	“The Great Wall of China is a series of fortifications that were built across the historical northern borders of ancient Chinese states and Imperial China as protection against various nomadic groups. Construction of the walls began as early as the 7th century BC, and”	they have a total length of 13,000 miles. Several sections of the wall are now in ruins or have disappeared. However, the wall is still one of the most attractive places in the world and was selected as one of the New Seven Wonders of the World in 2007.	the most recent construction was completed in the 18th century. The walls were constructed of stone, compacted earth, and compacted sand, depending on the area. The walls were constructed on high ground, and they overlook nearby plains. The walls were constructed with the idea of protecting the states from raiders, and they were constructed with the assumption that they could be defended by a minimal force.
Math	“Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market?”	Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg.$ 16-3-4=9 $.$ 9\times 2=18 $. The answer is$ 18. $\checkmark$	Janet’s ducks lay $16-3-4=9$ eggs per day. She sells 9 eggs per day. She makes $9\times 2=\$ 18 $per day. The answer is$ 18. $\checkmark$
Code	def is_prime(n: int) -> bool: """Return True if n is prime."""	if n < 2: return False for i in range(2,\ int(n ** 0.5) + 1): if n % i == 0: return False return True Optimal $O(\sqrt{n})$ . $\checkmark$	if n < 2: return False for i in range(2, n): if n % i == 0: return False return True Functionally correct but naive $O(n)$ ; the $\sqrt{n}$ optimization is lost. $\checkmark$
Knowledge reasoning	“In which year did World War II end? A. 1943 B. 1944 C. 1945 D. 1946 Answer:”	C $\checkmark$	C $\checkmark$

Appendix

Table 14: Case study on Llama-3.1-70B before and after dropping 25% of layers with SliMOO. We use greedy decoding for generation. For math, code, and reasoning, decoding is truncated after the final answer.

Asset	License	Source
Llama-3.1 [ 2 ]	Llama 3.1 Community	https://llama.com/llama3_1/license/
Qwen3 [ 3 ]	Apache-2.0	https://huggingface.co/Qwen/Qwen3-32B/blob/main/LICENSE ; https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/LICENSE
FineWeb-Edu [ 57 ]	ODC-BY-1.0	https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
FineWeb-2 [ 54 ]	ODC-BY-1.0	https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
GSM8K [ 30 ]	MIT	https://huggingface.co/datasets/openai/gsm8k
DeepScaleR [ 31 ]	MIT	https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset

Appendix

Table 15: Existing models, datasets, and benchmarks used in the paper.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Locality-Aware Redundancy Pruning for LLM Depth Compression
May 27, 2026Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3LLM Pruning LLM Compression
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
Apr 27, 2026Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3LLM Pruning LLM Inference Acceleration
Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Mar 25, 2026Jimyung Hong, Jaehyung KimLLM Pruning Structured Pruning
Yonsei University

May 27, 2026J/K move · Enter open · S save

Locality-Aware Redundancy Pruning for LLM Depth Compression

Vincent-Daniel Yun, Youngrae Kim, Woosang Lim, YoungJin Heo+2

University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea+1

LLM Pruning LLM Compression LLM Inference Acceleration

Open paper AskPDF arXiv

Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/

Abstract

LLM Pruning

Explore similar work

Locality-Aware Redundancy Pruning for LLM Depth Compression

Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score