Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
Authors: Muhammad Junaid Ali, Smail Niar, El-Ghazali Talbi
Organizations: Inria, University of Lille Lille, France · LAMIH UMR CNRS 8201, UPHF INSA Hauts-de-France Valenciennes, France · University of Lille CNRS, Inria, Centrale Lille Lille, France
Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.
Fig. 1: Overview of the proposed multi-objective structured pruning framework consisting of two different stages.
Fig. 2: Block-level pruning in LLMs. (a) Standard transformer block with MHA and MLP. (b) Pruned block where computation is skipped and replaced with an identity connection.
Parameter
Setting
Stage 1
NSGA-II (Pymoo) [ 14 ]
Population / budget
40 individuals / 40 generations
Crossover / mutation
Simulated binary / mask mutation
Stage-1 calibration
1,024 random WikiText-2 samples
Stage 2
BoTorch/GPyTorch; SingleTaskGP , RBF kernel
Acquisition
Log expected improvement; 256 raw samples
TABLE II: Core search settings.
Method
Mistral-v0.3
LLaMA-2 7B
Qwen-2.5
Phi-3 14B
37.5%
50%
37.5%
50%
37.5%
50%
37.5%
50%
Dense
67.08
67.08
60.71
60.71
68.72
68.72
72.19
72.19
ShortGPT [ 6 ]
38.13
36.83
42.63
36.89
41.28
32.33
48.34
34.83
Sliding Window [ 7 ]
38.79
33.65
41.69
34.96
42.20
31.36
47.80
32.97
Block Pruner [ 5 ]
42.14
37.16
42.96
37.19
44.78
35.14
48.16
40.61
EvoPress [ 10 ]
44.48
37.51
43.76
38.53
45.09
36.22
48.48
38.46
TABLE III: Average zero-shot benchmark results for four LLMs at 37.5 % and 50 % sparsity levels. Gold and silver shading denote the best and second-best pruned results, respectively. Proposed results are the mean and standard deviation across five independent runs. Mix denotes mixed calibration.
Fig. 3: Visualization of the Pareto front and non-Pareto solutions at a sparsity rate of 37.5 % .
Fig. 4: Layer-wise MHA and MLP pruning patterns under the two-stage strategy at 37.5 % sparsity.
Method
A100 (s)
Jetson Nano (s)
Sliding Window [ 7 ]
0.784
32.690
2SSP [ 3 ]
0.854
40.530
BlockPruner [ 5 ]
0.825
47.565
EvoPress [ 10 ]
0.891
44.625
ShortGPT [ 6 ]
0.792
49.875
CFSP [ 15 ]
0.753
31.325
TABLE IV: Comparison of latency between the proposed approach and existing studies. Results are shown in seconds, computed on 100-token text sequences over 10 iterations on an Nvidia A100 GPU and the evaluated Nvidia Jetson edge platform.
Fig. 5: Perplexity (PPL) comparison across Mistral and LLaMA2 on WikiText-2, C4, and FineWeb-Edu.
Fig. 6: Critical difference diagram showing the pairwise statistical comparison of structured pruning methods with the proposed approach.
Sparsity
Pred.
r
r2
ρ
ρ2
0.375
KL div.
-0.967
0.935
-0.794
0.630
0.375
PPL
-0.358
0.128
-0.006
0.000
0.5
KL div.
0.417
0.174
0.381
0.145
0.5
PPL
-0.828
0.685
-0.786
0.617
TABLE V: Correlation and squared-correlation analysis with downstream average accuracy.
Fig. 7: Effect of calibration sample size on latency, throughput, and downstream accuracy.
1st Objective
2nd Objective
Inference (s) ↓
Accuracy ↑
PPL ↓
Perplexity
Params
7.95
0.5821
8.2943
KL-Divergence
Params
5.86
0.5930
8.1393
Perplexity
Latency
6.60
0.5732
9.2695
KL-Divergence
Latency
4.86
0.5647
7.48
TABLE VI: Objective-combination comparison. Lower inference time and PPL are better; higher accuracy is better.
Fig. 8: Correlation between evaluation proxies and downstream accuracy across sparsity levels.
Fig. 9: Downstream accuracy and latency for stage-wise pruning configurations.
Method
Downstream Avg. ↑
Avg. PPL ↓
Latency (s) ↓
Importance [ 8 ]
0.4634
29.9688
3.7887
Wanda-SP [ 11 ]
0.4927
21.6250
3.7604
Taylor [ 1 ]
0.4821
23.6250
3.7659
Gradient [ 18 ]
0.4811
23.5312
3.7610
WActiGrad [ 9 ]
0.4413
26.3438
3.9623
TABLE VII: Comparison of Stage-2 importance estimators.
Fig. 10: Comparison of the perplexity values in log scale on different sparsity rates of the proposed approach with existing studies
Method
Avg. PPL ↓
Latency (s) ↓
Throughput (tokens/s) ↑
Downstream Avg. ↑
Uniform
15.8694
3.8216
133.98
0.4572
LSA [ 19 ]
13.8296
3.8948
131.60
0.4038
DSA [ 20 ]
12.0280
3.8965
131.47
0.4142
DLP [ 21 ]
10.3632
3.8641
132.53
0.4898
ER [ 23 ]
10.4251
3.9467
129.87
0.4929
OWL [ 22 ]
10.3745
3.8689
132.35
0.4898
TABLE VIII: Stage-2 allocation-method comparison. Lower PPL and latency are better; higher throughput and downstream average are better.
Fig. 11: Layer-wise pruning ratios for the selected LLaMA-2-7B candidate.
Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically consider only the total number of retained weights. Consequently, their pruning decisions do not account for how target hardware stores and processes those weights. These methods select a sparsity structure in advance and apply it uniformly to all projections, rather than accounting for how each structure trades pruning error against latency and memory across sublayers. We propose Heterogeneous Structured Pruning (HSP), a hardware-aware pruning framework that combines two components: (i) a heterogeneous search space of dense, unstructured, 2:4, block-sparse, and head / channel (H/C) pruning options and (ii) structure-regularising constraints based on latency and memory models. Our contribution generalises existing importance-based methods: a global solver selects one option per sublayer to minimise proxy pruning loss under a resource budget, thereby assigning each projection a structure and density. We evaluate HSP on OPT and LLaMA models for an NVIDIA Jetson Orin Nano Super edge device, where it improves the quality-resource frontiers over training-free baselines. A project page and updated links for the related code is maintained at: cameronbarker.me/publications/hsp.
Cameron Barker, Andreas Paraskeva, Jan N. van Rijn +2
School of Informatics, University of Edinburgh, UK · Leiden Institute of Advanced Computer Science, Leiden University, the Netherlands · School of Engineering, University of Edinburgh, UK
Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices. Existing approaches to model compression and optimization often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectInfer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1
Department of Computer Networks, Paderborn University, Paderborn, Germany
Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements. While some methods address this by developing small or tiny language models from scratch, these approaches demand extensive GPU training. Compressing pre-trained LLMs for edge devices offers a compelling alternative. Beyond pruning and quantization, Neural Architecture Search (NAS) enables effective compression, yet prior NAS approaches often limit the search space and decouple architecture from quantization. We introduce a differentiable NAS framework that explores the entire space and jointly optimizes architectural configurations alongside mixed-precision quantization for linear layers of LLMs. Experiments demonstrate superior accuracy-latency trade-offs: our models achieve up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy, or up to 6% higher average accuracy across seven reasoning tasks at equivalent latency.
Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1
UiT The Arctic University of Norway · University of Oslo, Norway