Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models

Cameron Barker; Andreas Paraskeva; Jan N. van Rijn; Elliot J. Crowley; Henry Gouk

cs.LGOct 11, 2026

Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models

Authors: Cameron Barker, Andreas Paraskeva, Jan N. van Rijn, Elliot J. Crowley, Henry Gouk

Organizations: School of Informatics, University of Edinburgh, UK · Leiden Institute of Advanced Computer Science, Leiden University, the Netherlands · School of Engineering, University of Edinburgh, UK

Abstract

Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically consider only the total number of retained weights. Consequently, their pruning decisions do not account for how target hardware stores and processes those weights. These methods select a sparsity structure in advance and apply it uniformly to all projections, rather than accounting for how each structure trades pruning error against latency and memory across sublayers. We propose Heterogeneous Structured Pruning (HSP), a hardware-aware pruning framework that combines two components: (i) a heterogeneous search space of dense, unstructured, 2:4, block-sparse, and head / channel (H/C) pruning options and (ii) structure-regularising constraints based on latency and memory models. Our contribution generalises existing importance-based methods: a global solver selects one option per sublayer to minimise proxy pruning loss under a resource budget, thereby assigning each projection a structure and density. We evaluate HSP on OPT and LLaMA models for an NVIDIA Jetson Orin Nano Super edge device, where it improves the quality-resource frontiers over training-free baselines. A project page and updated links for the related code is maintained at: cameronbarker.me/publications/hsp.

Figures & tables

Figure 1: Overview of HSP. (a) One option is selected per attention or MLP sublayer. H/C options couple projection masks, while independent options prune each projection separately. (b) Kernel profiles give latency, and an encoding model gives memory. (c) Density floors restrict candidates. Inner pruning constructs masks, outer scoring gives loss $e$ , and (b) gives cost $c$ . (d) Dominance filtering reduces the candidate sets. (e) MCKP chooses one option per sublayer. The bar shows variable cost against budget $B-C_{0}$ , set here to a fraction $\beta=0.7$ of dense cost. Numbers and density grids are illustrative.

Panel of Figure 2 — Figure 2: Structure-dependent weight memory. ( 2(a) ) Equal-density masks need different amounts of metadata. Orange denotes column indices and green denotes row pointers. ( 2(b) ) Analytical weight memory for the $2560\times 2560$ OPT-2.7B query projection, relative to dense FP16 weights. Compressed sparse row (CSR) and block sparse row (BSR) formats use 32-bit indices and row pointers. The $2{:}4$ format uses packed metadata and has fixed density $0.5$ . H/C pruning stores the narrowed matrix densely. Values below the dashed line save weight memory. Costs include weights and metadata only, excluding biases, cache, and runtime allocations.

Figure 3: Measured GPU-prefill kernel performance for OPT-2.7B on the Jetson Orin Nano Super. Panels show the Q/K/V, attention output (O), MLP up (U), and MLP down (D) projections. Linear-scaling efficiency is

E_{\mathrm{lin}}(d)=d\,T_{\mathrm{dense}}/T_{\mathrm{sparse}}

. Here

d

is retained density, and

T_{\mathrm{dense}}

and

T_{\mathrm{sparse}}

are measured latencies of the dense and sparse kernels for the same projection and workload. Points above the dashed

100d\%

line are faster than dense execution. A value of

100\%

denotes ideal inverse-density scaling. Each rectangular block shape is plotted separately and grouped into a block family in the legend.

Figure 4: WikiText-2 perplexity and mean seven-task zero-shot accuracy vs. whole-model density, profile-backed latency, and analytical memory (normalised to dense) for OPT-2.7B on the Jetson Orin Nano Super GPU under the mixed workload (1024-token prefill, 128-token decode). Resource axes are linear below dense cost and logarithmic above. Triangles (

\blacktriangle

) mark perplexities above the plot limit of

100

, with numeric annotations giving their values. The dashed rule marks random-guess accuracy (0.357).

Figure 5: WikiText-2 perplexity (left) and mean seven-task zero-shot accuracy (right) vs. density, latency, and memory for OPT-2.7B on the Jetson CPU. Plotting conventions follow Figure 4 .

2{:}4

baselines are excluded because the CPU does not support

2{:}4

instructions.

Figure 6: Structures selected by HSP for OPT-2.7B on the Jetson Orin Nano Super under the mixed workload at a budget ratio of

0.8

. Columns are the two devices and rows are the two objectives, so the panels show the GPU latency, CPU latency, GPU memory, and CPU memory objectives. Within each panel, rows correspond to the attention and MLP projections, while bars correspond to decoder layers. Bar height encodes retained density, and colour encodes the selected structure.

Method	PPL	Prefill		Decode		Mixed
Method	PPL	Mem	Lat	Mem	Lat	Mem	Lat
GPU
Dense	14.32*	OOM
Wanda- $2{:}4$	21.90	3447	0.91	3307	8.80	3748	9.62
FLAP	17.15	4519	0.64	4158	6.69	4519	9.97
HSP-Lat	15.31	4367	0.62	4065	6.52	4367	9.12

Table 1: On-device evaluation of OPT-2.7B on the Jetson Orin Nano Super GPU and CPU. Peak memory (

\downarrow

) is in megabytes (MB) and end-to-end latency (

\downarrow

) in seconds (s). Best and second-best pruned candidates within each device are bold and underlined respectively. Pruned candidates use a budget equal to

80\%

of the dense cost under their respective constraints, except Wanda-

2{:}4

which only supports a density constraint of

50\%

. The dense model exceeds GPU memory due to runtime overhead, so its perplexity in the GPU block is measured off-device (

*

) and its GPU costs are reported as OOM. The CPU runtime has no supported

2{:}4

kernel. CPU perplexity uses only the first 32 WikiText-2 windows of 1024 tokens each and is not directly comparable to the perplexities in the GPU block.

		Prefill		Decode		Mixed
Search	Constraint	PPL	Lat	PPL	Lat	PPL	Lat
Dense		14.32	0.755	14.32	9.890	14.32	11.7
Single	Density	18.40	0.525	18.40	7.530	18.40	8.995
Hetero.	Density	13.73	85.5	13.73	31.0	13.73	117.5
Single	Latency	18.85	0.537	20.95	7.045	22.61	8.292
Hetero.	Latency	16.62	0.537	20.75	7.045	21.28	8.291

Table 2: Ablation of search space heterogeneity and constraint type for OPT-2.7B on the Jetson Orin Nano Super GPU with

B=0.7

. Single uses coupled H/C options, while Hetero. uses the full heterogeneous search space. PPL (

\downarrow

) is WikiText-2 perplexity and Lat (

\downarrow

) is profile-backed latency in seconds (s). Marking follows Table 1 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Structure	Requested densities	OPT floor	Exceptions
Dense	$1$	–	Always available
Unstructured	$\{0.1,0.2,\ldots,0.9\}$	$0.3$	–
Rectangular blocks	$\{0.1,0.2,\ldots,0.9\}$	$0.5$	CPU memory: floor $0.7$
H/C	Profiled widths, increments of $1/H$	$0.5$	LLaMA 7B: floor $0.125$
$2{:}4$	$0.5$ (fixed)	–	GPU only

Appendix

Table A.1: Candidate density settings. Requested densities below the applicable floor are excluded.

H

is the number of attention heads. Coupled options enumerate every retained head count and every MLP width on the profiling grid at or above the floor.

Setting	Configuration
Target device	NVIDIA Jetson Orin Nano Super
CPU	Six-core ARM Cortex-A78AE with NEON
GPU	1024 CUDA cores and 32 Tensor Cores
Shared memory	8 GB LPDDR5, peak bandwidth 102.4 GB/s
Software	NVIDIA JetPack 6.2.1, L4T 36.4.7, CUDA 12.6, PyTorch 2.5.0 (prebuilt package)
Device controls	25 W mode, fans at 100%, $55\,^{\circ}\mathrm{C}$ temperature gating between profiling measurements

Appendix

Table A.2: Experimental setup. Pruning and search run offline on the server. On-device measurements use the Jetson Orin Nano Super.

Structure	Weight storage	CPU kernels	GPU kernels
Dense	Dense matrix	Custom C++	PyTorch dense
H/C	Narrowed dense matrix	Custom C++	PyTorch dense
Unstructured	CSR	Custom C++	Custom CUDA
Rectangular blocks	BSR	Custom C++	Custom CUDA
$2{:}4$	NV24	Not supported	CUTLASS and
Custom CUDA

Appendix

Table C.1: Projection storage and kernel families on the Jetson Orin Nano Super. H/C pruning uses dense kernels at reduced dimensions. Compatible kernels are compared during profiling.

Figure C.1 : GPU-prefill linear-scaling efficiency at retained density

d=0.5

across the OPT model family for the attention output (O) and MLP down (D) projections. “Best block” is the fastest profiled rectangular block shape for each model, which is

64\times 64

in every case shown. The dashed

50\%

line marks break-even with dense execution.

Model	Orin Nano CPU		Orin Nano GPU
	Configs	Time (min)	Configs	Time (min)
OPT-125M	2,004	10.4	2,964	4.7
OPT-350M	2,104	18.5	3,064	7.3
OPT-1.3B	2,464	77.4	3,424	25.8
OPT-2.7B	2,464	110.6	3,424	39.0

Appendix

Table C.2 : One-time kernel-profiling cost per model on the Jetson Orin Nano. “Configs” is the number of measured kernel configurations, and time is their total measurement wall-clock time.

Method	Time (s)
	Options	Solve	Select	Total
Wanda	–	–	–	1.15
FLAP	–	–	–	0.33
HSP-Lat (CPU)	19.22	1.93	0.89	22.05
HSP-Lat (GPU)	20.30	0.39	0.31	21.01
HSP-Mem (CPU)	4.16	0.86	3.98	9.00

Appendix

Table C.3 : Time to produce an OPT-2.7B pruning decision (

\downarrow

), using precomputed calibration statistics and excluding model loading. “Options” constructs the inner-problem solutions, “Solve” measures the MCKP runtime, and “Select” covers mask construction and bias compensation.

Figure D.1 : OPT-2.7B GPU prefill results. Plotting conventions follow Figure 4 .

Figure D.2 : OPT-2.7B GPU decode results. Plotting conventions follow Figure 4 .

Figure D.3 : OPT-2.7B CPU prefill results. Plotting conventions follow Figure 4 .

Figure D.4 : OPT-2.7B CPU decode results. Plotting conventions follow Figure 4 .

Figure D.5 : Perplexity versus density, latency, and memory on the GPU (left) and CPU (right) under the mixed workload. Plotting conventions follow Figure 4 .

Figure D.7 : Perplexity versus density, latency, and memory on the GPU (left) and CPU (right) under the mixed workload. Plotting conventions follow Figure 4 .

Figure D.9 : Perplexity versus density, latency, and memory on the GPU (left) and CPU (right) under the mixed workload. Plotting conventions follow Figure 4 .

Panel of Figure 23 — Figure D.11 : LLaMA 7B with the GPU search space under the mixed workload (1024-token prefill, 128-token decode). ( 11(a) ) Perplexity and mean seven-task zero-shot accuracy versus analytical memory, normalised to dense. Triangles mark off-scale perplexity, annotated with its true value. The dotted line marks random-guess accuracy (0.357). ( 11(b) , 11(c) ) Structures and retained densities selected for each projection in each decoder layer.

Figure D.12 : Block shapes selected by the OPT-2.7B candidates in Figure 6 . Each cell counts projections assigned that shape. Block rows and columns span output and input dimensions, respectively. Grey cells indicate zero selections.

Workload	Method	Latency budget
		0.6	0.7	0.8	0.9
Prefill	HSP (Wanda)	20.22	16.62	14.41	13.65
	HSP (Magnitude)	1.49k	539.36	47.16	15.05
	HSP (Wanda, no H/C)	–	18.26	15.07	13.69
Decode	HSP (Wanda)	39.21	20.75	16.14	14.15
	HSP (Magnitude)	16.58k	6.15k	5.41k	43.61

Appendix

Table D.1 : OPT-2.7B perplexity under latency constraints on the Jetson Orin Nano Super GPU. Best and second-best values within each workload and budget are bold and underlined, respectively. “–” indicates that no candidate meets the budget.

Calibration Samples	4	16	64	256
Zero Input-Feature Norms (%)	6.44	4.83	4.52	4.46
Perplexity ( $\downarrow$ )	16.947	15.129	14.880	15.299

Appendix

Table D.2: Calibration sample-count sensitivity for OPT-2.7B on the GPU under the mixed workload, with a latency budget ratio of

0.8

and all other settings fixed. Zero input-feature norms are the percentage of projection-input features with zero accumulated

\ell_{2}

norm.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models
Apr 21, 2026Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3LLM Pruning LLM Compression
Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University
Correlation-Aware Structured Pruning for Large Language Models
Aug 23, 2026Sicheng Xu, Hao Shi, Wei Zhang +6LLM Pruning Neural Network Pruning
Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sep 10, 2026Sankar Behera, Dhruv Singh, Anshika Agnihotri +3LLM Pruning LLM Compression
CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1

Apr 21, 2026J/K move · Enter open · S save

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li+2

Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University

LLM Pruning LLM Compression Structured Pruning

Open paper AskPDF arXiv

Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning.

Abstract

LLM Pruning

Explore similar work

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Correlation-Aware Structured Pruning for Large Language Models

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry