Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

Tianxiang Xia; Kai Lion; Niao He; Michael Muehlebach; Melanie Zeilinger; Hao Ma

cs.LGOct 22, 2026

Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

Authors: Tianxiang Xia, Kai Lion, Niao He, Michael Muehlebach, Melanie Zeilinger, Hao Ma

Organizations: ETH Zürich, Zürich · Max Planck Institute for Intelligent Systems, Tübingen · Vienna University of Economics and Business, Vienna · The University of Hong Kong

Abstract

Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unreliable network connectivity, communication delays, and privacy requirements can make such reliance impractical, motivating local execution on edge devices. Local deployment requires models to fit within limited memory and execute efficiently with the available computational resources. Recent work has shown that learning sparse-plus-low-rank representations enables elastic compression of a single checkpoint to multiple memory budgets. However, fewer parameters do not necessarily translate into faster inference. Indexing overhead and irregular memory access in unstructured sparse components can offset computational savings. To address this critical deployment barrier, we introduce S-SALAAD, a unified framework that jointly learns the model weights and their low-rank and structured sparse components during training. The framework supports a family of hardware-compatible structured sparsity patterns. It couples this structure learning with budget-aware compression and dedicated CPU and GPU kernels that exploit the learned structure for acceleration, enabling efficient inference from a single checkpoint across memory budgets. Experiments on Llama2 and Qwen3 models, covering both pretraining and distillation, show that S-SALAAD advances the empirical Pareto frontier in perplexity, memory, and inference latency across multiple hardware platforms relative to the evaluated baselines. These results show that structured learning can translate elastic compressibility into reduced memory requirements and lower inference latency, enabling practical deployment of large models on edge devices.

Figures & tables

Figure 2: Pipeline of Stage I.

Panel of Figure 3 — Figure 3: Llama-1B training and learned $S$ in a randomly selected layer. Log shading shows entry magnitudes for SALAAD ( 3(b) ) and $64\!\times\!64$ block Frobenius norms for S-SALAAD ( 3(c) , 3(d) ), clipped to the displayed range.

Panel of Figure 4 — Figure 4: Perplexity against throughput on a Quadro RTX 6000; marker radius linearly encodes peak GPU memory usage, with values annotated in GB. Lighter red denotes the $16\!\times\!16$ variant.

	Llama-1B					Qwen3-1.7B
Task	Vanilla	800M	700M	600M	500M	Vanilla	1273M	1200M	1000M	800M
PIQA	0.72	0.71	0.70	0.69	0.68	0.73	0.72	0.72	0.72	0.70
SciQ	0.79	0.76	0.76	0.76	0.77	0.92	0.92	0.91	0.90	0.89
WinoG.	0.54	0.54	0.54	0.54	0.54	0.57	0.58	0.59	0.58	0.56
HellaSwag	0.39	0.37	0.37	0.37	0.35	0.46	0.45	0.44	0.43	0.40
ARC-Easy	0.54	0.51	0.51	0.51	0.50	0.76	0.74	0.73	0.72	0.68

Table 1: Zero-shot accuracy of Vanilla and S-SALAAD (in blue) across budgets.

Figure 5: Perplexity vs. parameter count on Llama-350M.

Panel of Figure 6 — Figure 6: Perplexity versus post-compression parameter count for different block sizes on Llama.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Static weight memory and throughput of dense inference and the unstructured SLR representation used by SALAAD. Both metrics are normalized to their respective dense baselines. The benchmark uses a synthetic Llama-1B MLP down projection with a

15\text{\,}\mathrm{\%}

rank ratio and

5\text{\,}\mathrm{\%}

nonzero density.

Block size	$P_{\mathrm{tgt}}\!=\!500$ M	$P_{\mathrm{tgt}}\!=\!600$ M	$P_{\mathrm{tgt}}\!=\!700$ M
$1\!\times\!1$	14.11	13.86	13.78
$2\!\times\!2$	17.29	14.93	14.19
$4\!\times\!4$	87.44	28.89	18.28
$8\!\times\!8$	456.52	116.33	43.58
$16\!\times\!16$	1497.68	351.39	98.83

Appendix

Table 2: Post-compression perplexity of the same unstructured SALAAD Llama-1B checkpoint when block structure of size

p\!\times\!p

is imposed during compression, at three parameter budgets

P_{\mathrm{tgt}}

. The first row is the element-wise reference.

Figure 8: Post-compression perplexity on Llama-350M.

Figure 9: Post-compression perplexity for exact and iterative SVD on Llama-350M at matched budgets.

Panel of Figure 10 — Figure 10: $\Gamma_{L}$ trajectories for exact and iterative SVD on the down projections of three representative Llama-350M layers. Each inset magnifies the converged regime, where the two curves are otherwise indistinguishable.

Panel of Figure 13 — Figure 13: Kernel efficiency on a randomly sparsified Llama-1B MLP down projection at $15\text{\,}\mathrm{\%}$ rank ratio and $5\text{\,}\mathrm{\%}$ density, normalized to dense execution. Elem. and Block denote PyTorch’s torch.sparse_csr_tensor and torch.sparse_bsr_tensor backends, respectively. In the block-sparse setting both backends run at $16\!\times\!16$ on the GPU and $8\!\times\!8$ on the CPU, the smallest block size each platform supports.

Figure 14: Block-size sweep on random SLR weights for a

2048\!\times\!5632

Llama-1B down projection, measured on a Quadro RTX 6000 in FP16 with batch size 128 and sequence length 256. Throughput is normalized to dense.

Setting	Llama-350M	Llama-1B	Qwen3-1.7B
Hidden size	1024	2048	2048
Intermediate (FFN) size	2816	5632	6144
Layers	24	24	28
Attention heads (Q / KV)	16 / 16 (MHA)	32 / 32 (MHA)	16 / 8 (GQA)
Head dimension	64	64	128
Vocabulary size	32,128	32,128	131,072

Appendix

Table 3: Model architectures used for Llama 2 pretraining and Qwen3 distillation.

Table 16

Variant	Llama-350M		Llama-1B		Qwen3-1.7B
Variant	$\Delta_{\alpha}$	$\Delta_{\beta}$	$\Delta_{\alpha}$	$\Delta_{\beta}$	$\Delta_{\alpha}$	$\Delta_{\beta}$
Unstructured SALAAD	0.20	0.001	0.20	0.001	—	—
Block- $8\!\times\!8$	0.04	0.01	0.045	0.01	—	—
Block- $16\!\times\!16$	0.04	0.01	0.04	0.01	—	—
Block- $32\!\times\!32$	0.04	0.01	0.04	0.01	—	—
Block- $64\!\times\!64$	0.02	0.02	0.02	0.02	0.02	0.02

Appendix

Table 6: Controller step sizes across model scales and sparsity variants.

Component	Value
GPU	NVIDIA Quadro RTX 6000 (Turing, TU102)
GPU memory	24 GB GDDR6
GPU driver / CUDA	NVIDIA driver 550.x / CUDA 12.9
CPU	4 $\times$ AMD EPYC 7662 CPUs
RAM	32 GB
PyTorch	2.13.0 (cu129 build)

Appendix

Table 7: Profiling environment for the Quadro RTX 6000 and CPU measurements.

Hardware	Model	Batch $\times$ length	Warmup	Timed
Quadro RTX 6000	Llama-1B	$128\times 256$	10	20
Quadro RTX 6000	Qwen3-1.7B	$32\times 512$	10	20
RTX 2080 Ti	Llama-1B	$128\times 256$	10	20
RTX 2080 Ti	Qwen3-1.7B	$32\times 512$	10	20
4 $\times$ AMD EPYC 7662 CPUs	Llama-1B	$8\times 64$	10	20
4 $\times$ AMD EPYC 7662 CPUs	Qwen3-1.7B	$8\times 64$	3	6

Appendix

Table 8: Input shapes and repetition counts for inference profiling.

Panel of Figure 15 — Figure 15: Task-loss trajectories for (a) different sparsity patterns on Llama-350M, (b) different block sizes on Llama-1B, and (c) Qwen3-1.7B distillation. Curves use a moving average over approximately 1,000 training steps and end at the last available logged step.

Figure 16: Learned sparse components $S$ in layer 16 of Llama-350M. Rows show block- $16\!\times\!16$ , row-/column-wise, and $2\!:\!4$ training; columns show the Q, O, and down projections. Insets enlarge the red-boxed regions of each down projection on the same color scale; their labels give the crop dimensions in weights. The displayed matrices are taken directly from the trained checkpoints; the $2\!:\!4$ row shows the learned components before any additional structural projection.

Figure 17: Learned sparse components $S$ in layer 16 of Llama-1B across four block sizes. Columns show the Q, O, and down projections. Insets enlarge the red-boxed down-projection regions on the same color scale; labels give the crop dimensions in weights. Each inset contains $6\!\times\!6$ native blocks.

Figure 18: Learned sparse components $S$ in layer 16 of Qwen3-1.7B trained with block- $64\!\times\!64$ S-SALAAD. Columns show the Q, O, and down projections. The down-projection inset enlarges the red-boxed $384\!\times\!384$ region on the same color scale, showing $6\!\times\!6$ native blocks.

Figure 19: Paired sparsity patterns in a randomly selected layer of Llama-1B. Rows show unstructured sparsity ( $1\!\times\!1$ ) followed by four block sizes. Dark cells are nonzero and white cells are exactly zero. The O component is transposed so that its head slices align with those of V along rows.

Block size	Q/K correlation (layers)	V/O correlation (layers)
Unstructured ( $1\!\times\!1$ )	$0.971\pm 0.096$ (24)	$0.976\pm 0.024$ (24)
$8\!\times\!8$	$0.986\pm 0.025$ (23)	$0.962\pm 0.051$ (24)
$16\!\times\!16$	$0.979\pm 0.039$ (23)	$0.948\pm 0.076$ (24)
$32\!\times\!32$	$0.980\pm 0.018$ (21)	$0.937\pm 0.078$ (24)
$64\!\times\!64$	$0.967\pm 0.022$ (21)	$0.909\pm 0.127$ (24)

Appendix

Table 9: Mean

\pm

sample standard deviation of layer-wise Pearson correlations of head nonzero fractions. Parentheses give the number

n

of valid layers for each pair, excluding layers in which either projection has an entirely zero sparse component.

Granularity pair	Q	K	V	O
$1\!\times\!1$ / $8\!\times\!8$	$3.04\pm 0.77$ (23)	$2.73\pm 0.88$ (22)	$3.08\pm 0.72$ (24)	$2.88\pm 0.61$ (24)
$1\!\times\!1$ / $16\!\times\!16$	$3.11\pm 0.88$ (19)	$2.72\pm 0.83$ (18)	$2.92\pm 0.78$ (24)	$2.92\pm 0.72$ (24)
$1\!\times\!1$ / $32\!\times\!32$	$3.29\pm 0.77$ (17)	$3.00\pm 0.61$ (17)	$3.00\pm 0.63$ (16)	$2.91\pm 0.83$ (11)
$1\!\times\!1$ / $64\!\times\!64$	$3.29\pm 0.73$ (14)	$2.75\pm 0.68$ (16)	$3.12\pm 0.81$ (16)	$2.93\pm 0.73$ (14)
$8\!\times\!8$ / $16\!\times\!16$	$3.37\pm 0.60$ (19)	$3.06\pm 0.73$ (18)	$3.25\pm 0.61$ (24)	$3.25\pm 0.68$ (24)
$8\!\times\!8$ / $32\!\times\!32$	$3.12\pm 0.60$ (17)	$2.94\pm 0.66$ (17)	$3.38\pm 0.50$ (16)	$3.27\pm 0.65$ (11)

Appendix

Table 10: Number of shared head indices among the four heads with the largest nonzero fractions, comparing the same projection across each pair of sparsity granularities. Parentheses give the number of valid layers.

Figure 20: Effective-rank ratio and sparsity during Llama-350M pretraining. Rows correspond to block-

16\!\times\!16

, row-/column-wise, and

2\!:\!4

sparsity; columns show the O projection in layer 3, down projection in layer 7, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity (the zero fraction), respectively; red lines show the corresponding controller targets. The statistics are unsmoothed.

Figure 21: Effective-rank ratio and sparsity during Llama-1B pretraining. Rows correspond to block sizes

8

16

32

, and

64

; columns show the O projection in layer 3, down projection in layer 7, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity, respectively; red lines indicate their targets. The statistics are unsmoothed.

Figure 22: Effective-rank ratio and sparsity during Qwen3-1.7B distillation with block-

64\!\times\!64

S-SALAAD. Columns show the randomly selected K projection in layer 4, O projection in layer 17, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity, respectively; red lines indicate their targets. The statistics are unsmoothed.

Panel of Figure 23 — Figure 23: Perplexity against throughput on an RTX 2080 Ti; marker radius linearly encodes peak GPU memory usage, with values annotated in GB. Lighter red denotes the $16\!\times\!16$ variant. Both panels share the legend in the right panel.

Panel of Figure 24 — Figure 24: Perplexity against throughput on 4 $\times$ AMD EPYC 7662 CPUs (batch size 8, sequence length 64); marker radius is proportional to peak RAM usage, with values annotated in GB. Lighter red denotes the $16\!\times\!16$ variant. Both panels share the legend in the right panel.

Figure 25: Compressed Qwen3-1.7B: speedup and perplexity under a

3.4\text{\,}\mathrm{GB}

peak-memory cap. We increase the batch size until the cap.

Panel of Figure 26 — Figure 26: Perplexity against throughput on Llama for different block granularities, including the $L$ -only baseline. Marker radius encodes peak GPU memory with min–max scaling or peak RAM with proportional scaling, per panel.

Method	Budget	Params. $\downarrow$	Perplexity $\downarrow$	Peak mem. $\downarrow$	Latency $\downarrow$	Throughput $\uparrow$
	(M)	(M)		(GB)	(ms)	( $\mathrm{ktoken}\text{\,}{\mathrm{s}}^{-1}$ )
Llama-1B
Vanilla	–	1364.8	12.58	4.08	1116	29.35
	500	500.0	17.91	2.35	560	58.47
	550	550.0	15.54	2.45	607	53.99
	600	600.0	14.53	2.55	647	50.61

Appendix

Table 11: Numerical results for Figure 4 on a Quadro RTX 6000. FP16; batch size

\times

sequence length is

128\!\times\!256

for Llama and

32\!\times\!512

for Qwen. Arrows indicate preferred directions. Within each model, bold and underlined values mark the best and second-best distinct values per metric across the listed settings, at the displayed precision. Blue rows denote S-SALAAD, including its

L

-only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD

L

-only checkpoint, respectively.

Method	Budget	Params. $\downarrow$	Perplexity $\downarrow$	Peak mem. $\downarrow$	Latency $\downarrow$	Throughput $\uparrow$
	(M)	(M)		(GB)	(ms)	( $\mathrm{ktoken}\text{\,}{\mathrm{s}}^{-1}$ )
Llama-1B
Vanilla	–	1364.8	12.58	4.08	1683	19.47
	500	500.0	17.91	2.35	749	43.75
	550	550.0	15.54	2.45	820	39.98
	600	600.0	14.53	2.55	884	37.08

Appendix

Table 12: Numerical results for Figure 23 on an RTX 2080 Ti. FP16; batch size

\times

sequence length is

128\!\times\!256

for Llama and

32\!\times\!512

L

-only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD

L

-only checkpoint, respectively.

Method	Budget	Params. $\downarrow$	Perplexity $\downarrow$	Peak mem. $\downarrow$	Latency $\downarrow$	Throughput $\uparrow$
	(M)	(M)		(GB)	(ms)	( $\mathrm{token}\text{\,}{\mathrm{s}}^{-1}$ )
Llama-1B
Vanilla	–	1364.8	12.58	5.58	6705	76.36
	500	500.0	17.91	2.15	2410	212.46
	550	550.0	15.54	2.34	2681	190.96
	600	600.0	14.53	2.55	3016	169.77

Appendix

Table 13: Numerical results for Figure 24 on 4

\times

AMD EPYC 7662 CPUs. FP32; batch size 8 and sequence length 64 for both models. Arrows indicate preferred directions. Within each model, bold and underlined values mark the best and second-best distinct values per metric across the listed settings, at the displayed precision. Blue rows denote S-SALAAD, including its

L

-only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD

L

-only checkpoint, respectively.

Method	Budget	Params. $\downarrow$	Perplexity $\downarrow$	Peak mem. $\downarrow$	Latency $\downarrow$	Throughput $\uparrow$
	(M)	(M)		(GB)	(ms)
Llama-350M, Quadro RTX 6000 ( $\mathrm{ktoken}\text{\,}{\mathrm{s}}^{-1}$ )
Vanilla	–	374.1	15.83	0.92	84	97.98
	170	170.0	42.90	0.52	63	129.90
	180	180.0	31.78	0.54	65	126.32
	200	200.0	22.29	0.58	69	118.75

Appendix

Table 14: Additional numerical results for Figure 26 . Throughput units appear in the group headings. Arrows indicate preferred directions. Bold and underlined values mark the best and second-best distinct values at the displayed precision within each complete plotted panel, including entries referenced in Tables 11 – 13 .

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching
Aug 23, 2026Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4Parallel LLM Inference LLM Inference
Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA
Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Oct 5, 2026Arnav Kundu, Zhaoyang Xu, Bairu Hou +3On-Device Language Model Inference Dynamic Neural Networks
Apple
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
May 5, 2026Mohammed Sabry, Anya BelzLLM Compression Low-Rank Adaptation
ADAPT Centre, Dublin City University, Ireland

Aug 23, 2026J/K move · Enter open · S save

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

Nobel Dhar, Md Romyull Islam, Xuechen Zhang, Gongjin Sun+3

Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA

Parallel LLM Inference LLM Inference LLM Inference Acceleration On-Device Language Model Inference

Open paper AskPDF arXiv

Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.

Abstract

LLM Compression

Explore similar work

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference