Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
Authors: Tianxiang Xia, Kai Lion, Niao He, Michael Muehlebach, Melanie Zeilinger, Hao Ma
Organizations: ETH Zürich, Zürich · Max Planck Institute for Intelligent Systems, Tübingen · Vienna University of Economics and Business, Vienna · The University of Hong Kong
Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unreliable network connectivity, communication delays, and privacy requirements can make such reliance impractical, motivating local execution on edge devices. Local deployment requires models to fit within limited memory and execute efficiently with the available computational resources. Recent work has shown that learning sparse-plus-low-rank representations enables elastic compression of a single checkpoint to multiple memory budgets. However, fewer parameters do not necessarily translate into faster inference. Indexing overhead and irregular memory access in unstructured sparse components can offset computational savings. To address this critical deployment barrier, we introduce S-SALAAD, a unified framework that jointly learns the model weights and their low-rank and structured sparse components during training. The framework supports a family of hardware-compatible structured sparsity patterns. It couples this structure learning with budget-aware compression and dedicated CPU and GPU kernels that exploit the learned structure for acceleration, enabling efficient inference from a single checkpoint across memory budgets. Experiments on Llama2 and Qwen3 models, covering both pretraining and distillation, show that S-SALAAD advances the empirical Pareto frontier in perplexity, memory, and inference latency across multiple hardware platforms relative to the evaluated baselines. These results show that structured learning can translate elastic compressibility into reduced memory requirements and lower inference latency, enabling practical deployment of large models on edge devices.
Figures & tables
Figure 2: Pipeline of Stage I.
Figure 3: Llama-1B training and learned S in a randomly selected layer. Log shading shows entry magnitudes for SALAAD ( 3(b) ) and 64×64 block Frobenius norms for S-SALAAD ( 3(c) , 3(d) ), clipped to the displayed range.
Figure 4: Perplexity against throughput on a Quadro RTX 6000; marker radius linearly encodes peak GPU memory usage, with values annotated in GB. Lighter red denotes the 16×16 variant.
Llama-1B
Qwen3-1.7B
Task
Vanilla
800M
700M
600M
500M
Vanilla
1273M
1200M
1000M
800M
PIQA
0.72
0.71
0.70
0.69
0.68
0.73
0.72
0.72
0.72
0.70
SciQ
0.79
0.76
0.76
0.76
0.77
0.92
0.92
0.91
0.90
0.89
WinoG.
0.54
0.54
0.54
0.54
0.54
0.57
0.58
0.59
0.58
0.56
HellaSwag
0.39
0.37
0.37
0.37
0.35
0.46
0.45
0.44
0.43
0.40
ARC-Easy
0.54
0.51
0.51
0.51
0.50
0.76
0.74
0.73
0.72
0.68
Table 1: Zero-shot accuracy of Vanilla and S-SALAAD (in blue) across budgets.
Figure 5: Perplexity vs. parameter count on Llama-350M.
Figure 6: Perplexity versus post-compression parameter count for different block sizes on Llama.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Static weight memory and throughput of dense inference and the unstructured SLR representation used by SALAAD. Both metrics are normalized to their respective dense baselines. The benchmark uses a synthetic Llama-1B MLP down projection with a 15% rank ratio and 5% nonzero density.
Block size
Ptgt=500 M
Ptgt=600 M
Ptgt=700 M
1×1
14.11
13.86
13.78
2×2
17.29
14.93
14.19
4×4
87.44
28.89
18.28
8×8
456.52
116.33
43.58
16×16
1497.68
351.39
98.83
Appendix
Table 2: Post-compression perplexity of the same unstructured SALAAD Llama-1B checkpoint when block structure of size p×p is imposed during compression, at three parameter budgets Ptgt . The first row is the element-wise reference.
Figure 8: Post-compression perplexity on Llama-350M.
Figure 9: Post-compression perplexity for exact and iterative SVD on Llama-350M at matched budgets.
Figure 10: ΓL trajectories for exact and iterative SVD on the down projections of three representative Llama-350M layers. Each inset magnifies the converged regime, where the two curves are otherwise indistinguishable.
Figure 12
Figure 13: Kernel efficiency on a randomly sparsified Llama-1B MLP down projection at 15% rank ratio and 5% density, normalized to dense execution. Elem. and Block denote PyTorch’s torch.sparse_csr_tensor and torch.sparse_bsr_tensor backends, respectively. In the block-sparse setting both backends run at 16×16 on the GPU and 8×8 on the CPU, the smallest block size each platform supports.
Figure 14: Block-size sweep on random SLR weights for a 2048×5632 Llama-1B down projection, measured on a Quadro RTX 6000 in FP16 with batch size 128 and sequence length 256. Throughput is normalized to dense.
Setting
Llama-350M
Llama-1B
Qwen3-1.7B
Hidden size
1024
2048
2048
Intermediate (FFN) size
2816
5632
6144
Layers
24
24
28
Attention heads (Q / KV)
16 / 16 (MHA)
32 / 32 (MHA)
16 / 8 (GQA)
Head dimension
64
64
128
Vocabulary size
32,128
32,128
131,072
Appendix
Table 3: Model architectures used for Llama 2 pretraining and Qwen3 distillation.
Table 16
Variant
Llama-350M
Llama-1B
Qwen3-1.7B
Δα
Δβ
Δα
Δβ
Δα
Δβ
Unstructured SALAAD
0.20
0.001
0.20
0.001
—
—
Block- 8×8
0.04
0.01
0.045
0.01
—
—
Block- 16×16
0.04
0.01
0.04
0.01
—
—
Block- 32×32
0.04
0.01
0.04
0.01
—
—
Block- 64×64
0.02
0.02
0.02
0.02
0.02
0.02
Appendix
Table 6: Controller step sizes across model scales and sparsity variants.
Component
Value
GPU
NVIDIA Quadro RTX 6000 (Turing, TU102)
GPU memory
24 GB GDDR6
GPU driver / CUDA
NVIDIA driver 550.x / CUDA 12.9
CPU
4 × AMD EPYC 7662 CPUs
RAM
32 GB
PyTorch
2.13.0 (cu129 build)
Appendix
Table 7: Profiling environment for the Quadro RTX 6000 and CPU measurements.
Hardware
Model
Batch × length
Warmup
Timed
Quadro RTX 6000
Llama-1B
128×256
10
20
Qwen3-1.7B
32×512
10
20
RTX 2080 Ti
Llama-1B
128×256
10
20
Qwen3-1.7B
32×512
10
20
4 × AMD EPYC 7662 CPUs
Llama-1B
8×64
10
20
Qwen3-1.7B
8×64
3
6
Appendix
Table 8: Input shapes and repetition counts for inference profiling.
Figure 15: Task-loss trajectories for (a) different sparsity patterns on Llama-350M, (b) different block sizes on Llama-1B, and (c) Qwen3-1.7B distillation. Curves use a moving average over approximately 1,000 training steps and end at the last available logged step.
Figure 16: Learned sparse components S in layer 16 of Llama-350M. Rows show block- 16×16 , row-/column-wise, and 2:4 training; columns show the Q, O, and down projections. Insets enlarge the red-boxed regions of each down projection on the same color scale; their labels give the crop dimensions in weights. The displayed matrices are taken directly from the trained checkpoints; the 2:4 row shows the learned components before any additional structural projection.
Figure 17: Learned sparse components S in layer 16 of Llama-1B across four block sizes. Columns show the Q, O, and down projections. Insets enlarge the red-boxed down-projection regions on the same color scale; labels give the crop dimensions in weights. Each inset contains 6×6 native blocks.
Figure 18: Learned sparse components S in layer 16 of Qwen3-1.7B trained with block- 64×64 S-SALAAD. Columns show the Q, O, and down projections. The down-projection inset enlarges the red-boxed 384×384 region on the same color scale, showing 6×6 native blocks.
Figure 19: Paired sparsity patterns in a randomly selected layer of Llama-1B. Rows show unstructured sparsity ( 1×1 ) followed by four block sizes. Dark cells are nonzero and white cells are exactly zero. The O component is transposed so that its head slices align with those of V along rows.
Block size
Q/K correlation (layers)
V/O correlation (layers)
Unstructured ( 1×1 )
0.971±0.096 (24)
0.976±0.024 (24)
8×8
0.986±0.025 (23)
0.962±0.051 (24)
16×16
0.979±0.039 (23)
0.948±0.076 (24)
32×32
0.980±0.018 (21)
0.937±0.078 (24)
64×64
0.967±0.022 (21)
0.909±0.127 (24)
Appendix
Table 9: Mean ± sample standard deviation of layer-wise Pearson correlations of head nonzero fractions. Parentheses give the number n of valid layers for each pair, excluding layers in which either projection has an entirely zero sparse component.
Granularity pair
Q
K
V
O
1×1 / 8×8
3.04±0.77 (23)
2.73±0.88 (22)
3.08±0.72 (24)
2.88±0.61 (24)
1×1 / 16×16
3.11±0.88 (19)
2.72±0.83 (18)
2.92±0.78 (24)
2.92±0.72 (24)
1×1 / 32×32
3.29±0.77 (17)
3.00±0.61 (17)
3.00±0.63 (16)
2.91±0.83 (11)
1×1 / 64×64
3.29±0.73 (14)
2.75±0.68 (16)
3.12±0.81 (16)
2.93±0.73 (14)
8×8 / 16×16
3.37±0.60 (19)
3.06±0.73 (18)
3.25±0.61 (24)
3.25±0.68 (24)
8×8 / 32×32
3.12±0.60 (17)
2.94±0.66 (17)
3.38±0.50 (16)
3.27±0.65 (11)
Appendix
Table 10: Number of shared head indices among the four heads with the largest nonzero fractions, comparing the same projection across each pair of sparsity granularities. Parentheses give the number of valid layers.
Figure 20: Effective-rank ratio and sparsity during Llama-350M pretraining. Rows correspond to block- 16×16 , row-/column-wise, and 2:4 sparsity; columns show the O projection in layer 3, down projection in layer 7, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity (the zero fraction), respectively; red lines show the corresponding controller targets. The statistics are unsmoothed.
Figure 21: Effective-rank ratio and sparsity during Llama-1B pretraining. Rows correspond to block sizes 8 , 16 , 32 , and 64 ; columns show the O projection in layer 3, down projection in layer 7, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity, respectively; red lines indicate their targets. The statistics are unsmoothed.
Figure 22: Effective-rank ratio and sparsity during Qwen3-1.7B distillation with block- 64×64 S-SALAAD. Columns show the randomly selected K projection in layer 4, O projection in layer 17, and K projection in layer 0. Black solid and dashed curves show the rank ratio and sparsity, respectively; red lines indicate their targets. The statistics are unsmoothed.
Figure 23: Perplexity against throughput on an RTX 2080 Ti; marker radius linearly encodes peak GPU memory usage, with values annotated in GB. Lighter red denotes the 16×16 variant. Both panels share the legend in the right panel.
Figure 24: Perplexity against throughput on 4 × AMD EPYC 7662 CPUs (batch size 8, sequence length 64); marker radius is proportional to peak RAM usage, with values annotated in GB. Lighter red denotes the 16×16 variant. Both panels share the legend in the right panel.
Figure 25: Compressed Qwen3-1.7B: speedup and perplexity under a 3.4GB peak-memory cap. We increase the batch size until the cap.
Figure 26: Perplexity against throughput on Llama for different block granularities, including the L -only baseline. Marker radius encodes peak GPU memory with min–max scaling or peak RAM with proportional scaling, per panel.
Method
Budget
Params. ↓
Perplexity ↓
Peak mem. ↓
Latency ↓
Throughput ↑
(M)
(M)
(GB)
(ms)
( ktokens−1 )
Llama-1B
Vanilla
–
1364.8
12.58
4.08
1116
29.35
500
500.0
17.91
2.35
560
58.47
550
550.0
15.54
2.45
607
53.99
600
600.0
14.53
2.55
647
50.61
Appendix
Table 11: Numerical results for Figure 4 on a Quadro RTX 6000. FP16; batch size × sequence length is 128×256 for Llama and 32×512 for Qwen. Arrows indicate preferred directions. Within each model, bold and underlined values mark the best and second-best distinct values per metric across the listed settings, at the displayed precision. Blue rows denote S-SALAAD, including its L -only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD L -only checkpoint, respectively.
Method
Budget
Params. ↓
Perplexity ↓
Peak mem. ↓
Latency ↓
Throughput ↑
(M)
(M)
(GB)
(ms)
( ktokens−1 )
Llama-1B
Vanilla
–
1364.8
12.58
4.08
1683
19.47
500
500.0
17.91
2.35
749
43.75
550
550.0
15.54
2.45
820
39.98
600
600.0
14.53
2.55
884
37.08
Appendix
Table 12: Numerical results for Figure 23 on an RTX 2080 Ti. FP16; batch size × sequence length is 128×256 for Llama and 32×512 for Qwen. Arrows indicate preferred directions. Within each model, bold and underlined values mark the best and second-best distinct values per metric across the listed settings, at the displayed precision. Blue rows denote S-SALAAD, including its L -only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD L -only checkpoint, respectively.
Method
Budget
Params. ↓
Perplexity ↓
Peak mem. ↓
Latency ↓
Throughput ↑
(M)
(M)
(GB)
(ms)
( tokens−1 )
Llama-1B
Vanilla
–
1364.8
12.58
5.58
6705
76.36
500
500.0
17.91
2.15
2410
212.46
550
550.0
15.54
2.34
2681
190.96
600
600.0
14.53
2.55
3016
169.77
Appendix
Table 13: Numerical results for Figure 24 on 4 × AMD EPYC 7662 CPUs. FP32; batch size 8 and sequence length 64 for both models. Arrows indicate preferred directions. Within each model, bold and underlined values mark the best and second-best distinct values per metric across the listed settings, at the displayed precision. Blue rows denote S-SALAAD, including its L -only special case. For the Llama baselines, (V) and (L) denote compression of a vanilla-pretrained and of an S-SALAAD L -only checkpoint, respectively.
Method
Budget
Params. ↓
Perplexity ↓
Peak mem. ↓
Latency ↓
Throughput ↑
(M)
(M)
(GB)
(ms)
Llama-350M, Quadro RTX 6000 ( ktokens−1 )
Vanilla
–
374.1
15.83
0.92
84
97.98
170
170.0
42.90
0.52
63
129.90
180
180.0
31.78
0.54
65
126.32
200
200.0
22.29
0.58
69
118.75
Appendix
Table 14: Additional numerical results for Figure 26 . Throughput units appear in the group headings. Arrows indicate preferred directions. Bold and underlined values mark the best and second-best distinct values at the displayed precision within each complete plotted panel, including entries referenced in Tables 11 – 13 .
Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.
Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4
Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.
We study distillation for large language models under explicit compute constraints, with the goal of producing student models that are not only cheaper to train, but structurally efficient at inference time. While prior approaches to parameter-efficient distillation, such as LoRA, reduce adaptation cost, they leave the dense backbone unchanged and therefore fail to deliver meaningful inference savings. We propose Budgeted LoRA, a distillation framework that treats model compression as a structured compute allocation problem. Instead of using a fixed student architecture, we introduce a global compute budget that sets the final target fraction of dense computation retained. Under this constraint, the model redistributes capacity across dense and low-rank pathways via (i) module-level dense retention coefficients, (ii) adaptive low-rank allocation, and (iii) post-training compression that selectively removes, approximates, or preserves dense components. This formulation yields a family of students controlled by a single budget dial. Empirically, Budgeted LoRA matches standard LoRA perplexity at a moderate budget with a 1.74x compressed-module speedup; at an aggressive budget it achieves a 4.05x speedup with moderate perplexity degradation, and it preserves higher accuracy on function-style in-context learning probes. These results suggest that, under compute-constrained distillation, retaining behavior is less about matching perplexity or removing more parameters than it is about controlling how dense computation is transferred to low-rank pathways.