Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models

Andrew Kiruluta

cs.CLMar 22, 2026

Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models

Authors: Andrew Kiruluta

Organizations: UC Berkeley School of Information

Abstract

Large language models deliver strong generative performance but at the cost of massive parameter counts, memory use, and decoding latency. Prior work has shown that pruning and structured sparsity can preserve accuracy under substantial compression, while prompt-compression methods reduce latency by removing redundant input tokens. However, these two directions remain largely separate. Most model-compression methods are static and optimized offline, and they do not exploit the fact that different prompts and decoding steps activate different latent computational pathways. Prompt-compression methods reduce sequence length, but they do not adapt the executed model subnetwork. We propose a unified compressed-sensing-guided framework for dynamic LLM execution. Random measurement operators probe latent model usage, sparse recovery estimates task-conditioned and token-adaptive support sets, and the recovered supports are compiled into hardware-efficient sparse execution paths over blocks, attention heads, channels, and feed-forward substructures. The framework introduces five key contributions: task-conditioned measurements, so different prompts induce different sparse supports; token-adaptive recovery, so active substructures are re-estimated during decoding; formal sample-complexity bounds under restricted isometry or mutual incoherence assumptions; compile-to-hardware constraints that restrict recovery to GPU-efficient structures; and a joint objective that unifies prompt compression with model reduction. Together, these components recast LLM inference as a measurement-and-recovery problem with explicit approximation guarantees and deployment-oriented speedup constraints.

Figures & tables

Figure 1: Uncertainty-Driven Sensing (UDS) feedback loop for dynamic sparse LLM execution. At decoding step $t$ , the system uses the predictive entropy of the preceding token distribution to adapt the measurement budget $m_{t}$ , increasing the number of probes in high-uncertainty regimes and reducing sensing effort when the model is confident. The updated budget determines the size of the sensing matrix $A_{t}$ , which produces the compressed measurements used for sparse support recovery under hardware-aware constraints. The recovered support is then compiled into a sparse execution kernel that generates the next token, whose predictive distribution closes the dashed feedback loop by informing the entropy estimate for the following step.

Parameter	Description	Role in Stability
$L_{H}$	Entropy sensitivity constant	Measures how strongly predictive uncertainty increases in response to support recovery error. Large $L_{H}$ makes the loop more fragile.
$\gamma$	Budget gain	Controls how aggressively the sensing controller increases the measurement budget in response to uncertainty. Large $\gamma$ can induce oscillation.
$\dfrac{\partial\mathcal{G}}{\partial m}$	Sensing efficiency slope	Measures how much each additional probe reduces recovery error. A steeper magnitude improves loop damping and stabilizes adaptive sensing.

Table 1: Key parameters governing the local stability of the uncertainty-driven sensing loop.

Method	Compression axis	Representative reported gain	Quality retention	Dynamic at inference?	Relevance to proposed method
Dense baseline	None	$1.0\times$ speed, $1.0\times$ memory	Reference quality ceiling	No	Serves as the accuracy and latency reference point for all Pareto comparisons.
SparseGPT	Offline unstructured / semi-structured weight pruning	One-shot pruning to $\geq 50\%$ sparsity with minimal loss; up to $60\%$ sparsity with negligible perplexity increase on very large GPT-family models	Strong perplexity retention under offline pruning	No	Strong static pruning baseline; demonstrates that weight sparsity is feasible, but does not provide prompt-conditioned or token-adaptive execution.
Wanda	Offline activation-aware weight pruning	LLaMA-7B at unstructured $50\%$ sparsity: perplexity $7.26$ vs. dense $5.68$ ; mean zero-shot accuracy $54.21$ vs. dense $59.99$ ; structured 2:4 matmul speedups around $1.54\times$ – $1.63\times$ at layer level	Good quality retention, especially after fine-tuning	No	Strong activation-aware pruning reference; useful for comparing offline support estimation against online compressed-sensing recovery.
ZipLM	Structured inference-aware pruning	GPT-2 reported as $60\%$ smaller and $30\%$ faster than DistilGPT2	Strong accuracy-vs-speed trade-off	No	Closest static structured baseline; highlights the importance of compile-to-hardware constraints but remains offline and task-agnostic.
LLMLingua	Prompt compression	Up to $20\times$ prompt compression with little performance loss	Strong faithfulness under prompt reduction	No	Best prompt-only baseline; reduces input cost but does not adapt the executed subnetwork.
LongLLMLingua	Long-context prompt compression	Reported $1.6\times$ end-to-end latency speedup in long-context settings, with improved average score in the cited configuration	Strong in long-context retrieval regimes	No	Critical long-context baseline; tests whether joint prompt + model compression dominates prompt-only compression.

Table 2: Representative comparison against related state-of-the-art methods. The figures above are literature-reported headline results drawn from different experimental settings and should be used for positioning rather than as a substitute for matched-hardware comparisons. The central empirical hypothesis of the proposed method is that joint prompt–model compression with compressed-sensing-guided token-adaptive support recovery can achieve a strictly better quality–latency frontier than any single-axis compression strategy.

Method	Prompt compression ratio	Executed support fraction	Support type	Task quality	Prefill latency	Decode latency	Net speedup vs. dense
Dense baseline	$1.0\times$	$100\%$	Dense	$100\%$ normalized	$1.00\times$	$1.00\times$	$1.00\times$
SparseGPT (estimated)	$1.0\times$	$50\%$ – $60\%$	Static weight sparse	$97\%$ – $99\%$ of dense	$0.95\times$ – $1.00\times$	$1.05\times$ – $1.12\times$	$1.03\times$ – $1.10\times$
Wanda (estimated)	$1.0\times$	$50\%$ (or 2:4 structured equivalent)	Static weight sparse	$95\%$ – $98\%$ of dense	$1.00\times$	$1.20\times$ – $1.25\times$	$1.15\times$ – $1.22\times$
ZipLM (estimated)	$1.0\times$	$45\%$ – $55\%$	Static structured sparse	$96\%$ – $98\%$ of dense	$1.05\times$	$1.25\times$ – $1.35\times$	$1.20\times$ – $1.30\times$
LLMLingua (estimated)	$4.0\times$ – $8.0\times$	$100\%$	Dense model	$96\%$ – $99\%$ of dense	$1.35\times$ – $1.60\times$	$1.00\times$	$1.18\times$ – $1.35\times$
LongLLMLingua (estimated)	$3.0\times$ – $6.0\times$	$100\%$	Dense model	$97\%$ – $101\%$ of dense	$1.45\times$ – $1.75\times$	$1.00\times$ – $1.05\times$	$1.35\times$ – $1.60\times$

Table 3: Estimated matched-hardware comparison for a single-GPU serving stack. All entries except the dense baseline are literature-calibrated estimates rather than directly measured results, and should be replaced by empirical values in the final paper. “Task quality” is normalized relative to dense inference on the same benchmark, so

100\%

denotes parity with the dense baseline. Prefill and decode latency columns are shown as relative throughput improvements versus dense, where larger is better.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Oct 8, 2026Qitong Wang, Xinwei Niu, Mingluo Su +3LLM Pruning LLM Inference Acceleration
Westlake University · Ant Group
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sep 10, 2026Sankar Behera, Dhruv Singh, Anshika Agnihotri +3LLM Pruning LLM Compression
CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Oct 8, 2026J/K move · Enter open · S save

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao+2

Westlake University · Ant Group

LLM Pruning LLM Inference Acceleration Language Model Decoding Structured Sparsity

Open paper AskPDF arXiv

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

Abstract

LLM Inference Acceleration

Explore similar work

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression