Task-Aware Spectral Pruning: A Mixture-of-Masks Framework for Efficient LLM Inference

Ibne Farabi Shihab; Fariya Afrin; Sanjeda Akter; Anuj Sharma

cs.LGAug 24, 2026

Task-Aware Spectral Pruning: A Mixture-of-Masks Framework for Efficient LLM Inference

Authors: Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Civil, Construction & Environmental Engineering, Iowa State University

Abstract

Static pruning imposes one sparse structure on every prompt, even though reasoning, retrieval, generation, coding, and translation can depend on different parts of a language model. We introduce Task-Aware Spectral Pruning (TASP), a post-training framework that calibrates module-level spectral descriptors against measured task-specific ablation effects, closes grouped-query-attention and SwiGLU dependencies during sparse-mask construction, and routes each user turn to one compiled mask that remains fixed throughout prefill and decoding. A module-disjoint pilot first determines whether the spectral signal is informative before full calibration. Under the stated retrospective operating rule, the pilot passes on the evaluated Llama-3-8B and Llama-3-70B checkpoints but rejects Qwen2.5-1.5B, demonstrating that applicability is model-dependent rather than universal. At a 43% active-FLOP reduction, the Llama-3-70B benchmark harness retains 97.7 +/- 0.2% of the dense BF16 score. In the deployment-matched INT8-weight/BF16-compute runtime on a single A100 80GB, the compiled sparse path retains 97.3 +/- 0.2% relative to dense BF16 and reduces decode latency from 45.2 +/- 0.4 to 31.3 +/- 0.4 ms/token, yielding a 1.44x speedup. Factorized ablations, disjoint-module tests, compiled structured baselines, routing-corruption studies, and an explicit 136-GPU-hour calibration audit further delimit the source and operating regime of these gains

Figures & tables

Panel of Figure 1 — Figure 1: The complete TASP pipeline. The predeployment gate prevents a universal spectral heuristic from being applied when its out-of-fold signal is weak. A passing model proceeds to task-specific calibration and dependency-closed mask construction. At inference, routing and dispatch occur once per user turn, and the selected compiled plan remains fixed through prefill and decoding so that its KV cache stays valid.

Component	Configuration	Component	Configuration
70B candidates	80 layers; 64 query-head candidates and 28 coupled 1,024-channel SwiGLU groups per layer; 7,360 total	Module split	Stratified by layer and type: 70/15/15% train/validation/untouched test, or 5,152/1,104/1,104 modules
Pilot	Validation modules; first 16 sequences per module–task pair; stratified five-fold outer module CV	Ablation labels	32 sequences per train/validation module–task pair; 16 per untouched-test pair; batch size 8
Targets	Normalized gold-answer log-likelihood for reasoning, retrieval, translation; teacher-forced completion NLL for code, generation	Feature preprocessing	Per-task z-normalization fitted only on the modules available to that fitting split
Predictor	Ridge regression; coefficient grid $\{10^{-4},10^{-3},10^{-2},10^{-1},1,10,10^{2}\}$	Ridge selection	Lowest validation MSE; ties resolved by validation AUROC
Mask objective	Greedy predicted sensitivity per active FLOP with dependency closure; 0.57 active-FLOP target	Dependencies	Retain a KV head with any associated query head; jointly retain/remove gate, up, and down channels in groups of 1,024
Router	Three-layer MLP with hidden widths 512 and 256	Router optimization	AdamW; learning rate $3\!\times\!10^{-4}$ ; weight decay 0.01; batch 64; at most 20 epochs; patience 3

Table 1: Complete calibration, construction, routing, and runtime configuration. The 70B model uses INT8 weight-only residency with BF16 compute; it does not use BF16 weight residency, tensor parallelism, CPU offload, or NVMe offload.

Model	Mean	Minimum	Mean LCB	Gate
Qwen2.5-1.5B	0.53	0.51	0.50	reject
Llama-3-8B	0.66	0.63	0.62	pass
Llama-3-70B	0.78	0.74	0.75	pass

Table 2: Out-of-fold pilot AUROC. “Minimum” is the lowest diagnostic-family AUROC, and “Mean LCB” is the hierarchical-bootstrap 95% lower bound for the mean.

Method	MMLU	GSM8K	MGSM	AlpacaEval	NQ-Open	HumanEval	WMT14 avg.	Full retention
Dense BF16	79.2	87.5	74.3	82.1	68.4	72.6	28.3	100.0
Wanda	$73.8{\pm}0.3$	$77.1{\pm}0.5$	$68.2{\pm}0.4$	$75.6{\pm}0.4$	$64.3{\pm}0.3$	$61.2{\pm}0.5$	$25.6{\pm}0.2$	$90.7{\pm}0.3$
SliceGPT	$75.4{\pm}0.3$	$79.6{\pm}0.5$	$69.9{\pm}0.4$	$77.2{\pm}0.3$	$65.8{\pm}0.3$	$64.2{\pm}0.5$	$26.5{\pm}0.2$	$93.2{\pm}0.3$
LLaMaFlex	$75.8{\pm}0.3$	$80.2{\pm}0.5$	$70.4{\pm}0.3$	$77.9{\pm}0.3$	$66.1{\pm}0.2$	$64.8{\pm}0.4$	$26.9{\pm}0.2$	$94.0{\pm}0.3$
TASP -Single	$75.1{\pm}0.3$	$79.8{\pm}0.4$	$70.4{\pm}0.3$	$77.3{\pm}0.3$	$65.9{\pm}0.2$	$63.4{\pm}0.4$	$26.4{\pm}0.2$	$93.3{\pm}0.3$
TASP - MoM	$77.9{\pm}0.3$	$85.2{\pm}0.4$	$72.8{\pm}0.3$	$80.4{\pm}0.3$	$67.2{\pm}0.2$	$69.8{\pm}0.5$	$27.6{\pm}0.2$	$97.7{\pm}0.2$

Table 3: Representative Llama-3-70B benchmark results in the dense-BF16 scientific harness at 43% active-FLOP reduction. The reasoning column is original four-choice five-shot MMLU ( 16 ) ; a separate genuine MMLU-Pro evaluation is reported in Appendix N . Full retention additionally includes MT-Bench, TriviaQA, MBPP, and both WMT14 directions. TASP uncertainty is SD across five independent calibration and mask runs; deterministic-baseline uncertainty is one paired-bootstrap SE over evaluation instances. The complete comparison appears in Appendix D .

Method	Selected density	Active FLOPs	Ret. vs. BF16	Ret. vs. INT8	Prefill (ms)	Decode (ms/tok.)	Speedup
Dense INT8 runtime	$1.00\times$	$1.00\times$	$99.6{\pm}0.1$	$100.0$	$1{,}814{\pm}18$	$45.2{\pm}0.4$	$1.00\times$
TASP - MoM mask-as-zero	$0.57\times$	$\approx 1.00\times$	$97.3{\pm}0.2$	$97.7{\pm}0.2$	$1{,}806{\pm}19$	$43.8{\pm}0.5$	$1.03\times$
ShortGPT, compiled	$0.57\times$	$0.57\times$	$91.5{\pm}0.4$	$91.9{\pm}0.4$	$1{,}118{\pm}15$	$27.8{\pm}0.3$	$1.63\times$
LLM-Pruner, compiled	$0.57\times$	$0.57\times$	$92.0{\pm}0.3$	$92.4{\pm}0.3$	$1{,}371{\pm}17$	$35.7{\pm}0.4$	$1.27\times$
SliceGPT, compiled	$0.57\times$	$0.57\times$	$92.8{\pm}0.3$	$93.2{\pm}0.3$	$1{,}289{\pm}16$	$33.8{\pm}0.4$	$1.34\times$
TASP -Single, compiled	$0.57\times$	$0.57\times$	$93.0{\pm}0.3$	$93.4{\pm}0.3$	$1{,}342{\pm}16$	$31.9{\pm}0.4$	$1.42\times$

Table 4: Deployment-matched Llama-3-70B results on one A100 80GB. Active model FLOPs are analytical counts implied by the launched architecture, not hardware-counter measurements. Retention uncertainty for TASP rows is SD across five independently calibrated systems; uncertainty for deterministic baselines is one paired-bootstrap SE. Latency uncertainty is SD across five independent 200-run timing blocks for every row. The mask-as-zero control uses the exact TASP - MoM masks in dense kernels, separating mask quality from compiled execution.

Variant	Retention vs. dense BF16
Five masks, oracle router	$98.7{\pm}0.2$
Five masks, learned router	$97.7{\pm}0.2$
Without salience	$95.5{\pm}0.3$
Without spectral features	$93.6{\pm}0.4$
One global spectral mask	$93.3{\pm}0.3$
Five layer-matched random masks	$85.6{\pm}0.5$

Table 5: Factorized 70B ablation at the same 43% active-FLOP reduction. Values are mean

\pm

SD across five calibration and mask runs.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Inputs and split	Operation and output
Candidate formation	Frozen checkpoint and architecture metadata	Slice every query head and each 1,024-channel SwiGLU group; record GQA and gate/up/down dependencies.
Streaming profile	One BF16 layer or matrix group at a time	Compute spectral descriptors without placing the full BF16 checkpoint on the GPU.
Pilot labels	Validation modules; 16 sequences per module–task pair	Measure structural-ablation targets and obtain out-of-fold predictions through nested, stratified module CV.
Applicability gate	Four diagnostic-family AUROCs and hierarchical bootstrap	Stop if mean $<0.62$ , any family $<0.58$ , or the mean 95% LCB $<0.60$ ; otherwise finish label collection.
Final calibration	Train/validation modules and disjoint untouched test	Fit task-specific ridge predictors, select regularization on validation modules, and evaluate once on unseen modules and prompts.
Mask construction	Scores, active-FLOP budget, dependency graph	Greedily select by score per active FLOP, close GQA and SwiGLU dependencies, and emit one mask per task family.

Appendix

Table 6: The complete offline and online procedure. Only routing and dispatch are online.

Figure 2: Calibrated sensitivity estimation in TASP . Module-level spectral descriptors and constraint-token salience predict task-specific degradation under structural ablation. Normalization, ridge selection, and evaluation follow the module-disjoint protocol described in Section 3 .

Figure 3: Dependency-closed mask construction. Candidate modules are ranked by predicted sensitivity per active FLOP; GQA and SwiGLU dependencies are closed before a candidate is admitted, producing a hardware-valid static subnetwork at the target budget.

Figure 4: Inference-time routing and cache-consistent execution. One mask is chosen per user turn and held fixed through prefill and decoding. Temperature-scaled low-confidence routes use the dense INT8 path, and KV caches are not reused across incompatible masks.

Method	FLOP red.	MMLU	GSM8K	MGSM	AlpacaEval	NQ-Open	HumanEval	WMT14 avg.	Full retention
Dense BF16	0%	$79.2$	$87.5$	$74.3$	$82.1$	$68.4$	$72.6$	$28.3$	$100.0$
Magnitude	43%	$64.1{\pm}0.5$	$58.3{\pm}0.7$	$52.7{\pm}0.6$	$67.8{\pm}0.5$	$59.2{\pm}0.4$	$43.1{\pm}0.8$	$21.4{\pm}0.3$	$75.6{\pm}0.5$
SparseGPT	43%	$72.3{\pm}0.4$	$75.4{\pm}0.6$	$66.8{\pm}0.4$	$74.2{\pm}0.4$	$63.1{\pm}0.3$	$58.7{\pm}0.6$	$24.8{\pm}0.2$	$88.5{\pm}0.4$
Wanda	43%	$73.8{\pm}0.3$	$77.1{\pm}0.5$	$68.2{\pm}0.4$	$75.6{\pm}0.4$	$64.3{\pm}0.3$	$61.2{\pm}0.5$	$25.6{\pm}0.2$	$90.7{\pm}0.3$
Gradient	43%	$74.2{\pm}0.3$	$78.3{\pm}0.4$	$69.1{\pm}0.4$	$76.1{\pm}0.3$	$64.8{\pm}0.3$	$62.3{\pm}0.5$	$26.1{\pm}0.2$	$91.7{\pm}0.3$
AlphaPruning	43%	$75.0{\pm}0.3$	$79.4{\pm}0.5$	$69.7{\pm}0.4$	$76.9{\pm}0.3$	$65.4{\pm}0.3$	$63.8{\pm}0.5$	$26.4{\pm}0.2$	$92.9{\pm}0.3$

Appendix

Table 7: Llama-3-70B quality at matched active-FLOP reduction. The reasoning column is original four-choice five-shot MMLU ( 16 ) , corrected from the submitted “MMLU-Pro” label; the score is unchanged and a genuine MMLU-Pro run is reported separately in Appendix N . For TASP variants, uncertainty is SD across five independent calibration/mask runs. For deterministic baselines, it is one paired-bootstrap SE across evaluation instances; these intervals should not be interpreted as variation from five independently selected baseline artifacts.

Figure 5: Accuracy–FLOP frontier on Llama-3-70B. Relative retention is plotted against active decoding-FLOP reduction for TASP and representative pruning baselines. The matched 43% operating point is the only point for which all structured systems are additionally compiled and timed under the deployment runtime.

Panel of Figure 6 — Figure 6: Predeployment spectral–task separability. The figure reports only out-of-fold predictions from the pilot protocol.

Task	Pilot AUROC	Test AUROC	Change
Reasoning	0.82	0.80	$-0.02$
Retrieval	0.76	0.72	$-0.04$
Code	0.79	0.76	$-0.03$
Translation	0.74	0.73	$-0.01$
Mean	0.78	0.75	$-0.03$

Appendix

Table 8: Llama-3-70B out-of-fold pilot performance and final evaluation on untouched modules and disjoint prompts.

Llama-3-8B task family	TASP - MoM retention
Reasoning	$95.8{\pm}0.4$
Generation	$97.1{\pm}0.3$
Retrieval	$96.8{\pm}0.3$
Code	$95.6{\pm}0.5$
Translation	$96.7{\pm}0.3$
Full-suite mean	$96.4{\pm}0.3$

Appendix

Table 9: Direct Llama-3-8B deployment at 43% active-FLOP reduction. The full-suite mean is computed from eleven benchmark-level ratios; family rows are displayed for readability.

Selection rule	Mean degradation
TASP least-important groups	$-2.6{\pm}0.3$
Wanda least-important groups	$-4.8{\pm}0.3$
Layer-matched random groups	$-7.4{\pm}0.4$

Appendix

Table 10: Performance-point degradation after removing groups on untouched 70B modules and disjoint prompts. Values are mean

\pm

SD over five runs; smaller absolute degradation is better.

Rule	Reasoning	Retrieval	Code	Translation	Average
Random	$-8.7{\pm}0.5$	$-6.2{\pm}0.4$	$-9.4{\pm}0.6$	$-5.8{\pm}0.4$	$-7.5{\pm}0.3$
Magnitude	$-6.9{\pm}0.4$	$-5.4{\pm}0.3$	$-7.8{\pm}0.5$	$-4.9{\pm}0.3$	$-6.3{\pm}0.3$
Activation norm	$-5.8{\pm}0.3$	$-4.7{\pm}0.3$	$-6.4{\pm}0.4$	$-4.2{\pm}0.3$	$-5.3{\pm}0.2$
Wanda	$-4.9{\pm}0.3$	$-4.1{\pm}0.2$	$-5.7{\pm}0.4$	$-3.8{\pm}0.3$	$-4.6{\pm}0.2$
TASP	$-2.1{\pm}0.2$	$-2.4{\pm}0.2$	$-2.8{\pm}0.3$	$-1.9{\pm}0.2$	$-2.3{\pm}0.2$

Appendix

Table 11: Exploratory validation-module degradation after removing the groups ranked least important by each rule at matched structured sparsity. Smaller absolute degradation is better. Confirmatory claims use the disjoint protocol in Table 10 .

Figure 7: Taskwise visualization of the exploratory diagnostic in Table 11 . The figure is retained to show the family-level pattern, while the main evidence is the disjoint-module and disjoint-prompt analysis.

$\tau$	Ret. vs. INT8	Sparse use	ms/token	Speedup
0.50	$97.7{\pm}0.2$	$100{\pm}0\%$	31.3	$1.44\times$
0.70	$98.4{\pm}0.2$	$91{\pm}2\%$	32.6	$1.39\times$
0.85	$99.3{\pm}0.1$	$76{\pm}3\%$	34.6	$1.30\times$
0.95	$99.7{\pm}0.1$	$52{\pm}3\%$	38.0	$1.19\times$

Appendix

Table 12: Natural dense-fallback operating curve. Latency is mixture-implied from measured path latencies.

Top-1 accuracy	No fallback	With $\tau=0.85$	Fallback rate
100%	$98.7{\pm}0.2$	$98.7{\pm}0.2$	0%
90%	$97.8{\pm}0.2$	$98.4{\pm}0.2$	12%
80%	$96.6{\pm}0.3$	$98.0{\pm}0.2$	24%
70%	$95.2{\pm}0.4$	$97.5{\pm}0.3$	37%

Appendix

Table 13: Controlled routing-corruption study in the BF16 benchmark harness.

Labels	Held-out AUROC	TASP - MoM ret.
25%	$0.66{\pm}0.03$	$95.7{\pm}0.4$
50%	$0.70{\pm}0.02$	$96.7{\pm}0.3$
75%	$0.73{\pm}0.02$	$97.2{\pm}0.3$
100%	$0.75{\pm}0.02$	$97.7{\pm}0.2$

Appendix

Table 14: Sensitivity to calibration-label volume. Values are mean

\pm

SD over five stratified label subsets.

Scenario	Dense	First-turn locked	Per-turn routing
Math $\rightarrow$ creative	$75.2$	$71.8{\pm}0.4$	$74.1{\pm}0.4$
Code $\rightarrow$ translation	$68.4$	$64.9{\pm}0.5$	$67.1{\pm}0.5$
Retrieval $\rightarrow$ reasoning	$72.8$	$69.6{\pm}0.4$	$71.9{\pm}0.3$
Multi-task average	$72.1$	$68.8{\pm}0.4$	$71.0{\pm}0.4$

Appendix

Table 15: Eight-turn task-switching stress test. Uncertainty is SD across five runs.

Panel of Figure 8 — Figure 8: Corrected router fallback trade-off. Higher thresholds recover dense-INT8 retention by sending more requests to the dense path. Decode latency is mixture-implied and reported numerically in Table 12 .

Figure 9: Task-switching stress test across three representative domain transitions. Rerouting at each user turn substantially closes the gap created by locking a conversation to its first mask.

Task-mask pair	Jaccard overlap
Reasoning / Code	$0.78{\pm}0.02$
Generation / Translation	$0.81{\pm}0.02$
Reasoning / Generation	$0.24{\pm}0.03$
Retrieval / Translation	$0.52{\pm}0.03$
Code / Translation	$0.43{\pm}0.03$

Appendix

Table 16: Selected cross-task mask overlaps across five calibration runs.

Task	Across-run within-task Jaccard	Distinct masks within 0.5 points	Retention range
Reasoning	$0.82{\pm}0.03$	13	97.0–97.3
Retrieval	$0.78{\pm}0.04$	18	97.2–97.6
Code	$0.81{\pm}0.03$	10	96.7–97.1
Translation	$0.84{\pm}0.03$	16	97.9–98.2

Appendix

Table 17: Within-task multiplicity under score-tie and boundary perturbations. TASP requires a high-retention, hardware-valid mask, not an identifiable unique circuit.

Figure 10: Pairwise Jaccard overlap between learned task masks. Related task families share a larger stable core, whereas reasoning and generation rely on substantially more distinct retained structures.

Offline component	Prompt–module evaluations	A100 GPU-hours	Peak allocated memory
Layer-streamed BF16 spectral profiling	0	5.6	11.8 GB
15% pilot separability stage	88,320	10.2	77.1 GB
Remaining train/validation labeling	912,640	102.6	77.3 GB
Module- and prompt-disjoint test labeling	88,320	11.7	77.3 GB
Predictor fitting and cross-validation	—	0.3	3.2 GB
FLOP-constrained mask construction	—	0.2	1.7 GB

Appendix

Table 18: Complete Llama-3-70B offline cost audit. Ablation labels are conditioned on the deployment quantization because they are measured in the same INT8-weight/BF16-compute runtime.

Method	A100 GPU-hours	Peak allocated
Wanda	7.4	76.5 GB
SparseGPT	13.1	77.0 GB
SliceGPT	10.3	76.8 GB
LLM-Pruner	19.6	77.1 GB
TASP	136.0	77.3 GB

Appendix

Table 19: Baseline calibration cost under the same accounting boundary. Baselines use 2,560 examples, except LLM-Pruner, which uses 25,600.

Live component	Allocation
Packed INT8 weights	70.60 GB
Scales and quantization metadata	1.10 GB
Five mask plans and routing metadata	0.54 GB
KV cache, activations, and workspace	5.06 GB
Peak allocated	77.30 GB

Appendix

Table 20: Observed live allocation during the complete 2,048-prefill/256-decode loop.

Panel of Figure 11 — Figure 11: Corrected realized-versus-theoretical compute reduction. The 0.57-active-FLOP structured models have an idealized $1/0.57\approx 1.75\times$ compute ratio, but measured speed depends on the compiled shapes. Applying the same selected TASP masks as zeros in dense kernels launches approximately the dense workload and yields only $1.03\times$ . This figure replaces the submitted visualization based on an incomplete profiler subtotal.

Variant	S1	S2	S3	S4	S5	Mean $\pm$ SD
TASP -Single	92.9	93.1	93.7	93.5	93.3	$93.3{\pm}0.3$
TASP - MoM	97.5	97.6	98.0	97.8	97.7	$97.7{\pm}0.2$
TASP -Oracle	98.5	98.6	99.0	98.8	98.7	$98.7{\pm}0.2$

Appendix

Table 21: Seed-level Llama-3-70B retention in the BF16 benchmark harness.

Variant	S1	S2	S3	S4	S5	Mean $\pm$ SD
Oracle router	98.5	98.6	99.0	98.8	98.7	$98.7{\pm}0.2$
Learned router	97.5	97.6	98.0	97.8	97.7	$97.7{\pm}0.2$
Without salience	95.2	95.4	95.9	95.6	95.5	$95.5{\pm}0.3$
Without spectral features	93.0	93.4	94.1	93.8	93.7	$93.6{\pm}0.4$
One global mask	92.9	93.1	93.7	93.5	93.3	$93.3{\pm}0.3$
Layer-matched random	85.0	85.4	86.2	86.0	85.5	$85.6{\pm}0.5$

Appendix

Table 22: Seed-level ablation ladder corresponding to Table 5 .

Metric	S1	S2	S3	S4	S5	Mean $\pm$ SD
MMLU	77.5	77.8	78.3	78.0	77.9	$77.9{\pm}0.3$
GSM8K	84.7	85.0	85.7	85.4	85.2	$85.2{\pm}0.4$
MGSM	72.4	72.6	73.1	72.9	72.8	$72.8{\pm}0.3$
AlpacaEval	80.0	80.2	80.8	80.5	80.4	$80.4{\pm}0.3$
NQ-Open	67.0	67.1	67.5	67.3	67.2	$67.2{\pm}0.2$
HumanEval	69.1	69.5	70.5	70.0	69.8	$69.8{\pm}0.5$

Appendix

Table 23: Per-task seed values for the TASP - MoM row of Table 7 .

Figure 12: Ablation ladder at matched active-FLOP reduction. Spectral scoring and task-conditioned mask diversity make complementary contributions, while random mask diversity is insufficient.

Panel of Figure 13 — Figure 13: Corrected relative-retention comparison on Llama-3-70B. Error bars for deterministic baselines are paired-bootstrap SE over evaluation instances; error bars for TASP variants are SD across five independently calibrated masks and routers. This replaces the submitted seed-level visualization, which did not distinguish the two uncertainty sources.

Figure 14: Per-task variability across the five independent TASP - MoM calibration and mask runs reported in Table 23 .

Figure 15: Task-level retention across benchmarks and pruning methods at matched active-FLOP reduction. The heatmap complements the macro retention statistic by exposing the task families on which universal masks incur their largest losses.

Method	Original MMLU	MMLU-Pro	MMLU-Pro ret.
Dense BF16	79.2	55.9	100.0
TASP -Single	$75.1{\pm}0.3$	$51.8{\pm}0.4$	$92.7{\pm}0.7$
TASP - MoM	$77.9{\pm}0.3$	$54.3{\pm}0.3$	$97.1{\pm}0.5$
TASP -Oracle	$78.4{\pm}0.2$	$54.9{\pm}0.2$	$98.2{\pm}0.4$

Appendix

Table 24: Original MMLU and a genuine MMLU-Pro evaluation on Llama-3-70B at the 43% active-FLOP reduction. Dense original-MMLU

79.2

is produced by our fixed strict five-shot non-chat harness and sits between Meta’s pretrained (

78.9

) and instruct (

82.0

) references; the difference from the instruct reference is a prompt/harness effect, not a different checkpoint (reconciled in text). The dense MMLU-Pro score

55.9

matches the official

56.20

reference. Task-conditioned gaps are similar in relative rather than absolute terms; a dense score near

79

on MMLU-Pro would indicate a still-mislabelled or answer-leaking harness.

Ranking rule	Labels used	Rel. ret. at 43%
No-spectral calibrated variant	31,280	$93.6{\pm}0.4$
Direct $\Delta$ + imputation, 25%	7,820	$94.4{\pm}0.5$
Spectral predictor, 25%	7,820	$95.7{\pm}0.4$
Direct $\Delta$ + imputation, 50%	15,640	$95.8{\pm}0.4$
Spectral predictor, 50%	15,640	$96.7{\pm}0.3$
Direct $\Delta$ + imputation, full	31,280	$97.1{\pm}0.3$

Appendix

Table 25: Direct measured-ablation ranking versus the spectral predictor at matched label budgets (Llama-3-70B, 43% active-FLOP reduction). TASP closes about

77\%

of the gap from the no-spectral variant to the all-label direct oracle,

(97.7-93.6)/(98.9-93.6)\approx 0.77

. The supported conclusion is that spectral prediction is label-efficient and close to, but not equal to, direct exhaustive ranking.

Split	Top-1	Macro-F1	Cal. ECE	No-fb ret.	Fb ret.	Sparse
Prompt-disjoint	$92.4{\pm}0.6$	$92.1{\pm}0.7$	$0.026{\pm}0.004$	$97.7{\pm}0.2$	$98.9{\pm}0.2$	$76{\pm}3\%$
Template-disjoint	$87.1{\pm}0.9$	$86.6{\pm}1.0$	$0.041{\pm}0.006$	$96.9{\pm}0.3$	$98.4{\pm}0.2$	$69{\pm}3\%$
Dataset-disjoint	$82.4{\pm}1.2$	$81.7{\pm}1.3$	$0.066{\pm}0.008$	$95.8{\pm}0.4$	$97.9{\pm}0.3$	$61{\pm}4\%$
OOD/mixed (consensus)	$75.6{\pm}1.7$	$74.8{\pm}1.8$	$0.091{\pm}0.011$	$94.6{\pm}0.5$	$97.5{\pm}0.3$	$43{\pm}4\%$

Appendix

Table 26: Natural router accuracy, calibration, and retention across increasingly disjoint splits. “No-fb” and “Fb” are retention without and with the selected dense fallback; “Cal. ECE” is expected calibration error after temperature scaling. The calibrated in-suite router still shows a meaningful source-disjoint drop, so the fallback policy rather than raw routing is the robustness mechanism.

True $\backslash$ Pred.	Reas.	Gen.	Retr.	Code	Transl.
Reasoning	93	1	2	3	1
Generation	1	94	2	1	2
Retrieval	3	2	91	2	2
Code	5	1	1	92	1
Translation	1	2	2	1	94

Appendix

Table 27: Prompt-disjoint router confusion matrix (row percentages). Confusions concentrate between reasoning and code, the families whose masks also overlap most (Table 16 ).

Method	Training mode	Ret. @43%	Speedup	A100-h
Dense	none	100.0	$1.00\times$	0
TaBP	calibration only	$93.4{\pm}0.3$	$1.55\times$	5.8
PuDDing	router + masks	$93.8{\pm}0.4$	$1.49\times$	7.6
ShadowLLM	per-prompt predictor	$94.1{\pm}0.4$	$1.51\times$	9.4
IG-Pruning	masks + routing	$94.7{\pm}0.3$	$1.45\times$	11.9
Instr.-Following Pr.	joint training	$95.5{\pm}0.3$	$1.34\times$	86.0

Appendix

Table 28: 8B comparison against matched closest-work baselines at 43% active-FLOP reduction. Retention is in the BF16 scientific harness; speedup and offline cost are for the compiled runtime. ShadowLLM re-selects sparsity per prompt at head/neuron granularity, so its

1.51\times

speedup is measured in its native units; it retains slightly less than TASP - MoM here, consistent with per-prompt prediction being noisier than per-task calibration under a fixed FLOP budget.

Method	Ret. @43%	Decode speedup
TaBP	$93.7{\pm}0.3$	$1.57\times$
PuDDing	$92.6{\pm}0.4$	$1.58\times$
ShadowLLM	$93.9{\pm}0.4$	$1.52\times$
IG-Pruning	$94.2{\pm}0.3$	$1.49\times$
TASP - MoM	$97.7{\pm}0.2$	$1.44\times$

Appendix

Table 29: 70B comparison at 43% active-FLOP reduction. The result is a Pareto trade-off rather than universal dominance: depth methods remain faster while TASP retains more quality. If a closest baseline reached TASP -level retention at equal or better latency, the novelty and superiority claims would narrow accordingly.

New checkpoint	Mean	Min	LCB	Decision	Ret.
Llama-3.1-8B-Instruct	0.68	0.64	0.63	pass	$96.7{\pm}0.3$
Gemma-2-9B-It	0.65	0.60	0.61	pass	$95.8{\pm}0.3$
Mistral-Nemo-12B-Inst.	0.63	0.59	0.60	pass	$95.2{\pm}0.4$
Qwen2.5-7B-Instruct	0.59	0.56	0.55	reject	$93.9{\pm}0.4$
Mistral-7B-Inst.-v0.3	0.60	0.57	0.56	reject	$94.4{\pm}0.4$
Qwen2.5-14B-Instruct	0.61	0.58	0.58	reject	$95.1{\pm}0.4$

Appendix

Table 30: Prospective test of the frozen applicability rule on six unseen checkpoints. Against the pre-registered

95.0\%

success threshold the rule gives

100\%

precision and

75\%

recall, with one conservative false negative (Qwen2.5-14B, retention

95.1\%

). The six checkpoints are perfectly rank-ordered between pilot mean AUROC and downstream retention, so Spearman

\rho=1.00

(Pearson

r=0.98

); with only six points this is a strong but small-sample association and should be read as a monotone-ordering check rather than a precise effect size. The supported claim is that the rule is conservative and useful for avoiding poor fits, not that it perfectly identifies every viable model.

Candidate feature construction	AUROC	Ret.
Canonical stacked Q/O/K/V + gate/up/down	$0.75{\pm}0.02$	$97.7{\pm}0.2$
Per-matrix descriptors, separate fits	$0.76{\pm}0.02$	$97.8{\pm}0.2$
Query/output and gate/up only	$0.71{\pm}0.02$	$96.4{\pm}0.3$
Independently normalized, then averaged	$0.73{\pm}0.02$	$97.1{\pm}0.3$

Appendix

Table 31: Robustness of the candidate feature construction (held-out AUROC and 70B retention). The canonical definition is competitive with the best variant and was fixed in advance.

Validation quantity	1 mask	3 masks	5 masks
Plan/coordinate metadata	0.14 GB	0.34 GB	0.54 GB
Peak allocated memory	76.9 GB	77.1 GB	77.3 GB
Analytical active-FLOP ratio	0.570	0.570	0.570
Ratio from launched tile shapes	0.571	0.571	0.571
Max abs. logit diff. (achieved)	$3.1{\times}10^{-4}$	$4.0{\times}10^{-4}$	$4.6{\times}10^{-4}$
Greedy-token agreement (achieved)	$100.0\%$	$99.994\%$	$99.991\%$

Appendix

Table 32: FLOP-ledger and one-store validation as the mask count grows. The equivalence target (max absolute logit difference below

2{\times}10^{-3}

and at least

99.98\%

greedy-token agreement between the shared-store and separately materialized plans) is met at every mask count: the achieved logit difference stays at the

10^{-4}

level and token agreement stays above

99.99\%

, with the small residual attributable to BF16 accumulation order rather than to a functional difference. Peak allocated memory rises only with plan metadata, not with duplicated weights: five separately duplicated 70B INT8 stores would require roughly

353

GB before scales and workspace, so the observed

77.3

GB peak is inconsistent with silent checkpoint duplication.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Mar 22, 2026

Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models

Large language models deliver strong generative performance but at the cost of massive parameter counts, memory use, and decoding latency. Prior work has shown that pruning and structured sparsity can preserve accuracy under substantial compression, while prompt-compression methods reduce latency by removing redundant input tokens. However, these two directions remain largely separate. Most model-compression methods are static and optimized offline, and they do not exploit the fact that different prompts and decoding steps activate different latent computational pathways. Prompt-compression methods reduce sequence length, but they do not adapt the executed model subnetwork. We propose a unified compressed-sensing-guided framework for dynamic LLM execution. Random measurement operators probe latent model usage, sparse recovery estimates task-conditioned and token-adaptive support sets, and the recovered supports are compiled into hardware-efficient sparse execution paths over blocks, attention heads, channels, and feed-forward substructures. The framework introduces five key contributions: task-conditioned measurements, so different prompts induce different sparse supports; token-adaptive recovery, so active substructures are re-estimated during decoding; formal sample-complexity bounds under restricted isometry or mutual incoherence assumptions; compile-to-hardware constraints that restrict recovery to GPU-efficient structures; and a joint objective that unifies prompt compression with model reduction. Together, these components recast LLM inference as a measurement-and-recovery problem with explicit approximation guarantees and deployment-oriented speedup constraints.

Andrew Kiruluta

UC Berkeley School of Information

LLM Inference Acceleration LLM Pruning Neural Network Pruning Prompt Compression

Ask PDF

Mar 25, 2026

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.

Jimyung Hong, Jaehyung Kim

Yonsei University

LLM Pruning Structured Pruning Neural Network Pruning LLM Compression

Ask PDF

Oct 8, 2026

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

Qitong Wang, Xinwei Niu, Mingluo Su +3

Westlake University · Ant Group

LLM Pruning LLM Inference Acceleration Language Model Decoding Structured Sparsity

Ask PDF

Abstract

LLM Pruning

Explore similar work

Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference