Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Mehdi Makni; Natalia Ponomareva; Rahul Mazumder

cs.LGOct 13, 2026

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Authors: Mehdi Makni, Natalia Ponomareva, Rahul Mazumder

Organizations: Operations Research Center, Massachusetts Institute of Technology · Google Research

Abstract

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce $\texttt{CACTUS}$ , a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, $\texttt{CACTUS}$ can achieve over an order-of-magnitude speedup. At $50\%$ pruning of Qwen3-4B, $\texttt{CACTUS}$ reduces the C4 perplexity gap to the dense model by $45\%$ , improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from $0.670$ to $0.755$ , and runs over $28\times$ faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, $\texttt{CACTUS}$ supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the $5\%$ of data selected by $\texttt{CACTUS}$ achieves $49.44\%$ accuracy on BIG-Bench Hard, compared with $47.22\%$ using ALPS and $49.54\%$ using dense-model selection. Finally, integrating $\texttt{CACTUS}$ with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a $0.711$ cosine alignment with dense LoGRA-projected C4 gradients.

Figures & tables

Panel of Figure 1 — Figure 1: Each panel is a layer-level reconstruction problem centered on the same weight matrix $\mathbf{W}$ . Input activations from the previously compressed layer are represented by ${\mathbf{X}\in\mathbb{R}^{N_{\text{cal}}\times d_{\text{in}}}}$ ; the downstream gradients incoming from the next layer are represented by ${\mathbf{A}\in\mathbb{R}^{N_{\text{cal}}\times d_{\text{out}}}}$ . (Left) Data-free. Neither $\mathbf{X}$ nor $\mathbf{A}$ is available; both sides reduce to parallel gray passes and the objective collapses to ${\|\widehat{\mathbf{W}}-\mathbf{W}\|_{F}^{2}}$ . (Middle) Layerwise reconstruction (most common). Calibration activations ${\mathbf{X}}$ light up the left side, giving ${\|\mathbf{X}(\widehat{\mathbf{W}}-\mathbf{W})\|_{F}^{2}}$ ; outputs are matched but the method remains gradient-blind (SparseGPT, ALPS, 3BASiL, SVD-LLM). (Right) Curvature-aware. Both activations and the downstream gradients are used, yielding ${\|\mathbf{X}(\widehat{\mathbf{W}}-\mathbf{W})\mathbf{A}^{\!\top}\|_{F}^{2}}$ , as studied in this paper.

Method	C4 $\downarrow$	WT-2 $\downarrow$
Dense	11.27	7.05
SparseGPT	14.17	9.47
CACTUS -prune	13.42	8.88

Table 1: Qwen3.8-27B ppl at

2{:}4

sparsity.

	C4	WT-2	TracIn ( $\uparrow$ )				KFAC ( $\uparrow$ )				Compress Runtime
Method	PPL ( $\downarrow$ )	PPL ( $\downarrow$ )	$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$	$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	19.85	13.66	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	—
30% compression
Pure pruning
SparseGPT	21.05	14.53	0.852	0.862	0.959	0.746	0.739	0.750	0.498	0.292	24m 53s
ALPS	20.44	13.84	0.892	0.903	0.977	0.820	0.794	0.805	0.595	0.375	43m 07s

Table 2: One-shot compression of Qwen3-4B at

30\%

and

50\%

compression ratio. Bold = best within its constraint set; underline = best overall across all reported methods. Runtime results are reported using an L40 GPU; † denotes LLM-Surgeon runtimes on two NVIDIA A100 GPUs. More results are deferred to Appendix D .

	Llama-2-7B			Llama-3.2-3B
Selector	$\rho_{\mathrm{sel}}\uparrow$	Overlap $\uparrow$	EM $\uparrow$	$\rho_{\mathrm{sel}}\uparrow$	Overlap $\uparrow$	EM $\uparrow$
Dense LESS	$1.000$	$100.00$	$42.59$	$1.000$	$100.00$	$49.54$
CACTUS -prune	$\mathbf{0.638}$	$\mathbf{47.20}$	$41.57$	$\mathbf{0.678}$	$\mathbf{54.13}$	$\mathbf{49.44}$
ALPS	$0.594$	$44.07$	$40.46$	$0.621$	$52.24$	$47.22$
SparseGPT	$0.587$	$39.70$	$\mathbf{41.67}$	$0.598$	$49.60$	$46.02$
Random $5\%$	—	$4.45$	$39.81$	—	$4.94$	$48.89$

Table 3: LESS selection.

\rho_{\mathrm{sel}}

: Spearman correlation of aggregate scores. Overlap: selected examples shared with dense selected data. EM: BBH acc (%).

Method MLP-Layers	$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry}@5\%}$	Compression
LoGRA + Dense (oracle)	$1.000$	$1.000$	$1.000$	$1.000$	—
LoGRA + SparseGPT	$0.593$	$0.489$	$0.628$	$0.398$	19m 47s
LoGRA + LLM-Surgeon(1sh)	$0.630$	$0.557$	$0.708$	$0.445$	4h 02m †
LoGRA + LLM-Surgeon(5sh)	$0.635$	$0.570$	$0.706$	$0.499$	19h 55m †
LoGRA + CACTUS -prune	$0.645$	$0.584$	$0.730$	$0.583$	49m 36s
LoGRA + CACTUS -SLR	$\mathbf{0.711}$	$\mathbf{0.639}$	$\mathbf{0.804}$	$\mathbf{0.597}$	1h 33m

Figure 4: Influence Function preservation metrics for Qwen3-4B with MLP layers pruned using 2:4 sparsity. † denotes runtimes that use 2 A100 GPUs; other reported compression runtimes use a single L40 GPU.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: One objective, two derivations. LLM-Surgeon [ van der Ouderaa et al., 2024 ] and IProX’s IPSVD stage [ Chen et al., 2026 ] lead to Equation 1 . The influence-distortion bound uses the assumptions of IProX, Proposition 4.1 and Appendix D; $C_{\kappa}$ is the resulting finite constant. Here, $\mathbf{H}=\mathbf{X}^{\top}\mathbf{X}$ , and $\mathbf{G}=\mathbf{A}^{\top}\mathbf{A}$ .

	$\lambda_{G}$
$\lambda_{H}$	$0.005$	$0.01$	$0.1$	$0.25$
C4 perplexity ( $\downarrow$ )
$0.005$	19.729	19.647	19.448	19.449
$0.01$	19.719	19.596	19.432	19.392
$0.1$	19.626	19.601	19.536	19.395
$0.25$	19.681	19.780	19.548	19.509

Appendix

Table 4: Ridge-damping ablation for

\mathbf{H}

and

\mathbf{G}

using next-token cross-entropy on Llama-3.2-1B at

50\%

unstructured sparsity. Bold denotes the lowest perplexity within each dataset’s grid.

$C_{L}$	Accept (%)	Reject (%)	Avg. iterations	C4 $\downarrow$	WT-2 $\downarrow$
$\infty$	100.00	0.00	430	18.256	13.061
$10$ (default)	100.00	0.00	431	18.307	12.952
$3$	100.00	0.00	429	18.290	13.021
$1$	100.00	0.00	431	18.337	13.023
$0.9$	99.99	0.01	430	18.293	12.909
$0.8$	99.62	0.38	430	18.276	13.028

Appendix

Table 5: Low-rank movement-threshold ablation: CACTUS -

(\mathbf{S}+\mathbf{L})

on Llama-3.2-1B at

50\%

total compression with rank-budget allocation

\eta=0.3

. Rejection rates and iteration counts are averaged over all

112

compressed sublayers.

Setting	Max iter.	Support-change condition	$\rho$ multipliers
Pure pruning	$100$	$s_{t}/k>0.1,\;s_{t}/k>0.005,\;s_{t}>0.5$	$1.3,\;1.2,\;1.1$
$(\mathbf{S}+\mathbf{L})$	$2000$	$s_{t}/k>0.1,\;s_{t}/k>0.005,\;s_{t}>0.5$	$1.1,\;1.05,\;1.02$

Appendix

Table 6: Support-change penalty multipliers for compression. At the update period

m

, the multiplier is selected according to which support-change condition holds: the first matching threshold sets the multiplier

\rho^{(t+1)}/\rho^{(t)}

. The

(\mathbf{S}+\mathbf{L})

schedule is intentionally less aggressive as

(\mathbf{S}+\mathbf{L})

decompositions seem to be more sensitive in our experiments.

Method	C4 PPL ( $\downarrow$ )	WT-2 PPL ( $\downarrow$ )	TracIn ( $\uparrow$ )				Compress Runtime
			$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	14.01	9.75	1.000	1.000	1.000	1.000	—
30% compression
Pure pruning
SparseGPT	15.94	10.92	0.737 $\pm$ 0.005	0.730 $\pm$ 0.018	0.524 $\pm$ 0.014	0.293 $\pm$ 0.007	10m 24s
ALPS	15.53	10.71	0.773 $\pm$ 0.005	0.764 $\pm$ 0.017	0.580 $\pm$ 0.014	0.347 $\pm$ 0.007	14m 48s

Appendix

Table 7: Llama-3.2-1B. Influence-preservation results in the TracIn (unweighted) representation.

Method	C4 PPL ( $\downarrow$ )	WT-2 PPL ( $\downarrow$ )	KFAC ( $\uparrow$ )				Compress Runtime
			$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	14.01	9.75	1.000	1.000	1.000	1.000	—
30% compression
Pure pruning
SparseGPT	15.94	10.92	0.689 $\pm$ 0.005	0.690 $\pm$ 0.016	0.429 $\pm$ 0.011	0.277 $\pm$ 0.007	10m 24s
ALPS	15.53	10.71	0.729 $\pm$ 0.005	0.724 $\pm$ 0.016	0.482 $\pm$ 0.011	0.297 $\pm$ 0.008	14m 48s

Appendix

Table 8: Llama-3.2-1B. Influence-preservation results in the KFAC representation.

Method	C4 PPL ( $\downarrow$ )	WT-2 PPL ( $\downarrow$ )	TracIn ( $\uparrow$ )				Compress Runtime
			$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	19.85	13.66	1.000	1.000	1.000	1.000	—
30% compression
Pure pruning
SparseGPT	21.05	14.53	0.852 $\pm$ 0.004	0.862 $\pm$ 0.008	0.959 $\pm$ 0.002	0.746 $\pm$ 0.006	24m 53s
ALPS	20.44	13.84	0.892 $\pm$ 0.003	0.903 $\pm$ 0.006	0.977 $\pm$ 0.001	0.820 $\pm$ 0.004	43m 07s

Appendix

Table 9: Qwen3-4B influence-preservation results in the TracIn representation.

Method	C4 PPL ( $\downarrow$ )	WT-2 PPL ( $\downarrow$ )	KFAC ( $\uparrow$ )				Compress Runtime
			$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	19.85	13.66	1.000	1.000	1.000	1.000	—
30% compression
Pure pruning
SparseGPT	21.05	14.53	0.739 $\pm$ 0.004	0.750 $\pm$ 0.010	0.498 $\pm$ 0.008	0.292 $\pm$ 0.008	24m 53s
ALPS	20.44	13.84	0.794 $\pm$ 0.004	0.805 $\pm$ 0.009	0.595 $\pm$ 0.007	0.375 $\pm$ 0.006	43m 07s

Appendix

Table 10: Qwen3-4B. Influence-preservation results in the KFAC representation.

Method	C4 PPL ( $\downarrow$ )	WT-2 PPL ( $\downarrow$ )	TracIn ( $\uparrow$ )				Compress Runtime
			$\cos_{\text{tr}}$	$\cos_{\text{te}}$	$\rho_{\text{qry}}$	$R_{\text{qry@5\%}}$
Dense	9.45	6.14	1.000	1.000	1.000	1.000	—
$2{:}4+r256$ compression
$(\mathbf{S}+\mathbf{L})$
OATS	13.85	9.02	0.391 $\pm$ 0.004	0.393 $\pm$ 0.013	0.093 $\pm$ 0.009	0.093 $\pm$ 0.005	5h 28m
3BASiL	12.27	8.06	0.518 $\pm$ 0.005	0.523 $\pm$ 0.016	0.170 $\pm$ 0.016	0.148 $\pm$ 0.011	2h 42m

Appendix

Table 11: Llama-3-8B. Influence-preservation results at the

(2{:}4{+}r256)

(\mathbf{S}+\mathbf{L})

configuration in the TracIn (unweighted) representation. Each cell is annotated with

\pm\,\widehat{\mathrm{SEM}}

following the same convention as Tables 7 – 8 . Compression runtime is measured on a single A100 GPU.

Dense	C4 $=$ 9.45 WT-2 $=$ 6.14 PTB $=$ 11.18
Method	$2{:}4$ pruning			Method	$(2{:}4{+}r64)$			$(2{:}4{+}r256)$
	C4	WT-2	PTB		C4	WT-2	PTB	C4	WT-2	PTB
SparseGPT	22.27	15.98	24.30	OATS	22.28	15.36	23.33	13.85	9.02	14.47
ALPS	20.08	14.44	20.62	3BASiL	15.84	11.20	16.10	12.27	8.06	12.73
CACTUS-prune	19.74	14.37	20.12	CACTUS-SLR	15.46	10.75	15.73	12.21	8.04	12.80

Appendix

Table 12: Llama-3-8B perplexity (

\downarrow

) for one-shot

2{:}4

pruning, and

(2{:}4{+}r)

(\mathbf{S}+\mathbf{L})

compression. Bold : best across the three reported methods within each compression budget.

Dense	C4 $=$ 11.33 WT-2 $=$ 7.81
Method	$10\%$		$20\%$		$30\%$		$40\%$		$50\%$		$60\%$		$70\%$		$80\%$		$90\%$
	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2	C4	WT-2
SparseGPT	11.42	7.86	11.68	8.01	12.34	8.42	13.80	9.37	17.31	12.24	29.38	21.73	77.82	69.13	212.5	261.5	999.1	1510.6
ALPS	11.39	7.85	11.61	7.98	12.18	8.33	13.40	9.14	15.85	11.30	22.92	18.19	45.57	42.39	106.5	132.9	365.7	563.6
CACTUS-prune	11.38	7.84	11.56	7.94	12.07	8.27	13.17	9.01	15.36	11.12	21.49	17.37	41.92	35.58	101.2	125.2	354.7	482.7

Appendix

Table 13: Llama-3.2-3B perplexity (

\downarrow

) for one-shot unstructured pruning across nine sparsity levels. Bold : best across the three reported methods within each sparsity level.

Method	C4	WT-2	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	OBQA	RTE	Avg.	Time
Dense	11.27	7.05	81.72	73.06	58.96	82.91	75.85	46.40	82.31	71.60	—
SparseGPT	14.17	9.47	78.62	80.13	56.91	74.18	72.85	40.40	75.45	68.36	58m 15s
CACTUS -prune	13.42	8.88	79.27	82.07	58.11	75.51	72.77	41.80	78.34	69.70	9h 28m

Appendix

Table 14: Qwen3.8-27B at

2{:}4

sparsity across all

496

decoder linear matrices: C4/WT-2 perplexity (

\downarrow

), seven zero-shot accuracies (

\uparrow

). Bold : better compressed method. Total time (

\downarrow

) uses using

2

NVIDIA L40S GPUs for Qwen3.8-27B model.

Figure 6: LESS selection scores for dense Llama-3.2-3B and its SparseGPT, ALPS, and CACTUS -prune sparse proxies, with candidates ordered by the dense model’s aggregate score. The top panel spans the full instruction mixture; the bottom panel shows the first $1000$ candidates. The dashed line marks the top- $5\%$ cutoff.

Figure 7: Dense LESS and CACTUS -prune scores from Figure 6 , shown separately.

Figure 8: CACTUS relative-error trajectory for a representative layer. The KFAC objective can increase during the first ADMM iterations because the method is exploring feasible sparse candidates while the penalty schedule is still loose. As the penalty grows, primal feasibility tightens and the iterates settle, yielding the expected decrease and convergence of the objective.

	Llama-3.2-1B			Qwen3-4B
Sequences	C4	WT-2	PTB	C4	WT-2	PTB
32	21.113	15.455	24.121	22.429	17.235	28.703
64	20.105	14.566	23.181	21.952	16.811	28.576
128	19.465	14.211	22.613	21.743	16.102	27.924
256	19.132	13.842	22.399	21.663	16.057	27.660

Appendix

Table 15: Effect of calibration-set size on perplexity (

\downarrow

) for CACTUS -prune at

50\%

unstructured sparsity. All other settings are held fixed for each model.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 166 papers

Jul 20, 2026

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

Zhiren Gong, Zihao Zeng, Zijie Wang +3

College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

LLM Pruning LLM Compression Structured Pruning Attention Head Pruning Neural Network Pruning Transformer Pruning

Ask PDF

Aug 6, 2026

The Sparsity Whisperer

Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly sparsity-sensitive neurons in the MLP up and gate projections: separating similar inputs into dissimilar outputs. This suggests that effective pruning should preserve not only activations, but also the differences between outputs more broadly. We introduce a family of difference-informed pruning methods built upon this principle. Wisp is a first-order, update-free method that scores weights using input-difference norms, and Wisp+ refines this score neuronwise using the input pairs each neuron separates most strongly. Finally, Whisper is a second-order method that uses a lightly regularized difference Hessian as its reconstruction objective. Across Llama 2 and 3.1 models from 7B to 405B parameters, our second-order variant consistently improves over strong reconstruction-based baselines, while our update-free variants improve over activation-aware baselines, especially in constrained settings. The improvements over Wanda and SparseGPT extend to structured sparsity, downstream evaluations, and other model families. Augmenting stronger techniques such as RIA and ALPS with our difference-informed criteria yields further improvements, shifting the overall accuracy-runtime frontier outward at negligible additional cost. These results suggest that preserving output differences is a broadly useful and composable signal for post-training LLM sparsification.

Linghao Kong, Inimai Subramanian, Micah Adler +3

MIT · IST Austria · IBM +1

LLM Pruning LLM Inference Acceleration Neural Network Pruning

Ask PDF

Jul 8, 2026

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to

\pm 5\%

around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs,

p < 0.001

). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.

Yazdan Jamshidi, Alexey Shvets

Palo Alto Networks

LLM Pruning LLM Compression Neural Network Pruning Transformer Pruning

Ask PDF

Abstract

LLM Pruning

Explore similar work

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

The Sparsity Whisperer

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning