Candidate-Label Saliency for Efficient Conformal Pruning of Text Classifiers

Ibne Farabi Shihab; Joyanta Jyoti Mondal; Adria Binte Habib

cs.LGAug 24, 2026

Candidate-Label Saliency for Efficient Conformal Pruning of Text Classifiers

Authors: Ibne Farabi Shihab, Joyanta Jyoti Mondal, Adria Binte Habib

Organizations: Department of Computer Science, Iowa State University, USA · Department of Computer and Information Sciences & University of Delaware, USA · Department of Computer Science & Engineering, BRAC University, Bangladesh

Abstract

Recalibration can preserve conformal coverage after pruning while enlarging prediction sets and leaving more alternatives for downstream decisions. Directly searching for deletions that keep sets small is costly. We study whether a small candidate-gradient budget can approximate this search at lower cost. Candidate-Saliency Pruning (CSP) estimates the squared first-order change in two plausible labels' scores when each parameter is deleted. CSP averages these changes, combines the resulting importance with a base pruner, and selects the mixing weight under validation accuracy and coverage requirements. Independent split-conformal calibration supplies marginal coverage under exchangeability. We evaluate CSP on three classification tasks, two base pruners, and three sparsity levels against a squared true-label control that separates candidate choice from gradient squaring, under fixed recovery updates, equal total time, and restricted access to the training corpus. Before recovery, CSP reduces mean set size by 13.2% to 19.2% against the reference pruner and by 7.7% to 13.2% against the control in all six encoder cells at 50% sparsity. The advantage over the control shrinks with recovery to practical equivalence in all six cells at 1,000 updates, and at equal total time the cheaper reference pruner is practically equivalent in 18 of 18 settings. When recovery is limited to the retained pruning pool, CSP keeps a meaningful advantage in all six cells. Its value therefore depends on the time and data available for recovery.

Figures & tables

Figure 1: Candidate-Saliency Pruning (CSP). I: independent recalibration restores coverage after pruning but can enlarge prediction sets (schematic). CSP targets set size, and validation screens for accuracy. II: candidate saliency averages squared coordinate displacements over each pruning example’s Top-2 dense labels. III: validation selection mixes candidate and base-pruner importance over a coefficient grid, applies accuracy, coverage, and set-size rules, and falls back to the reference if needed. IV: the selected model and scorer are frozen before a single calibration on untouched data.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Data and Prediction
$\mathcal{X},\mathcal{Y},K$	Input space, finite label space, and number of labels $K=\|\mathcal{Y}\|$ .
$(x,y),(X,Y)$	An input-label pair and its random counterpart.
$\theta,\theta^{\prime}$	Dense-model parameters and parameters of a fixed pruned model.
$f_{\theta},p_{\theta}$	Normalized classifier and its class probabilities; decoder sequence probabilities are specified separately.
$s(x,y;\theta)$	Nonconformity score for a complete candidate label.

Appendix

Table 5: Notation for prediction, calibration, pruning, and analysis.

Figure 2: Calibration-size sensitivity for the fixed zero-shot Qwen scorer on DBpedia-14 with Wanda at 50% sparsity (Q1W50), using 16 fixed score matrices and nested independent calibration pools. Points show mean coverage (a) and mean set size (b), with 95% Student-

t

confidence intervals. GradRef is the absolute-gradient reference; Threshold-2 uses two candidate labels. Horizontal offsets separate methods at the same pool size. The dashed line marks nominal 90% coverage. The 1,024-example operating point reuses the Q1W50 aggregate; it is not an additional experiment. Only the calibration prefix changes, so the sweep measures threshold sensitivity rather than model changes.

Figure 3: Local threshold plateaus on one fixed 5,000-example reference score matrix. Each horizontal segment joins two observed thresholds within the same score gap; gaps between segments leave the unmeasured transition locations unspecified. Mean set size rises in increments of

0.0004=2/5000

across the reported plateaus, while coverage stays at 0.887800. The horizontal axis expresses threshold

q

through the offset

x

, where

q=6.3581+10^{-4}x

. These are fixed-matrix readouts, with no inferential intervals.

Figure 4: Top-2 encoder runtime by phase at 50% sparsity, with 256 pruning examples and 32 seeds. (a) Stacked phase means and separately reported total minutes with 95% Student- $t$ confidence intervals. (b) Each cell reports phase minutes and its 95% interval half-width; shading shows the phase mean as a percentage of the reported total. Loading/scoring includes checkpoint loading and dense scoring; selected mask constructs the chosen sparse model; grid/selection includes remaining candidates, provisional calibration, validation, and selection; final evaluation includes independent calibration and testing. Phases partition the pipeline; rounding can make their displayed sum differ slightly from the reported total. R1/R2/R3 denote DBpedia-14/Banking77/CLINC150 on RoBERTa; W/S denote Wanda/SparseGPT. Shared dense task fitting is reported separately.

Regime	Training min
R1	${11.00\pm 0.15}$
R2	${4.30\pm 0.06}$
R3	${7.90\pm 0.11}$
D1	${84.00\pm 1.12}$
D2	${31.00\pm 0.41}$
D3	${59.00\pm 0.79}$

Appendix

Table 22: Dense task-fitting time in minutes over 16 seeds on one A100 80GB. Full model-development cost adds this task-specific time to the incremental pruning pipeline. It is shared only when methods reuse the same trained checkpoint.

Dataset	Method	Accuracy $\uparrow$	Coverage	Avg. set size $\downarrow$
DBpedia-14	Wanda	$\mathbf{0.310}$	$0.901$	$11.2$
	TL-CEP-Wanda	$0.295$	$0.902$	$\mathbf{9.0}$
	SparseGPT	$0.347$	$0.901$	$10.1$
	TL-CEP-SparseGPT	$\mathbf{0.366}$	$0.902$	$\mathbf{8.6}$
Banking77	Wanda	$0.282$	$0.900$	$39.8$
	TL-CEP-Wanda	$\mathbf{0.316}$	$0.902$	$\mathbf{34.2}$

Appendix

Table 23: Qwen2.5-1.5B results at 50% sparsity, reported as descriptive means over five seeds. Coverage is the validity constraint; set sizes should be compared at matched coverage. Bold indicates the better value within each base/TL-CEP pair.

Table 8

Method	Wall time $\downarrow$	Relative cost $\downarrow$	Peak memory $\downarrow$
Wanda	7.8 min	$\mathbf{1.0}\times$	9.4 GB
Wanda++	15.1 min	$1.9\times$	11.2 GB
TL-CEP-Wanda	31.4 min	$4.0\times$	12.6 GB
Band-CEP-Wanda	68.7 min	$8.8\times$ Wanda; $2.2\times$ TL-CEP	14.8 GB
SparseGPT	24.6 min	$\mathbf{1.0}\times$	13.8 GB
TL-CEP-SparseGPT	47.9 min	$1.9\times$	15.1 GB

Appendix

Table 26: Offline pruning time and peak memory on one NVIDIA A100 80GB. Cost ratios use each method’s own base unless another reference is stated. Bold indicates the lowest value per base group and column;

\downarrow

indicates lower is better.

Dataset	Sparsity	Wanda	TL-CEP-Wanda	SparseGPT	TL-CEP-SparseGPT
AG News	30%	$0.901,\ 3.71$	$0.902,\ \mathbf{3.46}$	$0.900,\ 3.62$	$0.902,\ \mathbf{3.38}$
AG News	50%	$0.900,\ 3.86$	$0.902,\ \mathbf{3.52}$	$0.901,\ 3.77$	$0.902,\ \mathbf{3.41}$
AG News	70%	$0.901,\ 3.89$	$0.901,\ \mathbf{3.74}$	$0.900,\ 3.88$	$0.901,\ \mathbf{3.67}$
TREC	30%	$0.901,\ 5.34$	$0.902,\ \mathbf{4.92}$	$0.900,\ 5.17$	$0.902,\ \mathbf{4.78}$
TREC	50%	$0.900,\ 5.11$	$0.902,\ \mathbf{4.63}$	$0.901,\ 4.91$	$0.902,\ \mathbf{4.52}$
TREC	70%	$0.900,\ 5.61$	$0.901,\ \mathbf{5.33}$	$0.900,\ \mathbf{5.42}$	$0.901,\ 5.48$

Appendix

Table 28: Qwen2.5-1.5B results on AG News ( Zhang et al., 2015 ) and TREC ( Li and Roth, 2002 ) , reported as means over five seeds. Each entry lists empirical coverage followed by mean set size. No inferential interval is assigned. Bold indicates the smaller set size within each base/TL-CEP pair;

\downarrow

indicates lower is better. Coverage is not ranked.

Table 11

Table 12

Dataset	Sparsity	TL-CEP-W Acc	TL-CEP-SG Acc	TL-CEP-W $\|C\|$	TL-CEP-SG $\|C\|$
AG News	30%	W	L	W	W
AG News	50%	W	W	W	W
AG News	70%	L	W	W	W
TREC	30%	W	L	W	W
TREC	50%	L	W	W	W
TREC	70%	W	W	W	L

Appendix

Table 33: Direction of TL-CEP differences from the matched base in Qwen cell means. W/L indicate favorable/unfavorable differences and do not imply significance. In headers, W/SG denote Wanda/SparseGPT, Acc. denotes accuracy, and

|C|

denotes mean set size.

Method	Accuracy $\uparrow$	Coverage	Set size $\downarrow$
SparseGPT	$0.3492$	$0.9004$	$10.27$
TL-CEP-SparseGPT	$\mathbf{0.3638}$	$0.9026$	$\mathbf{8.79}$

Appendix

Table 34: Qwen results on DBpedia-14 at 50% sparsity under five independent split redraws. Bold indicates the better value per column. Coverage is not ranked.

$n_{\mathrm{prune}}$	SparseGPT	TL-CEP-SparseGPT
128	$0.341/0.900/10.2$	$\mathbf{0.354}/0.901/\mathbf{9.4}$
256	$0.344/0.900/10.1$	$\mathbf{0.359}/0.901/\mathbf{9.1}$
512	$0.346/0.901/10.1$	$\mathbf{0.363}/0.902/\mathbf{8.8}$
1024	$0.347/0.901/10.1$	$\mathbf{0.366}/0.902/\mathbf{8.6}$

Appendix

Table 35: Effect of pruning-pool size using nested subsets. Each entry lists accuracy, coverage, and mean set size, in that order;

n_{\mathrm{prune}}

is the number of pruning examples. Bold indicates the better value per row and metric;

\uparrow

\downarrow

indicate higher/lower is better. Coverage is not ranked.

Method	PPL $\downarrow$	Acc. $\uparrow$	Cov.	$\|C\|$ $\downarrow$
Dense	$6.12$	$.78$	$.902$	$4.5$
SparseGPT	$\mathbf{7.25}$	$.61$	$.901$	$8.9$
TL-CEP-SparseGPT	$7.28$	$\mathbf{.65}$	$.902$	$\mathbf{7.2}$

Appendix

Table 36: Llama-3-8B ( Dubey et al., 2024 ) results at 50% sparsity. Perplexity (PPL) is measured on WikiText-2 ( Merity et al., 2017 ) ; accuracy (Acc.), coverage (Cov.), and mean set size (

|C|

) are measured on DBpedia-14. Sparse results are descriptive means over three pruning seeds. Dense perplexity is a single point value. Bold indicates the better sparse value; Dense provides the unpruned reference.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage
Jun 2, 2026Hu Xu, Zhaolong Xing, Congcong Liu +5LLM Pruning Model Calibration
Shanghai Jiao Tong University
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
Oct 13, 2026Mehdi Makni, Natalia Ponomareva, Rahul MazumderLLM Pruning Neural Network Pruning
Operations Research Center, Massachusetts Institute of Technology · Google Research
OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning
Sep 15, 2026Ha Lan Nguyen, Huy Hoang Tran, Trac-Duy Tran +1LLM Pruning Large Reasoning Models
VinUniversity · Johns Hopkins University

Jun 2, 2026J/K move · Enter open · S save

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang+4

Shanghai Jiao Tong University

LLM Pruning Model Calibration

Open paper AskPDF arXiv

Post-training LLM pruning relies on a small calibration set, usually chosen heuristically and judged by WikiText perplexity or by mean accuracy on general-domain commonsense benchmarks. This single-axis view hides capability-specific effects: at 60% sparsity, strategies separated by

2.85

points in averaged commonsense accuracy diverge by

51.9

points in code retention, and across 15 sources no single one preserves all capabilities. We treat calibration as a capability-coverage problem. Balanced, a uniform four-way mixture under a fixed budget, tests the principle on real corpora: it outperforms every evaluated single source, and its margin over C4 grows from

+4.6

points at 30% sparsity to

+18.8

at 60%. The advantage holds across model scale and a second model family, and transfers to held-out benchmarks disjoint from calibration construction. When aligned corpora are unavailable, IGSP approximates that mixture from the base model and the evaluation taxonomy alone, outperforming prior self-generation methods by

7.23

12.79

points on the LLaMA main suite. On Qwen2.5-7B, IGSP exceeds Balanced on both the main suite and held-out benchmarks. Capability coverage is a first-order design variable for high-sparsity pruning.

Abstract

Neural Network Pruning

Explore similar work

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning