Static Pruning Across Sparse Retrieval Regimes: What Transfers, What Breaks, and What Still Helps

Zirui Song; Yuye Zhu; Yang Yang

cs.IRAug 17, 2026

Static Pruning Across Sparse Retrieval Regimes: What Transfers, What Breaks, and What Still Helps

Authors: Zirui Song, Yuye Zhu, Yang Yang

Organizations: Amazon Web Services Shanghai, China

Abstract

Static pruning is widely used to accelerate sparse neural retrieval, yet existing studies each validate their conclusions within a single custom pipeline, leaving it unclear which findings transfer to modern engines with different index organizations and dynamic pruning mechanisms. We present the first cross-engine pruning portability study, evaluating static pruning strategies across three engines - a controlled C++ pipeline (exhaustive inverted index), BMP (block-max pruning), and SEISMIC (clustered inverted indexes) - on two benchmarks (MS MARCO, Natural Questions) with two encoders spanning opposite query-density regimes (SPLADE: 44 avg. query terms; V3-GTE: 7 avg. query terms), totaling 1,140 experimental configurations, with an additional deep-judgment validation on TREC DL 2019/2020. We find that index-side pruning (document and posting-list) is portable: it consistently reduces latency (1.2-6.6 $\times$ ) and index size (18-82%) across all engines because sparse retrieval is memory-bound - a conclusion we support with cache-miss, TLB, and IPC profiling. In contrast, query pruning is already internalized by modern engines: it yields 4-11 $\times$ speedup on the exhaustive pipeline but is subsumed by BMP's $β$ and SEISMIC's query_cut. Static pruning complements dynamic pruning: on BMP, combining document and query reduction yields 2.5 $\times$ speedup with NDCG@10 within 0.003 of the exact baseline. Finally, NDCG@10 saturates while Recall@10 is still in the ${\sim}$ 85-95% range across all three engines, providing a portable stopping criterion: practitioners can push pruning to this knee without visible ranking degradation. Together, these findings answer what transfers (index-side pruning), what breaks (query pruning), and what still helps (static atop dynamic pruning).

Figures & tables

Configuration	Query nnz	Doc nnz	Docs	NDCG@10	DL’19/20
MS MARCO + SPLADE	43.95	119.96	8.8M	0.449	0.726
MS MARCO + V3-GTE	6.89	180.41	8.8M	0.428	0.720
NQ + SPLADE	46.97	147.52	2.7M	0.539	—
NQ + V3-GTE	7.20	185.00	2.7M	0.582	—

Table 1. Dataset and encoder statistics. Avg. non-zero counts (nnz) per query and document, corpus size, baseline NDCG@10 on the controlled pipeline, and baseline NDCG@10 on the deep-judgment TREC DL 2019/2020 topics (mean over both years; MS MARCO corpus).

Figure 1. Custom C++ Pareto frontiers on MS MARCO: speedup vs. Recall@10 for all six pruning families ({Doc, Query, Posting}

\times

{Alpha-Mass, Max-Ratio}). Left : SPLADE. Right : V3-GTE. Document pruning provides the most stable high-recall frontier across both encoders. Six Pareto curves comparing pruning families on two encoder regimes. Document pruning provides the most stable frontier in both.

Dataset+Enc.	Pruning	Lat. (ms)	Spd.	Idx (GB)	R@10	NDCG	MRR
MS+SPL	Baseline	70.1	1.0 $\times$	8.15	1.000	0.449	0.383
	Q AM 0.50	11.5	6.1 $\times$	8.15	0.964	0.448	0.383
	Doc AM 0.50	11.7	6.0 $\times$	1.53	0.928	0.443	0.379
	Doc MR 0.30	18.1	3.9 $\times$	2.16	0.974	0.447	0.381
	Post MR 0.30	15.6	4.5 $\times$	2.00	0.979	0.447	0.382
MS+GTE	Baseline	29.5	1.0 $\times$	12.23	1.000	0.428	0.362

Table 2. Controlled pipeline (

\Psi

): representative operating points with two-stage re-rank (

k^{\prime}

= 50). Speedup relative to unpruned baseline. Idx is the in-memory inverted-index size.

Figure 2. SEISMIC Pareto frontiers (qc = 5): speedup vs. Recall@10 under document and query pruning for both encoders. Document pruning provides consistent gains; query pruning overlaps with SEISMIC’s internal query_cut . SEISMIC Pareto curves showing document pruning provides consistent gains while query pruning overlaps with internal query cut.

		Speedup				Recall@10				$\Delta$ NDCG@10				$\Delta$ MRR@10
Config	Pruning	C++	C++ †	BMP	SEIS.	C++	C++ †	BMP	SEIS.	C++	C++ †	BMP	SEIS.	C++	C++ †	BMP	SEIS.
MS+SPL	Doc AM 0.90	1.49 $\times$	1.48 $\times$	1.43 $\times$	1.18 $\times$	1.000	.946	.948	.868	.000	$-$ .003	$-$ .003	$-$ .005	.000	$-$ .003	$-$ .002	$-$ .005
	Post MR 0.10	1.56 $\times$	1.54 $\times$	1.47 $\times$	1.17 $\times$	1.000	.947	.948	.871	.000	$-$ .002	$-$ .001	$-$ .004	.000	$-$ .002	$-$ .002	$-$ .004
	Query MR 0.10	1.68 $\times$	1.65 $\times$	1.91 $\times$ ∗	1.17 $\times$	1.000	.945	.935	.924	.000	.000	.000	+.001	.000	$-$ .001	.000	.000
MS+GTE	Doc AM 0.90	1.78 $\times$	1.61 $\times$	1.23 $\times$	1.42 $\times$	1.000	.969	.969	.937	.000	$-$ .001	$-$ .001	$-$ .002	.000	.000	$-$ .001	$-$ .001
	Post MR 0.10	1.86 $\times$	1.68 $\times$	1.27 $\times$	1.58 $\times$	1.000	.970	.969	.939	.000	$-$ .002	$-$ .002	$-$ .002	.000	$-$ .001	$-$ .002	$-$ .002

Table 3. Portability Matrix: three pruning methods

\times

four configurations, each evaluated on three engines. Speedup is relative to each engine’s unpruned baseline.

\Delta

NDCG and

\Delta

MRR show absolute change vs. the exact baseline (negative = degradation);

\Delta

MRR tracks

\Delta

NDCG throughout. C++ uses two-stage re-ranking (

k^{\prime}

= 50); C++ † shows single-stage (no re-ranking) for direct comparison with BMP/SEISMIC. ∗ BMP external query MR at

\beta

= 1.0.

Figure 3. Per-query latency histogram strips on MS MARCO+SPLADE (controlled pipeline) at four document-pruning levels. The P50/P90/P95 markers show progressive IQR compression: moderate pruning (

\alpha_{d}

= 0.9) narrows the distribution while aggressive pruning (

\alpha_{d}

= 0.5) concentrates it tightly with minimal tail degradation. Histogram strips showing per-query latency distributions at four pruning levels with KDE outlines and percentile markers.

Index	Query	Lat. ( $\mu$ s)	Spd.	R@10	NDCG	MRR	Idx
Full	Full	8,915	1.00 $\times$	1.000	0.449	0.382	15.7
Full	Q MR 0.10	4,680	1.91 $\times$	0.935	0.449	0.382	15.7
Pruned	Full	6,235	1.43 $\times$	0.948	0.446	0.380	10.1
Pruned	Q MR 0.10	3,537	2.52 $\times$	0.917	0.446	0.379	10.1

Table 4. BMP 2

\times

2 factorial on MS MARCO+SPLADE (

\alpha

= 1.0). Speedup relative to the full-index, full-query baseline. Index size in GB.

Figure 4. BMP 2

\times

2 factorial on MS MARCO+SPLADE: latency under four conditions (full/pruned index

\times

full/pruned queries). Recall@10 shown inside each bar. Combined pruning achieves the largest speedup (2.52

\times

) with minimal NDCG loss (

\Delta

NDCG =

-

0.003). Bar chart showing four BMP conditions with latency bars and recall values. Combined pruning achieves the best speedup.

Index	Query	Lat. ( $\mu$ s)	Spd.	R@10	NDCG	MRR	Idx
Full	Full	185	1.00 $\times$	0.977	0.443	0.379	8.1
Full	Q MR 0.10	161	1.15 $\times$	0.924	0.444	0.379	8.1
Pruned	Full	158	1.17 $\times$	0.868	0.438	0.374	6.0
Pruned	Q MR 0.10	133	1.39 $\times$	0.856	0.440	0.376	6.0

Table 5. SEISMIC on MS MARCO+SPLADE (qc = 5, hf = 1.0). Speedup relative to the unpruned baseline at qc = 5. Index size in GB.

Figure 5. Micro-architectural profiling of

\Phi

\Psi

\Xi

with Linux perf on multiple pruning configurations: (a) cache-miss rate, (b) IPC, (c) dTLB-miss rate, (d) per-query latency vs. cache-miss rate (

R^{2}

= 0.84). Latency is tightly coupled to memory locality, not arithmetic throughput. Four-panel profiling figure: bar charts for cache-miss rate, IPC, and dTLB-miss rate, plus a scatter plot of latency versus cache-miss rate with regression line.

Engine	Config	IPC	Cache%	dTLB%	CacheMiss/Q
C++ $\Phi$	scatter-add	1.58	30.0	97.0	740,000
C++ $\Psi$	window-switch	2.50	2.7	2.6	67,000
BMP	$\beta$ =1.0	3.37	1.3	28.4	114,000
BMP	$\beta$ =0.5	3.37	1.8	25.0	89,000
SEISMIC	qc=20	3.01	10.9	40.9	12,500
SEISMIC	qc=5	3.29	7.7	37.7	6,500

Table 6. Micro-architectural profile across engines (MS MARCO+SPLADE). Cache-miss and dTLB-miss are rates; CacheMiss/Q is the absolute count per query driving latency.

Figure 6. Latency vs. quality across three engines (columns) and two dataset–encoder regimes (rows: MS MARCO+SPLADE, top; NQ+V3-GTE, bottom). The dashed line marks the NDCG@10 knee (the most aggressive point within 0.005 of the unpruned baseline); the gold-shaded region to its right is where NDCG@10 has saturated while Recall@10 keeps rising, confirming the metric gap is engine-independent. Six panels: three engines (columns) by two dataset–encoder regimes (rows). A dashed line marks the NDCG knee in each panel; the gold-shaded region to its right highlights where NDCG has saturated while Recall keeps rising.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Pruning	Idx (GB)	Lat ( $\mu$ s)	R@10	NDCG	MRR	Spdup
MS+SPL	None	15.7	8,915	1.000	0.449	0.382	1.00 $\times$
	AM 0.50	4.3	2,869	0.685	0.398	0.335	3.11 $\times$
	AM 0.70	6.4	4,166	0.832	0.431	0.366	2.14 $\times$
	AM 0.90	10.1	6,235	0.948	0.446	0.380	1.43 $\times$
	AM 0.95	11.7	7,111	0.972	0.448	0.381	1.25 $\times$
	MR 0.10	10.4	6,281	0.951	0.446	0.380	1.42 $\times$

Appendix

Table 7. BMP document/posting/query pruning across all four dataset–encoder combinations (

\alpha

=1.0). Speedup relative to each config’s unpruned baseline.

Figure 7. Recall@10 under document pruning across the four dataset–encoder combinations. The two-stage C++ curve shows the effect of full-vector re-ranking, while the no-rerank C++ curve closely tracks BMP at the main operating point (AM=0.90). SEISMIC exhibits a larger Recall drop under the same static pruning threshold, indicating that pruning thresholds have different effective severity under cluster-based traversal. The gray band marks Recall@10

\geq 0.90

. Four-panel figure comparing Recall@10 under document alpha-mass pruning for C++ with reranking, C++ without reranking, BMP, and SEISMIC.

Config	Pruning	Idx (GB)	Lat ( $\mu$ s)	R@10	NDCG	MRR
MS+SPL	None	8.1	186	0.977	0.443	0.379
	AM 0.70	4.3	140	0.769	0.424	0.361
	AM 0.90	6.0	158	0.868	0.438	0.374
	AM 0.95	6.7	167	0.889	0.441	0.378
NQ+SPL	None	7.0	234	0.937	0.533	0.485
	AM 0.70	4.5	167	0.727	0.505	0.460

Appendix

Table 8. SEISMIC document pruning across all configurations (qc=5, hf=1.0).

	SPLADE			V3-GTE
Config	C++	BMP	SEIS.	C++	BMP	SEIS.
Baseline	.726	.726	.721	.720	.718	.721
Doc AM 0.90	.717	.719	.707	.721	.721	.718
Post MR 0.10	.720	.723	.716	.720	.719	.721
Doc AM 0.50	.680	.691	.675	.688	.685	.676
Query prune	.728	.728	.722	.723	.620	.720

Appendix

Table 9. TREC DL 2019/2020 mean NDCG@10 (DL19+DL20 pooled). Query pruning uses each engine’s native mechanism (C++ MR 0.10; BMP

\beta

=0.5; SEISMIC query_cut =5)

Learned Sparse Retrieval

Learned sparse retrieval trains models to expand queries and documents into weighted, indexable terms for efficient semantic search.

See all 18 papers

Mar 17, 2026

OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation

Domain-specific finetuning is essential for dense retrievers, yet not all data pairs contribute equally to the learning process. We introduce OPERA, a data pruning framework that exploits this heterogeneity to improve both the effectiveness and efficiency of retrieval model adaptation. We first investigate static pruning (SP), which retains only high-similarity query-document pairs, revealing an intrinsic quality-coverage tradeoff: ranking (NDCG) improves while retrieval (Recall) can degrade due to reduced query diversity. To resolve this tradeoff, we propose a two-stage dynamic pruning (DP) strategy that adaptively modulates sampling probabilities at both query and document levels throughout training, prioritizing high-quality examples while maintaining access to the full training set. Evaluations across eight datasets spanning six domains demonstrate the effectiveness of both approaches: SP improves ranking over standard finetuning (NDCG@10 +0.2 points), while DP achieves the strongest performance on both ranking (NDCG@10 +1.0 points) and retrieval (Recall@20 +0.4 points), with an average rank of 1.38 across all methods. These findings scale to Qwen3-Embedding, an LLM-based dense retriever, confirming architecture-agnostic benefits. Notably, DP reaches comparable performance in less than 50% of the training time required by standard finetuning.

Haoyang Fang, Shuai Zhang, Yifei Ma +5

Amazon Web Services

Dataset Pruning Dense Retrieval

Ask PDF

Jun 4, 2026

ColBERTSaR: Sparsified ColBERT Index via Product Quantization

While ColBERT is an effective neural retrieval architecture, it requires a heavy index structure to support candidate set retrieval based on approximated token embeddings, gathering and decompressing document token embeddings, and applying the MaxSim operation. Indexes in PLAID and similar ColBERT implementations require five to ten times the disk storage of the original raw text, which limits their scalability. Furthermore, prior work has identified that the gathering and decompression stages are the primary inefficiencies at query time. Limiting the number of document tokens that must be gathered by thresholding and score approximation does not eliminate the need for the entire index to support ad hoc queries. In this work, we propose an embedding quantization approach that turns a ColBERT index into a true inverted index. We show that, theoretically, ColBERT with embedding quantization is equivalent to learned-sparse retrieval except for the scoring mechanism. Empirically, we demonstrate that our index is 50-70% smaller than a one-bit PLAID index while retaining retrieval effectiveness.

Eugene Yang, Andrew Yates, Dawn Lawrie +3

Johns Hopkins University, Baltimore, MD, USA

Vector Quantization Multi-Vector Retrieval Embedding Compression Product Quantization Dense Retrieval

Ask PDF

Jun 8, 2026

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns. Despite sharing the same objective, these pruning approaches induce fundamentally different execution behaviors, causing realized speedups to depend heavily on hardware and kernel implementations. Consequently, the practical acceleration benefits of different pruning families remain poorly understood. In this work, we introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical M, N, and K dimensions of general matrix multiplication (GEMM). Leveraging this abstraction, we build a unified benchmarking framework that enables implementation-consistent comparison across the pruning design space and systematically characterizes the acceleration--quality Pareto frontier. Our results show that static depth pruning remains the strongest Pareto-optimal baseline and stays closest to its theoretical acceleration upper bound in memory-bounded scenarios. During prefill, the frontier transitions from static depth at low quality loss (0%--4%), to dynamic depth at moderate loss (5%--16%), and finally to static width pruning at higher loss levels (17%--26%). These findings establish the first unified view of the practical limits of pruning-based LLM acceleration and provide guidance for future pruning research.\footnote{Code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim}

Haozhe Hu, Hao Wu, Anhao Zhao +4

Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University · Munich Center for Machine Learning, LMU Munich

LLM Pruning LLM Inference Acceleration Structured Pruning

Ask PDF

Abstract

Learned Sparse Retrieval

Explore similar work

OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation

ColBERTSaR: Sparsified ColBERT Index via Product Quantization

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy