Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Guoming Ling; Yupei Lin; Zhongzhan Huang; Junxin Li; Junjie Liu; Shanshan Zhong; Hefeng Wu; Liang Lin

cs.CLOct 13, 2026

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Authors: Guoming Ling, Yupei Lin, Zhongzhan Huang, Junxin Li, Junjie Liu, Shanshan Zhong, Hefeng Wu, Liang Lin

Organizations: Sun Yat-sen University

Abstract

Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.

Figures & tables

Figure 1: Publication trends in benchmark compression research from 2023 to August 2026. Quarterly and cumulative paper counts show rapid growth since 2024.

Figure 2: Overall structure of this survey. Sections 3.1 and 3.2 introduce Single-Benchmark Compression and Cross-Benchmark Compression. Sections 4 and 5 cover Adaptive Evaluation and Benchmark Analysis and Foundations.

Figure 4: Illustration of single benchmark compression. Subsets are selected using item content, historical responses, or low-cost probes of the evaluated model.

Type	Method	Domain	Information			Selection	Estimation
Type	Method	Domain	Features	Responses	Target signal	Selection	Estimation
Content	PromptEval Polo et al. (2024a)	General	$\checkmark$	$\times$	$\times$	Balanced	Prediction
	Scales++ Bean et al. (2025)	General	$\checkmark$	$\times$	$\times$	Clustering	Weighted + Pred.
	EffiEval Wang et al. (2025b)	General	$\checkmark$	$\times$	$\times$	Coverage	Mean
History	Anchor Points Vivek et al. (2024)	General	$\times$	$\checkmark$	$\times$	Clustering	Weighted
	tinyBenchmarks Polo et al. (2024b)	General	$\times$	$\checkmark$	$\times$	Clustering	Weighted + Pred.

Table 1: Comparison of representative single benchmark compression methods. Pred. and Corr. denote prediction and residual correction. See Appendix A for the complete catalogue and Appendix D for detailed comparisons.

Type	Method	Domain	Information			Online control		Selection	Estimation
Type	Method	Domain	Features	Responses	Target signal	Update	Stop	Selection	Estimation
Selection	Fluid Hofmann et al. (2025)	General	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	Fisher info.	IRT ability
	AEA Li et al. (2025)	General	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	RL	Prediction
	AutoJudger Ding et al. (2026)	Multimodal	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	Agent	IRT ability
	FAQ-PAI Wu et al. (2026)	General	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	Uncertainty	Pred. + Corr.
	IRT-Agent Ge et al. (2026)	Code agents	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	Fisher info.	IRT ability

Table 2: Comparison of representative adaptive evaluation methods. Update and Stop denote feedback-driven selection and stopping. See Appendix A for the complete catalogue and Appendix D for detailed comparisons.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method label	Domain	Venue	Year	GitHub	Saved unit	Conclusion
Content-Based Compression (10)
Scales++ ( Bean et al., 2025 )	Gen/QA/Rsn/Math	arXiv	2025	N/A	Item	S
PromptEval ( Polo et al., 2024a )	Gen/QA/Rsn	NeurIPS	2024		Prompt	S
EffiEval ( Wang et al., 2025b )	Gen/QA/Rsn/Math	arXiv	2025	N/A	Item	S/R
MIS Select ( Kjorvezir et al., 2026 )	Gen/Rsn/Math	arXiv	2026	N/A	Prompt	R/CI
Agreement ( Purificato et al., 2026 )	Gen/NLG	arXiv	2026	N/A	Human	S

Appendix

Table 3: Catalogue of single-benchmark compression methods. S, R, D, and CI denote score, ranking, decision, and reliability interval. EMNLPF denotes Findings of EMNLP.

Method label	Domain	Venue	Year	GitHub	Saved unit	Conclusion
metabench ( Kipnis et al., 2025 )	Gen/QA/Rsn/Math	ICLR	2025		Item	S/R/CI
Adaptive Sampling ( Xu et al., 2024 )	Gen/Code	arXiv	2024	N/A	Item	S/R
LEGO-IRT ( Liao et al., 2025 )	Gen/QA/Know./MT	arXiv	2025		Item	S/R/CI
FPC ( Habba et al., 2026 )	Gen/QA/Rsn/Math	arXiv	2026		Item	S/R
Facility ( Yao et al., 2026 )	Gen	arXiv	2026	N/A	Item	S/R
Agent PACE ( Song et al., 2026 )	Rsn/Code/Agent	arXiv	2026	N/A	Agent	S/R

Appendix

Table 4: Catalogue of cross-benchmark compression methods. Conclusion abbreviations follow Table 3 . EMNLPF denotes Findings of EMNLP.

Method label	Domain	Venue	Year	GitHub	Saved unit	Conclusion
Adaptive Item Selection (21)
AEA ( Li et al., 2025 )	Gen/QA/Rsn	ICML	2025	N/A	Item+judge	S
Fluid ( Hofmann et al., 2025 )	Gen/QA/Rsn/Math	COLM	2025		Item	S/R
ATLAS ( Li et al., 2026c )	Gen/QA/Rsn/Math	ICML	2026		Item	S/R/CI
FactorCAT ( Truong et al. , 2025)	Gen/QA/Rsn	ICLR Sub.	2025	N/A	Item	S/R/CI
V-Optimal ( Krumdick et al., 2026 )	Gen/QA/Rsn/Math	arXiv	2026	N/A	Item	S

Appendix

Table 5: Catalogue of adaptive evaluation methods. Conclusion abbreviations follow Table 3 . ACLF denotes Findings of ACL.

Method label	Domain	Venue	Year	GitHub	Analysis focus
ONEBench ( Ghosh et al., 2025 )	Gen/MM	ACL	2025	N/A	Sparse ranking
DualEval ( Li et al., 2026a )	Gen/Math/Code/Know.	arXiv	2026		Item calibration
Redundancy ( Zhang et al., 2025c )	MM/Vision	ACL	2025	N/A	Benchmark redundancy
IRT Audit ( Jiang et al., 2026 )	Gen/Rsn/Math/Know.	arXiv	2026	N/A	IRT reliability
TaskBudget ( Huang, 2026 )	Code/Agent	arXiv	2026		Task sufficiency
cBMM ( Qu et al., 2026 )	Gen/Rsn/Math/Know.	arXiv	2026	N/A	Latent calibration

Appendix

Table 6: Catalogue of benchmark analysis and foundations papers. EMNLPF denotes Findings of EMNLP.

Paper	Category boundary	Classification basis
TAILORED BENCH ( Yuan et al., 2025 )	Possible overlap Target-Conditioned vs. Adaptive Primary label Target-Conditioned Compression	Deciding reason A small target probe chooses the source neighbors and one target-specific batch. Results from that main batch do not revise later tests or trigger stopping. Secondary attributes one target-conditioned batch, single benchmark, historical results and current model signal, item, score and ranking
metabench ( Kipnis et al., 2025 )	Possible overlap Cross-Benchmark vs. Historical-Response Primary label Cross-Benchmark Compression	Deciding reason Historical results guide selection and recovery, but one sparse design jointly covers a benchmark suite and recovers both constituent and suite results. Secondary attributes fixed, joint suite, historical results, item, score, ranking, and interval
PACE ( Song et al., 2026 )	Possible overlap Cross-Benchmark vs. Historical-Response Primary label Cross-Benchmark Compression	Deciding reason Historical model results train the predictor, but the evaluation plan replaces expensive target agent runs with a fixed set of cheaper source items. Secondary attributes fixed, source to target, historical results, agent run, score and ranking
BIS ( Shi et al., 2025 )	Possible overlap Cross-Benchmark vs. Foundations Primary label Cross-Benchmark Compression	Deciding reason Source-benchmark outcomes estimate a target-benchmark score. This replaces target tests, but it reduces marginal model inference only when the source is cached or cheaper. Secondary attributes fixed, source to target, prompt features, target test, score
Fluid ( Hofmann et al., 2025 )	Possible overlap Adaptive Item Selection vs. Sequential Decision and Stopping Primary label Adaptive Item Selection	Deciding reason Each target response updates ability and determines the next item. The main contribution is item selection and latent-score recovery, so stopping remains secondary. Secondary attributes adaptive, single benchmark, historical and online results, item, score and ranking
CELEUS ( Zhou et al., 2026b )	Possible overlap Adaptive Item Selection vs. Sequential Decision and Stopping Primary label Sequential Decision and Stopping	Deciding reason The main guarantee is an anytime-valid interval and a valid stop when the target precision is reached. Secondary attributes adaptive, single benchmark, current model and online results, item, score and interval

Appendix

Table 7: Examples at category boundaries. Each paper keeps one primary label. The final column gives the deciding reason, followed by plan timing, benchmark scope, main information, saved unit, and returned result.

Survey	Primary focus	Models	Design	Fixed reduction	Adaptive testing	Joint use	Cost	Reliability
LLM Psychometrics ( Ye et al., 2025 )	Measurement and validation	$\checkmark$	$\checkmark$	Opt.	Opt.	$\times$	$\times$	$\checkmark$
Computerized Adaptive Testing ( Zhuang et al., 2026 )	Adaptive tests and control	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
LLM-as-a-Judge ( Gu et al., 2026 )	LLM judging and reliability	$\times$	Opt.	$\times$	$\times$	$\times$	Opt.	$\checkmark$
This survey	Benchmark compression	Opt.	Opt.	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 8: Comparison with adjacent methodological surveys. The dimensions use general methodological definitions. Models denotes latent-trait, response, calibration, and diagnostic measurement models. Design covers instrument, item-bank, and protocol construction. Fixed reduction selects fewer items or labels before target outcomes are observed. Adaptive testing lets observed outcomes guide acquisition or stopping. Joint use shares evidence or allocation across benchmarks. Cost explicitly considers evaluation resources. Reliability covers uncertainty, calibration, bias, robustness, guarantees, validity, and reference quality. A check marks systematic treatment, Opt. marks explicit but partial treatment, and a cross marks no systematic treatment. Overall, our survey provides the broadest coverage of benchmark compression.

Survey	Primary focus	Coverage	Protocols	Evaluators	Quality	Adaptive	Efficiency	Reliability
LLM Evaluation ( Chang et al., 2024 )	Evaluation landscape	$\checkmark$	$\checkmark$	$\checkmark$	Opt.	Opt.	Opt.	Opt.
LLM Benchmarks ( Ni et al., 2025 )	Capabilities and limitations	$\checkmark$	Opt.	Opt.	$\checkmark$	$\times$	$\times$	Opt.
Static-to-Dynamic Evaluation ( Chen et al., 2025 )	Contamination resistance	Opt.	Opt.	$\times$	$\checkmark$	$\times$	$\times$	Opt.
Beyond the Leaderboard ( Sheikhi et al., 2026 )	Paradigms and lifecycle	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	Opt.	Opt.	$\checkmark$
This survey	Benchmark compression	Opt.	$\checkmark$	Opt.	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 9: Positioning relative to broader LLM evaluation and benchmark surveys. Coverage includes capability or application domains and benchmark instruments. Protocols covers execution, scoring, and reporting. Evaluators includes human assessment, automatic metrics, and LLM-based judging. Quality covers contamination, saturation, validity, robustness, reproducibility, and lifecycle concerns. Adaptive requires current-target outcomes to guide later acquisition or stopping. Efficiency covers methods designed primarily to reduce evaluation resources. Reliability includes uncertainty, calibration, stability, and formal error control. Symbols follow Table 8 . Overall, our survey complements their broad coverage with a focused and systematic treatment of efficient and reliable LLM evaluation.

Leaf category	Use this label when	Use another label when
Static single-benchmark compression
Content-Based Compression	The evaluated items are fixed in advance using item or dataset features, or target-independent sampling such as random, stratified, or unequal-probability sampling. The method uses neither past model responses nor a cheap signal from the model being tested.	Past model responses or a signal from the tested model determines which items are evaluated.
Historical-Response Compression	A reusable subset or estimator for one benchmark is built mainly from earlier model responses, score matrices, leaderboard records, or calibrated measurement parameters.	A signal from the model being tested, rather than past responses, determines the evaluated items.
Target-Conditioned Compression	A cheap signal or small pilot from the model being tested selects one fixed batch. Once the batch is fixed, later evaluation results cannot change the selected items or budget.	Evaluation results can change the next item, the remaining budget, or when evaluation ends.
Cross-benchmark compression
Cross-Benchmark Compression	One fixed method jointly reduces work across at least two benchmarks, datasets, or tasks. It returns separate scores or rankings, or a clearly defined suite-level result.	The method is run independently on each benchmark, or its required source runs cost about as much as the evaluations they replace.

Appendix

Table 10: Rules for assigning the seven leaf categories. Apply the scope requirements in Section C.1 first. The middle column gives the defining rule, and the right column identifies the closest competing label.

Dimension	Values
Timing	Fixed / one target-conditioned batch Adaptive
Scope	Single benchmark / joint suite Source-to-target
Information	Item content / historical results Target-model signal / online results
Saved unit	Items / benchmark runs / rollouts Human reviews / judge calls / comparisons
Output	Score / ranking / decision / interval

Appendix

Table 11: Secondary attributes. Multiple values may apply to each dimension.

Boundary	First category	Second category
Adaptivity	Target-Conditioned Compression A limited probe selects one fixed main batch.	Adaptive Evaluation Main-stage outcomes can change later tests, budget, or stopping.
Benchmark scope	Single-Benchmark Compression Benchmarks are evaluated independently.	Cross-Benchmark Compression Joint selection, sharing, skipping, or replacement reduces evaluation across a suite.
Main objective	Adaptive Item Selection Choosing the next item is the main contribution.	Sequential Decision and Stopping The main guarantee concerns stopping or a downstream action.
Contribution	Operational method The main contribution is a validated compressor.	Benchmark Analysis and Foundations Primarily studies the reliability of compression or benchmarks.
Reference result	Benchmark compression Approximates the predefined full-evaluation result.	Benchmark repair / related analysis Changes the target construct or the reference result.

Appendix

Table 12: Common category boundaries. For mixed information sources, the indispensable source determines the primary label; other sources are recorded as secondary attributes.

Workflow	Initial signal	Update selection	Outcome-based stop	Execution
Fixed batch	$\times$	$\times$	$\times$	Batch
Signal then batch	$\checkmark$	$\times$	$\times$	Batch
Selection only	Opt.	$\checkmark$	$\times$	Rounds
Stopping only	Opt.	$\times$	$\checkmark$	Rounds
Selection + stopping	Opt.	$\checkmark$	$\checkmark$	Rounds

Appendix

Table 13: Execution patterns. Checks and crosses mark the presence or absence of a feature. Opt. denotes an optional initial target signal. Updates and stopping use main-stage outcomes, excluding budget exhaustion.

Recovery	Predictions	Residual correction	Key check	Sensitivity
Sample mean	$\times$	$\times$	Sampling	Variance
Coreset aggregation	$\times$	$\times$	Coverage	Mismatch
Probability weighting	$\times$	$\times$	Support	Weights
Model prediction	$\checkmark$	$\times$	Fit	Shift
Prediction + correction	$\checkmark$	$\checkmark$	Sampling	Residuals

Appendix

Table 14: Recovery mechanisms and their checks. Checks and crosses mark the components used. Residual correction adjusts predictions using observed errors. The final columns summarize the main check and sensitivity.

Family	Item/design	History	Probe	Feedback	Suite	Reuse	Output
Content-Based	$\checkmark$	$\times$	$\times$	$\times$	$\times$	Subset	Estimate
Historical-Response	Opt.	$\checkmark$	$\times$	$\times$	$\times$	Model	Estimate
Target-Conditioned	Opt.	Opt.	$\checkmark$	$\times$	$\times$	Method	Estimate
Cross-Benchmark	Opt.	Opt.	Opt.	$\times$	$\checkmark$	Suite	Suite result
Adaptive Item Selection	Opt.	Opt.	Opt.	$\checkmark$	Opt.	Policy	Estimate
Sequential Decision	Opt.	Opt.	Opt.	$\checkmark$	Opt.	Rule	Decision

Appendix

Table 15: Information used by each family. A check marks defining information, a cross marks excluded information, and Opt. marks optional information.

Family	Build	Per target	Recovery	Refresh	Reused asset	Main risk	Fallback
Content-Based	Medium	Low	Low	Low	Fixed plan	Item mismatch	Stratified or full evaluation
Historical-Response	High	Low	Medium	High	Response model	Drift or weak support	Rebuild or sample directly
Target-Conditioned	Low	Medium	Medium	Low	Procedure	Weak target signal	Add random reference data
Cross-Benchmark	High	Low	Medium	High	Suite model	Suite or source drift	Keep unsupported parts
Adaptive Item Selection	Medium	Medium	High	Medium	Policy	Selection bias	Fixed design or full run
Sequential Decision and Stopping	Medium	Variable	Medium	Medium	Decision rule	Near ties or invalid stopping	Continue, tie, or abstain

Appendix

Table 16: Typical lifecycle cost and operational risk. Build covers construction and calibration. Per target covers mandatory signals and selected execution. Recovery covers estimation or decision logic. Refresh covers drift checks and asset updates. The levels summarize common structural demands across each family. Human time, accelerator time, latency, and money remain separate unless a study states a conversion.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
Sep 11, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3LLM Evaluation
Bosch Research · Sun Yat-sen University
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
May 25, 2026Sam Bowyer, Acyr Locatelli, Kris CaoLLM Evaluation Feature Selection
Cohere, University of Bristol · Work done during an internship at Cohere. · Cohere
Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Sep 23, 2026Kaifeng Tan, Yudong Li, Linlin ShenLLM Evaluation Data Contamination in Language Models
Shenzhen University · Tsinghua University

Sep 11, 2026J/K move · Enter open · S save

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

Zhongzhan Huang, Junxin Li, Guoming Ling, Yupei Lin+2

Bosch Research · Sun Yat-sen University

LLM Evaluation

Open paper AskPDF arXiv

Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Although recent benchmark compression methods (BCMs) can mitigate this cost, many strong BCMs rely on large collections of per-sample evaluation results from numerous LLMs to identify representative samples. Building such collections is also expensive unless they are already public, making these methods difficult to extend to newly released benchmarks. To address this challenge, we present ZipBench, a simple and low-cost BCM with theoretical error and rank-consistency guarantees. ZipBench evaluates only a small set of anchor LLMs, synthesizes pseudo evaluation results to broaden coverage, learns compact sample representations, and selects a small yet representative subset. Building on it, we create ZipBench Zoo, a collection of compact versions of 100+ benchmark proxies spanning text, multimodal, and agent tasks. These benchmark achieve mean absolute errors of 0.002--0.02 and average Spearman correlations of ~0.98 with the full benchmarks. Overall, ZipBench reduces the cost of both LLM evaluation and compact benchmark construction, lowering the barrier to broad LLM research for compute-constrained researchers. The code has been released in https://github.com/MilkThink-Lab/ZipBench.

Abstract

LLM Evaluation

Explore similar work

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models