HoloCosm: Beyond Single-Metric Benchmark Compression Across Multiple Metrics

Yupei Lin; Zhongzhan Huang; Junxin Li; Guoming Ling; Shanshan Zhong; Hefeng Wu; Yukai Shi; Liang Lin

cs.CLOct 19, 2026

HoloCosm: Beyond Single-Metric Benchmark Compression Across Multiple Metrics

Authors: Yupei Lin, Zhongzhan Huang, Junxin Li, Guoming Ling, Shanshan Zhong, Hefeng Wu, Yukai Shi, Liang Lin

Organizations: Sun Yat-sen University · Guangdong University of Technology

Abstract

Benchmark compression reduces evaluation cost by selecting a small set of samples. Existing methods typically guide selection using a single metric, although benchmarks often assess models along multiple dimensions. We find that model rankings are often weakly correlated across metrics, indicating that different metrics capture distinct aspects of model performance. Samples selected for one metric do not reliably preserve scores and rankings on others when used for compressed evaluation. At 80% compression, each of the five single-metric baselines yields higher reconstruction error than equal-size random sampling in over 70% of cross-metric comparisons. To address this challenge, we present HoloCosm, which jointly minimizes normalized score-reconstruction error across source models and metrics while constraining ranking distortion. We evaluate HoloCosm across multiple compression rates on 15 language, multimodal, VLA, and agent benchmark configurations with 85 component metrics. With 80% compression, HoloCosm outperforms Random, the strongest baseline overall: average NMAE drops by 35.3% to 0.236, worst-component NMAE drops by 39.5% to 0.330, and average Spearman correlation rises from 0.884 to 0.915. In addition, HoloCosm remains competitive with existing baselines in single-metric evaluation. We will release our code and experimental results.

Figures & tables

Figure 1: Model-ranking consistency across metrics. Each point shows Spearman

\rho

between rankings induced by two official metrics, all direction-aligned so higher is better. Colors show correlation ranges, the dashed line marks

\rho\geq 0.85

, and right-side annotations give metric counts and high-correlation pairs. Weak and negative correlations are common.

Figure 2: Single-metric selection fails across metrics. Left: Cross-metric NMAE win, tie, and loss rates against Random across 15 benchmarks at 80% compression. HoloCosm wins in 89.6% of comparisons. Right: NMAE relative to Random on metrics other than the selection metric, with compressed axis tails. Red points show cross-metric failure despite selected-metric gains.

Method	Selection basis	Objective	Metric setting
AnchorPoints	Response correlation	Coverage	Single
mRMR	Mutual information	Score reconstruction	Single
tinyBenchmarks	IRT embeddings	Score estimation	Single
SubLIME	Predicted rank correlation	Ranking preservation	Single
EssenceBench	Redundancy + subset fitness	Score & rank fidelity	Single
MiniLongBench	Learned sample representations	Ranking preservation	Single

Table 1: Benchmark-compression methods by selection basis, objective, and metric setting.

Type	Benchmark	Task	Models	Items	Metrics
LLM	ConStory ( Li et al., 2026 )	Story evaluation	33	1,179	5
	LiveIdea-Core3 ( Ruan et al., 2026 )	Ideation	41	869	3
	LiveIdea-Full4 ( Ruan et al., 2026 )	Ideation	41	235	4
	LongForm ( Paech, 2025 )	Writing	59	108	14
	MMDocRAG-Text ( Dong et al., 2025 )	Retrieval	51	1,955	8
	RouterArena ( Lu et al., 2026 )	Routing	12	8,400	2

Table 2: Benchmark configurations used in the cross-metric transfer analysis. The 15 configurations contain 85 component metrics across LLM, multimodal, VLA, and agent evaluations.

Figure 3: Multi-metric fidelity across retention ratios. Mean NMAE, worst-component NMAE, and mean SP are computed across evaluation metrics within each benchmark, averaged over runs, then equally across 15 benchmarks. For each single-target baseline, the target with lowest run-averaged mean NMAE is selected at each benchmark and retention level and used for all panels.

		All-metric / random selection				Single-metric selection (best across metrics)
		Ours		Random		tinyBenchmarks		ZipBench		mRMR		AnchorPoints		EssenceBench
Benchmark	$K$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$
ConStory	5	0.30	0.88	0.29	0.88	0.43	0.84	0.45	0.81	0.67	0.69	1.24	0.52	0.44	0.88
Bench2Drive	4	0.27	0.84	0.39	0.84	0.46	0.78	0.53	0.76	0.37	0.83	0.59	0.79	0.34	0.85
Fail2Drive-ID	3	0.28	0.80	0.58	0.59	0.74	0.79	0.51	0.57	0.37	0.49	0.28	0.71	1.00	0.51
Fail2Drive-OOD	3	0.39	0.86	0.79	0.79	0.68	0.69	0.73	0.65	0.50	0.51	0.61	0.80	0.58	0.76

Table 3: Multi-metric score and ranking fidelity at 20% retention. Entries average over

K

metrics and runs with benchmarks weighted equally; each baseline’s best-NMAE target supplies the measures. W/T/L denotes wins, ties, and losses against Random. Bold indicates best results.

Method	Selection metric	NMAE $\downarrow$					SP $\uparrow$
		Efficiency	Score	Smoothness	Success	Mean	Efficiency	Score	Smoothness	Success
Ours	All metrics	0.053	0.435	0.159	0.448	0.274	0.782	0.845	0.911	0.835
ZipBench	Efficiency	0.111	0.872	0.343	0.818	0.536	0.758	0.748	0.862	0.787
	Score	0.100	0.831	0.338	0.860	0.532	0.668	0.750	0.869	0.747
	Smoothness	0.174	0.944	0.299	0.973	0.598	0.847	0.783	0.845	0.825
	Success	0.229	0.807	0.315	0.821	0.543	0.689	0.815	0.872	0.726

Table 4: Cross-metric fidelity on Bench2Drive at 20% retention. Underlining marks selection-metric results; Mean averages NMAE, and blue-gray marks each baseline's lowest-mean target.

Variant	Selection rule	Mean NMAE $\downarrow$	Worst NMAE $\downarrow$	Mean SP $\uparrow$
Random + correction	Uniform sampling	0.2812	0.3828	0.8840
Multi-metric reconstruction	Minimize $R_{\mathrm{mean}}$	0.2404	0.3390	0.9077
HoloCosm	Minimize $R_{\mathrm{mean}}$ s.t. $R_{\mathrm{rank}}\leq\epsilon_{\mathrm{rank}}$	0.2362	0.3299	0.9151

Table 5: Selection ablation at 20% retention. All variants use the same source-assisted estimator. Results are three-seed means with benchmarks weighted equally.

Method	Strategy	Mean NMAE $\downarrow$	Worst NMAE $\downarrow$	Mean SP $\uparrow$
Random	—	0.3650	0.5450	0.8840
AnchorPoints	Single metric	0.5533	0.9256	0.8361
	AverageMetric	0.6106	1.1302	0.8497
	Vote	0.6288	0.9719	0.9038
K-means	Multi-metric responses	0.3242	0.4288	0.8842
Ours	AverageMetric	0.2822	0.3896	0.8946

Table 6: Simple multi-metric compression strategies at 20% retention. Results are averaged equally over 15 configurations; mean and worst-metric errors are computed per configuration.

	MMStar		SimpleQA		SimpleVQA		Macro-average
Method	MAE $\downarrow$	SP $\uparrow$	MAE $\downarrow$	SP $\uparrow$	MAE $\downarrow$	SP $\uparrow$	MAE $\downarrow$	SP $\uparrow$
Ours	0.017	0.964	0.009	1.000	0.013	1.000	0.013	0.988
Random	0.014	0.988	0.009	1.000	0.022	0.992	0.015	0.993
tinyBenchmarks	0.032	0.952	0.022	0.988	0.018	0.984	0.024	0.975
ZipBench	0.018	0.964	0.018	0.988	0.031	0.937	0.023	0.963
mRMR	0.067	0.893	0.030	1.000	0.022	0.976	0.040	0.956

Table 7: Single-metric evaluation at 20% retention. MAE measures score reconstruction and SP measures ranking preservation. Results are averaged over runs; Macro-average weights the three benchmarks equally. Bold marks the best result in each column.

Figure 4: Weak-to-strong generalization. Six weaker models guide selection for disjoint stronger models. Curves show mean SP

\pm

standard deviation over three seeds.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Random-selection variability across 10 seeds. Results cover Bench2Drive, LongForm, and SoB-Image at 10%, 20%, 30%, 40%, and 50% retention under fixed model splits. Curves show ten-seed means; shaded bands indicate one standard deviation. Lower NMAE and higher SP indicate better fidelity.

Figure 6: A high-correlation control on DeepResearch Bench. Pairwise model-ranking Spearman correlations among four component dimensions and Overall Score. Overall Score is highlighted.

Method	Selection metric	Comp.	Insight	Inst. Follow.	Read.	Overall	Avg.
HoloCosm	All metrics	0.12 / 0.99	0.10 / 0.99	0.21 / 0.98	0.10 / 0.98	0.10 / 0.98	0.13 / 0.98
Random	–	0.10 / 0.99	0.09 / 0.99	0.27 / 0.97	0.15 / 0.99	0.09 / 0.99	0.14 / 0.99
mRMR	Comprehensiveness	0.13 / 0.98	0.08 / 0.99	0.26 / 0.98	0.22 / 0.98	0.11 / 0.99	0.16 / 0.98
	Insight	0.10 / 0.98	0.08 / 0.99	0.32 / 0.94	0.11 / 0.98	0.09 / 0.98	0.14 / 0.98
	Instruction Following	0.15 / 0.99	0.09 / 0.99	0.30 / 0.94	0.18 / 0.98	0.10 / 1.00	0.16 / 0.98
	Readability	0.11 / 0.98	0.08 / 0.99	0.27 / 0.96	0.13 / 0.96	0.10 / 0.99	0.14 / 0.98

Appendix

Table 8: Cross-metric fidelity on DeepResearch at 20% retention. Each mRMR and AnchorPoints row uses one component for selection and evaluates the selected items on all five metrics. HoloCosm uses all metrics, while Random uses no guiding metric. Entries report NMAE

\downarrow

/ SP

\uparrow

, averaged over three seeds.

Method	Strategy	Mean NMAE	Worst NMAE	Mean SP	Worst SP
AnchorPoints	AverageMetric	0.2955	0.4285	0.9137	0.8613
AnchorPoints	Official Overall	0.3781	0.6402	0.9142	0.8707
Ours	All metrics	0.1573	0.2420	0.9561	0.9006
Ours	AverageMetric	0.1809	0.3055	0.9530	0.8942
Ours	Official Overall	0.1729	0.2679	0.9534	0.9009
Random	Unguided	0.1747	0.2813	0.9549	0.9199

Appendix

Table 9: Aggregate-guided compression at 20% retention on five existing configurations. Official Overall includes the restricted LiveIdea-Core3 average described in the text.

Method	Strategy	Mean NMAE	Worst NMAE	Mean SP	Worst SP
AnchorPoints	AverageMetric	0.3951	0.5538	0.8756	0.8200
AnchorPoints	Official Overall	0.5118	0.7924	0.8454	0.7867
Ours	All metrics	0.2156	0.3050	0.9105	0.8582
Ours	AverageMetric	0.2711	0.3878	0.8845	0.8282
Random	Unguided	0.3295	0.4558	0.8748	0.8215
ZipBench	AverageMetric	0.4429	0.6315	0.8030	0.7382

Appendix

Table 10: Aggregate-guided compression at 20% retention on eight configurations with common coverage. Official Overall includes the restricted LiveIdea-Core3 average described in the text.

Figure 7: Source-model count sensitivity on LiveIdea-Full4. Results at 20% retention are averaged over three random seeds. All configurations use the same target models. Lower NMAE and higher SP indicate better fidelity.

	NMAE $\downarrow$		SP $\uparrow$
Metric	Random-6	HoloCosm	Random-6	HoloCosm
Hierarchical path recall	0.6434	0.5412	0.7500	0.8389
Leaf-value EM	0.9410	0.6525	0.5333	0.7944
Path-set F1	0.7366	0.5841	0.6389	0.8444
Schema compliance	0.7012	0.5230	0.7608	0.8507
Value-token F1	0.5520	0.4463	0.7667	0.9333

Appendix

Table 11: Random-6 versus diverse source models on SoB-Audio at 20% retention. Results are averaged over three seeds.

Variant	Mean NMAE $\downarrow$	Worst NMAE $\downarrow$	Mean SP $\uparrow$
Without swap refinement	0.2500	0.3544	0.9130
Corrected-residual objective	0.2817	0.3802	0.8983
HoloCosm	0.2362	0.3299	0.9151

Appendix

Table 12: Swap refinement and reconstruction-objective ablations at 20% retention. All variants use matched seeds and the same source-assisted estimator. Results are averaged equally over 15 configurations.

Variant	HM	DR	SR	Avg.
Ours	0.2940 / 0.7845	0.2246 / 0.8345	0.3101 / 0.7703	0.2762 / 0.7964
Single-metric selection	0.4507 / 0.6663	0.3309 / 0.6968	0.4960 / 0.6636	0.4259 / 0.6756
Without ranking constraint	0.3710 / 0.6429	0.3099 / 0.6714	0.3911 / 0.6640	0.3573 / 0.6594

Appendix

Table 13: Ablation on Fail2Drive-ID at 20% retention. Three-seed NMAE

\downarrow

/ SP

\uparrow

means; Avg. averages evaluation metrics. Single-metric results also average guiding metrics.

	NMAE $\downarrow$			SP $\uparrow$
Metric	Recon	Ours	Reduction	Recon	Ours	Gain
Correct	0.1384	0.1288	6.9%	0.9790	0.9855	+0.0065
Exact Match	0.1561	0.1304	16.5%	0.9814	0.9837	+0.0022
Safety	0.1696	0.1578	6.9%	0.9592	0.9614	+0.0022

Appendix

Table 14: Per-metric constrained-selection ablation on WorkBench. Results at 20% retention are three-run means. Reconstruction and Ours use the same source-assisted correction. NMAE reduction is

(\mathrm{Recon}-\mathrm{Ours})/\mathrm{Recon}

; SP gain is

\mathrm{Ours}-\mathrm{Recon}

. Positive values favor Ours.

	NMAE $\downarrow$			SP $\uparrow$
Metric	Recon	Ours	Reduction	Recon	Ours	Gain
Clarity	0.1054	0.1074	$-1.9$ %	0.9725	0.9738	+0.0012
Feasibility	0.1564	0.1303	16.7%	0.9641	0.9771	+0.0130
Originality	0.0690	0.0707	$-2.6$ %	0.9947	0.9947	0.0000

Appendix

Table 15: Per-metric constrained-selection ablation on LiveIdea-Core3. Results at 20% retention are three-run means. Reconstruction and Ours use the same source-assisted correction. NMAE reduction is

(\mathrm{Recon}-\mathrm{Ours})/\mathrm{Recon}

; SP gain is

\mathrm{Ours}-\mathrm{Recon}

. Positive values favor Ours.

Estimator / method	Mean NMAE $\downarrow$	Worst NMAE $\downarrow$	Mean SP $\uparrow$
ZipBench: original weighted	0.4920	0.7376	0.8145
ZipBench: uniform + correction	0.3423	0.4720	0.8705
HoloCosm	0.2362	0.3299	0.9151

Appendix

Table 16: Common-estimator comparison with AverageMetric-guided ZipBench at 20% retention. Results average three runs per method and weight the 15 benchmarks equally. All ZipBench variants use the same selected items.

Setting	Fail2Drive-ID	SoB-Audio	LiveIdea-Core3
$q=0.10$	0.3996 / 0.4671 / 0.5782	0.6236 / 0.7902 / 0.7960	0.1028 / 0.1336 / 0.9819
$q=0.25$ (default)	0.2762 / 0.3219 / 0.7964	0.5798 / 0.7520 / 0.8293	0.1028 / 0.1336 / 0.9819
$q=0.50$	0.3573 / 0.3922 / 0.6594	0.5798 / 0.7520 / 0.8293	0.1102 / 0.1564 / 0.9771

Appendix

Table 17: Sensitivity to the ranking-constraint quantile at 20% retention. Each entry reports mean/worst-component NMAE / mean SP.

Benchmark	$B$	$\|S_{\mathrm{union}}\|$	$I_{\mathrm{union}}$
LongForm	22	101	4.591
Widget2Code	58	263	4.534
MMDocRAG-MM	395	1,624	4.111
SoB-Text	1,000	3,320	3.320
RouterArena	1,680	3,020	1.798

Appendix

Table 18: Representative actual union sizes for tinyBenchmarks. At 20% nominal retention, independent component-wise selections can require substantially more than

B

unique evaluations.

Benchmark	Items	Metrics	Selection size	Time (s)
RouterArena	8,400	2	1,680	$2.53\pm 1.12$
SoB-Text	5,000	6	1,000	$2.38\pm 0.28$
MMDocRAG-Multimodal	1,974	8	395	$1.32\pm 0.17$

Appendix

Table 19: Offline item-selection time at 20% retention. Times are means and one standard deviation over three seeds.

	NMAE $\downarrow$		SP $\uparrow$		Kendall's $\tau_{b}\uparrow$
Method	Mean	Worst	Mean	Worst	Mean
HoloCosm	0.2362	0.3299	0.9151	0.8624	0.8173
Random	0.3650	0.5450	0.8840	0.8302	0.7859
tinyBenchmarks	0.4421	0.6297	0.8483	0.7545	0.7252
ZipBench	0.4123	0.6364	0.8467	0.7646	0.7288
mRMR	0.4321	0.5997	0.7894	0.5952	0.6734

Appendix

Table 20: Aggregate fidelity across 15 benchmarks at 20% retention. Mean and worst-component fidelity are aggregated within benchmarks before averaging benchmarks equally. Each scalar method uses its best selection target by mean NMAE, with the same target used for all fidelity measures. Bold and underline indicate the best and second-best values in each column.

Configuration	Random		AnchorPoints: Single		AnchorPoints: AverageMetric		AnchorPoints: Vote
	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$
Widget2Code	0.2979	0.9335	0.3280	0.8628	0.3148	0.9167	0.4707	0.8911
ConStory	0.2938	0.8813	1.2431	0.5209	0.8375	0.7038	0.3320	0.8606
Bench2Drive	0.3881	0.8418	0.5910	0.7899	0.5200	0.8120	0.9476	0.8800
Fail2Drive-ID	0.5805	0.5918	0.2839	0.7051	0.4309	0.7524	0.8388	0.8696
Fail2Drive-OOD	0.7939	0.7903	0.6111	0.7968	0.7323	0.8720	0.7760	0.7794

Appendix

Table 21: Per-configuration simple-strategy results at 20% retention: Random and AnchorPoints. NMAE and SP are means over metrics.

Configuration	K-means		Ours: AverageMetric		Ours: All metrics
	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$
Widget2Code	0.2231	0.9196	0.2086	0.9276	0.2178	0.9444
ConStory	0.5427	0.7896	0.4015	0.8618	0.2972	0.8754
Bench2Drive	0.2938	0.8346	0.3234	0.8677	0.2736	0.8432
Fail2Drive-ID	0.2370	0.7865	0.3761	0.7031	0.2762	0.7964
Fail2Drive-OOD	0.4327	0.7447	0.5646	0.7401	0.3889	0.8642

Appendix

Table 22: Per-configuration simple-strategy results at 20% retention: K-means and Ours. NMAE and SP are means over metrics.

Configuration	tinyBenchmarks		ZipBench		mRMR		AnchorPoints		Ours: AverageMetric		Ours: All metrics
	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$	NMAE $\downarrow$	SP $\uparrow$
Widget2Code	0.4120	0.8958	0.3892	0.8562	0.2476	0.9167	0.3148	0.9167	0.2086	0.9276	0.2178	0.9444
ConStory	0.4377	0.8259	0.5151	0.8100	0.7738	0.4386	0.8375	0.7038	0.4015	0.8618	0.2972	0.8754
Bench2Drive	0.7194	0.7078	0.6091	0.7980	0.5039	0.7149	0.5200	0.8120	0.3234	0.8677	0.2736	0.8432
Fail2Drive-ID	0.6051	0.7966	0.6112	0.5426	0.3388	0.5753	0.4309	0.7524	0.3761	0.7031	0.2762	0.7964
Fail2Drive-OOD	0.7430	0.5717	0.9218	0.4478	0.4691	0.5431	0.7323	0.8720	0.5646	0.7401	0.3889	0.8642

Appendix

Table 23: Additional AverageMetric-guided results at 20% retention. Ours: All metrics uses the full metric set. NMAE and SP are metric means; Average weights 15 configurations equally.

Fail2Drive-ID
Method	Selection metric	NMAE $\downarrow$				SP $\uparrow$
		HM	DR	SR	Mean	HM	DR	SR
Ours	All metrics	0.294	0.225	0.310	0.276	0.785	0.835	0.770
Random	None	0.611	0.488	0.643	0.580	0.571	0.646	0.558
tinyBenchmarks	HM	0.804	0.702	0.820	0.775	0.482	0.608	0.296
	DR	0.766	0.681	0.814	0.754	0.477	0.583	0.506

Appendix

Table 24: Cross-metric fidelity on Fail2Drive-ID and Fail2Drive-OOD at 20% retention. Ours and Random do not use metric-specific selection. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline's lowest-mean selection target.

LiveIdea-Core3
Method	Selection metric	NMAE $\downarrow$				SP $\uparrow$
		Originality	Feasibility	Clarity	Mean	Originality	Feasibility	Clarity
Ours	All metrics	0.071	0.130	0.107	0.103	0.995	0.977	0.974
Random	None	0.071	0.149	0.103	0.108	0.995	0.976	0.979
tinyBenchmarks	Originality	0.103	0.382	0.208	0.231	0.990	0.963	0.956
	Feasibility	0.117	0.187	0.184	0.162	0.985	0.952	0.946

Appendix

Table 25: Cross-metric fidelity on LiveIdea-Core3 and WorkBench at 20% retention. Ours and Random do not use metric-specific selection. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline's lowest-mean selection target.

Bench2Drive
Method	Selection metric	NMAE $\downarrow$					SP $\uparrow$
		Efficiency	Score	Smoothness	Success	Mean	Efficiency	Score	Smoothness	Success
Ours	All metrics	0.053	0.435	0.159	0.448	0.274	0.782	0.845	0.911	0.835
Random	None	0.139	0.631	0.253	0.529	0.388	0.733	0.857	0.933	0.844
tinyBenchmarks	Efficiency	0.077	0.714	0.339	0.706	0.459	0.766	0.770	0.824	0.741
	Score	0.143	0.782	0.346	0.929	0.550	0.845	0.755	0.824	0.714

Appendix

Table 26: Cross-metric fidelity on benchmarks with 4 metrics at 20% retention. Each baseline selects using the indicated metric and is evaluated on every component. Underlining marks diagonal entries. Ours and Random do not use metric-specific selection. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline's lowest-mean selection target.

Method	Selection metric	Acc.	Cost	Mean
Ours	All metrics	0.11/0.96	0.07/0.98	0.09
Random	None	0.19/1.00	0.10/0.98	0.15
tinyBenchmarks	Acc.	0.43/0.89	0.14/0.94	0.28
	Cost	0.55/0.90	0.16/0.98	0.35
ZipBench	Acc.	0.37/0.96	0.12/0.96	0.25
	Cost	0.43/0.92	0.11/0.98	0.27

Appendix

Table 27: Cross-metric fidelity on RouterArena at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Char.	Fact.	Narr.	Time.	World	Mean
Ours	All metrics	0.51/0.74	0.20/0.97	0.27/0.79	0.23/0.96	0.28/0.92	0.30
Random	None	0.46/0.79	0.23/0.96	0.24/0.79	0.19/0.97	0.36/0.90	0.29
tinyBenchmarks	Char.	0.75/0.67	0.31/0.93	0.50/0.71	0.27/0.93	0.55/0.71	0.48
	Fact.	0.89/0.72	0.44/0.87	0.60/0.77	0.31/0.92	0.71/0.74	0.59
	Narr.	0.71/0.74	0.33/0.87	0.38/0.77	0.25/0.96	0.50/0.85	0.43
	Time.	1.04/0.64	0.38/0.88	0.43/0.78	0.34/0.91	0.62/0.79	0.56

Appendix

Table 28: Cross-metric fidelity on ConStory at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Schema	LeafEM	ValueF1	PathRec	PathF1	Mean
Ours	All metrics	0.55/0.77	0.73/0.83	0.54/0.92	0.49/0.84	0.58/0.79	0.58
Random	None	0.56/0.73	1.14/0.75	0.76/0.86	0.50/0.82	0.57/0.73	0.70
tinyBenchmarks	Schema	0.82/0.70	2.02/0.65	1.45/0.75	0.90/0.73	0.96/0.63	1.23
	LeafEM	0.89/0.64	1.18/0.59	0.93/0.73	0.98/0.69	0.97/0.58	0.99
	ValueF1	0.88/0.66	1.53/0.57	0.97/0.79	0.76/0.67	0.94/0.55	1.02
	PathRec	0.71/0.74	2.37/0.73	1.68/0.78	0.75/0.73	0.89/0.70	1.28

Appendix

Table 29: Cross-metric fidelity on SoB-Audio at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	PathRec	LeafEM	PathF1	Schema	ValueF1	Mean
Ours	All metrics	0.51/0.77	0.38/0.88	0.56/0.74	0.54/0.80	0.58/0.67	0.51
Random	None	1.01/0.68	1.20/0.77	1.10/0.57	1.00/0.62	1.57/0.62	1.17
tinyBenchmarks	PathRec	0.95/0.60	1.37/0.69	1.11/0.54	0.95/0.63	1.90/0.52	1.25
	LeafEM	0.87/0.64	1.19/0.71	0.94/0.51	0.93/0.59	1.20/0.64	1.03
	PathF1	1.16/0.62	1.65/0.70	1.30/0.52	1.10/0.62	1.98/0.59	1.44
	Schema	1.06/0.60	1.15/0.69	1.21/0.48	1.02/0.55	1.37/0.32	1.16

Appendix

Table 30: Cross-metric fidelity on SoB-Image at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Schema	LeafEM	ValueF1	PathRec	PathF1	JSONEM	Mean
Ours	All metrics	0.14/0.97	0.16/0.95	0.18/0.96	0.14/0.97	0.14/0.97	0.17/0.97	0.16
Random	None	0.19/0.95	0.21/0.97	0.19/0.96	0.18/0.96	0.18/0.95	0.32/0.96	0.21
tinyBenchmarks	Schema	0.36/0.91	0.45/0.83	0.44/0.83	0.36/0.87	0.39/0.89	0.65/0.90	0.44
	LeafEM	0.37/0.91	0.34/0.90	0.35/0.88	0.36/0.89	0.35/0.89	0.70/0.91	0.41
	ValueF1	0.31/0.92	0.45/0.79	0.46/0.79	0.31/0.93	0.31/0.90	0.50/0.87	0.39
	PathRec	0.35/0.84	0.53/0.86	0.50/0.83	0.38/0.81	0.37/0.78	0.56/0.88	0.45

Appendix

Table 31: Cross-metric fidelity on SoB-Text at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	QuoteP	QuoteR	QuoteF1	Fluency	CitQual	TxtImg	Logic	Fact.	Mean
Ours	All metrics	0.09/0.99	0.05/0.99	0.07/0.99	0.06/1.00	0.06/0.99	0.06/0.99	0.07/1.00	0.06/0.99	0.06
Random	None	0.11/0.99	0.06/1.00	0.07/0.99	0.07/0.99	0.06/0.99	0.05/0.99	0.08/0.99	0.09/0.99	0.07
tinyBenchmarks	QuoteP	0.19/0.97	0.12/0.99	0.15/0.98	0.12/0.97	0.12/0.98	0.11/0.98	0.15/0.97	0.13/0.98	0.14
	QuoteR	0.21/0.95	0.13/0.98	0.16/0.98	0.14/0.98	0.15/0.97	0.15/0.98	0.13/0.98	0.14/0.97	0.15
	QuoteF1	0.15/0.98	0.10/0.99	0.11/0.98	0.13/0.98	0.14/0.98	0.12/0.98	0.18/0.97	0.19/0.98	0.14
	Fluency	0.27/0.93	0.15/0.98	0.18/0.97	0.16/0.96	0.17/0.97	0.14/0.98	0.20/0.98	0.16/0.98	0.18

Appendix

Table 32: Cross-metric fidelity on MMDocRAG-MM at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	QuoteP	QuoteR	QuoteF1	Fluency	CitQual	TxtImg	Logic	Fact.	Mean
Ours	All metrics	0.09/0.99	0.08/0.99	0.10/0.98	0.08/0.98	0.10/0.98	0.10/0.98	0.11/0.98	0.12/0.97	0.10
Random	None	0.12/0.99	0.09/0.99	0.13/0.98	0.09/0.98	0.11/0.98	0.11/0.97	0.12/0.98	0.12/0.98	0.11
tinyBenchmarks	QuoteP	0.13/0.98	0.13/0.98	0.15/0.95	0.13/0.97	0.16/0.96	0.15/0.96	0.14/0.97	0.17/0.95	0.14
	QuoteR	0.16/0.97	0.15/0.98	0.19/0.95	0.16/0.97	0.19/0.96	0.17/0.96	0.21/0.95	0.29/0.94	0.19
	QuoteF1	0.13/0.98	0.12/0.98	0.14/0.97	0.12/0.97	0.14/0.97	0.14/0.96	0.14/0.98	0.20/0.95	0.14
	Fluency	0.19/0.98	0.16/0.98	0.24/0.96	0.15/0.96	0.17/0.97	0.19/0.96	0.23/0.94	0.30/0.92	0.20

Appendix

Table 33: Cross-metric fidelity on MMDocRAG-Text at 20% retention. Each cell reports NMAE / SP. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Margin	Content	Area	TextJac	Contrast	Local	Palette	Vibrancy	Polarity	SSIM	LP	Geom.	Mean
Ours	All metrics	0.27	0.10	0.10	0.10	0.51	0.24	0.32	0.30	0.33	0.19	0.10	0.04	0.22
Random	None	0.35	0.23	0.13	0.12	0.56	0.32	0.82	0.37	0.29	0.21	0.13	0.05	0.30
tinyBenchmarks	Margin	0.49	0.24	0.23	0.18	0.86	0.47	1.08	0.68	1.07	0.50	0.21	0.07	0.51
	Content	0.49	0.20	0.29	0.19	0.95	0.32	1.14	0.40	0.47	0.36	0.20	0.08	0.42
	Area	0.53	0.31	0.27	0.18	0.92	0.47	1.07	0.48	0.57	0.38	0.24	0.08	0.46
	TextJac	0.51	0.28	0.21	0.14	1.05	0.40	0.90	0.88	0.71	0.26	0.25	0.09	0.47

Appendix

Table 34: Cross-metric NMAE on Widget2Code at 20% retention. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Margin	Content	Area	TextJac	Contrast	Local	Palette	Vibrancy	Polarity	SSIM	LP	Geom.
Ours	All metrics	0.90	0.98	0.99	0.95	0.86	0.94	0.92	0.93	0.99	0.94	0.96	0.98
Random	None	0.92	0.99	0.95	0.93	0.87	0.98	0.83	0.88	0.99	0.98	0.93	0.96
tinyBenchmarks	Margin	0.71	0.96	0.99	0.86	0.79	0.89	0.89	0.93	0.87	0.92	0.95	0.88
	Content	0.88	0.93	0.95	0.85	0.75	0.94	0.61	0.89	0.93	0.96	0.95	0.94
	Area	0.86	0.82	0.89	0.89	0.73	0.88	0.80	0.87	0.90	0.92	0.95	0.87
	TextJac	0.88	0.85	0.93	0.92	0.86	0.85	0.73	0.76	0.90	0.94	0.96	0.90

Appendix

Table 35: Cross-metric SP on Widget2Code at 20% retention. Underlining marks the selection metric; pale blue-gray marks each single-metric baseline’s selection target with the lowest mean NMAE.

Method	Selection metric	Amat.	Profile	Coher.	Plot	Engage	Prompt	Plan	Poetry	Nuance	Purple	Tell	Uncreat.	Dialogue	Tone	Mean
Ours	All metrics	0.11	0.11	0.12	0.11	0.10	0.12	0.11	0.16	0.10	0.14	0.11	0.13	0.11	0.10	0.12
Random	None	0.12	0.13	0.14	0.13	0.13	0.14	0.15	0.20	0.13	0.18	0.12	0.13	0.13	0.13	0.14
tinyBenchmarks	Amat.	0.17	0.18	0.22	0.22	0.16	0.23	0.20	0.27	0.16	0.24	0.17	0.22	0.17	0.18	0.20
	Profile	0.17	0.19	0.19	0.20	0.18	0.19	0.21	0.26	0.17	0.24	0.18	0.17	0.18	0.20	0.19
	Coher.	0.19	0.17	0.23	0.20	0.18	0.20	0.22	0.33	0.18	0.35	0.18	0.22	0.19	0.19	0.22
	Plot	0.18	0.16	0.17	0.17	0.16	0.18	0.18	0.21	0.16	0.22	0.17	0.18	0.17	0.17	0.18

Appendix

Table 36: Cross-metric NMAE on LongForm at 20% retention. Underlining marks the selection metric. The Mean column averages NMAE across metrics; pale blue-gray marks each single-metric baseline’s lowest-mean selection target.

Method	Selection metric	Amat.	Profile	Coher.	Plot	Engage	Prompt	Plan	Poetry	Nuance	Purple	Tell	Uncreat.	Dialogue	Tone
Ours	All metrics	0.99	0.99	0.97	0.98	0.99	0.98	0.98	0.98	0.99	0.98	0.99	0.99	0.99	0.99
Random	None	0.99	0.98	0.97	0.97	0.98	0.98	0.98	0.97	0.98	0.98	0.99	0.98	0.98	0.98
tinyBenchmarks	Amat.	0.98	0.97	0.94	0.96	0.97	0.97	0.97	0.93	0.97	0.96	0.98	0.97	0.97	0.97
	Profile	0.98	0.96	0.95	0.97	0.97	0.97	0.97	0.91	0.98	0.95	0.98	0.97	0.97	0.97
	Coher.	0.95	0.97	0.93	0.95	0.97	0.95	0.97	0.95	0.97	0.95	0.97	0.96	0.96	0.97
	Plot	0.98	0.97	0.97	0.96	0.98	0.97	0.97	0.95	0.98	0.96	0.97	0.98	0.97	0.97

Appendix

Table 37: Cross-metric SP on LongForm at 20% retention. Underlining marks the selection metric; pale blue-gray marks each single-metric baseline’s selection target with the lowest mean NMAE.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
Oct 13, 2026Guoming Ling, Yupei Lin, Zhongzhan Huang +5LLM Evaluation Benchmark Design
Sun Yat-sen University
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
Sep 11, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3LLM Evaluation
Bosch Research · Sun Yat-sen University
Hierarchical Compression of Vision-Language Model Benchmarks
Sep 29, 2026Hyunjong Ok, Seunggu Kang, Jaeho LeeVLM Evaluation
Pohang University of Science and Technology · Upstage AI

Oct 13, 2026J/K move · Enter open · S save

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Guoming Ling, Yupei Lin, Zhongzhan Huang, Junxin Li+4

Sun Yat-sen University

LLM Evaluation Benchmark Design

Open paper AskPDF arXiv

Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.

Abstract

LLM Evaluation

Explore similar work

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

Hierarchical Compression of Vision-Language Model Benchmarks