Hierarchical Compression of Vision-Language Model Benchmarks

Hyunjong Ok; Seunggu Kang; Jaeho Lee

cs.LGSep 29, 2026

Hierarchical Compression of Vision-Language Model Benchmarks

Authors: Hyunjong Ok, Seunggu Kang, Jaeho Lee

Organizations: Pohang University of Science and Technology · Upstage AI

Abstract

Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace. Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for language models, but for VLMs the question remains under-explored. We present PRIMEBench (Pruning Redundant Items for Multimodal Evaluation), a vision-aware hierarchical benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one benchmark per capability category, item pruning with Vision-Aware Variance (VAW), and category-count pruning. VAW combines inter-model variance with a vision-dependence score computed from multimodal embeddings alone, while encouraging coverage of diverse items within each benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side pruning.

Figures & tables

Method	Mod.	Level	Scale
tinyBenchmarks ( Polo et al., 2024 )	T	I	${\sim}$ 300 LLMs
metabench ( Kipnis et al., 2025 )	T	I	${\sim}$ 5000 LLMs
DISCO ( Rubinstein et al., 2026 )	T	I	—
EssenceBench ( Wang et al., 2026 )	T	I	—
SparseEval ( Zhang et al., 2026 )	T	I	—
SubLIME ( Saranathan et al., 2025 )	T	I	—

Table 1: Selected benchmark compression methods, organised by modality, compression level, and validation panel size. Mod. denotes the demonstration setting in the original work (T=text-only LLMs, V=vision–language). Level : B=benchmark, I=item. Scale : largest validation panel. PRIMEBench combines benchmark- and item-level compression, releases a compact suite, and validates item selection on

30

VLMs and benchmark-level ranking preservation against

231

VLMs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	$r{=}2\%$	$r{=}5\%$	$r{=}10\%$	$\Delta\rho$
Qwen3-VL-Embedding-8B (default)	0.907	0.943	0.949	–
Qwen3-VL-Embedding-2B	0.911	0.931	0.950	$-0.012$
VLM2Vec-V2	0.923	0.932	0.956	$-0.011$
Gemini-Embedding-2	0.909	0.932	0.954	$-0.011$
Variance	0.897	0.914	0.940	$-0.029$

Appendix

Table 14: Swapping the encoder lowers VAW by

0.011

–

0.012

r{=}5\%

. Macro Spearman

\rho

(mean over categories) on held-out models, mean over the

20

random

15/15

splits, on the four categories that every encoder covers (Comprehensive, Document Understanding, Text Recognition, Science Knowledge). Each encoder supplies both

\delta^{\mathrm{VL}}

and the cells;

\Delta\rho

is against the default encoder at

r{=}5\%

. Variance (item-index tie-breaking) is a reference on the same splits and categories. Best in bold, second best underlined.

$k$	Stage 3	End-to-end	Items	Selected categories
	$\rho$
$3$	0.976	0.919	7,391	Academic Knowledge, Chart Understanding, Visual Reasoning
$5$	0.988	0.959	9,377	Academic Knowledge, Chart Understanding, Hallucination Detection, Visual Reasoning, Web/UI Understanding
$8$	0.996	0.962	1,136	Academic Knowledge, Chart Understanding, Commonsense, Document Understanding, Math Reasoning, Object Recognition, Science Knowledge, Spatial Reasoning
$10$	0.995	0.958	9,882	Academic Knowledge, Comprehensive, Hallucination Detection, Math Reasoning, Object Grounding, Object Recognition, Science Knowledge, Text Recognition, Visual Reasoning, Web/UI Understanding

Appendix

Table 15: Exhaustive Stage 3

k

-sweep on the Stage 2-pruned (

r{=}5\%

, VAW) category matrix. Top-

1

of the

\binom{18}{k}

subsets by Spearman

\rho

against the

18

-category ranking on the pruned matrix (Stage 3 convention), also evaluated against the category means over all Stage 0-cleaned benchmarks (end-to-end).

$k$	$\rho$	$95\%$ CI	$\rho$ , mean imputation
3	0.978	$[0.973,\,0.982]$	0.951
5	0.988	$[0.984,\,0.990]$	0.978
8	0.994	$[0.993,\,0.995]$	0.990

Appendix

Table 18: Five of the

14

OpenVLM benchmarks recover the full ranking at

\rho{=}0.988

\rho

of exhaustive Stage 3 selection against the

14

-benchmark ranking on the

231

models that cover at least

7

benchmarks, with

95\%

CIs from

10{,}000

paired bootstrap resamples stratified over

95

model families inferred from model names (

53

of them singletons); the CI is computed on the selected subset and omits selection uncertainty. The last column fills missing scores with the benchmark mean over all

265

models and re-selects the subset.

Release era	Models	Benchmarks	Components for $95\%$	Participation ratio	Top- $5$ energy (%)
Before Jun 2024	73	14	8	7.2	88.8
Jun–Dec 2024	114	14	9	7.7	88.1
From Jan 2025	78	13	8	8.1	86.0

Appendix

Table 19: The top five components hold

86

–

89\%

of the OpenVLM spectral energy in every release era. Spectrum of each era’s

z

-normalised model-by-benchmark matrix; the last era has

13

benchmarks because one has no scores, and each model’s era is inferred from its name. This concentration is what lets a few benchmarks stand in for all of them.

Target	Models	$M{=}1$	$2$	$4$	$8$	$16$	$32$	$64$	$128$
MMT-Bench	$200$	$0.114$	$0.089$	$0.074$	$0.061$	$0.051$	$0.044$	$0.039$	$0.039$
ScienceQA	$214$	$0.168$	$0.126$	$0.117$	$0.103$	$0.086$	$0.071$	$0.065$	$0.062$
TextVQA	$126$	$0.211$	$0.194$	$0.169$	$0.139$	$0.123$	$0.111$	$0.102$	–
BLINK	$177$	$0.119$	$0.092$	$0.079$	$0.065$	$0.058$	$0.054$	$0.049$	$0.047$

Appendix

Table 22: Left-out benchmark error falls with the number of training models and flattens beyond

M{=}64

. Mean MAE over

30

random splits of a Ridge regressor that predicts each target benchmark from the other OpenVLM aggregate benchmarks, by the number

M

of training models. Models = models scored on the target, with a missing feature score replaced by its mean over the training models; TextVQA has no tested training size above

M{=}64

Explore similar work

Aug 7, 2026cs.CV

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3

Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea

Visual Token Pruning Large Language Model Compression Sparsity Importance

Ask PDF

Sep 23, 2026cs.CL

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a frontier-centric design: each model is reduced to a single accuracy number, narrowing the inter-model gap on saturated suites and pressing models into low-score bands on harder ones. We introduce PRISM-VLM, a multi-axis discriminative benchmark that scores every item along seven axes covering the recurring failure modes (task quality, behavioral robustness, and capability bottlenecks) and combines them into a single PScore, with items recycled from fifteen public benchmarks. Across compact VLMs from the past two years, PScore separates model pairs more reliably than prior single-axis benchmarks under an item-level paired bootstrap, and surfaces behavioral differences these benchmarks average away. Even models with statistically indistinguishable PScores diverge sharply along the per-axis profile, particularly on sycophancy, which is nearly orthogonal to single-prompt accuracy. We will release the full pipeline, prompts, and per-item annotations.

Sanghee Park, Kee-Eung Kim

NAVER Cloud AI · KAIST AI

Recent Vision-Language Models Multimodal Benchmarks Multi-Modal Data

Ask PDF

May 28, 2026cs.LG

AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference

Vision-Language Models (VLMs) process thousands of visual tokens per image alongside comparatively few text tokens, yet existing compression methods treat both modalities uniformly. We observe that the two modalities have fundamentally different properties: vision tokens are spatially redundant and dominate prefill, while text tokens are causally dependent and accumulate during decoding. Based on this asymmetry, we propose and empirically evaluate AsymVLM, which applies aggressive pruning to vision tokens before prefill using a learned importance scorer with per-sample adaptive budgeting, and temporal threshold-based eviction to text tokens only when they exceed a fixed budget. Our experiments indicate that AsymVLM achieves the highest FLOPs savings (up to 54%) among state-of-the-art methods while outperforming existing approaches by 2--3% on document and chart understanding tasks where visual information is spatially localized and query-specific, and maintaining competitive accuracy on holistic benchmarks. In text-dominated scenarios, our eviction strategy substantially outperforms standard LLM cache compression methods by adapting to the short-context nature of VLM.

Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan Kandemir

The Pennsylvania State University

Visual Token Pruning Large Language Model Compression Structured Pruning

Ask PDF