Prompt-Guided Prefiltering for VLM Image Compression

Bardia Azizian; Ivan V. Bajic

eess.IVMar 31, 2026

Prompt-Guided Prefiltering for VLM Image Compression

Authors: Bardia Azizian, Ivan V. Bajic

Organizations: School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada

Abstract

The rapid progress of large Vision-Language Models (VLMs) has enabled a wide range of applications, such as image understanding and Visual Question Answering (VQA). Query images are often uploaded to the cloud, where VLMs are typically hosted, hence efficient image compression becomes crucial. However, traditional human-centric codecs are suboptimal in this setting because they preserve many task-irrelevant details. Existing Image Coding for Machines (ICM) methods also fall short, as they assume a fixed set of downstream tasks and cannot adapt to prompt-driven VLMs with an open-ended variety of objectives. We propose a lightweight, plug-and-play, prompt-guided prefiltering module to identify image regions most relevant to the text prompt, and consequently to the downstream task. The module preserves important details while smoothing out less relevant areas to improve compression efficiency. It is codec-agnostic and can be applied before conventional and learned encoders. Experiments on several VQA benchmarks show that our approach achieves a 25-50% average bitrate reduction while maintaining the same task accuracy. Our source code is available at https://github.com/bardia-az/pgp-vlm-compression.

Figures & tables

Fig. 1: The overall block diagram of the proposed method

Fig. 2: Visualization of the prefiltered versions of a single image generated by our method under three different prompts.

Panel of Figure 3 — Fig. 3: Rate–Accuracy curves for InternVL3-9B (first row) and LLaVA-OV-7B (second row) across SEEDBench, MME, and MMBench benchmarks.

VLM	Compression Method	BD-Rate (%) $\downarrow$
VLM	Compression Method	MMBench	MME	SEEDBench	Average
InternVL	JPEG	$-29.30$	$-24.55$	$-23.78$	-25.88
	HEVC	$-30.60$	$-51.93$	$-48.54$	-43.69
	VVC	$-38.27$	$-49.02$	$-52.05$	-46.45
	Cheng [ 11 ]	$-29.65$	$-45.64$	$-53.50$	-42.93
LLaVA	JPEG	$-33.22$	$-33.83$	$-25.66$	-30.90

TABLE I: BD-Rate (%) comparing our prefiltered approach against the corresponding baseline (same codec without prefiltering). Negative values indicate bitrate savings.

Fig. 4: Rate–accuracy curves for InternVL3-9B on the MME dataset, illustrating the impact of different components of our method.

Method Variant	InternVL	LLaVA	Average
Ours-57M	0	0	0
HEVC-Only	$+82.40$	$+72.73$	$+77.57$
w/o TinyCLIP	$+71.41$	$+70.42$	$+70.91$
Ours-23M	$+7.13$	$+4.51$	$+5.82$
Ours-120M	$+6.69$	$-2.06$	$+2.31$
NoOverlap	$+4.12$	$+6.15$	$+5.13$

TABLE II: Average BD-Rate (%) across SeedBench, MME, and MMBench for method ablations, using “Ours-57M” as the anchor.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 5: Rate–accuracy curves for InternVL3-9B on SEEDBench dataset

Method Variant	InternVL	LLaVA	Average
Ours-57M	0	0	0
HEVC-Only	$+82.40$	$+72.73$	$+77.57$
w/o TinyCLIP (Gaussian 0.2)	$+71.41$	$+70.42$	$+70.91$
Global Gaussian ( $\sigma=1.0)$	$+54.99$	$+47.81$	$+51.40$
Global Gaussian ( $\sigma=3.0)$	$+155.89$	$+115.65$	$+135.77$
Downsampling (1/2)	$+49.50$	$+61.26$	$+55.38$

Appendix

TABLE III: Average BD-Rate (%) across SeedBench, MME, and MMBench for different baselines, using “Ours-57M” as the anchor

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Aug 3, 2026Long Qian, Jiaqi Wei, Bingke Zhu +2Efficient VLM Inference Coreset Selection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1
LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
May 7, 2026Roy Xie, Dan Friedman, Donghan Yu +7Document Understanding Tool Use in VLMs
Duke University · Apple · OpenAI +1
Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity
Mar 10, 2026Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3Visual Token Pruning Visual Token Compression
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory

Aug 3, 2026J/K move · Enter open · S save

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

Long Qian, Jiaqi Wei, Bingke Zhu, Yingying Chen+1

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China+1

Efficient VLM Inference Coreset Selection Faithfulness in VLMs Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.

Abstract

Vision-Language Models

Explore similar work

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity