ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding

Yuanhao Sun; Huawei Ji; Jiaxin Ding; Luoyi Fu; Xinbing Wang

cs.CVAug 24, 2026

ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding

Authors: Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, Xinbing Wang

Organizations: Shanghai Jiao Tong University, Shanghai, China

Abstract

Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text entropy of cross-modal attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose ENCORE, an entropy-guided framework with two components: At inference, an Entropy-based Cropping Strategy (ECS) evaluates a small set of candidate crops and selects the one with minimal entropy, preserving contiguous regions relevant to the prompt. At training, Entropy Regularization Training (ERT) augments next-token prediction with an entropy term that sharpens attention on key visual tokens while down-weighting irrelevant ones. Experiments on ten VQA benchmarks show that ENCORE, fine-tuning only 0.14% of parameters, achieves an average 1.43% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.

Figures & tables

Figure 1: Insights into image–text entropy. (a) Visualization of prompt-relevant visual-token probability distributions from VLM’s second layer; (b) Images cropped by ratio with the n-th smallest $H_{S}$ and the maximum IoU(intersection over union) between patches and answer-grounding boundary (baseline: original aspect ratio) on VizWiz [ 1 ] . Results show that lower $H_{S}$ indicates higher coverage ratio, corresponding to more accurate visual understanding; (c) Average $H_{S}$ distributions over first four layers for correctly answered samples on TextVQA [ 16 ] , HallB [ 6 ] , SEED-2 [ 10 ] , RWQA [ 20 ] are lower than that of incorrectly answered samples, with a significantly statistic difference in T-test ( $t=-11,p=3.33\times 10^{-28}<0.05$ ).

Figure 2: ENCORE framework. (a) ECS: Compute early-layer cross-modal similarities between visual tokens and prompt, derive prompt-relevant distribution, compute its entropy, select minimal-entropy crop ratio to preserve contiguous prompt-relevant semantics; (b) ERT: During fine-tuning, augment next-token prediction loss with entropy regularizer to encourage sharpening cross-modal attention for accurate key visual token extraction from fragmented high-res inputs.

Model	Size	SEED-2	Info	Text	OCRB	CCB	RWQA	HallB	POPE	$\text{MMB}_{\text{v1.1}}^{\text{EN}}$	MMStar
Deepseek-VL [ 14 ]	2B	43.3	21.0	57.7	414	37.6	49.7	27.6	85.9	63.8	39.9
Qwen2-VL [ 17 ]	2B	62.4	65.5	79.4	809	53.5	62.6	41.7	87.8	72.7	47.8
mPLUG-Owl3 [ 22 ]	2B	44.6	29.0	59.2	434	33.7	55.5	25.1	85.8	-	41.2
InternVL-2 [ 3 ]	2B	60.0	58.0	74.7	779	74.3	57.3	37.9	85.2	70.2	50.2
Mini-monkey [ 8 ]	2B	61.2	60.0	75.7	802	75.5	57.5	38.7	86.7	68.9	50.4
InternVL-3 [ 2 ]	2B	64.6	66.1	77.0	831	74.9	64.3	41.8	89.6	78.6	60.7

Table 1: Comparison results with SOTA VLMs. ENCORE achieves the best results among 2B-parameter VLMs and surpass some larger VLMs in several benchmarks.

Strategy	TextVQA	OCRBench	RWQA	CCBench
DHR [ 3 ]	77.0	831	64.3	74.9
FHR [ 11 ]	76.9	825	63.6	73.5
MSAC [ 8 ]	78.8	834	63.0	75.0
RTC [ 13 ]	77.2	827	63.9	74.8
ECS (ours)	79.1	837	65.1	75.3

Table 2: Comparison of ECS and other cropping strategies. Results are tested with InternVL3-2B.

Figure 3: Ablation study of $H_{S}$ under varying temperature $\tau$ . (a) Visualization of $\tau$ ’s impact. (b) shows the $\tau$ ’s impact on average score.

Figure 4: Ablation study of Entropy Regularization Training (ERT). (a) shows average score on POPE, OCRBench, RWQA and CCBench; In (a)–(b), NTP denotes next‑token‑prediction fine‑tuning.

Figure 5: Visualization of cross-attention matrices in layer 2 after Entropy Regularization Training (ERT).

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models
Sep 17, 2026Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5VLM Robustness Prompt Sensitivity
Sapienza University of Rome · Université Côte d’Azur · CNRS +3
SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
Jun 18, 2026Bo Yin, Xiaobin Hu, Chengming Xu +6VLM Robustness VLM Adaptation
National University of Singapore · Fudan University · Technical University of Munich +4
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
Apr 19, 2026Lijie ZhouCross-Modal Learning VLM Robustness
School of Computer Science University of Nottingham Ningbo China Ningbo, China

Sep 17, 2026J/K move · Enter open · S save

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza, Oleksandr Pryymak+4

Sapienza University of Rome · Université Côte d’Azur · CNRS+3

VLM Robustness Prompt Sensitivity Cross-Modal Attention

Open paper AskPDF arXiv

Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust---e.g., rephrasing "Is there a cat?" into "Please look carefully and answer: is there a cat?". Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., "what colour is the cup left of the chair?" instead of "is there a cup?". Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70--81% on the 8B models. The practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.

Abstract

VLM Adaptation

Explore similar work

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability