Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level

Alexandra Schild; Gerard de Melo

cs.CVSep 20, 2026

Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level

Authors: Alexandra Schild, Gerard de Melo

Organizations: HPI/University of Potsdam Potsdam, Germany

Abstract

Visual Language Models (VLMs) have achieved remarkable success across diverse tasks, yet they struggle with high-resolution inputs where critical information resides in small regions or detailed, cluttered scenes. While several approaches address this limitation, a systematic understanding of why models fail at high resolutions is lacking. We introduce a controlled evaluation framework that disentangles resolution-related performance degradation from task difficulty through semantics-preserving transformations. We propose two simple metrics: Area Under the Scaling Curve (AUSC), which quantifies scaling robustness independent of baseline accuracy, and Prediction Variance Score (PVS), which measures resolution-induced prediction instability. Through comprehensive experiments across 5 model families and 5 benchmarks, we identify three primary failure modes: (1) information loss from downsampling at vision token limits, (2) tokenization artifacts from patch boundary shifts and positional encoding fragility under non-standard aspect ratios, and (3) attention dilution as token counts increase. Our analysis reveals that even state-of-the-art models suffer from performance drops when processing high-resolution images, with degradation patterns varying systematically by architectural family. We provide actionable insights for model architecture design and data augmentation strategies to mitigate these limitations.

Figures & tables

Figure 1: Information loss may occur due to various factors, including downsizing, patch artifacts, and attention dilution.

Figure 2: On the left: Illustration of attention dilution in Qwen2-VL-2B. Despite identical visual content within the cyan bounding box, attention weights diffuse across uninformative blank regions, reducing concentration on task-relevant content and degrading performance; On the right: Centered Kernel Alignment (CKA) calculated for all layers of the Qwen2-VL-2B VLM comparing internal representations for two images: (1) original image padded with white field, (2) upscaled image

Figure 3: Pixel- and Token- Resolution Scaling Example

Pattern	Trajectory	Attribution	Fingerprint
A	$1\cdots 1\,0\cdots 0$ (threshold)	Downsampling loss	Pixel Resolution scaling only, at token budget
B	Alternating	Boundary / density	Token Resolution Scaling $\to$ boundary; Pixel Resolution Scaling $\to$ density
C	$0\cdots 0\,1\cdots$	Tokenization masking	PVS signal; both scenarios
D	$1\,1\,0\,0\cdots$	Attention dilution	Both scenarios, same token count
E	$0\cdots 0$	Reasoning failure	Excluded from metrics
F	$1\cdots 1$	Robust correct	AUSC numerator

Table 1: Answer trajectory patterns, their diagnostic interpretation, and the cross-scenario fingerprint enabling attribution.

Figure 4: Accuracy degradation under different scaling factors.

		AUSC					PVS
Model	Orig. Acc.	1D-C	1D-T	2D-C	2D-T	Intra	1D-C	1D-T	2D-C	2D-T	Intra
Qwen2-VL-2B	0.448	0.752	0.716	0.845	0.826	0.827	0.368	0.368	0.290	0.307	0.300
Qwen2-VL-7B	0.562	0.786	0.733	0.850	0.830	0.851	0.385	0.397	0.353	0.348	0.334
Qwen2.5-VL-3B	0.536	0.830	0.814	0.832	0.823	0.846	0.368	0.352	0.361	0.319	0.352
Qwen2.5-VL-7B	0.604	0.858	0.853	0.870	0.864	0.871	0.382	0.397	0.359	0.351	0.349
Qwen3-VL-2B	0.466	0.803	0.799	0.827	0.817	0.836	0.414	0.376	0.416	0.394	0.445

Table 2: Orig. Acc., AUSC, and PVS across augmentation settings for different VLMs.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Failure Mode	Img.	VL	LLM	Architecture	Training Tasks	Training Data
(Pattern)	Enc.	Align.		Changes	Changes	Changes
1. Down- sampling Loss	$\checkmark\checkmark\checkmark$			Adaptive resolution; hierarchical/pyramid encoder; multi-scale feature extraction; increase native resolution (1024 $\times$ 1024+)	Pre-train on high-res tasks; multi-scale prediction tasks	High-resolution datasets; preserve original resolutions
2a. Patch Boundary Sensitivity	$\checkmark\checkmark\checkmark$			Overlapping patches; continuous tokenization (Perceiver); sliding window approaches	Tasks requiring cross-patch reasoning; random patch offset augmentation	Annotations spanning patch boundaries
2b. Information Density Mismatch	$\checkmark\checkmark$	$\checkmark$	$\checkmark\checkmark$	Adaptive token allocation; variable patch sizes; learned token pruning/expansion	Detail-focused pretraining; dense region description; small object detection	Variable-density images; sparse/dense scene mixtures; fine-grained datasets
3. Attention Dilution			$\checkmark\checkmark\checkmark$	Hierarchical attention; sparse/local attention; region-of-interest modules; cross-attention gating	Spatial attention supervision; localization-aware tasks; progressive detail focusing	Ground-truth attention maps; saliency-annotated data
4. Vision–Language Alignment Gap	$\checkmark$	$\checkmark\checkmark\checkmark$	$\checkmark$	Richer projection layers; multi-layer alignment; feature-preserving projectors	Fine-grained VQA; contrastive attribute learning; pixel-word grounding; referring expression tasks	Detailed attribute annotations; fine-grained VQA; grounding datasets (RefCOCO); descriptive region captions

Appendix

Table 3: Failure mode attribution to pipeline components and recommended interventions. Checkmark weight indicates strength of component involvement.

Figure 5: All models perform better when irrelevant content is replaced with white background, showing that white padding itself is not the primary cause of performance degradation.

Figure 6: Distribution of prediction outcomes across the evaluated image transformations. Samples are divided into those answered incorrectly under all tested conditions ( all scores $=0$ ), those whose predictions vary and are incorrect for the original image ( mixed, originally wrong ), those whose predictions vary and are correct for the original image ( mixed, originally correct ), and those answered correctly under all tested conditions ( all scores $=1$ ). A large proportion of benchmark errors are therefore recoverable under an alternative, semantically equivalent image representation.

Model	$\mathrm{Acc}_{\mathrm{orig}}$	AUSC	PVS	ACE	RCE	Flip +	Flip -
Qwen2.5-VL-7B-Instruct	0.604	0.863	0.368	0.054	0.091	0.086	0.067
Qwen2.5-VL-3B-Instruct	0.536	0.829	0.351	0.070	0.137	0.096	0.073
Qwen3-VL-4B-Instruct	0.600	0.827	0.457	0.086	0.151	0.109	0.079
Qwen3-VL-8B-Instruct	0.627	0.816	0.408	0.109	0.195	0.121	0.063
Qwen3-VL-2B-Instruct	0.466	0.816	0.409	0.080	0.174	0.090	0.085
Qwen2-VL-7B-Instruct	0.562	0.810	0.363	0.109	0.214	0.112	0.070

Appendix

Table 4: Comparison of AUSC and PVS with existing robustness metrics. Higher values indicate better robustness for AUSC, whereas lower values indicate greater robustness for ACE, RCE, Flip + , and Flip - . For PVS, higher values indicate greater prediction variability among samples that are initially answered incorrectly.

Figure 7: AUSC disaggregated by benchmark and transformation type. Each panel corresponds to one model, and each axis to one benchmark. Higher values indicate stronger retention of predictions that are correct on the original images. The five curves represent centered and tokenization-aligned 1D and 2D padding, as well as interpolation.

Figure 8: PVS disaggregated by benchmark and transformation type. Higher values indicate greater prediction variability among samples that are answered incorrectly on the original images. Differences between transformation types reveal that the recoverability of these errors depends on how the image representation is modified.

Figure 9: AUSC and PVS stratified by original image size. AUSC varies only modestly across size strata, whereas PVS generally decreases from small to large images, indicating stronger transformation sensitivity among initially incorrect predictions on smaller inputs. This trend is model dependent and is not observed uniformly across architectures.

Figure 10: Illustration of the Area Under the Scaling Curve (AUSC) metric. AUSC is computed on $\mathcal{B}_{\text{correct}}$ and measures the area under the accuracy-vs-scale curve, normalized to $[0,1]$ .

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
Oct 1, 2026Xinye Zhao, Yunkai Dang, Yunchen Wu +1Vision-Language Models Efficient VLM Inference
School of Intelligence Science and Technology, Nanjing University
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
Mar 14, 2026Nimrod Shabtay, Moshe Kimhi, Artem Spector +5Tool Use in VLMs Efficient VLM Inference
IBM Research · Tel-Aviv University · Technion +1
UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs
May 12, 2026Shuo Ni, Tong Wang, Jing Zhang +5VLM Evaluation Remote Sensing Image Understanding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology, Beijing, China · Zhongguancun Academy, Beijing, China · School of Computer Science, Wuhan University, Wuhan, China +2

Oct 1, 2026J/K move · Enter open · S save

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Xinye Zhao, Yunkai Dang, Yunchen Wu, Wenbin Li

School of Intelligence Science and Technology, Nanjing University

Vision-Language Models Efficient VLM Inference Language Model Scaling Laws

Open paper AskPDF arXiv

Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.

Abstract

VLM Robustness

Explore similar work

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs

UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs