Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap Digital Agnosia. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
Figures & tables
Figure 1: Samples from G2M. Difficulty ranges from (a) simple tests to (b) dense settings that exceed standard patch resolution.
Grid
GPT-5-mini
Gemini-3-Flash
3×3
100 / 100
100 / 100
6×6
97.3 / 99.9
99.0 / 100.0
9×9
0.0 / 65.6
97.3 / 100.0
12×12
0.0 / 41.5
69.7 / 99.2
20×20
0.0 / 33.7
0.0 / 87.8
32×32
0.0 / 0.0
0.0 / 38.5
Table 1: Zero-shot Exact / Cell accuracies (%) of proprietary models on 3-color grids.
Figure 2: Cell accuracy of open-weight models under zero-shot generation and VE probing. While zero-shot performance collapses rapidly as grid size increases, substantially higher accuracy remains recoverable from frozen VE features under supervised probing.
Figure 3: Diagnostic spatial heatmaps for proprietary models for grid sizes from 9×9 to 32×32 . Red indicates high cell accuracy and blue indicates lower accuracy.
Figure 4: Diagnostic spatial heatmaps for open-weight models for grid sizes from 4×4 to 12×12 . Red indicates high cell accuracy and blue indicates lower accuracy.
Figure 5: Diagnostic spatial heatmaps for open-weight model VEs (via Spatial Probing) for grid sizes from 20×20 to 64×64 . Red indicates high cell accuracy and blue indicates lower accuracy.
Figure 6: Grid-patch alignment across varying grid sizes. Performance generally peaks at patch-aligned resolutions (e.g., 32×32 , 48×48 ), but this trend breaks at 64×64 , which underperforms its immediate misaligned neighbors.
Figure 7: Zero-shot scaling on open-weight models for 9×9 and 12×12 grids.
Model Family
VLM Size
Vision Encoder Backbone
Spatial Probe Accuracy
48×48
64×64
Qwen3-VL
2B / 4B
SigLIP2-Large (300M)
81.59%
57.47%
8B / 32B
SigLIP2-SO (400M)
72.31%
54.81%
InternVL3.5
1B–14B
InternViT (300M)
98.75%
69.27%
38B
InternViT (6B)
100.00%
73.34%
Table 2: Spatial probe Cell Accuracy on dense grids. We only show the results for 48×48 and 64×64 here, since accuracies are saturated for smaller grids.
Figure 8: Error heatmaps from spatial probes on open-weight models at 48×48 and 64×64 .
Figure 9: Error heatmaps comparing vanilla and extracted vision encoders at 64×64 . Alignment improves overall accuracy, but many of the base encoder’s blind spots remain visible.
Model Family
Vision Encoder
Spatial Probe Accuracy
20×20
32×32
48×48
64×64
Qwen3-VL-8B
SigLIP2-SO (400M)
95.49%
75.39%
57.94%
47.23%
Extracted
99.93%
100.00%
72.31%
54.81%
InternVL3.5-8B
InternViT (300M)
100.00%
100.00%
70.89%
56.20%
Extracted
100.00%
100.00%
98.75%
69.27%
InternVL3.5-38B
InternViT (6B)
100.00%
100.00%
99.97%
67.99%
Table 3: Spatial probe Cell Accuracy on dense grids. Comparing vanilla and extracted vision encoders shows that multimodal alignment substantially improves spatial recoverability under probing.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Error heatmaps for 47×47 , 48×48 , 64×64 , and 65×65 . Yellow lines show the patch boundaries. For better visibility, we show only the center 8×8 patches. See Figures 14 and 15 for the full heatmaps.
Figure 11: Cell accuracy across six boundary-interaction types near the 64×64 grid size. Because a 64×64 grid yields exactly 2×2 cells in each patch, 100% of its cells fall into the Edg-Edg configuration. Nearby misaligned grids distribute their cells across all categories, revealing a strict degradation in accuracy as cells cross more patch boundaries.
Grid Size
InternVL3.5-8B (Probe VE)
Qwen3-VL-8B-Instruct (Probe VE)
3C
4C
5C
6C
8C
10C
3C
4C
5C
6C
8C
10C
20 × 20
100.00
100.00
100.00
100.00
100.00
100.00
99.93
99.91
99.84
99.76
99.55
99.49
32 × 32
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
99.99
99.99
99.96
99.97
48 × 48
98.75
97.60
96.68
95.80
92.88
90.73
72.31
67.80
64.12
61.70
56.36
54.40
64 × 64
69.27
62.23
56.63
52.46
46.45
42.55
54.81
47.41
41.97
38.08
32.31
28.59
Appendix
Table 4: Spatial Probing Cell Accuracy (%) of the frozen VE across varying color diversities. The notation 3C to 10C denotes the number of distinct colors present in the grid.
Grid Size
InternVL3.5-8B
Qwen3-VL-8B-Instruct
3C
4C
5C
6C
8C
10C
3C
4C
5C
6C
8C
10C
3 × 3
100.00
100.00
100.00
100.00
100.00
99.33
99.78
99.78
99.78
100.00
100.00
100.00
4 × 4
100.00
99.84
100.00
99.99
100.00
99.99
95.86
98.30
98.88
99.35
99.69
99.78
6 × 6
97.94
97.90
98.02
98.59
99.06
99.10
69.59
74.69
78.22
80.89
81.76
82.22
9 × 9
65.07
62.07
63.14
64.47
65.36
63.14
40.00
38.59
37.34
36.98
35.20
34.03
12 × 12
35.74
28.43
24.55
22.13
19.94
17.39
34.58
26.53
22.62
19.39
16.42
14.67
Appendix
Table 5: Zero-Shot Cell Accuracy (%) of Open-Weight VLMs across varying color diversities. The notation 3C to 10C denotes the number of distinct colors present in the grid.
Figure 12: Color analysis heatmap illustrating per-color localization performance in InternVL3.5-8B.
Grid
GPT-5-mini
GPT-5.2
Gemini-3-Flash
Gemini-3-Pro
6×6
97.3 / 99.9
100.0 / 100.0
99.0 / 100.0
100.0 / 100.0
9×9
0.0 / 65.6
97.3 / 100.0
97.3 / 100.0
100.0 / 100.0
12×12
0.0 / 41.5
94.3 / 100.0
69.7 / 99.2
100.0 / 100.0
20×20
0.0 / 33.7
0.0 / 64.4
0.0 / 87.8
47.3 / 84.3
32×32
0.0 / 0.0
0.0 / 34.1
0.0 / 38.5
0.0 / 35.0
Appendix
Table 6: Zero-shot Exact / Cell accuracies (%) of proprietary VLMs on G2M for grid densities 6×6 to 32×32 .
Figure 13: Scaling on proprietary models on 12×12 and 20×20 grids.
Grid
InternVL3.5-8B
Qwen3-VL-8B-Instruct
2×2
100.0 / 100.0
100.0 / 100.0
3×3
100.0 / 100.0
66.0 / 99.8
4×4
100.0 / 100.0
0.0 / 96.6
6×6
62.0 / 98.1
0.0 / 71.2
9×9
0.0 / 64.8
0.0 / 40.0
12×12
0.0 / 35.6
0.0 / 35.0
Appendix
Table 7: Zero-shot Exact / Cell accuracies (%) of open-weight models on 3-color grids.
Figure 14: Grid-patch interaction patterns in InternVL3.5-8B. We present error heatmaps for grid sizes around 32×32 , 48×48 , and 64×64 . Below each heatmap, yellow lines illustrate how the vision encoder’s patch boundaries overlay the original image. Because the fixed patch size does not evenly divide the grid cells, the boundaries progressively misalign across the image. As detailed in Section 4.2 , the cell that occupies the majority of a patch dictates the feature representation and achieves higher accuracy. This gradual shift in patch-to-cell overlap produces the distinct macroscopic interference patterns visible in the error heatmaps.
Figure 15: Grid-patch interaction patterns in Qwen3-VL-8B-Instruct. We present error heatmaps for grid sizes around 32×32 , 48×48 , and 64×64 . Below each heatmap, yellow lines illustrate how the vision encoder’s patch boundaries overlay the original image. Because the fixed patch size does not evenly divide the grid cells, the boundaries progressively misalign across the image. As detailed in Section 4.2 , the cell that occupies the majority of a patch dictates the feature representation and achieves higher accuracy. This gradual shift in patch-to-cell overlap produces the distinct macroscopic interference patterns visible in the error heatmaps. The impact of grid-patch interactions appears to be more pronounced compared to InternVL3.5-8B, which is especially visible in the first row (around 32×32 ).
Figure 16: Zero-shot spatial error heatmaps for all proprietary models, a full version of Figure 7 from Section 5.1 .
Figure 17: Zero-shot spatial error heatmaps for Qwen3-VL-8B-Instruct across varying grid sizes and color complexities. The autoregressive degradation pattern remains as the color vocabulary increases (see Appendix B ).
Figure 18: Zero-shot spatial error heatmaps for InternVL3.5-8B. Similar to Qwen3-VL, the autoregressive degradation pattern remains as the color vocabulary increases (see Appendix B ).
Figure 19: VE spatial probing heatmaps for Qwen3-VL-8B-Instruct. While the blind spots persist across different color counts (see Appendix B ), the overarching performance patterns fluctuate across grid sizes due to the cells being divided by patch boundaries and area dominance effects analyzed in Appendix 4.2 .
Figure 20: VE spatial probing heatmaps for InternVL3.5-8B. While the blind spots persist across different color counts (see Appendix B ), the overarching performance patterns fluctuate across grid sizes due to the cells being divided by patch boundaries and area dominance effects analyzed in Appendix 4.2 .
Figure 21: Color analysis heatmap illustrating per-color localization performance in Qwen3-VL-8B-Instruct. We observe a similar pattern as for InternVL3.5-8B in Figure 12 from Appendix B .
Figure 22: Color analysis heatmap illustrating per-color localization performance in GPT-5-mini versus GPT-5.2 (see Appendix B ). We observe that both models exhibit a bias toward overpredicting blue relative to other colors. This is not immediately visible in this plot since we processed the raw accuracies with IoU. Furthermore, white represents the earliest failure mode as grid complexity increases, suffering from severe underprediction. This phenomenon is likely tied to the models’ pre-training distributions, where white often serves as negative space or lacks semantic salience in natural images.
Figure 23: Color analysis heatmap illustrating per-color localization performance in Gemini 3 Flash versus Gemini 3 Pro (see Appendix B ). Similar to the GPT-5 family, both Gemini models demonstrate a tendency to overpredict blue. Additionally, white is typically the first color to degrade in performance, becoming significantly underpredicted at larger grid sizes. We hypothesize this bias also stems from natural image distributions, where pure white patches are rarely the primary informative target.
Figure 24: Zero-shot spatial error heatmaps for the InternVL3.5 family across varying model and grid sizes. While scaling the LLM parameter count generally improves spatial retention on less dense grids, we observe a distinct spatial degradation when scaling from 14B to the 38B model on the 12×12 grid, which we hypothesize is due to the projection bandwidth bottleneck as discussed in Section 5.1 .
Figure 25: Zero-shot spatial error heatmaps for the Qwen3-VL family. In contrast to InternVL, Qwen exhibits strictly monotonic improvement with end-to-end model scale. Notably, at 32B, the model successfully maintains spatial fidelity across almost the entire left half of the 12×12 grid.
Figure 26: VE spatial probing heatmaps for the InternVL3.5 family across varying grid sizes. The 8B model uses the InternViT-300M encoder, while the 38B model upgrades to the massive InternViT-6B. As detailed in Section C , scaling the VE significantly enhances the spatial fidelity captured in the raw visual representations.
Figure 27: VE spatial probing heatmaps for the Qwen3-VL family. The 4B model uses the SigLIP2-Large-300M encoder, whereas the 8B model uses the SigLIP2-SO-400M. Contrary to conventional scaling expectations, the larger VE actually exhibits a slight degradation in spatial fidelity compared to its smaller counterpart (discussed further in Section C ).
Figure 28: VE spatial probing heatmaps for the Qwen3-VL family, comparing the isolated pre-trained vision encoder against the aligned vision encoder extracted from the full VLM. The structural blind spots remain remarkably consistent across both settings, suggesting that multimodal alignment might not fully correct these inherent spatial vulnerabilities (discussed further in Section 5.2 ).
Figure 29: VE spatial probing heatmaps for the InternVL3.5 family, comparing the pre-trained vision encoder with its aligned counterpart. In contrast to Qwen3-VL, the spatial error landscape shifts noticeably not only across different model scales but also before and after multimodal alignment, suggesting that its spatial mechanics might be more sensitive to the alignment process (discussed further in Section 5.2 ).
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored. A natural extension of ``How many r are there in Strawberry?'' asks: how small a visual pattern can a VLM reliably perceive? As such, we introduce FineSightBench, a new benchmark that systematically probes this limit by separating perception tasks (pixel-level recognition of letters, shapes, objects) from reasoning tasks (spatial reasoning, counting, ordering over small targets) across controlled scales of 4--48px. Through comprehensive experiments and detailed failure mode analysis on state-of-the-art models, we reveal a sharp dissociation: perception saturates around 12px, while reasoning remains limited even at larger scales, with persistent numeracy and sequence errors. These findings expose fundamental deficiencies in VLMs' fine-scale visual reasoning that demand more rigorous evaluation.
Lujun Li, Lama Sleem, Niccolo Gentile +4
University of Luxembourg · Foyer S.A. · Université Paris-Saclay
We present a simple experiment that exposes a fundamental limitation in vision-language models (VLMs): the inability to accurately localize filled cells in binary grids when those cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled cells) and render each as two image types -- text symbols (. and #) and filled squares without gridlines -- then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder -- the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition -- systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini) -- but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University