Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models

Yunkai Zhang; Linda Li; Yingxin Cui; Xiyuan Ruan; Zeyu Zheng; Kezhen Chen; Yi Zhang; Diji Yang

cs.CVApr 6, 2026

Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models

Authors: Yunkai Zhang, Linda Li, Yingxin Cui, Xiyuan Ruan, Zeyu Zheng, Kezhen Chen, Yi Zhang, Diji Yang

Organizations: BAIR, UC Berkeley · UC Santa Cruz · Analogy AI

Abstract

Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap Digital Agnosia. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.

Figures & tables

Panel of Figure 1 — Figure 1: Samples from G2M. Difficulty ranges from (a) simple tests to (b) dense settings that exceed standard patch resolution.

Grid	GPT-5-mini	Gemini-3-Flash
$3\times 3$	100 / 100	100 / 100
$6\times 6$	97.3 / 99.9	99.0 / 100.0
$9\times 9$	0.0 / 65.6	97.3 / 100.0
$12\times 12$	0.0 / 41.5	69.7 / 99.2
$20\times 20$	0.0 / 33.7	0.0 / 87.8
$32\times 32$	0.0 / 0.0	0.0 / 38.5

Table 1: Zero-shot Exact / Cell accuracies (%) of proprietary models on 3-color grids.

Figure 2: Cell accuracy of open-weight models under zero-shot generation and VE probing. While zero-shot performance collapses rapidly as grid size increases, substantially higher accuracy remains recoverable from frozen VE features under supervised probing.

Figure 3: Diagnostic spatial heatmaps for proprietary models for grid sizes from $9\times 9$ to $32\times 32$ . Red indicates high cell accuracy and blue indicates lower accuracy.

Figure 4: Diagnostic spatial heatmaps for open-weight models for grid sizes from $4\times 4$ to $12\times 12$ . Red indicates high cell accuracy and blue indicates lower accuracy.

Figure 5: Diagnostic spatial heatmaps for open-weight model VEs (via Spatial Probing) for grid sizes from $20\times 20$ to $64\times 64$ . Red indicates high cell accuracy and blue indicates lower accuracy.

Figure 6: Grid-patch alignment across varying grid sizes. Performance generally peaks at patch-aligned resolutions (e.g.,

32\times 32

48\times 48

), but this trend breaks at

64\times 64

, which underperforms its immediate misaligned neighbors.

Figure 7: Zero-shot scaling on open-weight models for

9\times 9

and

12\times 12

grids.

Model Family	VLM Size	Vision Encoder Backbone	Spatial Probe Accuracy
Model Family	VLM Size	Vision Encoder Backbone	$48\times 48$	$64\times 64$
Qwen3-VL	2B / 4B	SigLIP2-Large (300M)	81.59%	57.47%
Qwen3-VL	8B / 32B	SigLIP2-SO (400M)	72.31%	54.81%
InternVL3.5	1B–14B	InternViT (300M)	98.75%	69.27%
InternVL3.5	38B	InternViT (6B)	100.00%	73.34%

Table 2: Spatial probe Cell Accuracy on dense grids. We only show the results for

48\times 48

and

64\times 64

here, since accuracies are saturated for smaller grids.

Figure 8: Error heatmaps from spatial probes on open-weight models at $48\times 48$ and $64\times 64$ .

Figure 9: Error heatmaps comparing vanilla and extracted vision encoders at $64\times 64$ . Alignment improves overall accuracy, but many of the base encoder’s blind spots remain visible.

Model Family	Vision Encoder	Spatial Probe Accuracy
Model Family	Vision Encoder	$20\times 20$	$32\times 32$	$48\times 48$	$64\times 64$
Qwen3-VL-8B	SigLIP2-SO (400M)	95.49%	75.39%	57.94%	47.23%
Qwen3-VL-8B	Extracted	99.93%	100.00%	72.31%	54.81%
InternVL3.5-8B	InternViT (300M)	100.00%	100.00%	70.89%	56.20%
InternVL3.5-8B	Extracted	100.00%	100.00%	98.75%	69.27%
InternVL3.5-38B	InternViT (6B)	100.00%	100.00%	99.97%	67.99%

Table 3: Spatial probe Cell Accuracy on dense grids. Comparing vanilla and extracted vision encoders shows that multimodal alignment substantially improves spatial recoverability under probing.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Error heatmaps for $47\times 47$ , $48\times 48$ , $64\times 64$ , and $65\times 65$ . Yellow lines show the patch boundaries. For better visibility, we show only the center $8\times 8$ patches. See Figures 14 and 15 for the full heatmaps.

Figure 11: Cell accuracy across six boundary-interaction types near the

64\times 64

grid size. Because a

64\times 64

grid yields exactly

2\times 2

cells in each patch, 100% of its cells fall into the Edg-Edg configuration. Nearby misaligned grids distribute their cells across all categories, revealing a strict degradation in accuracy as cells cross more patch boundaries.

Grid Size	InternVL3.5-8B (Probe VE)						Qwen3-VL-8B-Instruct (Probe VE)
Grid Size	3C	4C	5C	6C	8C	10C	3C	4C	5C	6C	8C	10C
20 $\times$ 20	100.00	100.00	100.00	100.00	100.00	100.00	99.93	99.91	99.84	99.76	99.55	99.49
32 $\times$ 32	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.99	99.99	99.96	99.97
48 $\times$ 48	98.75	97.60	96.68	95.80	92.88	90.73	72.31	67.80	64.12	61.70	56.36	54.40
64 $\times$ 64	69.27	62.23	56.63	52.46	46.45	42.55	54.81	47.41	41.97	38.08	32.31	28.59

Appendix

Table 4: Spatial Probing Cell Accuracy (%) of the frozen VE across varying color diversities. The notation 3C to 10C denotes the number of distinct colors present in the grid.

Grid Size	InternVL3.5-8B						Qwen3-VL-8B-Instruct
Grid Size	3C	4C	5C	6C	8C	10C	3C	4C	5C	6C	8C	10C
3 $\times$ 3	100.00	100.00	100.00	100.00	100.00	99.33	99.78	99.78	99.78	100.00	100.00	100.00
4 $\times$ 4	100.00	99.84	100.00	99.99	100.00	99.99	95.86	98.30	98.88	99.35	99.69	99.78
6 $\times$ 6	97.94	97.90	98.02	98.59	99.06	99.10	69.59	74.69	78.22	80.89	81.76	82.22
9 $\times$ 9	65.07	62.07	63.14	64.47	65.36	63.14	40.00	38.59	37.34	36.98	35.20	34.03
12 $\times$ 12	35.74	28.43	24.55	22.13	19.94	17.39	34.58	26.53	22.62	19.39	16.42	14.67

Appendix

Table 5: Zero-Shot Cell Accuracy (%) of Open-Weight VLMs across varying color diversities. The notation 3C to 10C denotes the number of distinct colors present in the grid.

Figure 12: Color analysis heatmap illustrating per-color localization performance in InternVL3.5-8B.

Grid	GPT-5-mini	GPT-5.2	Gemini-3-Flash	Gemini-3-Pro
$6\times 6$	97.3 / 99.9	100.0 / 100.0	99.0 / 100.0	100.0 / 100.0
$9\times 9$	0.0 / 65.6	97.3 / 100.0	97.3 / 100.0	100.0 / 100.0
$12\times 12$	0.0 / 41.5	94.3 / 100.0	69.7 / 99.2	100.0 / 100.0
$20\times 20$	0.0 / 33.7	0.0 / 64.4	0.0 / 87.8	47.3 / 84.3
$32\times 32$	0.0 / 0.0	0.0 / 34.1	0.0 / 38.5	0.0 / 35.0

Appendix

Table 6: Zero-shot Exact / Cell accuracies (%) of proprietary VLMs on G2M for grid densities

6\times 6

32\times 32

Figure 13: Scaling on proprietary models on $12\times 12$ and $20\times 20$ grids.

Grid	InternVL3.5-8B	Qwen3-VL-8B-Instruct
$2\times 2$	100.0 / 100.0	100.0 / 100.0
$3\times 3$	100.0 / 100.0	66.0 / 99.8
$4\times 4$	100.0 / 100.0	0.0 / 96.6
$6\times 6$	62.0 / 98.1	0.0 / 71.2
$9\times 9$	0.0 / 64.8	0.0 / 40.0
$12\times 12$	0.0 / 35.6	0.0 / 35.0

Appendix

Table 7: Zero-shot Exact / Cell accuracies (%) of open-weight models on 3-color grids.

Grid	InternVL3.5-8B	Qwen3-VL-8B-Instruct
$20\times 20$	100.0 / 100.0	75.1 / 99.9
$32\times 32$	100.0 / 100.0	96.2 / 100.0
$48\times 48$	0.0 / 98.8	0.0 / 72.3
$64\times 64$	0.0 / 69.3	0.0 / 54.8

Appendix

Table 8: Spatial Probe Exact / Cell accuracies (%) on high-density 3-color grids.

Figure 14: Grid-patch interaction patterns in InternVL3.5-8B. We present error heatmaps for grid sizes around $32\times 32$ , $48\times 48$ , and $64\times 64$ . Below each heatmap, yellow lines illustrate how the vision encoder’s patch boundaries overlay the original image. Because the fixed patch size does not evenly divide the grid cells, the boundaries progressively misalign across the image. As detailed in Section 4.2 , the cell that occupies the majority of a patch dictates the feature representation and achieves higher accuracy. This gradual shift in patch-to-cell overlap produces the distinct macroscopic interference patterns visible in the error heatmaps.

Figure 15: Grid-patch interaction patterns in Qwen3-VL-8B-Instruct. We present error heatmaps for grid sizes around $32\times 32$ , $48\times 48$ , and $64\times 64$ . Below each heatmap, yellow lines illustrate how the vision encoder’s patch boundaries overlay the original image. Because the fixed patch size does not evenly divide the grid cells, the boundaries progressively misalign across the image. As detailed in Section 4.2 , the cell that occupies the majority of a patch dictates the feature representation and achieves higher accuracy. This gradual shift in patch-to-cell overlap produces the distinct macroscopic interference patterns visible in the error heatmaps. The impact of grid-patch interactions appears to be more pronounced compared to InternVL3.5-8B, which is especially visible in the first row (around $32\times 32$ ).

Figure 16: Zero-shot spatial error heatmaps for all proprietary models, a full version of Figure 7 from Section 5.1 .

Figure 17: Zero-shot spatial error heatmaps for Qwen3-VL-8B-Instruct across varying grid sizes and color complexities. The autoregressive degradation pattern remains as the color vocabulary increases (see Appendix B ).

Figure 18: Zero-shot spatial error heatmaps for InternVL3.5-8B. Similar to Qwen3-VL, the autoregressive degradation pattern remains as the color vocabulary increases (see Appendix B ).

Figure 19: VE spatial probing heatmaps for Qwen3-VL-8B-Instruct. While the blind spots persist across different color counts (see Appendix B ), the overarching performance patterns fluctuate across grid sizes due to the cells being divided by patch boundaries and area dominance effects analyzed in Appendix 4.2 .

Figure 20: VE spatial probing heatmaps for InternVL3.5-8B. While the blind spots persist across different color counts (see Appendix B ), the overarching performance patterns fluctuate across grid sizes due to the cells being divided by patch boundaries and area dominance effects analyzed in Appendix 4.2 .

Figure 21: Color analysis heatmap illustrating per-color localization performance in Qwen3-VL-8B-Instruct. We observe a similar pattern as for InternVL3.5-8B in Figure 12 from Appendix B .

Figure 22: Color analysis heatmap illustrating per-color localization performance in GPT-5-mini versus GPT-5.2 (see Appendix B ). We observe that both models exhibit a bias toward overpredicting blue relative to other colors. This is not immediately visible in this plot since we processed the raw accuracies with IoU. Furthermore, white represents the earliest failure mode as grid complexity increases, suffering from severe underprediction. This phenomenon is likely tied to the models’ pre-training distributions, where white often serves as negative space or lacks semantic salience in natural images.

Figure 23: Color analysis heatmap illustrating per-color localization performance in Gemini 3 Flash versus Gemini 3 Pro (see Appendix B ). Similar to the GPT-5 family, both Gemini models demonstrate a tendency to overpredict blue. Additionally, white is typically the first color to degrade in performance, becoming significantly underpredicted at larger grid sizes. We hypothesize this bias also stems from natural image distributions, where pure white patches are rarely the primary informative target.

Figure 24: Zero-shot spatial error heatmaps for the InternVL3.5 family across varying model and grid sizes. While scaling the LLM parameter count generally improves spatial retention on less dense grids, we observe a distinct spatial degradation when scaling from 14B to the 38B model on the $12\times 12$ grid, which we hypothesize is due to the projection bandwidth bottleneck as discussed in Section 5.1 .

Figure 25: Zero-shot spatial error heatmaps for the Qwen3-VL family. In contrast to InternVL, Qwen exhibits strictly monotonic improvement with end-to-end model scale. Notably, at 32B, the model successfully maintains spatial fidelity across almost the entire left half of the $12\times 12$ grid.

Figure 26: VE spatial probing heatmaps for the InternVL3.5 family across varying grid sizes. The 8B model uses the InternViT-300M encoder, while the 38B model upgrades to the massive InternViT-6B. As detailed in Section C , scaling the VE significantly enhances the spatial fidelity captured in the raw visual representations.

Figure 27: VE spatial probing heatmaps for the Qwen3-VL family. The 4B model uses the SigLIP2-Large-300M encoder, whereas the 8B model uses the SigLIP2-SO-400M. Contrary to conventional scaling expectations, the larger VE actually exhibits a slight degradation in spatial fidelity compared to its smaller counterpart (discussed further in Section C ).

Figure 28: VE spatial probing heatmaps for the Qwen3-VL family, comparing the isolated pre-trained vision encoder against the aligned vision encoder extracted from the full VLM. The structural blind spots remain remarkably consistent across both settings, suggesting that multimodal alignment might not fully correct these inherent spatial vulnerabilities (discussed further in Section 5.2 ).

Figure 29: VE spatial probing heatmaps for the InternVL3.5 family, comparing the pre-trained vision encoder with its aligned counterpart. In contrast to Qwen3-VL, the spatial error landscape shifts noticeably not only across different model scales but also before and after multimodal alignment, suggesting that its spatial mechanics might be more sensitive to the alignment process (discussed further in Section 5.2 ).

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
Jun 5, 2026Lujun Li, Lama Sleem, Niccolo Gentile +4VLM Evaluation Large Vision-Language Models
University of Luxembourg · Foyer S.A. · Université Paris-Saclay
Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
Feb 17, 2026Yuval LeventalVision-Language Models VLM Reasoning
Rochester Institute of Technology
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

Jun 5, 2026J/K move · Enter open · S save

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

Lujun Li, Lama Sleem, Niccolo Gentile, Yangjie Xu+3

University of Luxembourg · Foyer S.A. · Université Paris-Saclay

VLM Evaluation Large Vision-Language Models VLM Reasoning Fine-Grained Visual Perception

Open paper AskPDF arXiv

Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored. A natural extension of ``How many r are there in Strawberry?'' asks: how small a visual pattern can a VLM reliably perceive? As such, we introduce FineSightBench, a new benchmark that systematically probes this limit by separating perception tasks (pixel-level recognition of letters, shapes, objects) from reasoning tasks (spatial reasoning, counting, ordering over small targets) across controlled scales of 4--48px. Through comprehensive experiments and detailed failure mode analysis on state-of-the-art models, we reveal a sharp dissociation: perception saturates around 12px, while reasoning remains limited even at larger scales, with persistent numeracy and sequence errors. These findings expose fundamental deficiencies in VLMs' fine-scale visual reasoning that demand more rigorous evaluation.

Abstract

VLM Evaluation

Explore similar work

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?