Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
Organizations: Rochester Institute of Technology
Abstract
We present a simple experiment that exposes a fundamental limitation in vision-language models (VLMs): the inability to accurately localize filled cells in binary grids when those cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled cells) and render each as two image types -- text symbols (. and #) and filled squares without gridlines -- then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder -- the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition -- systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini) -- but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.
Figures & tables
| Condition | Cell Acc. | Precision | Recall | F1 |
|---|---|---|---|---|
| Text symbols | 90.6% | 83.3% | 83.4% | 83.4% |
| Segmented squares | 64.9% | 33.6% | 31.7% | 32.6% |
| Unsegmented squares | 66.9% | 34.5% | 25.9% | 29.6% |
| Model | Condition | Cell Acc. | Prec. | Recall | F1 |
|---|---|---|---|---|---|
| Claude Opus | Text | 90.6% | 83.3% | 83.4% | 83.4% |
| Unseg. Squares | 66.9% | 34.5% | 25.9% | 29.6% | |
| Gap | 53.8 | ||||
| ChatGPT 5.2 | Text | 91.3% | 85.8% | 82.3% | 84.0% |
| Unseg. Squares | 65.9% | 37.1% | 41.5% | 39.2% | |
| Gap | 44.8 |
| Model | Condition | G1 | G7 | G14 | Agg. |
|---|---|---|---|---|---|
| Claude Opus | Text ( .# ) | 95 | 85 | 75 | 84 |
| Text-in-squares | 100.0 | 65.4 | 82.4 | 79.6 | |
| Unicode squares | 70.8 | 75.0 | 64.3 | 68.8 | |
| Filled squares | 45 | 30 | 20 | 35 | |
| Gemini 3 Think | Text ( .# ) | 100 | 85 | 63 | 63 |
| Text-in-squares | 100.0 | 88.3 | 80.2 | 85.7 |
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 724 papers