cs.CVApr 12, 2026
SaveBareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs
Organizations: (Independent Researcher) · University of Calgary
Abstract
While Vision-Language Models (VLMs) demonstrate remarkable zero-shot recognition capabilities across a diverse spectrum of multimodal tasks, it yet remains an open question whether these architectures genuinely comprehend geometric structure or merely exploit RGB textures and contextual priors as statistical shortcuts. Existing evaluations fail to isolate this mechanism, conflating semantic reasoning with texture mapping and relying on imprecise annotations that inadvertently leak environmental cues. To address this gap, we introduce , a zero-shot benchmark designed to stress-test pure geometric shape comprehension. We curate pixel-level silhouettes of geometrically distinct classes across six datasets: five established segmentation sources (ImageNet-S, DIS5K, ThinObject5K, PASCAL VOC, CUB-200) and our novel flagship collection, WTP-Bench, establishing a noise-free geometric taxonomy. WTP-Bench is an extreme, fine-grained visual puzzle that forces models to identify inter-class geometric concepts from boundary contours alone. Our evaluation of 26 state-of-the-art proprietary and open-weight VLMs (eg. GPT-4.1, Gemini, Claude Sonnet 4.5, LLaVA) reveals a consistent, severe performance collapse under RGB deprivation, a phenomenon we term the . By documenting universal structural blindspots, BareBones establishes a rigorous yardstick for genuine geometric grounding. Project Page: https://eternal-f1ame.github.io/WTP-Bench/
Figures & tables
Figure 2: Evaluation pipeline : HQ images and their segmentation masks are binarized to pure structural silhouettes, eliminating all color, texture, and background cues. Models are queried in two modes (HQ and Silhouette), and exact-match accuracy is recorded.
| Dataset | Targets | Geometric Focus / Visual Challenge | HQ (Texture) | Sil (Shape) | Avg Drop |
| Mean Std (%) | Mean Std (%) | (%) | |||
| ImageNet-S [ 8 ] | 100 | Base geometries with distinct outlines. | 35.3 | ||
| DIS5K [ 25 ] | 125 | Complex boundaries, fine appendages. | 13.5 | ||
| ThinObject5K [ 17 ] | 193 | Continuous, narrow geometries. | 0.9 | ||
| PASCAL VOC [ 7 ] | 20 | Articulated objects, non-canonical views. | 19.3 | ||
| CUB-200 [ 29 ] | 200 | Fine-grained species (anatomy geometry). | 9.8 |
Table 1: Comprehensive Dataset Taxonomy & Texture Bias Performance. Summary of the 6 high-fidelity segmentation datasets utilizing our shape-only canonicalization. We report mean standard deviation of performance for 16 leading open-weight models in High-Quality (HQ) vs. Silhouette (Sil) modes. Consistent degradation confirms universal texture dependence.
| WTP-Bench Accuracy | |||
| Model | Size | HQ (%) | Sil (%) |
| Proprietary Models | |||
| Claude Haiku 4.5 [ 3 ] | – | 27.59 | 27.41 |
| GPT-4o mini [ 22 ] | – | 49.31 | 8.53 |
| Gemini 2.5 Flash [ 11 ] | – | 77.24 | 30.00 |
| Claude Sonnet 4.5 [ 3 ] | – | 65.09 | 53.10 |
Table 2: Evaluated VLMs & The Texture Bias Cliff on WTP-Bench. Master Ball (exact-match) accuracy under HQ (full-color) and Silhouette (shape-only) conditions. Relaxed tiers (Ultra Ball, Great Ball) in Table 7 . Open-weight rows are representative; full results in the Supplementary.
Figure 3: Semantic Collapse: Performance Plummets Without Texture. 16 open-weight models across all 5 datasets. Left: High-quality (texture) performance. Right: Silhouette (shape-only) performance. Sweep of red from left to right maps the Texture Bias Cliff.
Figure 4: Parameter Scaling Fails to Resolve Geometric Blindness. Silhouette-mode accuracy vs. parameter count across all 19 open-weight models (1B–26B). The near-horizontal trend confirms the failure is architectural, not a function of scale.
Figure 5: Widespread Statistical Hallucinations. When stripped of RGB patterns, models disregard vision encoder inputs and default to pretraining priors ( e.g . afghan hound , american crow ) or outright refuse the prompt tens of thousands of times.
Figure 6: Texture Bias Cliff Scatter Plot. True geometric understanding demands high accuracy regardless of RGB texture (ideal models lie along the dashed diagonal). Instead, all evaluated models cluster at the bottom right, confirming systematic over-reliance on texture over global shape. Each point represents one model on one dataset.
Figure 7: Dataset Mosaic. Qualitative zero-shot evaluation samples from the five repurposed segmentation datasets. Each column shows the HQ image (top) and its binarized silhouette (bottom), illustrating the diversity in boundary complexity from fine-grained biological forms (CUB-200) to rigid man-made structures (DIS5K, ThinObject5K).
| Proprietary Models | |||
|---|---|---|---|
| ID | Size | Model | Developer |
| PRM-1 | – | Claude Haiku 4.5 [ 3 ] | Anthropic |
| PRM-2 | – | GPT-4o mini [ 22 ] | OpenAI |
| PRM-3 | – | Gemini 2.5 Flash [ 11 ] | |
| PRM-4 | – | Claude Sonnet 4.5 [ 3 ] | Anthropic |
| PRM-5 | – | Grok 4 Fast [ 30 ] | xAI |
Table 3: Complete Model Registry. All 26 evaluated VLMs with their identifiers, sizes, and developers. PRM identifiers denote proprietary models; OSM identifiers denote open-weight models, ordered by parameter count.
Figure 8: WTP-Bench Qualitative Pairs. High-quality artwork (left) vs. pure structural silhouette (right) for a representative selection of targets spanning all generational tiers, highlighting the full spectrum of geometric difficulty: simple quadrupedal outlines, intricate multi-appendage designs, and amorphous Gigantamax forms.
Figure 9: Macro-Category Vulnerabilities. Average performance across primary visual domains on WTP-Bench. Fine-grained biological targets (Birds, Animals) collapse to near-zero on silhouettes, while coarse object categories (Vehicles, Indoor) degrade more gracefully.
| Class | HQ Acc (%) |
|---|---|
| WTP-Bench | |
| Eternatus (Eternamax) | 54.2 |
| Dugtrio (Alolan) | 62.1 |
| Slowbro (Galarian) | 71.6 |
| ImageNet-S | |
| Green Mamba | 48.3 |
Table 4: Universally Hard Targets. A selection of classes achieving exact-match accuracy across all 26 evaluated models in silhouette format. The Sil column is uniformly ; HQ accuracy confirms these are not ambiguous classes, but genuine geometric blindspots.
| Generation 1 | Generation 8 | |||
| Model | HQ (%) | Sil (%) | HQ (%) | Sil (%) |
| Proprietary Models | ||||
| GPT-4.1 | 85.4 | 58.2 | 71.2 | 38.5 |
| Gemini 2.5 Pro | 81.3 | 29.1 | 73.4 | 15.2 |
| Claude Sonnet 4.5 | 74.2 | 33.6 | 55.1 | 18.4 |
| Open-Weight Models | ||||
Table 5: Pre-training Memorization Bias. Exact-match accuracy on Generation 1 vs. Generation 8 WTP-Bench targets. The consistent regression from Gen 1 (extensive internet presence) to Gen 8 (recent release) confirms models rely on memorized priors rather than geometric reasoning.
| Confused Pair | HQ Acc (%) | Sil Confusions |
| ImageNet-S ( n=100 ) | ||
| Admiral Cabbage Butterfly | 71.0 | 380+ |
| Walker Hound Beagle | 84.2 | 310+ |
| WTP-Bench ( n=1,160 ) | ||
| Plusle Minun | 82.5 | 540+ |
| Nidoran (Male) Nidoran (Female) | 79.8 | 480+ |
Table 6: Top Geometric Confusion Pairs. Target pairs sharing near-identical silhouette geometry that are trivially discriminable under HQ conditions. Models confuse each pair hundreds of times in silhouette mode despite high HQ accuracy, confirming the confusion is driven by geometric similarity rather than semantic ambiguity.
Figure 10: Typological and Morphological Breakdown. Left: Accuracy stratified by Pokémon elemental type. Ghost-, Bug-, and Dragon-type silhouettes (amorphous geometry) are consistently hardest. Right: Accuracy by morphological form. Alternate forms (Mega, Gigantamax) incur the steepest silhouette drops as added geometric complexity compounds the baseline difficulty.
| High-Quality (Texture) | Silhouette (Pure Shape) | |||||
| Model ID | Master | Ultra | Great | Master | Ultra | Great |
| Proprietary Models (PRM) | ||||||
| PRM-1 | 27.59 | 35.78 | 42.07 | 27.41 | 35.86 | 41.90 |
| PRM-2 | 49.31 | 61.98 | 68.71 | 8.53 | 12.50 | 15.52 |
| PRM-3 | 77.24 | 83.02 | 86.21 | 30.00 | 35.95 | 42.07 |
| PRM-4 | 65.09 | 79.22 | 81.90 | 53.10 | 62.84 | 68.45 |
Table 7: WTP-Bench tiered accuracy (%). Performance under three progressively relaxed matching criteria—Master Ball (exact species + form), Ultra Ball (correct species, wrong form), and Great Ball (correct evolutionary lineage)—for both high-quality (HQ) and silhouette conditions. The Texture Bias Cliff persists across all tiers: even at the most lenient criterion, silhouette accuracy lags far behind HQ accuracy for every model.
Figure 11: Pre-training Distribution Bias. Both proprietary and open-weight models achieve their highest exact matches on Generation 1 targets, dropping sharply on modern targets with identical structural complexity.
Figure 12: Generation 1 Over-representation in Failures. When models fail on a silhouette, they hallucinate a Generation 1 target at a rate far exceeding the 19.6% baseline prevalence.
| Open-Weight Silhouette Errors | |||
| Generation | Wrong Guesses | % of Errors | Dataset % |
| 1 | 13,959 | 79.2% | 19.6% |
| 2 | 1,156 | 6.6% | 9.6% |
| 3 | 613 | 3.5% | 14.2% |
| 4 | 868 | 4.9% | 11.2% |
| 5 | 371 | 2.1% | 15.9% |
Table 8: Hallucination Bias Distribution. Frequency of incorrect predictions by the 16 open-weight models mapped to generational taxonomies, revealing a massive, statistically disproportionate bias towards Generation 1 textual priors when visual signals fail.
More in this topic
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 989 papers