cs.CVOct 12, 2026
SaveGeometric Parrots: Rule Absence vs. Rule Fragility in Spatial and Physical Reasoning for Vision-Language Models
Organizations: Independent Researcher · Simon Fraser University, Canada · Salim Habib University, Karachi, Pakistan · The University of Texas MD Anderson Cancer Center, Houston, TX, USA ·
Abstract
Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for example, abandoning a correct answer when confronted with a confident but incorrect user assertion. Standard single-turn accuracy cannot distinguish such capitulations from beneficial corrections because it scores each response in isolation. We introduce GRIP, a controlled benchmark with 34 procedurally generated reasoning domains, 100,000 images, and 600,000 questions spanning geometric, spatial, visual, and physical reasoning. Each scene is generated by deterministic code, and every answer is derived directly from the underlying scene state, providing exact ground truth without human annotation. On GRIP, we propose Dual-Loop Evaluation. A closed loop probes each image with a five-step compositional question ladder to localize where reasoning fails. An open loop then re-elicits the model's answer under controlled challenges, including peer-model responses and unsupported user assertions. Because ground truth is known exactly, answer changes can be signed as corrections (wrong-to-right) or capitulations (right-to-wrong), separating two failure modes: rule absence and rule fragility. The dataset is available at https://huggingface.co/datasets/bilaljawaid980/GRIP-Benchmark, with further details at https://bilaljawaid980.github.io/grip-benchmark/.
Figures & tables
Figure 1: (a) 34 procedurally generated domains in nine reasoning families, wedge width proportional to domain count. (b) Test-1 accuracy (%) for all ten models: every one is weakest on topological or optical reasoning.
| Benchmark | Domains | Images | Questions | Format | Derived | Proc. | Levels | Challenge |
|---|---|---|---|---|---|---|---|---|
| GeoQA ( Chen and others, 2021 ) | 3 | – | 4,998 | Short | ✗ | ✗ | ✗ | ✗ |
| MathVista ( Lu and others, 2024 ) | 31 | – | 6,141 | MCQ, Short | ✗ | ✗ | ✗ | ✗ |
| MathVerse ( Zhang and others, 2024 ) | 12 | – | 2,612 | Short, OE | ✗ | ✗ | ✗ | ✗ |
| PhysBench ( Chow and others, 2025 ) | 4 | – | 10,002 | MCQ | ✗ | ✗ | ✗ | ✗ |
| PENDULUM ( Rahman et al., 2025 ) | 6 | 751 | 2,000 | OE | ✗ | ✗ | ✗ | ✓ |
| CLEVRER ( Yi et al., 2020 ) | 4 | – | 305,280 | MCQ, OE | ✓ | ✓ | ✗ | ✗ |
Table 1: Comparison of visual reasoning benchmarks. Derived indicates ground truth computed from the generating program. Proc. indicates procedurally generated data. Levels indicates a shared compositional difficulty ladder. Challenge indicates answer re-elicitation after controlled intervention.
Figure 2: The GRIP evaluation pipeline. Scenes and answers are produced together by a single seeded generator, so no annotation step exists. The harness issues one request per item and stores every response verbatim before any parsing, enabling scoring to be re-run without re-querying any model. Scoring is a three-stage offline pass: a deterministic typed comparator; a semantic validator applied only to items the comparator rejected; and a human audit of the validator’s decisions. Because stage 2 only processes rejected items, it can promote an item from incorrect to correct but never demote it.
| Model | Organisation |
|---|---|
| Gemini 3.8 Flash | |
| Claude Opus 5 | Anthropic |
| Grok 4.6 | xAI |
| Claude Sonnet 5 | Anthropic |
| GPT-5.6 Sol | OpenAI |
| Muse Glimmer 30B 30B | Meta |
Table 2: Evaluated models. Blue annotations denote published parameter counts; act. denotes activated parameters in mixture-of-experts models. Thinking releases are marked † .
| Model | Q1 | Q2 | Q3 | Q4 | Q5 | Mean |
|---|---|---|---|---|---|---|
| Hosted | ||||||
| Gemini 3.8 Flash | 82.8 | 76.6 | 65.3 | 52.4 | 67.3 | 68.9 |
| Claude Opus 5 | 80.2 | 76.0 | 60.9 | 47.3 | 65.4 | 66.0 |
| Grok 4.6 | 67.4 | 69.3 | 52.2 | 38.8 | 57.6 | 57.1 |
| GPT-5.6 Sol | 71.4 | 65.5 | 50.9 | 35.9 | 48.3 | 54.4 |
| Claude Sonnet 5 | 67.1 | 68.8 | 50.2 | 37.7 | 47.4 | 54.2 |
Table 3: Closed-loop accuracy by question position (E1). Accuracy across the five reasoning levels; means match overall model accuracy. (Table 9 ).
Figure 3: Mean closed-loop accuracy across the five reasoning levels. Bars show raw accuracy before baseline adjustment; the dashed line indicates the constant-answer baseline. Several models approach the baseline, motivating the baseline-adjusted analysis in Table 3.
Figure 4: Grain robustness (E2). Accuracy under increasing visual noise; all comparisons are within-image, with showing the change from clean to highest noise level.
Figure 5: Answer sycophancy (E3). Raw and control-adjusted transition rates under challenge; capitulation and correction are reported separately because they use different denominators.
Figure 6: Open-ended cross-examination (E4). Peer effects under natural and constructed peer conditions; capitulation and correction are reported separately due to different denominators.
Figure 7: One assertion, two outcomes. Unsupported challenges can preserve or reverse a correct answer; GRIP identifies the latter as capitulation using closed-form ground truth.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | # | Competence demanded | Domains |
|---|---|---|---|
| Plane | 5 | Relations among 2D figures | Nested Squares, Nested Triangles, Nested Hexagons, Line Intersection, Angle Estimation |
| Solid | 6 | 3D structure from 2D projections | Cube Nets, Cube Structure, 3D Combination, Orthographic, Polyhedra, Depth & Height |
| Transformational | 6 | Rigid motion, reflection, symmetry | Rotation Matching, Symmetry Patterns, Fold & Punch, 2D Combination, Embedded Figures, Overlapping Circles |
| Physical | 7 | Mechanical and physical reasoning | Physical Stability, Gear Trains, Free-Body Diagrams, Projectile Motion, Laser & Mirrors, Clock Reading, Gauge Reading |
| Topological | 3 | Connectivity, topology, constrained paths | Surface Topology, Route Puzzles, Hex Pathfinding |
| Projective | 2 | Reasoning from partial observations | Occluded Patterns, Shadow Inference |
Table 4: Reasoning families and domains in GRIP. Each family groups domains according to the reasoning capability the generated tasks require.
| # | Competence demanded | Example (Physical Stability) |
|---|---|---|
| 1 | Direct observation, recognition or counting | How many blocks are in this stack? |
| 2 | Basic spatial relationship or classification | Is block C’s centre of mass to the left, right, or directly above the centre of block B? |
| 3 | Precise measurement or multi-element inference | Which block has the largest absolute horizontal centre offset from the block directly beneath it? |
| 4 | Multi-step integration or derived comparison | Is this entire stack stable, or will it tip? If it tips, state the lowest joint at which it first becomes unstable. |
| 5 | Counterfactual reasoning after modifying part of the scene | If topmost block D were removed, would the remaining stack be stable or unstable? |
Table 5: The five questions carried by every image. The competence demanded at each position is constant across all 34 domains; the question instantiating it is domain-specific. Examples are drawn from Physical Stability.
Figure 8: Sample items from the GRIP dataset. One generated scene from each of the nine reasoning families, shown with its question and ground truth. Every scene is drawn by deterministic code from a seeded generator, and every answer ( GT ) is computed in closed form from the same coordinates the renderer uses, so ground truth is derived rather than annotated. Ground truth is never shown to the model; answer-format instructions, appended verbatim to every prompt, are omitted for space.
Figure 9: Further sample items on the same nine scenes. The scenes are unchanged; each is paired with a question demanding a spatial relation or a classification rather than a direct reading. Every scene is drawn by deterministic code from a seeded generator, and every answer ( GT ) is computed in closed form from the same coordinates the renderer uses, so ground truth is derived rather than annotated. Ground truth is never shown to the model; answer-format instructions, appended verbatim to every prompt, are omitted for space.
Figure 10: Further sample items on the same nine scenes. Here each scene is paired with a question requiring a measurement or an inference drawn across several elements of the scene. Every scene is drawn by deterministic code from a seeded generator, and every answer ( GT ) is computed in closed form from the same coordinates the renderer uses, so ground truth is derived rather than annotated. Ground truth is never shown to the model; answer-format instructions, appended verbatim to every prompt, are omitted for space.
Figure 11: Further sample items on the same nine scenes. Here each scene is paired with a question that modifies the scene and asks for the consequence, so the answer cannot be read off the image as drawn. Every scene is drawn by deterministic code from a seeded generator, and every answer ( GT ) is computed in closed form from the same coordinates the renderer uses, so ground truth is derived rather than annotated. Ground truth is never shown to the model; answer-format instructions, appended verbatim to every prompt, are omitted for space.
| Experiment | Measures | Design | Queries/model |
|---|---|---|---|
| E1 Closed-loop accuracy | Reasoning across five levels | Fresh single-turn evaluation per item | 8,500 hosted / 252,500 open-weight |
| E2 Grain robustness | Sensitivity to visual degradation | Matched images with | 10,200 |
| E3 Answer sycophancy | Response to conflicting assertions | Second-turn challenge on stored responses | 3,300 |
| E4 Cross-examination | Stability under peer disagreement | Three controlled peer-answer conditions | 4,300 |
Table 6: Experimental configuration.
| Model | Served via | Requested identifier |
| Gemini 3.8 Flash | routed API | google/gemini-3.8-flash |
| Claude Opus 5 | routed API | anthropic/claude-opus-5 |
| Grok 4.6 | routed API | x-ai/grok-4.6 |
| Claude Sonnet 5 | routed API | anthropic/claude-sonnet-5 |
| GPT-5.6 Sol | routed API | openai/gpt-5.6-sol |
| Muse Glimmer 30B | routed API | meta/muse-glimmer-30b:nitro |
Table 7: Model access. Hosted models were reached through a single routing provider, so request formatting, retry policy and timeout were identical across vendors; open-weight models were served from the released checkpoint. The identifier shown is the string sent with each request. The provider’s response payload carries a more specific served revision, but the runners serialised only the normalised response fields, so that returned revision is not retained in our stored results.
| Model | Context | Generation | Batched tok. | Note |
|---|---|---|---|---|
| Qwen3-VL-8B-Instruct | 16,384 | 4,096 | 8,192 | — |
| Qwen3-VL-8B-Thinking | 16,384 | 4,096 | 8,192 | — |
| Molmo2-8B | 16,384 | 4,096 | 16,384 | raised; no system role |
| InternVL3.5-8B | 16,384 | 4,096 | 8,192 | — |
| Kimi-VL-A3B-Thinking | 16,384 | 4,096 | 8,192 | — |
| DeepSeek-VL2-Small | 4,096 | 1,024 | 8,192 | native 4K context |
Table 8: Local serving configuration (open-weight models). All six ran on one GPU under vLLM, with no tensor parallelism. Context is max_model_len and Generation is max_tokens .
Figure 12: Deterministic grain perturbation. The same GRIP scene under increasing Gaussian grain levels, with structure remaining visually identifiable across the evaluated range.
| Question position | Hosted | Open-weight |
|---|---|---|
| Q1 direct observation | 31.6 | 32.7 |
| Q2 spatial relation | 44.8 | 41.9 |
| Q3 measurement | 25.9 | 23.3 |
| Q4 multi-step integration | 27.5 | 23.5 |
| Q5 counterfactual | 41.4 | 41.9 |
| Mean | 34.2 | 32.7 |
Table 9: Constant-answer baselines by question position (%). The accuracy obtainable without reading the image, computed from the answer key. Reported per tier because the two tiers draw different item samples.
Figure 13: Accuracy above baseline at each question position, hosted models. Colour encodes the direction of the Q4–Q5 step. Model identity is carried by the labels, not by colour. The dashed line is the score obtained by constant answering; the corresponding raw baselines are in Table 9 .
| Model | Phys- ical | Induc- tive | Ana- lytic | Trans- form. | Solid | Projec- tive | Plane | Opti- cal | Topo- log. | Mean |
|---|---|---|---|---|---|---|---|---|---|---|
| Hosted | ||||||||||
| Gemini 3.8 Flash | 79.4 | 83.2 | 79.2 | 69.7 | 61.5 | 68.0 | 65.0 | 62.0 | 57.7 | 68.9 |
| Claude Opus 5 | 77.8 | 83.2 | 78.6 | 64.3 | 62.1 | 67.4 | 63.3 | 51.4 | 48.4 | 66.0 |
| Grok 4.6 | 72.8 | 65.2 | 64.4 | 55.2 | 56.9 | 51.4 | 52.1 | 39.4 | 40.8 | 57.1 |
| GPT-5.6 Sol | 63.1 | 63.2 | 59.8 | 53.3 | 56.6 | 47.8 | 51.8 | 39.4 | 44.3 | 54.4 |
| Claude Sonnet 5 | 68.3 | 66.4 | 58.6 | 50.6 | 55.6 | 47.8 | 45.0 | 51.4 | 40.5 | 54.2 |
Table 10: Raw accuracy by reasoning family (E1), all sixteen models (%). Families are ordered by hosted-tier mean, strongest first. In each row the model’s strongest family is bold and its weakest underlined . Mean is the domain-count-weighted mean over the nine families and reproduces each model’s overall accuracy in Table 3 to within . Open-weight rows were scored under the original comparator rather than the typed comparator used for the hosted tier, so they are internally consistent but not comparable across the tier rule; see Appendix B.6 .
Figure 14: Accuracy by reasoning family, hosted models (%). The ten hosted rows of Table 10 drawn as bars; families are ordered by fleet mean, and within each family the models are ordered by overall accuracy, with shade encoding that overall rank. A group whose shades run dark to light without interruption is one in which the family ordering matches the overall ordering. Most do: the per-family rank correlation with the overall ordering averages and never falls below . Optical reasoning is the exception, where Claude Sonnet 5 outscores three models that are ahead of it overall. The six open-weight models are omitted here because they were scored under a different comparator.
Figure 15: Accuracy at each noise level, hosted models (E2). One spoke per model, one ring per noise level, on the raw accuracy scale. Spoke order is by clean accuracy, mirrored down both sides of the circle so that rank 1 is at the top and rank 10 at the bottom; position on a radar carries no meaning, and this ordering only prevents the sorted sequence from winding the polygon to one side. A value is printed only where it clears the neighbouring value on the same spoke, so a spoke showing one number is one on which all four levels agree to within a point. The full grid is in Figure 4 (a).
| Model | Q1 | Q2 | Q3 | Q4 | Q5 |
|---|---|---|---|---|---|
| Grok 4.6 | 2.31 / 29.33 | 13.13 / 36.36 | 14.04 / 37.24 | 13.51 / 34.07 | 26.36 / 21.03 |
| Gemini 3.8 Flash | 4.80 / 33.93 | 8.48 / 35.40 | 7.51 / 49.61 | 12.44 / 46.50 | 10.63 / 22.28 |
| GPT-5.6 Sol | 19.76 / 58.87 | 42.75 / 73.89 | 46.93 / 68.97 | 44.14 / 64.84 | 53.06 / 44.36 |
| Claude Opus 5 | 29.25 / 58.06 | 38.55 / 65.25 | 44.44 / 61.88 | 44.13 / 55.23 | 39.31 / 29.32 |
| Perplexity Sonar Pro | 35.37 / 63.29 | 52.61 / 56.16 | 57.06 / 60.07 | 59.78 / 53.68 | 61.68 / 44.41 |
| Muse Glimmer 30B | 41.82 / 67.37 | 47.54 / 53.85 | 51.19 / 56.02 | 56.41 / 59.29 | 63.35 / 29.46 |
Table 11: Capitulation / correction by question position (E3), %. Each cell gives capitulation before the slash and correction after it. Rates are raw rather than control-adjusted, so they are comparable down a column but not with the adjusted figures of Figure 5 (a). Models are ordered by capitulation at Q1. The two rates have different denominators and are not comparable with each other.
Figure 16: Capitulation and correction across the five question positions (E3). Bars are grouped by model, one bar per question position, with models ordered by capitulation at Q1. Colour encodes question position, which is ordered rather than nominal, so the light-to-dark ramp carries the variable itself. The two rates are kept in separate panels because their denominators differ; the equal vertical scales are for legibility and do not license a comparison between panels. Upper panel: the four models on the right are already near their ceiling at Q1, while the two on the left climb steeply. Lower panel: the Q5 bar is the shortest in every one of the ten groups.
Figure 17: Both directions of a signed transition. Each card shows the exact response template the model was given, then the values it returned in each round; the repeated field names live in the template rather than on every line. In the constructed-peer arm the three peers always contradict the model’s own round-one answer, so the peer block is wrong exactly when the model was right (a) and right exactly when it was wrong (b) . This is why the two transitions are counted separately and over different denominators: capitulation over items answered correctly in round one, correction over those answered incorrectly. Both models within a card received the identical peer block. Reported confidence does not mark the difference — in (a) the model that abandons a correct answer and the model that keeps it both report before and after — and the capitulating model writes a justification for the adopted claim, describing three neighbouring holes that are not in the image.
| Capitulation | Correction | |||
|---|---|---|---|---|
| Model | Raw | Peer effect (95% CI) | Raw | Peer effect (95% CI) |
| Gemini 3.8 Flash | 8.36 | 5.74 (3.51–7.92) | 12.56 | 8.70 (5.50–11.84) |
| GPT-5.6 Sol | 12.59 | 0.20 ∗ ( 4.37–3.82) | 28.12 | 23.48 (20.21–26.72) |
| Claude Opus 5 | 5.98 | 5.98 (4.53–7.56) | 11.19 | 10.86 (8.55–13.29) |
| Grok 4.6 | 2.92 | 2.58 (1.19–3.95) | 12.15 | 10.81 (8.67–13.03) |
| Muse Glimmer 30B | 6.38 | 1.62 ∗ ( 5.65–2.22) | 13.60 | 12.94 (10.52–15.33) |
Table 12: Natural-peer effects (E4, arm 1). Raw rates (%) and the peer effect (pp) — the raw rate minus that model’s own control rate — with bootstrap intervals over image-level resamples. ∗ marks an interval containing zero. Adjusted correction is significantly positive for all ten models; adjusted capitulation is indistinguishable from zero for five. The two rates have different denominators and are not compared with each other.
| Capitulation | Correction | |||
|---|---|---|---|---|
| Model | Raw | Peer effect (95% CI) | Raw | Peer effect (95% CI) |
| Gemini 3.8 Flash | 12.48 | 9.86 (7.40–12.27) | 80.81 | 76.95 (73.32–80.51) |
| GPT-5.6 Sol | 24.77 | 11.98 (7.30–16.52) | 83.18 | 78.54 (75.64–81.28) |
| Claude Opus 5 | 16.16 | 16.16 (14.01–18.37) | 82.51 | 82.18 (79.57–84.66) |
| Grok 4.6 | 16.78 | 16.44 (14.01–18.91) | 78.85 | 77.51 (74.81–80.10) |
| Muse Glimmer 30B | 41.96 | 33.96 (28.94–38.78) | 85.50 | 84.83 (82.34–87.33) |
Table 13: Constructed-consensus effects (E4, arm 2). As Table 12 , for three synthetic peers asserting one answer unanimously. Every interval here excludes zero. Comparing a row with the same row of Table 12 is valid: same items, same stored first responses, same metric and denominator. The intervals are disjoint between the two tables for nine of the ten models; Gemini 3.8 Flash overlaps over – .
| Model | Capitulation (95% CI) | Correction (95% CI) |
|---|---|---|
| GPT-5.6 Sol | 12.79 (8.84–16.94) | 4.65 (2.72–6.85) |
| Perplexity Sonar Pro | 10.00 (5.45–15.13) | 3.69 (2.15–5.40) |
| Muse Glimmer 30B | 8.00 (4.43–12.00) | 0.66 (0.00–1.69) |
| GPT-5.6 Luna | 4.97 (1.89–8.56) | 4.19 (2.54–6.05) |
| DeepSeek V4.1 Flash | 2.70 (0.62–5.59) | 4.67 (2.77–6.70) |
| Gemini 3.8 Flash | 2.62 (1.08–4.36) | 3.86 (1.64–6.37) |
Table 14: Control instability (E4, arm 3). Second-turn behaviour when no peer response is shown and the model is simply asked for its final answer, with bootstrap intervals. This is the quantity subtracted to obtain the peer effect in Tables 12 and 13 , and it is why the adjusted rates rather than the raw ones are the primary comparison: GPT-5.6 Sol abandons a correct answer in of items with nothing shown to it, and Perplexity Sonar Pro in , while Grok 4.6 and Claude Opus 5 are near zero.
| Model | Arm | paired | Before | After | |
|---|---|---|---|---|---|
| Gemini 3.8 Flash ∗ | Natural peers | 918/1658 | 0.982 | 0.978 | 0.003 |
| Constructed | 977/1925 | 0.982 | 0.986 | 0.005 | |
| Control | 361/641 | 0.980 | 0.983 | 0.003 | |
| Claude Opus 5 | Natural peers | 1661/1678 | 0.756 | 0.775 | 0.018 |
| Constructed | 1952/1975 | 0.751 | 0.737 | 0.014 | |
| Control | 656/661 | 0.750 | 0.741 | 0.009 |
Table 15: Confidence movement (E4). Mean reported confidence before and after the second turn, over rows carrying a valid confidence value in both rounds; n is that paired count against the arm’s total. Coverage is above for eight models. It is poor for Gemini 3.8 Flash ( – ) and Muse Glimmer 30B ( – ), whose responses were truncated at their output-token ceiling, so those two rows describe a selected subset and are marked ∗ . Confidence is the model’s own CONFIDENCE field, not a calibrated probability, and does not depend on the answer comparator. Plotted in Figure 18 .
Figure 18: Direction of change and confidence movement by arm (E4). Change in mean reported confidence between the first and second answer, by arm, over rows carrying a confidence value in both rounds. Ordered by the natural-peer value. The vertical scale spans : no model’s mean confidence moves more than in any arm, whatever happens to its answers. ∗ marks the two models whose paired coverage is poor (Table 15 ).
Figure 19: Model verdicts on single items. Five generated scenes, each shown at a different rung of the closed-loop ladder, with the verdict of all ten hosted models on that one question. A tick marks a correct answer and a cross an incorrect one, scored by the same typed comparator used throughout; models are ordered by overall E1 accuracy, strongest first, reading down the left column and then the right. Each panel is a different scene , so the panels illustrate what each level asks rather than tracing a difficulty curve on a fixed stimulus — level and scene vary together and cannot be separated here. Per-position accuracy over the full evaluated subset is in Figure 13 . The remaining four figure scenes were never evaluated and so carry no verdicts.