Evaluating Physically Grounded Procedural Reasoning in VLMs through Real-World Execution: Crochet as a Testbed
Organizations: University of York / York, UK
Abstract
Vision-Language Models (VLMs) can generate procedures that look convincing on paper, but do they work in practice? In this paper, we propose crochet as an ideal testbed for physically grounded procedural reasoning in VLMs. Crochet cannot be reliably simulated or automated: the only way to know what a pattern produces is to make it by hand, yet verification is tangible and cheap, requiring nothing but a hook and a ball of yarn. We introduce FoundationChain, a deceptively simple test of procedural reasoning: given an image of a basic crochet object, can VLMs infer how it was made and generate a pattern that produces the same result? We crochet 40 patterns generated by four VLMs by hand. Even at this scale, physical execution not only reveals failures that automated proxies cannot detect, but also enables a more precise diagnosis of their causes and consequences: while models write fluent, convincing patterns, only four are free of errors, and our analysis exposes deficiencies in visual, geometric and procedural reasoning.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Shared -gram | Pairs (%) | |
|---|---|---|
| 1 | in | 99.9 |
| 1 | of | 97.0 |
| 1 | ch | 95.5 |
| 1 | 2 | 90.9 |
| 2 | in next | 79.8 |
| 2 | in each | 77.3 |
| CB | Enhanced | + post-proc. | |
|---|---|---|---|
| Model | prompt | prompt | + 1 retry |
| Gemini 3.7 Flash | 7 | 28 | 64 |
| GPT-5.6 Terra | 0 | 21 | 40 |
| Kimi K3 | 1 | 20 | 34 |
| Inkling | 3 | 22 | 29 |
| Claude Sonnet 5 | 1 | 12 | 20 |
| DINO | DINOv2 | |
|---|---|---|
| Renders / photos | 134 / 56 | |
| Matched pairs | 0.188 | 0.106 |
| Mismatched pairs | 0.176 | 0.095 |
| 0.11 | 0.12 | |
| Perm. | 0.007 | 0.001 |
| Rank-1 (chance 1.8%) | 1.5% | 2.2% |
| Thinking off | Thinking low | Thinking medium | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | All | All | All | |||||||||
| Gemini 3.7 Flash † | – | – | – | – | 0.66 | 0.71 | 0.64 | 0.67 | – | – | – | – |
| GPT-5.6 Terra ‡ | 1.74 | 1.86 | 1.86 | 1.84 | – | – | 2.03 | 2.03 | – | – | 1.95 | 1.95 |
| Kimi K3 | 1.38 | 1.45 | 1.82 | 1.60 | 1.88 | 1.93 | 2.18 | 2.03 | 2.00 | 1.88 | 1.95 | 1.93 |
| Inkling | 1.84 | 1.77 | 1.90 | 1.83 | 2.06 | 2.22 | 1.99 | 2.10 | 2.02 | 2.03 | 1.97 | 2.00 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,058 papers