Vision-Language Models (VLMs) can generate procedures that look convincing on paper, but do they work in practice? In this paper, we propose crochet as an ideal testbed for physically grounded procedural reasoning in VLMs. Crochet cannot be reliably simulated or automated: the only way to know what a pattern produces is to make it by hand, yet verification is tangible and cheap, requiring nothing but a hook and a ball of yarn. We introduce FoundationChain, a deceptively simple test of procedural reasoning: given an image of a basic crochet object, can VLMs infer how it was made and generate a pattern that produces the same result? We crochet 40 patterns generated by four VLMs by hand. Even at this scale, physical execution not only reveals failures that automated proxies cannot detect, but also enables a more precise diagnosis of their causes and consequences: while models write fluent, convincing patterns, only four are free of errors, and our analysis exposes deficiencies in visual, geometric and procedural reasoning.
Figures & tables
Figure 2: A–D: making a ‘double crochet’ stitch. Yarn is wrapped over the hook (A), the hook is inserted into the target stitch and yarn is drawn back through (B), then through two of the loops on the hook (C), and finally through the last two loops (D), completing the stitch and returning to the state in A. E: making a ‘foundation chain’, which acts as the base for the first row of stitches in a pattern. F: working ‘in the round’ to make a circle.
Figure 3: A: Our own crocheted Hexagon B: A render of a hand-written CrochetPARADE DSL translation of the same pattern. Their cosine similarity is just 0.184.
Figure 4: Results. Heatmaps showing error categories across each model’s output patterns. Filled cells mark whether an error of a given type is present in a generated pattern (G geometry, C construction, X stitches, S size, N count, T state tracking, H hallucination, A ambiguity). The final column, 1st , denotes the round in which the first error occurs, with D and P representing the description and planning stages. Dindicates an error-free pattern.
Figure 5: Stitch counts per round for the Circle target. Coloured lines show each model’s pattern, and the black line shows the target. Kimi K3 is doubly wrong: by skipping alternating stitches, its second round creates half as many stitches (solid line) as it claims (dashed line), though neither method would give a flat circle.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
n
Shared n -gram
Pairs (%)
1
in
99.9
1
of
97.0
1
ch
95.5
1
2
90.9
2
in next
79.8
2
in each
77.3
Appendix
Table 1: Word n -grams shared between a pattern and an unrelated pattern (one random pairing of 6,084 patterns; tokenised on whitespace and case-sensitive, as in CrochetBench’s BLEU). Rows are selected from the ten most shared n -grams of each length (four for n≤2 , six for n≥3 ); a strict top- k is dominated by overlapping fragments of the same few phrases (e.g. in 2nd ch , 2nd ch from , sc in 2nd ch ), so we show a range of phrases instead. “Pairs” is the percentage of pairs in which the n -gram occurs in both patterns.
CB
Enhanced
+ post-proc.
Model
prompt
prompt
+ 1 retry
Gemini 3.7 Flash
7
28
64
GPT-5.6 Terra
0
21
40
Kimi K3
1
20
34
Inkling
3
22
29
Claude Sonnet 5
1
12
20
Appendix
Table 2: Valid Pattern Rate (VPR) (%) on 100 CrochetBench Task D-proj test patterns. The ’enhanced’ prompt adds basic syntax guidance. The final column adds post-processing and one round of compiler feedback.
DINO
DINOv2
Renders / photos
134 / 56
Matched pairs
0.188
0.106
Mismatched pairs
0.176
0.095
d
0.11
0.12
Perm. p
0.007
0.001
Rank-1 (chance 1.8%)
1.5%
2.2%
Appendix
Table 3: DINO and DINOv2 similarity (mean cosine similarity unless noted), renders vs. product photos. Ranks are out of 56 photos.
Figure 6: The FoundationChain targets.
Thinking off
Thinking low
Thinking medium
Model
T=0
0.7
1.0
All
T=0
0.7
1.0
All
T=0
0.7
1.0
All
Gemini 3.7 Flash †
–
–
–
–
0.66
0.71
0.64
0.67
–
–
–
–
GPT-5.6 Terra ‡
1.74
1.86
1.86
1.84
–
–
2.03
2.03
–
–
1.95
1.95
Kimi K3
1.38
1.45
1.82
1.60
1.88
1.93
2.18
2.03
2.00
1.88
1.95
1.93
Inkling
1.84
1.77
1.90
1.83
2.06
2.22
1.99
2.10
2.02
2.03
1.97
2.00
Appendix
Table 4: Mean absolute error of row/round counts when models are asked only to count the rows or rounds in each target photo. The targets have median 3.5, mean 5.4 and maximum 17 rows/rounds. All pools the three temperatures. † Gemini 3.7 Flash cannot turn thinking off; low is its lowest setting. ‡ GPT-5.6 Terra accepts only T=1.0 with thinking enabled.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
While multimodal large language models can describe visual content, their ability to generate executable procedures remains underexplored. CrochetBench presented in this paper evaluates this shift from describing to doing through fine-grained procedural reasoning in crochet: models must recognize stitches, select structurally appropriate instructions, and generate compilable procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply decreases as the evaluation shifts from surface-level similarity to executable correctness, revealing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. Our proposed CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.
Vision-language models (VLMs) generate fluent causal explanations, but current evaluations cannot distinguish linguistic plausibility from faithful causal reasoning. We introduce a dual-probe methodology that isolates these properties. The Text-Only Probe measures linguistic quality. The Chain-Text Probe requires models to first generate explicit causal chains. The Abstraction Gap (AG) metric quantifies the normalized performance difference. Evaluating eight VLMs on CAGE (Causal Abstraction Gap Evaluation), a benchmark of 49,500 questions across 5,500 images spanning Pearl's causal hierarchy, we find seven models exhibit AG exceeding 0.50 with text scores of 6--8 but chain scores below 2.5. Fine-tuning on 45,000 chain-annotated examples fails to close the gap. However, one model achieves near-zero AG. The capability exists within current VLM architectures and depends on pretraining and architectural choices. CAGE provides a diagnostic tool for assessing faithful causal reasoning in VLMs.
Chinh Hoang, Mohammad Rashedul Hasan
Department of Electrical and Computer Engineering, University of Nebraska–Lincoln, Lincoln, Nebraska, USA.
Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.