Solving topological grid puzzles requires reasoning over global spatial invariants such as connectivity, loop closure, and region symmetry and remains challenging for even the most powerful large language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a benchmark of six puzzle families across three difficulty levels. We evaluate strong reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated reasoning is a benign effect of search. Finally we study mitigation strategies including prompt guidance, cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in reasoning over them. Code and data are available at github.com/mayug/topobench-benchmark.
Figures & tables
Figure 1: The six TopoBench puzzle families organized by the global spatial constraint each targets: path connectivity (Flow Free), network connectivity (Bridges), loop closure (Loopy), region partitioning under rotational symmetry (Galaxies), visibility through reflection (Undead), and contiguity across intersecting axes (Pattern).
Puzzle
Task description
Spatial constraint
Flow Free
Route non-intersecting paths between color-matched endpoints, filling every cell.
Path connectivity
Bridges (Hashiwokakero)
Connect numbered islands with bridges; degree, crossing, and connectivity constraints.
Network connectivity
Galaxies (Tentai Show)
Partition grid into regions each rotationally symmetric around a marked center.
Rotational symmetry
Undead
Place monsters so line-of-sight counts through mirrors match clues.
Reflection & visibility
Pattern (Nonogram)
Fill a binary grid to match row/column run-length clues.
Contiguity across axes
Loopy (Slitherlink)
Draw a single closed loop on grid edges satisfying per-cell edge-count clues.
Loop closure
Table 1: TopoBench puzzle families and the global spatial constraint each targets.
Model
Source
Params (Active)
Arch.
Open-weight
Reasoning
Multimodal
Reasoning models
GPT-5-mini-high
OpenAI
†
†
–
✓
✓
Gemini-3-Flash-Preview
Google
†
†
–
✓
✓
DeepSeek V3.2
DeepSeek
671B (37B)
MoE
✓
✓
–
Qwen3-32B
Alibaba
32B
D
✓
✓
–
OLMo-3.1-32B-Think
AI2
32B
D
✓
✓
–
Table 2: Models evaluated on TopoBench. Active denotes active parameters for MoE models. Architecture: D = dense, MoE = mixture-of-experts, Diff = diffusion, †= Not publicly disclosed.
Model
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
Avg
Easy (n=50)
Gemini 3 Flash
0.72
0.86
0.18
0.16
0.78
0.90
0.60
GPT-5 Mini
0.66
0.90
0.32
0.58
0.88
0.92
0.71
DeepSeek V3.2
0.64
0.94
0.26
0.04
0.76
0.82
0.58
Qwen3-235b
0.48
0.72
0.00
0.00
0.24
0.44
0.31
GLM-4.7 Flash
0.14
0.00
0.04
0.00
0.00
0.02
0.03
Table 3: Accuracy (proportion of solved puzzles) on TopoBench by model and puzzle type across difficulties. Cell colors use a low-to-high heat scale over accuracy values (lighter to darker/greener indicates higher accuracy). Rows with avg accuracy ≤ 0.01 are omitted for readability. Full results in Tables 18 , 19 , 20 (Appendix).
Abbr.
Error type
Definition
RR
Repeated Reasoning
Near-identical move sequences repeated from the same board position without meaningful variation.
STF
State-Tracking Failure
Claimed board state diverges from the cumulative effect of the model’s own stated actions.
CF
Constraint Forgetting
An explicit action that directly violates structural puzzle rules (overcounting, crossing, overwriting fixed cells).
PC
Premature Commitment
Commits to a provably incorrect configuration and persists for 3+ steps.
ES
Explicit Surrender
Explicitly gives up, requests a solver, or states it cannot proceed.
IO
Incomplete Output
Fails to produce a complete, valid final answer.
Table 4: Error taxonomy for chain-of-thought analysis. The first four categories (above the mid-rule) are plausibly causal contributors to failure; the remaining categories are symptomatic of an already-degraded reasoning state. Four additional low-frequency categories (UE, HV, TD, OTH) are reported with the full taxonomy in Appendix D .
Figure 2: Prevalence of the seven main error categories among incorrect traces, pooled across difficulty tiers: all five 2 2 2 Loopy omitted because even frontier models score zero on medium and hard. puzzle types ( n=455 , left), Bridges ( n=45 , center), and Undead ( n=84 , right). ES dominates the aggregate and Undead panels as a downstream symptom of failure. STF is the leading category on Bridges (47%), while PC (37%) and RD (43%) are prominent on Undead. CF is the rarest category across all panels. Four additional low-frequency categories are reported in Appendix D .
Figure 3: Intervention effects on Bridges (circles, N=300 ) and Undead (squares, N=300 ). Points show total accuracy with 95% Wilson CIs; shaded bands mark the baseline. PC and CF produce large, significant drops on both puzzles; STF reaches significance only on Undead; RR variants are indistinguishable from baseline.
Figure 4: Tokenization of a Bridges puzzle with GPT-5-mini’s tokenizer ( OpenAI, 2022 ) . ASCII (left) produces ragged boundaries that straddle grid cells; IntFormat (center) and IntFormat-JSON (right) yield uniform, cell-aligned tokens preserving board structure.
Model / Format
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
DeepSeek V3.2
28.7
74.0
8.7
1.3
44.0
52.7
Δ Int-Format
+5.3
+17.3
+7.3
+28.0
-26.0
-8.7
Δ Int-Format JSON
+3.3
+14.0
+3.3
+23.3
-36.7
-8.0
GPT-5 Mini
30.0
68.7
10.7
19.3
73.3
75.3
Δ With Image
+0.7
-0.7
+2.0
-0.7
-5.3
-18.0
Δ Int-Format
+8.0
+12.0
+12.0
+15.3
-8.7
-22.0
Table 5: Per-model accuracy (%) by input format, average of all difficulties. Each model’s Δ rows show change relative to its own Plain ASCII baseline. ‡ In each section the biggest delta is marked (positive in green and negative in red). Full table in the Appendix in Figure 8 .
Condition
Acc.
Valid
Baseline (no tools)
40%
50%
Structured only
46%
100%
+ render_board
42%
100%
Full structured suite
50%
100%
Table 6: Tool ablation, hard Bridges ( N=50 ). Full table in Appendix I .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
Avg
Easy
Gemini 3 Flash
0.72 ± 0.12
0.86 ± 0.10
0.18 ± 0.11
0.16 ± 0.10
0.78 ± 0.11
0.90 ± 0.09
0.60 ± 0.06
GPT-5 Mini
0.66 ± 0.13
0.90 ± 0.09
0.32 ± 0.13
0.58 ± 0.13
0.88 ± 0.09
0.92 ± 0.08
0.71 ± 0.05
DeepSeek V3.2
0.64 ± 0.13
0.94 ± 0.07
0.26 ± 0.12
0.04 ± 0.06
0.76 ± 0.12
0.82 ± 0.11
0.58 ± 0.06
qwen3-235b-a22b-thinking-2507
0.48 ± 0.13
0.72 ± 0.12
0.00 ± 0.04
0.00 ± 0.04
0.24 ± 0.12
0.44 ± 0.13
0.31 ± 0.05
GLM-4.7 Flash
0.14 ± 0.10
0.00 ± 0.04
0.04 ± 0.06
0.00 ± 0.04
0.00 ± 0.04
0.02 ± 0.05
0.03 ± 0.02
Appendix
Table 7: Accuracy by model and puzzle type across difficulties. Rows with all-zero accuracy are omitted.
Model / Format
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
DeepSeek V3.2
28.7 ± 7.2
74.0 ± 7.0
8.7 ± 4.6
1.3 ± 2.2
44.0 ± 7.8
52.7 ± 7.9
Δ With Image
–
–
–
–
–
–
Δ Int-Format
+5.3 ± 10.4
+17.3 ± 8.4
+7.3 ± 7.5
+28.0 ± 7.6
-26.0 ± 10.0
-8.7 ± 11.1
Δ Int-Format JSON
+3.3 ± 10.3
+14.0 ± 8.8
+3.3 ± 7.1
+23.3 ± 7.3
-36.7 ± 9.0
-8.0 ± 11.1
Gemini 3 Flash
42.7 ± 7.8
58.0 ± 7.8
6.0 ± 3.9
5.3 ± 3.7
29.3 ± 7.2
66.0 ± 7.5
Δ With Image
+4.7 ± 11.1
+7.3 ± 10.9
-3.3 ± 5.1
+0.0 ± 5.5
-4.7 ± 10.0
+6.7 ± 10.3
Appendix
Table 8: Per-model accuracy (%) by input format. Each model’s Δ rows show change relative to its own Plain ASCII baseline.
Puzzle
Easy
Medium
Hard
Bridges
5x5d0
7x7d1
10x10d2
Galaxies
5x5dn
7x7du
10x10du
Loopy
5x5t0de
7x7t0dt
10x10t0dh
Pattern
5x5
7x7
10x10
Undead
4x4de
5x5dn
7x7dt
Flow Free
5 × 5–6 × 6
7 × 7–8 × 8
9 × 9–12 × 12
Appendix
Table 9: Engine parameter strings for each puzzle family and difficulty tier. Notation follows the Simon Tatham generator conventions: dimensions precede the difficulty code. For Loopy, t0 denotes the square grid type. Difficulty codes: Bridges d0 / d1 / d2 = increasing constraint complexity; Galaxies dn / du = normal/unreasonable solver; Loopy de / dt / dh = easy/tricky/hard deduction; Undead de / dn / dt = easy/normal/tricky deduction.
Figure 5: Spearman rank correlations between model performance on TopoBench, existing puzzle benchmarks (KORGym, Enigmata), and general reasoning benchmarks (ARC-AGI-1/2, AIME 2025, AA Intelligence). All puzzle benchmarks correlate with existing benchmarks.
Abbr.
Error type
Definition
RR
Repeated Reasoning
Near-identical move sequences repeated from the same board position without meaningful variation.
STF
State-Tracking Failure
Claimed board state diverges from the cumulative effect of the model’s own stated actions.
CF
Constraint Forgetting
An explicit action that directly violates structural puzzle rules (overcounting, crossing, overwriting fixed cells).
PC
Premature Commitment
Commits to a provably incorrect configuration and persists for 3+ steps.
ES
Explicit Surrender
Explicitly gives up, requests a solver, or states it cannot proceed.
IO
Incomplete Output
Fails to produce a complete, valid final answer.
Appendix
Table 10: Full eleven-category error taxonomy. Categories above the first double rule are plausibly causal ; those below are symptomatic . The last four rows (below the second double rule) are omitted from the main-body figures.
Figure 6: Error category prevalence by puzzle type and difficulty tier (50 traces per cell). Galaxies and FlowFree are dominated by ES even at lower difficulties. FlowFree has the strongest PC scaling. Pattern exhibits elevated RR and extreme RD across all tiers. Bridges has the lowest ES/RR rates but shows concentrated STF and PC at hard.
Group
N
Mean tokens
Median tokens
Std
Correct
308
21,801
18,368
12,174
Other failure
128
34,632
34,924
15,423
ES failure
464
42,425
42,326
8,713
Appendix
Table 11: Output token counts and reasoning trace length by outcome group. ES traces cluster near the token ceiling with notably low variance; no ES trace occurs below 22k tokens.
Figure 7: Difference in error prevalence between incorrect and correct traces (incorrect rate − correct rate), aggregated across difficulty tiers. Positive bars indicate errors more common in failed solves. ES dominates the aggregate because it is a downstream symptom of failure. On Bridges, STF is the strongest discriminator (+44 pp); on Undead, ES (+69 pp) and STF (+27 pp) lead. CF is near zero across all panels despite its large causal effect in interventions (Section 4.2 ), reinforcing that observational association—like raw frequency—is a poor proxy for causal impact.
Condition
Easy
Medium
Hard
Total
Δ
Baseline
91.7
82.0
32.0
66.2 [60.4, 71.5]
–
RR Gold Ctrl
91.7
85.0
32.0
67.3 [61.5, 72.6]
+ 1.1
RR Rand Ctrl
93.5
80.0
30.0
65.7 [59.9, 71.0]
− 0.5
RR Standard
94.7
76.0
24.0
62.2 [56.3, 67.7]
− 4.0
STF k=3
88.9
73.0
19.0
58.4 [52.5, 64.0]
− 7.8
PC Half
93.9
43.0
8.0
45.4 [39.7, 51.2]
− 20.8 ∗
Appendix
Table 12: Intervention results on Bridges (accuracy %, 100 puzzles per difficulty). 95% Wilson CIs shown for the Total column. ∗ Significant at α=0.05 (95% Newcombe CI for Δ excludes zero).
Condition
Easy
Medium
Hard
Total
Δ
Baseline
81.0
44.0
9.0
44.7 [39.1, 50.3]
–
RR Gold Ctrl
82.0
45.0
6.0
44.3 [38.8, 50.0]
− 0.3
RR Rand Ctrl
71.0
33.0
3.0
35.7 [30.5, 41.2]
− 9.0 ∗
RR Standard
70.0
45.0
2.0
39.0 [33.7, 44.6]
− 5.7
STF k=3
63.0
35.0
1.0
33.0 [27.9, 38.5]
− 11.7 ∗
PC Half
74.0
26.0
0.0
33.3 [28.2, 38.8]
− 11.3 ∗
Appendix
Table 13: Intervention results on Undead (accuracy %, 100 puzzles per difficulty). 95% Wilson CIs shown for the Total column. ∗ Significant at α=0.05 (95% Newcombe CI for Δ excludes zero).
Condition
Bridges ( N=300 )
Undead ( N=300 )
Total
Δ
Total
Δ
Baseline
66.2 [60.4, 71.5]
–
44.7 [39.1, 50.3]
–
RR Gold Ctrl
67.3 [61.5, 72.6]
+ 1.1
44.3 [38.8, 50.0]
− 0.3
RR Standard
62.2 [56.3, 67.7]
− 4.0
39.0 [33.7, 44.6]
− 5.7
STF k=3
58.4 [52.5, 64.0]
− 7.8
33.0 [27.9, 38.5]
− 11.7 ∗
PC Half
45.4 [39.7, 51.2]
− 20.8 ∗
33.3 [28.2, 38.8]
− 11.3 ∗
Appendix
Table 14: Cross-puzzle comparison of intervention effects (accuracy %). Both puzzles: 100 puzzles per difficulty ( N=300 ). 95% Wilson CIs for individual proportions; Newcombe hybrid score CIs for Δ . ∗ Significant at α=0.05 (95% CI for Δ excludes zero).
Puzzle
Condition
Easy
Medium
Hard
Total
Bridges
Standard prompt ( N=150 )
94.0
88.0
40.0
74.0
Bridges
Intervention baseline ( N=300 )
91.7
82.0
32.0
66.2
Undead
Standard prompt ( N=150 )
76.0
46.0
10.0
44.0
Undead
Intervention baseline ( N=300 )
81.0
44.0
9.0
44.7
Appendix
Table 15: Comparison of DeepSeek-v3.2 accuracy (%) under the standard benchmark prompt (no prefix) and the intervention baseline (gold prefix at 15% depth, no error injection). Grid sizes are identical across evaluations. Differences are within the range expected from sampling variation and minor prompt format differences.
#
Condition
Medium
Hard
Total
Δ
1
Baseline (rules + minimal I/O)
80%
40%
60%
–
2
Text backtrack instruction
84%
28%
56%
− 4
3
1-shot gold path (no NL reasoning)
76%
42%
59%
− 1
4
1-shot worked example (verbose NL)
86%
28%
57%
− 3
5
2-shot worked example (verbose NL)
76%
30%
53%
− 7
6
Text planning instruction
78%
30%
54%
− 6
Appendix
Table 16: Prompt-based intervention results on Bridges (DeepSeek-v3.2). Each condition is evaluated on 50 medium and 50 hard puzzles ( N=100 ). Δ is change relative to baseline.
Condition
Tools available
Accuracy
Board valid
Avg tokens
Baseline (no tools)
—
40%
50%
25,640
Additive ablation
State mutation only
make_move , render_board
26%
96%
28,308
+ degree bookkeeping
+ state_summary
42%
100%
22,784
+ move enumeration
+ neighbors
44%
100%
19,570
+ connectivity
+ components
48%
98%
18,030
Appendix
Table 17: Full tool ablation on hard Bridges ( N=50 ). Top : additive ablation adding one tool per row. Bottom : grid removal variants testing whether render_board helps when structured tools are available.
Model
Format
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
Avg
Gemini 3 Flash
ascii
0.72
0.86
0.18
0.16
0.78
0.90
0.60
Gemini 3 Flash
intformat
0.70
1.00
0.40
0.98
0.60
0.94
0.77
Gemini 3 Flash
intf. json
0.72
1.00
0.54
0.98
0.12
0.96
0.72
Gemini 3 Flash
ascii + img
0.74
0.92
0.08
0.16
0.60
0.92
0.57
GPT-5 Mini
ascii
0.66
0.90
0.32
0.58
0.88
0.92
0.71
GPT-5 Mini
intformat
0.72
0.98
0.66
0.88
0.98
0.90
0.85
Appendix
Table 18: Easy accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy ≤0.01 .
Model
Format
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
Avg
Gemini 3 Flash
ascii
0.54
0.66
0.00
0.00
0.10
0.78
0.35
Gemini 3 Flash
intformat
0.70
1.00
0.00
0.14
0.10
0.78
0.45
Gemini 3 Flash
intf. json
0.68
1.00
0.00
0.06
0.00
0.86
0.43
Gemini 3 Flash
ascii + img
0.62
0.70
0.00
0.00
0.14
0.82
0.38
GPT-5 Mini
ascii
0.22
0.72
0.00
0.00
0.80
0.90
0.44
GPT-5 Mini
intformat
0.38
0.90
0.02
0.16
0.72
0.70
0.48
Appendix
Table 19: Medium accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy ≤0.01 .
Model
Format
FlowFree
Bridges
Loopy
Galaxies
Undead
Pattern
Avg
Gemini 3 Flash
ascii
0.02
0.22
0.00
0.00
0.00
0.30
0.09
Gemini 3 Flash
intformat
0.12
0.86
0.00
0.00
0.00
0.36
0.22
Gemini 3 Flash
intf. json
0.18
0.90
0.00
0.00
0.00
0.30
0.23
Gemini 3 Flash
ascii + img
0.06
0.34
0.00
0.00
0.00
0.44
0.14
GPT-5 Mini
ascii
0.02
0.44
0.00
0.00
0.52
0.44
0.24
GPT-5 Mini
intformat
0.04
0.54
0.00
0.00
0.24
0.00
0.14
Appendix
Table 20: Hard accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy ≤0.01 .
Model
Tokens
Price per 1M tokens (USD)
Full price (USD)
gpt-5-mini
49,607,682
2.00
99.22
qwen3-235b
36,252,844
2.30
83.38
gemini-3-flash-preview
27,014,575
3.00
81.04
deepseek-v3.2
30,766,175
0.42
12.92
glm-4.7-flash
26,939,558
0.40
10.78
olmo-3.1-32b-think
19,736,186
0.50
9.87
Appendix
Table 21: Cost of running the main benchmark
Model
Tokens
Price per 1M tokens (USD)
Full price (USD)
gpt-5-mini
191,749,525
2.00
383.50
gemini-3-flash-preview
109,343,674
3.00
328.03
qwen3-235b
108,633,225
2.30
249.86
deepseek-v3.2
91,074,375
0.42
38.25
glm-4.7-flash
82,757,329
0.40
33.10
olmo-3.1-32b-think
58,457,735
0.50
29.23
Appendix
Table 22: Cost of running the input format ablations
Figure 8: The same Bridges puzzle represented in three input formats: ASCII, intformat, and intformat_json.
Figure 11: Prompt used for evaluating FlowFree puzzles
Figure 12: Prompt used for evaluating Galaxies puzzles
Figure 13: Prompt used for evaluating Pattern puzzles
Figure 14: Prompt used for evaluating Undead puzzles
Figure 15: Prompt used for evaluating Loopy puzzles
Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots. MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We benchmark 14 MLLMs and study agent configurations augmented with image and video generation, including 3 video generative models in planning settings. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions. Our website is at https://mind-topo.github.io/
Yunfei Ge, Anbang Liu, Qineng Wang +9
Northwestern University · Microsoft Research · Stanford University
We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles, a family of constraint-satisfaction problems closely related to NP-complete problems, with deterministic, step-level verification. From a database of 62,231 puzzles across 94 varieties with verified unique solutions, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers in two modes: direct ask (single-shot) and agentic (multi-turn with iterative verification). A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints, localizing errors to the exact rule violated, providing the infrastructure for dense, per-move reward signals for process supervision and reinforcement learning. Our evaluation reveals two distinct axes of capability: (1) reasoning effort scaling, where GPT-5.2 improves 81x from no reasoning to maximum effort; and (2) agentic iteration, where Claude Opus 4.6 rises from 0.3% to 30.0% through iterative checking, while GPT-5.2@xhigh improves from 20.2% to 56.0%. Agentic attempts span a median of 29 turns over 17 minutes, with the longest exceeding 1,221 turns and 14.3 hours - a demanding test of long-context utilization, not just reasoning.
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.