TopoBench: Benchmarking LLMs on Hard Topological Reasoning

Mayug Maniparambil; Nils Hoehing; Janak Kapuriya; Arjun Karuvally; Ellen Rushe; Anthony Ventresque; Noel O'Connor; Fergal Reid

cs.AIMar 12, 2026

TopoBench: Benchmarking LLMs on Hard Topological Reasoning

Authors: Mayug Maniparambil, Nils Hoehing, Janak Kapuriya, Arjun Karuvally, Ellen Rushe, Anthony Ventresque, Noel O'Connor, Fergal Reid

Organizations: Intercom Research · University College Dublin · University of Galway · Salk Institute ·

Abstract

Solving topological grid puzzles requires reasoning over global spatial invariants such as connectivity, loop closure, and region symmetry and remains challenging for even the most powerful large language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a benchmark of six puzzle families across three difficulty levels. We evaluate strong reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated reasoning is a benign effect of search. Finally we study mitigation strategies including prompt guidance, cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in reasoning over them. Code and data are available at github.com/mayug/topobench-benchmark.

Figures & tables

Figure 1: The six TopoBench puzzle families organized by the global spatial constraint each targets: path connectivity (Flow Free), network connectivity (Bridges), loop closure (Loopy), region partitioning under rotational symmetry (Galaxies), visibility through reflection (Undead), and contiguity across intersecting axes (Pattern).

Puzzle	Task description	Spatial constraint
Flow Free	Route non-intersecting paths between color-matched endpoints, filling every cell.	Path connectivity
Bridges (Hashiwokakero)	Connect numbered islands with bridges; degree, crossing, and connectivity constraints.	Network connectivity
Galaxies (Tentai Show)	Partition grid into regions each rotationally symmetric around a marked center.	Rotational symmetry
Undead	Place monsters so line-of-sight counts through mirrors match clues.	Reflection & visibility
Pattern (Nonogram)	Fill a binary grid to match row/column run-length clues.	Contiguity across axes
Loopy (Slitherlink)	Draw a single closed loop on grid edges satisfying per-cell edge-count clues.	Loop closure

Table 1: TopoBench puzzle families and the global spatial constraint each targets.

Reasoning models
Model	Source	Params (Active)	Arch.	Open-weight	Reasoning	Multimodal
GPT-5-mini-high	OpenAI	$\dagger$	$\dagger$	–	✓	✓
Gemini-3-Flash-Preview	Google	$\dagger$	$\dagger$	–	✓	✓
DeepSeek V3.2	DeepSeek	671B (37B)	MoE	✓	✓	–
Qwen3-32B	Alibaba	32B	D	✓	✓	–
OLMo-3.1-32B-Think	AI2	32B	D	✓	✓	–

Table 2: Models evaluated on TopoBench. Active denotes active parameters for MoE models. Architecture: D = dense, MoE = mixture-of-experts, Diff = diffusion,

\dagger=

Not publicly disclosed.

Easy (n=50)
Model	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern	Avg
Gemini 3 Flash	0.72	0.86	0.18	0.16	0.78	0.90	0.60
GPT-5 Mini	0.66	0.90	0.32	0.58	0.88	0.92	0.71
DeepSeek V3.2	0.64	0.94	0.26	0.04	0.76	0.82	0.58
Qwen3-235b	0.48	0.72	0.00	0.00	0.24	0.44	0.31
GLM-4.7 Flash	0.14	0.00	0.04	0.00	0.00	0.02	0.03

Table 3: Accuracy (proportion of solved puzzles) on TopoBench by model and puzzle type across difficulties. Cell colors use a low-to-high heat scale over accuracy values (lighter to darker/greener indicates higher accuracy). Rows with avg accuracy

\leq

0.01 are omitted for readability. Full results in Tables 18 , 19 , 20 (Appendix).

Abbr.	Error type	Definition
RR	Repeated Reasoning	Near-identical move sequences repeated from the same board position without meaningful variation.
STF	State-Tracking Failure	Claimed board state diverges from the cumulative effect of the model’s own stated actions.
CF	Constraint Forgetting	An explicit action that directly violates structural puzzle rules (overcounting, crossing, overwriting fixed cells).
PC	Premature Commitment	Commits to a provably incorrect configuration and persists for 3+ steps.
ES	Explicit Surrender	Explicitly gives up, requests a solver, or states it cannot proceed.
IO	Incomplete Output	Fails to produce a complete, valid final answer.

Table 4: Error taxonomy for chain-of-thought analysis. The first four categories (above the mid-rule) are plausibly causal contributors to failure; the remaining categories are symptomatic of an already-degraded reasoning state. Four additional low-frequency categories (UE, HV, TD, OTH) are reported with the full taxonomy in Appendix D .

Figure 2: Prevalence of the seven main error categories among incorrect traces, pooled across difficulty tiers: all five 2 2 2 Loopy omitted because even frontier models score zero on medium and hard. puzzle types ( $n{=}455$ , left), Bridges ( $n{=}45$ , center), and Undead ( $n{=}84$ , right). ES dominates the aggregate and Undead panels as a downstream symptom of failure. STF is the leading category on Bridges (47%), while PC (37%) and RD (43%) are prominent on Undead. CF is the rarest category across all panels. Four additional low-frequency categories are reported in Appendix D .

Figure 3: Intervention effects on Bridges (circles, $N{=}300$ ) and Undead (squares, $N{=}300$ ). Points show total accuracy with 95% Wilson CIs; shaded bands mark the baseline. PC and CF produce large, significant drops on both puzzles; STF reaches significance only on Undead; RR variants are indistinguishable from baseline.

Panel of Figure 4 — Figure 4: Tokenization of a Bridges puzzle with GPT-5-mini’s tokenizer ( OpenAI, 2022 ) . ASCII (left) produces ragged boundaries that straddle grid cells; IntFormat (center) and IntFormat-JSON (right) yield uniform, cell-aligned tokens preserving board structure.

Model / Format	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern
DeepSeek V3.2	28.7	74.0	8.7	1.3	44.0	52.7
$\Delta$ Int-Format	+5.3	+17.3	+7.3	+28.0	-26.0	-8.7
$\Delta$ Int-Format JSON	+3.3	+14.0	+3.3	+23.3	-36.7	-8.0
GPT-5 Mini	30.0	68.7	10.7	19.3	73.3	75.3
$\Delta$ With Image	+0.7	-0.7	+2.0	-0.7	-5.3	-18.0
$\Delta$ Int-Format	+8.0	+12.0	+12.0	+15.3	-8.7	-22.0

Table 5: Per-model accuracy (%) by input format, average of all difficulties. Each model’s

\Delta

rows show change relative to its own Plain ASCII baseline. ‡ In each section the biggest delta is marked (positive in green and negative in red). Full table in the Appendix in Figure 8 .

Condition	Acc.	Valid
Baseline (no tools)	40%	50%
Structured only	46%	100%
+ render_board	42%	100%
Full structured suite	50%	100%

Table 6: Tool ablation, hard Bridges (

N{=}50

). Full table in Appendix I .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Easy
Model	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern	Avg
Gemini 3 Flash	0.72 $\pm$ 0.12	0.86 $\pm$ 0.10	0.18 $\pm$ 0.11	0.16 $\pm$ 0.10	0.78 $\pm$ 0.11	0.90 $\pm$ 0.09	0.60 $\pm$ 0.06
GPT-5 Mini	0.66 $\pm$ 0.13	0.90 $\pm$ 0.09	0.32 $\pm$ 0.13	0.58 $\pm$ 0.13	0.88 $\pm$ 0.09	0.92 $\pm$ 0.08	0.71 $\pm$ 0.05
DeepSeek V3.2	0.64 $\pm$ 0.13	0.94 $\pm$ 0.07	0.26 $\pm$ 0.12	0.04 $\pm$ 0.06	0.76 $\pm$ 0.12	0.82 $\pm$ 0.11	0.58 $\pm$ 0.06
qwen3-235b-a22b-thinking-2507	0.48 $\pm$ 0.13	0.72 $\pm$ 0.12	0.00 $\pm$ 0.04	0.00 $\pm$ 0.04	0.24 $\pm$ 0.12	0.44 $\pm$ 0.13	0.31 $\pm$ 0.05
GLM-4.7 Flash	0.14 $\pm$ 0.10	0.00 $\pm$ 0.04	0.04 $\pm$ 0.06	0.00 $\pm$ 0.04	0.00 $\pm$ 0.04	0.02 $\pm$ 0.05	0.03 $\pm$ 0.02

Appendix

Table 7: Accuracy by model and puzzle type across difficulties. Rows with all-zero accuracy are omitted.

Model / Format	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern
DeepSeek V3.2	28.7 $\pm$ 7.2	74.0 $\pm$ 7.0	8.7 $\pm$ 4.6	1.3 $\pm$ 2.2	44.0 $\pm$ 7.8	52.7 $\pm$ 7.9
$\Delta$ With Image	–	–	–	–	–	–
$\Delta$ Int-Format	+5.3 $\pm$ 10.4	+17.3 $\pm$ 8.4	+7.3 $\pm$ 7.5	+28.0 $\pm$ 7.6	-26.0 $\pm$ 10.0	-8.7 $\pm$ 11.1
$\Delta$ Int-Format JSON	+3.3 $\pm$ 10.3	+14.0 $\pm$ 8.8	+3.3 $\pm$ 7.1	+23.3 $\pm$ 7.3	-36.7 $\pm$ 9.0	-8.0 $\pm$ 11.1
Gemini 3 Flash	42.7 $\pm$ 7.8	58.0 $\pm$ 7.8	6.0 $\pm$ 3.9	5.3 $\pm$ 3.7	29.3 $\pm$ 7.2	66.0 $\pm$ 7.5
$\Delta$ With Image	+4.7 $\pm$ 11.1	+7.3 $\pm$ 10.9	-3.3 $\pm$ 5.1	+0.0 $\pm$ 5.5	-4.7 $\pm$ 10.0	+6.7 $\pm$ 10.3

Appendix

Table 8: Per-model accuracy (%) by input format. Each model’s

\Delta

rows show change relative to its own Plain ASCII baseline.

Puzzle	Easy	Medium	Hard
Bridges	5x5d0	7x7d1	10x10d2
Galaxies	5x5dn	7x7du	10x10du
Loopy	5x5t0de	7x7t0dt	10x10t0dh
Pattern	5x5	7x7	10x10
Undead	4x4de	5x5dn	7x7dt
Flow Free	5 $\times$ 5–6 $\times$ 6	7 $\times$ 7–8 $\times$ 8	9 $\times$ 9–12 $\times$ 12

Appendix

Table 9: Engine parameter strings for each puzzle family and difficulty tier. Notation follows the Simon Tatham generator conventions: dimensions precede the difficulty code. For Loopy, t0 denotes the square grid type. Difficulty codes: Bridges d0 / d1 / d2 = increasing constraint complexity; Galaxies dn / du = normal/unreasonable solver; Loopy de / dt / dh = easy/tricky/hard deduction; Undead de / dn / dt = easy/normal/tricky deduction.

Figure 5: Spearman rank correlations between model performance on TopoBench, existing puzzle benchmarks (KORGym, Enigmata), and general reasoning benchmarks (ARC-AGI-1/2, AIME 2025, AA Intelligence). All puzzle benchmarks correlate with existing benchmarks.

Abbr.	Error type	Definition
RR	Repeated Reasoning	Near-identical move sequences repeated from the same board position without meaningful variation.
STF	State-Tracking Failure	Claimed board state diverges from the cumulative effect of the model’s own stated actions.
CF	Constraint Forgetting	An explicit action that directly violates structural puzzle rules (overcounting, crossing, overwriting fixed cells).
PC	Premature Commitment	Commits to a provably incorrect configuration and persists for 3+ steps.
ES	Explicit Surrender	Explicitly gives up, requests a solver, or states it cannot proceed.
IO	Incomplete Output	Fails to produce a complete, valid final answer.

Appendix

Table 10: Full eleven-category error taxonomy. Categories above the first double rule are plausibly causal ; those below are symptomatic . The last four rows (below the second double rule) are omitted from the main-body figures.

Figure 6: Error category prevalence by puzzle type and difficulty tier (50 traces per cell). Galaxies and FlowFree are dominated by ES even at lower difficulties. FlowFree has the strongest PC scaling. Pattern exhibits elevated RR and extreme RD across all tiers. Bridges has the lowest ES/RR rates but shows concentrated STF and PC at hard.

Group	N	Mean tokens	Median tokens	Std
Correct	308	21,801	18,368	12,174
Other failure	128	34,632	34,924	15,423
ES failure	464	42,425	42,326	8,713

Appendix

Table 11: Output token counts and reasoning trace length by outcome group. ES traces cluster near the token ceiling with notably low variance; no ES trace occurs below 22k tokens.

Figure 7: Difference in error prevalence between incorrect and correct traces (incorrect rate $-$ correct rate), aggregated across difficulty tiers. Positive bars indicate errors more common in failed solves. ES dominates the aggregate because it is a downstream symptom of failure. On Bridges, STF is the strongest discriminator (+44 pp); on Undead, ES (+69 pp) and STF (+27 pp) lead. CF is near zero across all panels despite its large causal effect in interventions (Section 4.2 ), reinforcing that observational association—like raw frequency—is a poor proxy for causal impact.

Condition	Easy	Medium	Hard	Total	$\bm{\Delta}$
Baseline	91.7	82.0	32.0	66.2 [60.4, 71.5]	–
RR Gold Ctrl	91.7	85.0	32.0	67.3 [61.5, 72.6]	$+$ 1.1
RR Rand Ctrl	93.5	80.0	30.0	65.7 [59.9, 71.0]	$-$ 0.5
RR Standard	94.7	76.0	24.0	62.2 [56.3, 67.7]	$-$ 4.0
STF $k{=}3$	88.9	73.0	19.0	58.4 [52.5, 64.0]	$-$ 7.8
PC Half	93.9	43.0	8.0	45.4 [39.7, 51.2]	$-$ 20.8 ∗

Appendix

Table 12: Intervention results on Bridges (accuracy %, 100 puzzles per difficulty). 95% Wilson CIs shown for the Total column. ∗ Significant at

\alpha{=}0.05

(95% Newcombe CI for

\Delta

excludes zero).

Condition	Easy	Medium	Hard	Total	$\bm{\Delta}$
Baseline	81.0	44.0	9.0	44.7 [39.1, 50.3]	–
RR Gold Ctrl	82.0	45.0	6.0	44.3 [38.8, 50.0]	$-$ 0.3
RR Rand Ctrl	71.0	33.0	3.0	35.7 [30.5, 41.2]	$-$ 9.0 ∗
RR Standard	70.0	45.0	2.0	39.0 [33.7, 44.6]	$-$ 5.7
STF $k{=}3$	63.0	35.0	1.0	33.0 [27.9, 38.5]	$-$ 11.7 ∗
PC Half	74.0	26.0	0.0	33.3 [28.2, 38.8]	$-$ 11.3 ∗

Appendix

Table 13: Intervention results on Undead (accuracy %, 100 puzzles per difficulty). 95% Wilson CIs shown for the Total column. ∗ Significant at

\alpha{=}0.05

(95% Newcombe CI for

\Delta

excludes zero).

Condition	Bridges ( $N{=}300$ )		Undead ( $N{=}300$ )
	Total	$\bm{\Delta}$	Total	$\bm{\Delta}$
Baseline	66.2 [60.4, 71.5]	–	44.7 [39.1, 50.3]	–
RR Gold Ctrl	67.3 [61.5, 72.6]	$+$ 1.1	44.3 [38.8, 50.0]	$-$ 0.3
RR Standard	62.2 [56.3, 67.7]	$-$ 4.0	39.0 [33.7, 44.6]	$-$ 5.7
STF $k{=}3$	58.4 [52.5, 64.0]	$-$ 7.8	33.0 [27.9, 38.5]	$-$ 11.7 ∗
PC Half	45.4 [39.7, 51.2]	$-$ 20.8 ∗	33.3 [28.2, 38.8]	$-$ 11.3 ∗

Appendix

Table 14: Cross-puzzle comparison of intervention effects (accuracy %). Both puzzles: 100 puzzles per difficulty (

N{=}300

). 95% Wilson CIs for individual proportions; Newcombe hybrid score CIs for

\Delta

. ∗ Significant at

\alpha{=}0.05

(95% CI for

\Delta

excludes zero).

Puzzle	Condition	Easy	Medium	Hard	Total
Bridges	Standard prompt ( $N{=}150$ )	94.0	88.0	40.0	74.0
Bridges	Intervention baseline ( $N{=}300$ )	91.7	82.0	32.0	66.2
Undead	Standard prompt ( $N{=}150$ )	76.0	46.0	10.0	44.0
Undead	Intervention baseline ( $N{=}300$ )	81.0	44.0	9.0	44.7

Appendix

Table 15: Comparison of DeepSeek-v3.2 accuracy (%) under the standard benchmark prompt (no prefix) and the intervention baseline (gold prefix at 15% depth, no error injection). Grid sizes are identical across evaluations. Differences are within the range expected from sampling variation and minor prompt format differences.

#	Condition	Medium	Hard	Total	$\bm{\Delta}$
1	Baseline (rules + minimal I/O)	80%	40%	60%	–
2	Text backtrack instruction	84%	28%	56%	$-$ 4
3	1-shot gold path (no NL reasoning)	76%	42%	59%	$-$ 1
4	1-shot worked example (verbose NL)	86%	28%	57%	$-$ 3
5	2-shot worked example (verbose NL)	76%	30%	53%	$-$ 7
6	Text planning instruction	78%	30%	54%	$-$ 6

Appendix

Table 16: Prompt-based intervention results on Bridges (DeepSeek-v3.2). Each condition is evaluated on 50 medium and 50 hard puzzles (

N{=}100

\Delta

is change relative to baseline.

Additive ablation
Condition	Tools available	Accuracy	Board valid	Avg tokens
Baseline (no tools)	—	40%	50%	25,640
State mutation only	make_move , render_board	26%	96%	28,308
+ degree bookkeeping	+ state_summary	42%	100%	22,784
+ move enumeration	+ neighbors	44%	100%	19,570
+ connectivity	+ components	48%	98%	18,030

Appendix

Table 17: Full tool ablation on hard Bridges (

N{=}50

). Top : additive ablation adding one tool per row. Bottom : grid removal variants testing whether render_board helps when structured tools are available.

Model	Format	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern	Avg
Gemini 3 Flash	ascii	0.72	0.86	0.18	0.16	0.78	0.90	0.60
Gemini 3 Flash	intformat	0.70	1.00	0.40	0.98	0.60	0.94	0.77
Gemini 3 Flash	intf. json	0.72	1.00	0.54	0.98	0.12	0.96	0.72
Gemini 3 Flash	ascii + img	0.74	0.92	0.08	0.16	0.60	0.92	0.57
GPT-5 Mini	ascii	0.66	0.90	0.32	0.58	0.88	0.92	0.71
GPT-5 Mini	intformat	0.72	0.98	0.66	0.88	0.98	0.90	0.85

Appendix

Table 18: Easy accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy

\leq 0.01

Model	Format	FlowFree	Bridges	Loopy	Galaxies	Undead	Pattern	Avg
Gemini 3 Flash	ascii	0.54	0.66	0.00	0.00	0.10	0.78	0.35
Gemini 3 Flash	intformat	0.70	1.00	0.00	0.14	0.10	0.78	0.45
Gemini 3 Flash	intf. json	0.68	1.00	0.00	0.06	0.00	0.86	0.43
Gemini 3 Flash	ascii + img	0.62	0.70	0.00	0.00	0.14	0.82	0.38
GPT-5 Mini	ascii	0.22	0.72	0.00	0.00	0.80	0.90	0.44
GPT-5 Mini	intformat	0.38	0.90	0.02	0.16	0.72	0.70	0.48

Appendix

Table 19: Medium accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy

\leq 0.01

Model	Format	FlowFree	Bridges	Undead	Pattern	Avg
Gemini 3 Flash	ascii	0.02	0.22	0.00	0.30	0.09
Gemini 3 Flash	intformat	0.12	0.86	0.00	0.36	0.22
Gemini 3 Flash	intf. json	0.18	0.90	0.00	0.30	0.23
Gemini 3 Flash	ascii + img	0.06	0.34	0.00	0.44	0.14
GPT-5 Mini	ascii	0.02	0.44	0.52	0.44	0.24
GPT-5 Mini	intformat	0.04	0.54	0.24	0.00	0.14

Appendix

Table 20: Hard accuracy results across format types. intf. json = IntFormat JSON, ascii + img = ASCII + Image. Gray rows indicate average accuracy

\leq 0.01

Model	Tokens	Price per 1M tokens (USD)	Full price (USD)
gpt-5-mini	49,607,682	2.00	99.22
qwen3-235b	36,252,844	2.30	83.38
gemini-3-flash-preview	27,014,575	3.00	81.04
deepseek-v3.2	30,766,175	0.42	12.92
glm-4.7-flash	26,939,558	0.40	10.78
olmo-3.1-32b-think	19,736,186	0.50	9.87

Appendix

Table 21: Cost of running the main benchmark

Model	Tokens	Price per 1M tokens (USD)	Full price (USD)
gpt-5-mini	191,749,525	2.00	383.50
gemini-3-flash-preview	109,343,674	3.00	328.03
qwen3-235b	108,633,225	2.30	249.86
deepseek-v3.2	91,074,375	0.42	38.25
glm-4.7-flash	82,757,329	0.40	33.10
olmo-3.1-32b-think	58,457,735	0.50	29.23

Appendix

Table 22: Cost of running the input format ablations

Panel of Figure 8 — Figure 8: The same Bridges puzzle represented in three input formats: ASCII, intformat, and intformat_json.

Panel of Figure 11 — Figure 11: Prompt used for evaluating FlowFree puzzles

Panel of Figure 12 — Figure 12: Prompt used for evaluating Galaxies puzzles

Panel of Figure 13 — Figure 13: Prompt used for evaluating Pattern puzzles

Panel of Figure 14 — Figure 14: Prompt used for evaluating Undead puzzles

Panel of Figure 15 — Figure 15: Prompt used for evaluating Loopy puzzles

Constraint-Guided Reasoning in Language Models

See all 15 papers

MindTopo: Can Foundation Models Reason in Topological Space?
Sep 10, 2026Yunfei Ge, Anbang Liu, Qineng Wang +9Spatial Reasoning Benchmarks AI Agent Benchmarks
Northwestern University · Microsoft Research · Stanford University
Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning
Mar 2, 2026Justin WaughLLM Evaluation LLM Reasoning
Approximate Labs
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangLanguage Model Reasoning Evaluation LLM Evaluation
Georgia Institute of Technology

Sep 10, 2026J/K move · Enter open · S save

MindTopo: Can Foundation Models Reason in Topological Space?

Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica+8

Northwestern University · Microsoft Research · Stanford University

Spatial Reasoning Benchmarks AI Agent Benchmarks VLM Reasoning

Open paper AskPDF arXiv

Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots. MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We benchmark 14 MLLMs and study agent configurations augmented with image and video generation, including 3 video generative models in planning settings. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions. Our website is at https://mind-topo.github.io/

Abstract

Constraint-Guided Reasoning in Language Models

Explore similar work

MindTopo: Can Foundation Models Reason in Topological Space?

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models