Grid-Bench: Can LLMs Predict How Discrete Worlds Change?

Zixiang Xu; Zirui Song; Wanrong Zheng; Chiyu Zhang; Yue Huang; Yuyang Peng; Kexin Zhang; Jinkun Chen

cs.CLOct 15, 2026

Grid-Bench: Can LLMs Predict How Discrete Worlds Change?

Authors: Zixiang Xu, Zirui Song, Wanrong Zheng, Chiyu Zhang, Yue Huang, Yuyang Peng, Kexin Zhang, Jinkun Chen,

Organizations: University of Southern California · Mohamed bin Zayed University of Artificial Intelligence · Meta · University of Notre Dame ·

Abstract

Spatial instructions describe how to move objects, transform regions, or modify their surroundings. When several operations are applied in sequence, each acts on a configuration shaped by earlier changes. How accurately can large language models (LLMs) predict the resulting spatial state? We introduce Grid-Bench, an executable benchmark in which eight families of operations transform discrete worlds represented as two-, three-, and four-dimensional grids. Given an initial grid, explicit rules, and an event history expressed as text, a model predicts the complete final grid. Reference execution makes every predicted value checkable, supporting comparisons across spatial operations, history lengths, and workloads. Evaluating six LLMs shows that complete-state accuracy falls sharply as histories grow, with substantial divergence among models that perform similarly on single events. Models with similar exact-match accuracy can nevertheless differ substantially in the number of incorrect cells on shared instances. Even over long histories, some models recover many correct changes beyond what copying the initial state would yield. Grid-Bench makes both complete recovery and these differences in returned states directly measurable across a range of spatial conditions.

Figures & tables

Figure 1: Reference execution in 2D (A) and 3D (B), from

W_{0}

through

W_{1}

W_{2}

, and

W_{3}

. Values 0, 1, and 2 denote empty cells, the moving entity, and obstacles. MOVE stops at an obstacle or its step limit; SET places an obstacle at the labeled coordinate. The new obstacle limits the final move to one cell. Coordinates are zero-based: (row, column) in (A) and

(x,y,z)

in (B). The model receives

W_{0}

, rules, and all three events as text and predicts the complete

W_{3}

Rule	Spatial selection	State transformation
PAINT	Cells with a specified source value inside a Manhattan ball.	Simultaneously replace the source value with a target value.
REFLECT	An axis-aligned box.	Apply a bijective reflection along one axis, including zero-valued cells.
STENCIL	A Manhattan ball, with reads from adjacent cells beyond the ball.	Replace each selected value by a modular sum over its old local neighborhood.
MIX	Cells reachable within a bounded number of steps through nonzero cells.	Apply synchronous neighborhood aggregation while keeping updated cells nonzero.
RELABEL	The maximal equal-valued connected component containing a seed.	Assign one target value to the entire component.
LAYER	A component boundary or the component cells exposed to empty space.	Grow the component by one layer or erode its exposed layer.

Table 1: The eight core transition families. Each trajectory uses core events from one family, interleaved with single-cell SET assignments. Full definitions, valid parameter ranges, and edge cases appear in Appendix B .

Evaluation	Instances per model	Six-model outcomes
Main	1,152	6,912
Additional workload	336	2,016
Unique total	1,488	8,928

Table 2: Evaluation coverage. Lengths are

T=0,1,8,16,32,64

. The workload analysis combines seven additional settings with four reused main settings, for 528 instances per model; reused instances are counted only once in the total.

Figure 2: Complete-state recovery across event histories. Each point reports full-world EM on 192 main instances. Colors and markers identify models; shading gives marginal 95% trajectory-bootstrap intervals, omitting degenerate intervals. Lengths are categorical.

Figure 3: Complete-state recovery across eight task families. Each cell gives exact answers out of 96 instances, comprising four prefixes ( $T=8,16,32,64$ ) of 24 trajectories across three dimensions. Shading encodes counts on a shared scale; label colors identify models.

Figure 4: Composition of valid incorrect states at

T=1

and

T=64

. Each bar shows the proportion with errors on reference-changed cells only (

C

), reference-unchanged cells only (

U

), or both. Segments display counts where space permits;

n

counts the conditional subset at each length. Changed and unchanged compare final reference values with initial values.

Figure 5: Error extent in returned states. (a) Median wrong-cell counts among each model’s valid incorrect answers at

T=1,16,64

on a logarithmic axis. Marker shapes identify lengths, and the

n

column lists their sample sizes in that order. Each length uses its own conditional subset. (b) Wrong-cell counts on 84 shared instances where Kimi K2.5 and MiniMax M2.5 both return valid incorrect states at

T=8

. Both axes are logarithmic with equal scales; points above the equality line indicate fewer errors for Kimi K2.5.

Model	$n/N$	Pred.	Copy	Zero
GPT-5.6 Sol	191/192	81.6	51.0	48.8
Claude 5 Sonnet	58/192	90.5	49.5	56.5
GPT-5.4 Mini	151/192	56.1	50.6	49.8
GLM-5	148/192	62.4	52.5	50.6
Kimi K2.5	138/192	55.6	51.1	50.2
MiniMax M2.5	137/192	52.4	50.9	49.4

Table 3: Cell accuracy (percent) at

T=64

n/N

gives valid grids, including exact ones, out of all main requests. Pred. denotes the model prediction; Copy and Zero denote copy-initial and all-zero. All three use the same instances within each row; subsets differ across models.

	$T=1$			$T=16$
Model	$d{=}2$	$d{=}3$	$d{=}4$	$d{=}2$	$d{=}3$	$d{=}4$
GPT-5.6 Sol	62	60	61	49	21	17
Claude 5 Sonnet	53	52	32	34	16	11
GPT-5.4 Mini	29	26	14	4	1	0
GLM-5	47	30	19	6	2	0
Kimi K2.5	56	53	39	13	5	0

Table 4: Exact answers out of 64 main instances per cell, retaining all failures. The two lengths show early and intermediate structural contrasts. Worlds at

d=2,3,4

contain 256, 252, and 256 cells; geometry and coordinate representation change together. Complete length curves appear in Appendix Figure 11 .

Figure 6: Within-family workload comparisons at

d=2

. Each panel shows all six models; marker shapes denote the task-specific targets listed above it. Points report exact answers out of 32 instances: four prefixes (

T=8,16,32,64

) from eight trajectories per target. Targets refer to ball size ( PAINT ), box volume ( REFLECT ), component-plus-boundary size ( LAYER ), and reached-region size ( MIX ).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Setting	Supplied information	Evaluated output	Evaluation focus
SPARTQA ( Mirzaee et al., 2021 )	Textual scenes	Description and question	Relation, object, or containment answer	Spatial relations and reasoning steps
CLEVR/NLVR/VSR ( Johnson et al., 2017 ; Suhr et al., 2017 ; Liu et al., 2023b )	Visual scenes	Image and language	Answer or statement truth	Compositional and spatial relations
SCONE ( Long et al., 2016 )	Executable domains	Initial world and instructions	Executable logical forms	Final-world denotation accuracy
ProPara/OpenPI ( Dalvi et al., 2018 ; Tandon et al., 2020 )	Natural processes	Procedural text	Entity states or change tuples	Changes through process steps
Entity tracking ( Kim and Schuster, 2023 )	Containers and objects	Initial contents and events	Queried container contents	Update and retention
Interactive worlds ( Côté et al., 2018 ; Shridhar et al., 2021 ; Wang et al., 2022 )	Textual or embodied environments	Observations and goals	Agent-selected actions	Goal success and efficiency

Appendix

Table 5: Representative evaluation settings related to Grid-Bench. The columns describe what information a task supplies, what prediction or action it scores, and the main focus of that evaluation. Rows group closely related datasets when they share a role in the discussion; the table is not a numerical comparison.

Rule	Initial values	Event parameters	Final values
PAINT	$[0,1,1,2,0]$	$a=[2],r=1,u=1,v=3$	$[0,3,3,2,0]$
REFLECT	$[1,0,2,3,0]$	$\mathrm{lo}=[1],\mathrm{hi}=[4],j=0$	$[1,0,3,2,0]$
STENCIL	$[1,2,0,3,0]$	$a=[2],r=1,M=4$	$[1,3,1,3,0]$
MIX	$[1,2,0,3,1]$	$p=[0],r=2,M=4$	$[1,1,0,3,1]$
RELABEL	$[1,1,0,1,2]$	$p=[0],v=2$	$[2,2,0,1,2]$
LAYER	$[0,1,1,0,2]$	$p=[1],s=+1$	$[1,1,1,1,2]$

Appendix

Table 6: Single-event illustrations with explicit expected states. All positions not changed by the event retain their initial value. The COMPRESS alphabet includes 8; other examples use an alphabet containing their displayed labels.

Figure 7: Four hand-calculated two-dimensional rule examples, verified by the production and dictionary executors. Cell color encodes the printed integer value; solid blue old-state outlines mark write support, dashed gray outlines mark other read coordinates, and orange new-state outlines mark actual changes. RELABEL preserves disconnected cells with the same label. LAYER grows only one boundary layer and does not overwrite label 2. STENCIL reads beyond its five-cell update ball and uses only old values. MOVE updates each row front to back, with label-2 obstacles preserved. Read counts include repeated incidences, whereas outlines show unique coordinates. These small examples are explanatory fixtures, not formal samples or model predictions.

$d$	Shape	$N$	$E$	$\overline{\deg}$	Boundary	$N+2E$
2	$[16,16]$	256	480	3.750	23.44%	1216
3	$[6,6,7]$	252	636	5.048	68.25%	1524
4	$[4,4,4,4]$	256	768	6.000	93.75%	1792

Appendix

Table 7: Exact structural quantities for the three evaluated shapes. Boundary is the percentage of cells on the geometric world boundary. The final column is full-world neighborhood-aggregation scan, not the realized workload of the selected STENCIL or MIX events.

Rule	Target	Unit	Alphabet / initializer	Density band	SET rate
PAINT	100	reads	$0{:}3$ / uniform	.45–.65	.2
REFLECT	64	reads	$0{:}3$ / uniform	.45–.65	.2
STENCIL	10	support	$0{:}3$ / uniform	.55–.85	.2
MIX	12	support	$0{:}3$ / uniform	.55–.70	.2
RELABEL	100	reads	$0{:}6$ / regions	.50–.65	.2
LAYER	60	reads	$0{:}6$ / regions	.45–.60	.2

Appendix

Table 8: Main generation conditions. Targets guide candidate selection; realized operation sizes vary with the state and geometry. A cells target selects the shape tier, whose volume is 252 or 256. Density bands guide initialization and event selection.

Model	Requested identity	Recorded route	Recorded request parameters
GPT-5.6 Sol	gpt-5.6-sol	azure/gpt-5.6-sol	max_completion_tokens = 128000 reasoning_effort = "high"
Claude 5 Sonnet	claude-sonnet	vertex_ai/claude-sonnet-5	max_completion_tokens = 128000 reasoning_effort = "high"
GPT-5.4 Mini	gpt-5.4-mini	azure/gpt-5.4-mini	max_completion_tokens = 128000 reasoning_effort = "high"
GLM-5	zai.glm-5	zai.glm-5	max_tokens = 128000 reasoning_effort = "high"
Kimi K2.5	moonshotai.kimi-k2.5	moonshotai.kimi-k2.5	max_tokens = 128000 reasoning_effort = "high"
MiniMax M2.5	minimax.minimax-m2.5	minimax.minimax-m2.5	max_tokens = 128000 Effort absent; native thinking.

Appendix

Table 9: Six evaluated models, each with 1,488/1,488 scored instances, prompt 0.4, a 128,000-token generation allowance, and no tools. Parameter keys and values are copied from the retained request ledger. For Bedrock, ledger max_tokens maps to wire field inferenceConfig.maxTokens ; gateway requests retain max_completion_tokens . MiniMax M2.5 sends no effort field and uses native always-on thinking. Identifiers and routes follow the frozen collection metadata.

Model	$T=0$	$T=1$	$T=8$	$T=16$	$T=32$	$T=64$
GPT-5.6 Sol	192/0/0	183/9/0	145/47/0	87/105/0	57/135/0	25/166/1
Claude 5 Sonnet	192/0/0	137/54/1	91/98/3	61/111/20	35/102/55	14/44/134
GPT-5.4 Mini	191/1/0	69/120/3	18/157/17	5/166/21	2/153/37	1/150/41
GLM-5	191/0/1	96/95/1	17/165/10	8/169/15	3/167/22	0/148/44
Kimi K2.5	192/0/0	148/43/1	33/138/21	18/153/21	6/151/35	0/138/54
MiniMax M2.5	188/2/2	150/40/2	31/117/44	11/133/48	5/144/43	1/136/55

Appendix

Table 10: Answer composition at each main history length. Each cell lists exact / valid incorrect / invalid or empty counts, which sum to 192. The first two categories form the valid-state population; the second is used for error-extent and location diagnostics. All six lengths use separately queried prefixes of the same trajectories.

Figure 10: Full task-specific history profiles. Each point is exact answers out of 24 main instances: eight trajectories in each of three dimensions. Colors and markers identify the six models. Lines connect the tested categorical lengths; the curves are descriptive. All outcomes remain in the denominator.

Figure 11: Dimension profiles for all six models. Each model has two vertically adjacent panels: full-world EM and valid-state rate. Circles/solid, squares/dashed, and triangles/dotted denote

d=2,3,4

. Each point includes 64 main instances from eight tasks and eight trajectories per task. Valid states include exact and incorrect answers. Lines connect categorical lengths and carry no uncertainty bands.

Figure 12: PAINT workload profiles at

d=2

. Six model panels compare the two ball-size targets. Each point is EM over eight trajectories at the specified length, including all outcomes. Markers and line styles identify targets; lengths are categorical. The main-condition level reuses main instances.

Figure 13: REFLECT workload profiles at

d=2

. Each model panel compares box-volume targets 16, 64, and 256 using the legend’s markers and line styles. Each point is EM over eight trajectories at a categorical tested length, with all outcomes retained. Curves are descriptive.

Figure 14: LAYER workload profiles at

d=2

, comparing component-plus-boundary targets 15, 60, and 150. Each model panel reports EM over eight trajectories per categorical length, with all outcomes retained. Markers and line styles identify targets; curves are descriptive.

Figure 15: MIX workload profiles at

d=2

, comparing reached-region targets 6, 12, and 30. Each model panel reports EM over eight trajectories per categorical length, with all outcomes retained. Markers and line styles identify targets; curves are descriptive.

Model	Original /384	Additional /384	Combined /768
GPT-5.6 Sol	160	154	314
Claude 5 Sonnet	103	98	201
GPT-5.4 Mini	13	13	26
GLM-5	10	18	28
Kimi K2.5	23	34	57
MiniMax M2.5	21	27	48

Appendix

Table 11: Exact main-set answers over

T=8,16,32,64

, separated by collection cohort. The original and additional cohorts each contain 96 trajectories, with four prefixes per trajectory. They use disjoint seeds under the same selected generation and evaluation conditions. The combined column is their sum.

Model	Including $T=0$	Positive lengths only
GPT-5.6 Sol	20	20
Claude 5 Sonnet	51	51
GPT-5.4 Mini	13	12
GLM-5	5	4
Kimi K2.5	15	15
MiniMax M2.5	9	6

Appendix

Table 12: Trajectories with at least one exact later answer after an earlier failed answer, out of 192 per model. The second column includes the reconstruction control; the third uses only positive lengths. Each prefix is a separate request.

Model	$T$	$n$	Median	P90	$\leq 5$ cells
GPT-5.6 Sol	1	9	2.0	2.2	9
GPT-5.6 Sol	16	105	4.0	24.6	62
GPT-5.6 Sol	64	166	33.0	123.0	12
Claude 5 Sonnet	1	54	2.0	8.0	44
Claude 5 Sonnet	16	111	6.0	34.0	53
Claude 5 Sonnet	64	44	14.0	88.9	6

Appendix

Table 13: Error extent at early, intermediate, and long histories, conditional on valid incorrect main answers. Median and P90 count wrong cells in worlds of 252 or 256 cells; P90 uses linear interpolation. The last column counts predictions with at most five wrong cells. Each row has its own subset of size

n

Model	$T$	$n_{C}$	$C$ acc. (%)	$n_{U}$	$U$ acc. (%)
GPT-5.6 Sol	16	192	95.3	192	99.0
GPT-5.6 Sol	32	192	88.6	192	96.1
GPT-5.6 Sol	64	191	73.6	191	88.7
Claude 5 Sonnet	16	172	92.7	172	98.6
Claude 5 Sonnet	32	137	89.3	137	96.1
Claude 5 Sonnet	64	58	88.6	58	91.8

Appendix

Table 14: Changed- and unchanged-cell accuracy for valid main answers. Each value is a macro-average over the

n_{C}

n_{U}

answers with a nonempty reference subset.

C

denotes final-reference-changed cells;

U

denotes final-reference-unchanged cells. Accuracy is undefined for an empty contributing subset. This table covers

T=16,32,64

Model	$T$	$n$	Prediction	Copy	Zero	Above copy
GPT-5.6 Sol	16	192	98.0	63.0	48.5	192
GPT-5.6 Sol	32	192	93.1	56.6	48.6	191
GPT-5.6 Sol	64	191	81.6	51.0	48.8	180
Claude 5 Sonnet	16	172	96.8	63.5	48.9	172
Claude 5 Sonnet	32	137	93.4	57.3	49.3	134
Claude 5 Sonnet	64	58	90.5	49.5	56.5	57

Appendix

Table 15: Cell accuracy (percent) for model predictions, copy-initial, and all-zero.

n

is the number of valid main answers; all three predictions in a row use the same instances. Above copy counts model answers with strictly greater cell accuracy than copy-initial. Valid-answer subsets can differ across models. This table covers

T=16,32,64

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

Mar 18, 2026

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

We introduce GSU, a text-only grid dataset to evaluate the spatial reasoning capabilities of LLMs over 3 core tasks: navigation, object localization, and structure composition. By forgoing visual inputs, isolating spatial reasoning from perception, we show that while most models grasp basic grid concepts, they struggle with frames of reference relative to an embodied agent and identifying 3D shapes from coordinate lists. We also find that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Finally, we show that while the very latest frontier models can solve the provided tasks (though harder variants may still stump them), fully fine-tuning a small LM or LORA fine-tuning a small LLM show potential to match frontier model performance, suggesting an avenue for specialized embodied agents.

Risham Sidhu, Julia Hockenmaier

University of Illinois at Urbana-Champaign

Spatial Reasoning in Language Models Spatial Reasoning Spatial Reasoning Benchmarks 3D Spatial Reasoning Embodied Reasoning

Ask PDF

May 17, 2025

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: Predict asks for the state that follows an action sequence, Plan asks for an action sequence that reaches a goal, and Decompose asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.

Idriss Malek, Omar Choukrani, Daniil Orel +5

Tandon School of Engineering New York University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Abu Dhabi, UAE

LLM Planning LLM Agent Evaluation AI Agent Benchmarks Long-Horizon LLM Agents

Ask PDF

Apr 12, 2026

Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks

Foundation models have shown remarkable performance across diverse tasks, yet their ability to construct internal spatial world models for reasoning and planning remains unclear. We systematically evaluate the spatial understanding of large language models through maze tasks, a controlled testing context requiring multi-step planning and spatial abstraction. Across comprehensive experiments with Gemini-2.5-Flash, GPT-5-mini, Claude-Haiku-4.5, and DeepSeek-Chat, we uncover significant discrepancies in spatial reasoning that challenge assumptions about LLM planning capabilities. Using chain-of-thought prompting, Gemini achieves 80-86% accuracy on smaller mazes (5x5 to 7x7 grids) with tokenized adjacency representations, but performance collapses to 16-34% with visual grid formats, which is a 2-5x difference, suggesting representation-dependent rather than format-invariant spatial reasoning. We further probe spatial understanding through sequential proximity questions and compositional distance comparisons. Despite achieving 96-99% semantic coverage in reasoning traces, models fail to leverage this understanding for consistent spatial computations, indicating that they treat each question independently rather than building cumulative spatial knowledge. Our findings based on the maze-solving tasks suggest that LLMs do not develop robust spatial world models, but rather exhibit representation-specific and prompting-dependent reasoning that succeeds only under narrow conditions. These results have critical implications for deploying foundation models in applications requiring spatial abstraction.

Weijiang Li, Yilin Zhu, Rajarshi Das +1

University of Notre Dame · Columbia University · MQube Cognition

Spatial Reasoning in Language Models LLM World Models Spatial Reasoning Spatial Reasoning Benchmarks

Ask PDF

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks