LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

Bingo Zhang; Haochuan Lu; Zongjie Li; Genjian Li; Ari Yu Zhang; Chaozheng Wang

cs.CLSep 28, 2026

LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

Authors: Bingo Zhang, Haochuan Lu, Zongjie Li, Genjian Li, Ari Yu Zhang, Chaozheng Wang

Organizations: Vera Praxis · Tencent · The Hong Kong University of Science and Technology · Independent Researcher · Department of Computer Science and Engineering, The Chinese University of Hong Kong

Abstract

GUI agents need long-horizon visual reasoning: they must interpret a changing interface while keeping a multi-step plan viable as earlier actions constrain later ones. Existing benchmarks evaluate grounding, computer use, and game play, but rarely test whether agents stay coherent across long chains of coupled decisions. Long-horizon visual puzzles expose this capability directly: a legal move that looks like progress can make the puzzle unsolvable, and the loss shows only several moves later. We introduce LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced. With Native GUI Actions alone, the strongest agents solve most objectives, but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve nothing harder than Medium, and none completes Bolt Unscrew Hard, which a human solves along with every other objective. Code Execution CUA does not close this gap, and its scores mix visual solving with algorithmic search. Controlled diagnostics trace these failures to one limitation that neither rules, state hints, nor failure memory removes: agents judge each move by the visible progress it makes, not by the future options it leaves.

Figures & tables

		What solving requires			How the agent must act
Benchmark	Environment	Visual puzzle state	Delayed consequences	Dead-end states	Agent-owned recovery	Native GUI actions
PuzzleBench ( Zhang et al., 2025 )	Puzzle VQA	$\checkmark$	$\times$	–	–	–
VisualWebArena ( Koh et al., 2024 )	Web apps	$\times$	$\times$	$\times$	–	$\times$
AndroidWorld ( Rawles et al., 2025 )	Mobile apps	$\times$	$\times$	$\times$	–	$\checkmark$
BALROG ( Paglieri et al., 2025 )	Text/grid games	$\checkmark^{\dagger}$	$\checkmark$	$\checkmark$	$\times$	$\times$
VideoGameBench ( Zhang et al., 2026 )	Video games	$\checkmark^{\dagger}$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$

Table 1: Structural comparison with adjacent benchmarks.

\checkmark

: yes;

\times

: no;

\dagger

: in a subset of tasks or one track; –: not applicable or not reported. Property definitions and per-benchmark notes: Appendix A.4 .

	Completion			Horizon		Difficulty	Behavior
Agent	OSR $\uparrow$	LC $\uparrow$	FGP $\uparrow$	LPR $\uparrow$	H 50 $\uparrow$	LC hard $\uparrow$	PER $\downarrow$	NGAR $\uparrow$
Human reference	100.0	100.0	100.0	100.0	10+	100.0	0.0	–
Native GUI Actions general-purpose, Codex
GPT-6-Astra	91.7	94.8	94.8	97.9	10+	87.5	0.0	–
GPT-6-Sol	82.8	91.0	92.2	95.4	10+	80.3	0.0	–
GPT-6-Luna	33.9	40.0	43.8	46.7	2	9.0	68.8	–

Table 2: Main results. Swatch colors match Figure 3 ; ⋄ open-weight. Bold: best within each setting. Values in %; H 50 in levels (10+: at least half the episodes clear all ten). –: undefined or not applicable.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Game	Difficulties	Levels	Level completion
Bolt Unscrew	Easy / Hard	8 / 8	Remove every board.
Rush Hour	Easy / Medium / Hard	10 / 10 / 10	Move the target vehicle through the exit.
Nut and Bolt	Easy / Medium / Hard / Extreme / Nightmare	3 / 3 / 3 / 3 / 1	Make every nonempty stack full and monochromatic.
Truck Escape	Default	5	Remove all vehicles.
Maze Paint	Easy / Medium / Hard	10 / 10 / 10	Paint every traversable cell.
Color Connect	Easy / Hard	10 / 10	Connect every matching color pair with non-overlapping paths.

Appendix

Table 3: Games and objectives. Each game–difficulty pair is one objective; level counts follow the listed difficulty order. The completion condition applies to each level.

Objective	Levels	Pointer presses	Restarts
Bolt Unscrew / Easy	8	100	1
Bolt Unscrew / Hard	8	2,500	52
Rush Hour / Easy	10	89	0
Rush Hour / Medium	10	102	0
Rush Hour / Hard	10	442	12
Nut and Bolt / Easy	3	45	0

Appendix

Table 4: Human reference episodes: target levels, pointer presses, and recovery actions per objective.

Native GUI Actions	Code Execution CUA
Action space: one GUI-action tool	Action space: code runtime and GUI input
click(x,y) primary click	Run JavaScript in a persistent runtime
double_click(x,y) double click	over screenshots and derived data
drag(x0,y0,x1,y1) drag pointer	Request a screenshot
press_key(key) game key	Issue the same coordinate GUI inputs,
type_text(text) text entry	directly or from code

Appendix

Table 5: The two evaluation settings. Both play the same game in a dedicated Chromium browser at

1280\times 900

and change it only through coordinate-based GUI input (bottom row); they differ in what the agent may do between inputs.

Agent	Endpoint	Effort	Limit	Price (in / cached / out)
General-purpose, Codex scaffold
GPT-6-Astra	OpenAI	medium	258	10.00 / 1.00 / 50.00
GPT-6-Sol	OpenAI	medium	258	2.00 / 0.20 / 10.00
GPT-6-Luna	OpenAI	medium	258	0.10 / 0.01 / 0.50
GPT-5.6-Sol	OpenAI	medium	258	4.00 / 0.40 / 20.00
GPT-5.6-Terra	OpenAI	medium	258	2.00 / 0.20 / 12.00

Appendix

Table 6: Model access and inference settings. Effort: requested reasoning effort, identical in both settings and matched by the recorded value in every episode. Limit: context window configured in the Codex scaffold (general-purpose) or maximum output tokens per request (GUI-specialized), in thousands of tokens. Price: list price in USD per million uncached input, cached input, and output tokens, retrieved in September 2026 from each model’s first-party API or the listed endpoint; DeepSeek 4.1 Flash at peak-hour rates.

Native GUI Actions
Agent	Requests	Input (M)	Cached (%)	Output (M)	Cost (USD)
GPT-6-Astra	1,286	100.7	97.9	0.19	129.70
GPT-6-Sol	2,592	237.5	98.6	0.48	58.41
GPT-6-Luna	2,713	288.4	98.2	0.37	3.53
GPT-5.6-Sol	3,077	327.5	98.3	0.55	161.87
GPT-5.6-Terra	2,132	218.6	98.1	0.44	56.32

Appendix

Table 7: Token usage and estimated API cost of the 384 episodes of Table 2 . Input counts all prompt tokens, including cache hits; output includes reasoning tokens. Cost applies the list prices of Table 6 to uncached input, cached input, and output; it is an estimate, not a billed amount.

Outcomes
	Native GUI Actions	Code Execution CUA
Episodes solved / timed out / abandoned	84 / 41 / 35	92 / 31 / 37
Timeouts that solve a level in the final 15 min	15 of 41	7 of 31
Level restarts per episode (mean)	2.4	3.6
Interaction
Tool calls per episode (median)	100.0	80.5

Appendix

Table 8: Paired behavioral comparison of the two settings over the ten Codex backbones (160 episodes per setting). Inputs are pointer and key inputs recorded by the game. Matched-level ratios are medians of per-level ratios of Code Execution CUA to Native GUI Actions for time, calls, screenshots, and executed inputs. Drag slips are non-adjacent jumps and wrong-endpoint releases.

Puzzle	Diagnostic	Human	Astra, Native	Astra, Code
Bolt Unscrew Hard L4	Dead ends with no board removed	1	1	5
	Attempt that clears the level	3	–	8
	Most boards in one attempt	14	0	14
Nut and Bolt Nightmare	Stack-state changes	431	65	67
	Revisited states	115	4	0
	Solved-stack regressions	7	0	0

Appendix

Table 9: Trajectory diagnostics for the human reference and the GPT-6-Astra episodes of Table 2 in both settings (Native: Native GUI Actions; Code: Code Execution CUA). Bolt Unscrew values describe visible level 4 (14 boards, four free holes at the start), the first level Astra does not clear with Native GUI Actions, which it reaches near the deadline. Maze Paint and Rush Hour values sum moves over ten levels per objective; E/M/H: Easy/Medium/Hard.

Agent	Wins/reached	Conditional (%)	Unreached (%)	Exits/failures	Exit time
GPT-6-Astra	109/110	99.1	80.0	0/1	N/A
GPT-6-Sol	104/107	97.2	70.0	0/3	N/A
GPT-6-Luna	37/49	75.5	79.7	11/12	19.5 [5.4, 33.7]
GPT-5.6-Sol	97/102	95.1	70.4	1/5	20.6
GPT-5.6-Terra	68/77	88.3	74.3	9/9	17.2 [13.7, 21.0]
GPT-5.6-Luna	58/68	85.3	76.3	10/10	7.0 [5.5, 8.6]

Appendix

Table 10: RQ1 denominators and stopping times with Native GUI Actions (Table 2 episodes, 16 per agent). Reached and won levels are pooled episode–level observations. Conditional completion is wins/reached and differs from the macro-weighted LC in Figure 9 . Unreached share: fraction of the LC deficit in levels never reached. Exits/failures counts premature exits among unsuccessful episodes. Exit time: minutes, median [first quartile, third quartile], conditional on premature exit.

Agent	Maze LC (%)	Valid/attempted	Productive/valid (%)	Revisited/observed (%)
GPT-6-Astra	100.0	557/557	441/557 (79.2)	6/587 (1.0)
GPT-6-Sol	90.0	505/508	386/505 (76.4)	7/537 (1.3)
GPT-6-Luna	50.0	276/385	169/276 (61.2)	143/408 (35.0)
GPT-5.6-Sol	96.7	687/714	446/687 (64.9)	64/745 (8.6)
GPT-5.6-Terra	56.7	361/541	245/361 (67.9)	192/566 (33.9)
GPT-5.6-Luna	66.7	451/863	256/451 (56.8)	466/890 (52.4)

Appendix

Table 11: Maze Paint move diagnostics with Native GUI Actions (three episodes per agent, 260 attempts in total). Productive moves paint at least one new cell. State observations comprise the initial state and the states after game moves, pooled within attempts; revisits repeat an already observed position and painted-cell configuration. Counters include retries and stop at each level win or attempt end.

Condition/state	Appended information
B/D	State information: The current puzzle state cannot be completed through normal forward play. Recovery is required before progress can continue.
B/N	State information: The current puzzle state remains solvable through normal forward play.
C/D	B’s D message, followed by: Recovery information: Use the visible button labelled Restart to restart the current level.

Appendix

Table 12: RQ2 intervention text. A appends no message. The restart button carries a Chinese label, given here by its English translation.

State	Condition	Timely restart	Verified recovery	Same-level win
D	A	1/24 (4.2%)	8/24 (33.3%)	0/24 (0.0%)
D	B	22/24 (91.7%)	23/24 (95.8%)	0/24 (0.0%)
D	C	22/24 (91.7%)	22/24 (91.7%)	0/24 (0.0%)
N	A	0/12 (0.0%)	—	10/12 (83.3%)
N	B	0/12 (0.0%)	—	9/12 (75.0%)

Appendix

Table 13: RQ2 outcomes for GPT-5.6-Sol. Each cell gives a count and percentage over all continuations in that state and condition. Timely restart must occur within 120 seconds and the first ten actions; verified recovery and same-level wins use the 300-second limit. No N continuation restarts, including after the response window.

Hard level	D checkpoints	Selection $B-A$	Recovery $B-A$	Wins (A/B/C)
1	10	+80.0	+50.0	0/0/0
2	2	+100.0	+100.0	0/0/0
3	2	+100.0	+50.0	0/0/0
5	2	+100.0	+50.0	0/0/0
6	4	+75.0	+75.0	0/0/0
7	2	+100.0	+50.0	0/0/0

Appendix

Table 14: Within-level effects of state information on D, in percentage points. Every source level improves on timely selection and verified recovery. Checkpoints sharing a level remain together in the paired bootstrap. Win counts use the listed checkpoint count as the denominator for each condition.

Game	Model	A	B	C
Bolt Unscrew	Luna	0/6	5/6	5/6
Bolt Unscrew	Sol	2/6	5/6	6/6
Bolt Unscrew	Terra	5/6	6/6	5/6
Nut and Bolt	All three	0/9	0/9	0/9

Appendix

Table 15: Verified recovery in the RQ2 follow-up. Each Bolt Unscrew cell has six certified dead ends per condition; Nut and Bolt pools three dead ends per model, nine trials per condition. Same-level completion is zero in every cell.

Contrast or cell	Completion	$\Delta$ LC	$\Delta$ FGP
K0P0	5/18	–	–
K0P1	6/18	–	–
K1P0	8/18	–	–
K1P1	8/18	–	–
K1–K0 under P0	–	$+8.6$	$+7.8$
K1–K0 under P1	–	$+9.6$	$+9.7$

Appendix

Table 16: Cross-model RQ4 outcomes over 18 matched model–objective units per K/P cell. LC and FGP entries are mean paired effects in percentage points.

Configuration	$n$	S	PER	Native	Other	Cells
GPT-6-Astra / Code Execution CUA	16	15	0	729	381	16/16
GPT-6-Sol / Code Execution CUA	16	13	3	486	696	16/16
GPT-6-Luna / Code Execution CUA	16	4	12	1620	858	16/16
GPT-5.6-Sol / Code Execution CUA	16	14	0	122	1316	16/16
GPT-5.6-Terra / Code Execution CUA	16	6	10	528	1067	16/16
GPT-5.6-Luna / Code Execution CUA	16	5	11	629	568	16/16

Appendix

Table 17: Metric denominators by evaluation setting. PER and SMR use all

n

episodes. S is the number of successful episodes. Native counts visual-observation and GUI-input steps and Other all remaining model action steps (Code Execution CUA only). Cells: objectives with a valid episode.

Objective	S	LC	FGP	P	M	NGAR	$\|L_{o}\|$
GPT-6-Astra / Code Execution CUA
Bolt / easy	1	100.0	100.0	0	0	100.0	8
Bolt / hard	0	50.0	60.7	0	0	65.2	8
Color / easy	1	100.0	100.0	0	0	38.6	10
Color / hard	1	100.0	100.0	0	0	25.0	10
Maze / easy	1	100.0	100.0	0	0	92.0	10

Appendix

Table 18: Per-objective results for every agent and setting. S: objective solved; P: premature exit; M: setting mismatch. LC, FGP, and NGAR are percentages; NGAR applies to Code Execution CUA only (–).

|L_{o}|

: number of target levels.

Explore similar work

Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?
May 11, 2026Maximilian Triebel, Marco Menner, Dominik HelfensteinPuzzle Weak Visual Grounding
Institute of Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.
EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Jul 19, 2026Yaohan Yang, Minglei Shi, Borui Zhang +2
Department of Automation, Tsinghua University
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Jul 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +10Long-Horizon Agents Long-Horizon Task Planning
Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +5

May 11, 2026, cs.AIJ/K move · Enter open · S save

Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?

Maximilian Triebel, Marco Menner, Dominik Helfenstein

Institute of Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.

Puzzle Weak Visual Grounding Vision-Language-Action Framework

Open paper AskPDF arXiv

Vision-Language(-Action) Models (VLMs) are increasingly applied to interactive environments, yet existing benchmarks often overlook the complex physical reasoning required for point-and-click puzzle games. This paper introduces Vision-Language Against The Incredible Machine (VLATIM), a benchmark designed to evaluate human-like logical problem-solving capabilities within the classic physics puzzle game The Incredible Machine 2 (TIM). Unlike existing benchmarks, VLATIM specifically targets the critical gap between high-level logical reasoning and continuous action spaces requiring precise mouse interactions. This benchmark is structured into five progressive parts, assessing capabilities that range from basic visual grounding and domain understanding to multi-step manipulation and full puzzle solving. Our results reveal a significant disparity between reasoning and execution. While large proprietary models demonstrate superior planning abilities, they struggle with precise visual grounding. Consequently, they do not yet show human-like problem-solving capabilities.

Abstract

Explore similar work

Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading