While language-model agents excel at long-horizon coding and terminal tasks, this success relies on two conditions: explicit text specifications and immediate execution feedback. However, neither condition holds in real-world problems where critical information exists only in images, leaving it unclear whether long-horizon capability transfers from text to visual reasoning. In such visual settings, because rules and clues resist text transcription and lack test-case feedback, agents must repeatedly inspect images, simulate spatial transformations, and autonomously cross-check clues. Despite this challenge, existing evaluations fail to capture such capability: visual reasoning benchmarks largely reduce tasks to single-turn, short-chain question answering, whereas agent benchmarks focus on text-based environments or interactive states that permit trial-and-error. To bridge this gap, we introduce Visual Cipher Bench, a collection of 40 original visual puzzles across four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding. Each puzzle provides only one to three black-and-white line drawings and a minimal submission instruction, requiring agents to discover the underlying solving mechanism entirely from the images. Across 29 reasoning-effort configurations of eight frontier models, even the strongest model, GPT-6 Astra (max), achieves only 26.7% accuracy, while five of the eight models score below 1%. Yet on a ten-task subset, providing a single non-spoiler mechanism hint boosts accuracy from 7.8% to 55.6%. This sharp contrast shows that the primary bottleneck of current agents lies not in executing multi-step plans, but in autonomously discovering rules from visual evidence. Visual Cipher Bench thus provides a rigorous benchmark for measuring long-horizon visual reasoning in agents.
Figures & tables
Figure 1: Overview of Visual Cipher Bench. Left: 40 original visual puzzles (67 images in total), divided by primary difficulty into spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding, with 10 puzzles each. Middle: mean accuracy of each model’s best reasoning-effort configuration over three runs. Right: the evaluation pipeline.
Table 1: Results of all 29 configurations on Visual Cipher Bench, with models ordered by their best configuration. P1–P4 give the category accuracy over the 30 attempts in each category, Acc. gives the mean accuracy over all 120 attempts, and Mean gives the corresponding number of tasks solved per run (out of 40). All values average three runs; the best value in each column is in bold. Each model’s best configuration is shaded.
Table 2: Mechanism-hint ablation: accuracy (%) on the ten selected tasks. The upper block reports each model; the lower block pools the three models by category. Each model runs each task three times.
03
+ horn
+ box
SHOE
SHOEHORN
SHOEBOX
POST
POSTHORN
POSTBOX
Figure 2: Solving Closed Stock (knowledge-based decoding). (a) Seal 03 points to a horn on page I and to a box on page II; both SHOE and POST satisfy these two local clues. (b) Seal 10 takes part in five compounds, and only STONE satisfies all of them, fixing seals 10, 04, and 08. (c) RETURN requires the 15 words to form a closed loop of compounds; only one such loop exists, and it also rules out POST. Reading the seal numbers from 01 gives the answer. Icons are taken from the puzzle; the full puzzle is shown in Appendix B .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Harness
Harbor v0.24.0 [ 75 ]
Agent runner
Codex CLI v0.163.0 for GPT-family models; Claude Code CLI v2.1.296 for all other models
Time limit
60 minutes per run
Compute
2 CPUs, 4 GB memory, no GPU
Environment
Network-disabled container with Python 3.12, NumPy, and Pillow; puzzle images are read-only
GUI agents need long-horizon visual reasoning: they must interpret a changing interface while keeping a multi-step plan viable as earlier actions constrain later ones. Existing benchmarks evaluate grounding, computer use, and game play, but rarely test whether agents stay coherent across long chains of coupled decisions. Long-horizon visual puzzles expose this capability directly: a legal move that looks like progress can make the puzzle unsolvable, and the loss shows only several moves later. We introduce LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced. With Native GUI Actions alone, the strongest agents solve most objectives, but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve nothing harder than Medium, and none completes Bolt Unscrew Hard, which a human solves along with every other objective. Code Execution CUA does not close this gap, and its scores mix visual solving with algorithmic search. Controlled diagnostics trace these failures to one limitation that neither rules, state hints, nor failure memory removes: agents judge each move by the visible progress it makes, not by the future options it leaves.
Bingo Zhang, Haochuan Lu, Zongjie Li +3
Vera Praxis · Tencent · The Hong Kong University of Science and Technology +2
This paper argues that a systemic lack of Agency constrains the implicit reasoning capabilities of current Vision-Language Models (VLMs). Implicit reasoning refers to the ability to autonomously discover and utilize hidden visual evidence to bridge information gaps, rather than merely relying on explicitly specified targets. This capacity underlies human visual understanding and everyday reasoning. We argue that this limitation arises from a tendency to approach visual reasoning primarily as passive semantic retrieval, rather than as active, situated reasoning that depends on autonomous visual exploration. As a result, most existing benchmarks primarily assess Passive Capacity, leaving this aspect of reasoning largely unmeasured. To address this gap, we introduce the Visual Implicit Reasoning Diagnosing Benchmark (V-IRD), which targets this missing quadrant by requiring models to derive answers strictly through autonomous visual analysis. Our results show that, despite strong retrieval abilities, prominent VLMs struggle to utilize reference objects and to attend to visual evidence that requires self-directed inquiry. Simply put, strong semantic recognition does not equate to active visual exploration, revealing a critical gap in current VLMs. More information can be found at https://haoychen.github.io/Implicit-Reasoning/
Yizhao Huang, Haoyang Chen, Shiqin Wang +6
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. +2
Large Vision-Language Models (LVLMs) have achieved remarkable proficiency in explicit visual recognition, effectively describing what is directly visible in an image. However, a critical cognitive gap emerges when the visual input serves only as a clue rather than the answer. We identify that current models struggle with the complex, multi-step reasoning required to solve problems where information is not explicitly depicted. Successfully solving a rebus puzzle requires a distinct cognitive workflow: the model must extract visual and textual attributes, retrieve linguistic prior knowledge (such as idioms), and perform abstract mapping to synthesize these elements into a meaning that exists outside the pixel space. To evaluate this neurosymbolic capability, we introduce RebusBench, a benchmark of 1,164 puzzles designed to test this specific integration of perception and knowledge. Our evaluation of state-of-the-art models (including Qwen, InternVL, and LLaVA) shows a severe deficiency: performance saturates below 10% Exact Match and 20% semantic accuracy, with no significant improvement observed from model scaling or In-Context Learning (ICL). These findings suggest that while models possess the necessary visual and linguistic components, they lack the cognitive reasoning glue to connect them. Project page available at https://amirkasaei.com/rebusbench/.
Seyed Amir Kasaei, Arash Marioriyad, Mahbod Khaleti +3
Department of Computer Engineering Sharif University of Technology