Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning
Organizations: Navers Lab, Einsia.AI, Tsinghua University
Abstract
While language-model agents excel at long-horizon coding and terminal tasks, this success relies on two conditions: explicit text specifications and immediate execution feedback. However, neither condition holds in real-world problems where critical information exists only in images, leaving it unclear whether long-horizon capability transfers from text to visual reasoning. In such visual settings, because rules and clues resist text transcription and lack test-case feedback, agents must repeatedly inspect images, simulate spatial transformations, and autonomously cross-check clues. Despite this challenge, existing evaluations fail to capture such capability: visual reasoning benchmarks largely reduce tasks to single-turn, short-chain question answering, whereas agent benchmarks focus on text-based environments or interactive states that permit trial-and-error. To bridge this gap, we introduce Visual Cipher Bench, a collection of 40 original visual puzzles across four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding. Each puzzle provides only one to three black-and-white line drawings and a minimal submission instruction, requiring agents to discover the underlying solving mechanism entirely from the images. Across 29 reasoning-effort configurations of eight frontier models, even the strongest model, GPT-6 Astra (max), achieves only 26.7% accuracy, while five of the eight models score below 1%. Yet on a ten-task subset, providing a single non-spoiler mechanism hint boosts accuracy from 7.8% to 55.6%. This sharp contrast shows that the primary bottleneck of current agents lies not in executing multi-step plans, but in autonomously discovering rules from visual evidence. Visual Cipher Bench thus provides a rigorous benchmark for measuring long-horizon visual reasoning in agents.
Figures & tables
| 03 | + horn | + box |
|---|---|---|
| SHOE | SHOEHORN | SHOEBOX |
| POST | POSTHORN | POSTBOX |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
|---|---|
| Harness | Harbor v0.24.0 [ 75 ] |
| Agent runner | Codex CLI v0.163.0 for GPT-family models; Claude Code CLI v2.1.296 for all other models |
| Time limit | 60 minutes per run |
| Compute | 2 CPUs, 4 GB memory, no GPU |
| Environment | Network-disabled container with Python 3.12, NumPy, and Pillow; puzzle images are read-only |
| Runs | 40 tasks 3 runs 120 attempts per model–effort configuration |
Visual Reasoning
Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.
See all 169 papers