Organizations: Nanyang Technological University, Singapore. · the School of Cyber Security, University of Chinese Academy of Sciences, China. · the School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, Guangdong 518107, P.R. China. · Beihang University, Beijing, China. · the University of Leeds, Leeds, U.K.
Modern reasoning CAPTCHAs require fine-grained perception, compositional relation reasoning, and grounded browser interaction. End-to-end vision-language model agents remain brittle in this setting because a missed object, incorrect count, or confused relation can propagate directly to the final action. To address this problem, we present CaptchaAgent, a structured visual reasoning framework for interactive CAPTCHA solving. Instead of predicting an action directly from a screenshot, CaptchaAgent first constructs a visual intermediate representation (V-IR) that explicitly records entities, attributes, and spatial or topological relations. A symbol-only ASCII protocol further requires the parser to commit to a concrete object set and count. Building on this global representation, localized evidence refinement revisits ambiguous regions at higher effective resolution, while counterfactual consistency verification tests whether the resulting action remains stable under task-preserving transformations. Across three proprietary backbones on Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over a commercially available browser agent baseline evaluated with the same backbones. On the separate offline Oedipus-data benchmark, it further improves average Pass@1 by 9.2-12.3 percentage points over the Oedipus solver across three matched backbones. The intermediate traces show that exact counting, continuous geometry, and ordered execution remain major sources of automation difficulty for the evaluated agents, providing practical guidance for designing more resilient compositional CAPTCHAs.
Figures & tables
Fig. 1: Comparison of action-generation paradigms. Direct agents map observations to actions, chain-of-thought (CoT) agents introduce a free-form textual rationale, and CaptchaAgent instead constructs an explicit V-IR before action prediction. Unlike direct prediction or free-form CoT, the V-IR commits the agent to an explicit object set and relation structure that can be checked before execution.
Fig. 2: Overall framework of CaptchaAgent. At each step, the agent extracts the puzzle image, constructs a V-IR, refines ambiguous local evidence, and applies bounded consistency voting before selecting an action.
Fig. 3: Visual reasoning for a Click Order CAPTCHA. From left to right: the web GUI, a text-only CoT that captures object attributes and coarse object locations but may remain verbose or miss fine-grained spatial cues, and the proposed ASCII-box V-IR that retains the object count, spatial layout, key attributes, and relations.
Fig. 4: Visual reasoning for a Place Dot CAPTCHA. From left to right: the web GUI, a text-only CoT that captures object attributes and coarse object locations but may miss critical spatial relations and distances, and the proposed ASCII-box V-IR that retains the essential object count, spatial layout, attributes, and relations.
Family
Action
Acceptance rule
Main challenge
Family
Action
Acceptance rule
Main challenge
Dice Count
Click
Exact total/option
Counting
Object Match
Click
Exact option
Cardinality
Geometry Click
Click
Region or 25-px tol.
Grounding
Select Animal
Mark
Exact set
Recognition
Rotation Match
Drag
Exact angle mod. 360
Orientation
Coordinates
Click
Exact option
Coordinates
Slide Puzzle
Drag
Position, default 10 px
Translation
Path Finder
Click
Exact option
Route tracing
Unusual Detection
Mark
Exact set
Anomaly
Place Dot
Click
Endpoint, default 15 px
Localization
Image Recognition
Mark
Exact set
Recognition
Connect Icon
Click
Exact option
Relations
TABLE I: Action-centric taxonomy and answer checking rules for the 20 Open CaptchaWorld families. “Exact” denotes equality with the stored answer; tolerances are instance-specific when supplied by the benchmark.
Agent
MLLM backbone
Pass@1 (%)
Model-call expenditure ($)
BU-Agent [ 13 ]
OpenAI-o1 [ 63 ]
5.0
94.6
GPT-4o [ 64 ]
5.7
25.8
OpenAI-o3 [ 65 ]
40.0
66.4
Claude-3.5-Haiku [ 66 ]
15.0
9.3
Claude-3.5-Sonnet [ 67 ]
10.0
21.9
DeepSeek-V3 [ 68 ]
20.0
7.3
TABLE II: Open CaptchaWorld results. Expenditure covers recorded model calls only, not detector, browser, network, or local-compute overhead. Legacy BU-Agent rows are reported by Open CaptchaWorld [ 12 ] ; shared-backbone comparisons are author runs.
Fig. 5: Qualitative comparison on misleading click CAPTCHA puzzles: the text-only BU agent [ 13 ] describes the scene but clicks the default region; CaptchaAgent enumerates key layout cues and refines the click direction with localized evidence refinement (LER) and counterfactual consistency verification (CCF) to hit the correct target.
Agent
Backbone
Angular
Gobang
Space
Space Reasoning
Average
Oedipus [ 11 ]
GPT-4V [ 69 ]
37.90
79.70
71.30
65.00
63.50
Gemini [ 70 ]
31.00
56.30
59.50
58.00
51.20
miniGPT-4 [ 19 ]
7.50
16.90
18.00
15.10
14.40
VTT [ 54 ]
✗
✗
55.70
44.50
25.10
Oedipus [ 11 ]
Gemini-2.0 [ 71 ]
35.10
79.10
82.70
62.90
65.00
CaptchaAgent
Gemini-2.0 [ 71 ]
44.00
96.00
100.00
68.00
77.00 (+ 12.0 )
TABLE III: Pass@1 (%) on Oedipus-data [ 11 ] . Oedipus rows are values reported by the released solver evaluation; CaptchaAgent rows are author runs under its offline protocol.
ASCII-based V-IR
Localized Refinement
Counterfactual Consistency
Pass@1 [%]
✓
✓
29.2
✓
✓
36.8
✓
✓
34.8
✓
✓
✓
46.8
TABLE IV: Ablation on Open CaptchaWorld [ 12 ] using Gemini-2.5 Pro [ 61 ] : impact of ASCII-based V-IR, localized refinement, and counterfactual consistency on Pass@1.
Fig. 6: Structured trace for a Connect Icon instance. The reference region and two candidate regions are paired with their ASCII V-IRs, which expose the icon inventories, spatial layouts, and directed connections before action selection.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Action
Success criterion
Main challenge
Dice Count
Click
exact option or total
count top faces correctly
Geometry Click
Click
inside valid region or tolerance
geometric grounding
Rotation Match
Drag
exact angle modulo 360
orientation alignment
Slide Puzzle
Drag
final position within tolerance
translation estimation
Unusual Detection
Mark
exact selected set
anomaly detection
Image Recognition
Mark
exact selected set
category recognition
Appendix
TABLE I: Compact taxonomy of the 20 Open CaptchaWorld CAPTCHA families.
Fig. 1: A full-page screenshot of a Connect Icon puzzle in Open CaptchaWorld. The webpage contains the textual instruction, the left-side reference icon set, and the right-side candidate regions from which CaptchaAgent must identify the matching option.
Fig. 2: Local crops from the same Connect Icon instance. The left crop is the reference icon set to be matched. The two right crops are candidate regions containing both matching icons and distractor icons, making the task depend on explicit icon inventory rather than coarse global similarity alone.
CAPTCHAs are widely deployed as human verification mechanisms and frequently block intelligent agents from completing end-to-end automation in real-world web environments. Solving modern CAPTCHAs requires robust multi-step visual reasoning and interaction capabilities, yet training-based approaches have remained absent due to the lack of large-scale training data and process-level annotations. We introduce CaptchaBench, the first CAPTCHA benchmark designed to support large-scale training, comprising 16,000 programmatically generated samples across eight task categories with detailed region and process-level annotations. Systematic evaluation on CaptchaBench reveals that existing methods fail consistently on tasks requiring fine-grained visual detail capture and region-level comparison. We therefore present CaptchaMind, an RL-based solver trained with explicit reasoning process supervision, achieving 82.9% average success rate across eight tasks and 71.0% on real-world instances, substantially outperforming all existing methods without closed-source APIs.
Pengcheng Wang, Haoxiang Liu, Yang Dai +5
1Alibaba Group · 2Southern University of Science and Technology · 3Harbin Institute of Technology (Shenzhen)
GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base agents, while maintaining strong performance on general GUI-agent benchmarks.
Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.
Zhenhao Zhang, Zhaoyu Fan, Haohan Ying +5
Columbia University · Zhejiang University · University of Rochester +1