Organizations: Nanyang Technological University, Singapore. · the School of Cyber Security, University of Chinese Academy of Sciences, China. · the School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, Guangdong 518107, P.R. China. · Beihang University, Beijing, China. · the University of Leeds, Leeds, U.K.
Modern reasoning CAPTCHAs require fine-grained perception, compositional relation reasoning, and grounded browser interaction. End-to-end vision-language model agents remain brittle in this setting because a missed object, incorrect count, or confused relation can propagate directly to the final action. To address this problem, we present CaptchaAgent, a structured visual reasoning framework for interactive CAPTCHA solving. Instead of predicting an action directly from a screenshot, CaptchaAgent first constructs a visual intermediate representation (V-IR) that explicitly records entities, attributes, and spatial or topological relations. A symbol-only ASCII protocol further requires the parser to commit to a concrete object set and count. Building on this global representation, localized evidence refinement revisits ambiguous regions at higher effective resolution, while counterfactual consistency verification tests whether the resulting action remains stable under task-preserving transformations. Across three proprietary backbones on Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over a commercially available browser agent baseline evaluated with the same backbones. On the separate offline Oedipus-data benchmark, it further improves average Pass@1 by 9.2-12.3 percentage points over the Oedipus solver across three matched backbones. The intermediate traces show that exact counting, continuous geometry, and ordered execution remain major sources of automation difficulty for the evaluated agents, providing practical guidance for designing more resilient compositional CAPTCHAs.
Figures & tables
Fig. 1: Comparison of action-generation paradigms. Direct agents map observations to actions, chain-of-thought (CoT) agents introduce a free-form textual rationale, and CaptchaAgent instead constructs an explicit V-IR before action prediction. Unlike direct prediction or free-form CoT, the V-IR commits the agent to an explicit object set and relation structure that can be checked before execution.
Fig. 2: Overall framework of CaptchaAgent. At each step, the agent extracts the puzzle image, constructs a V-IR, refines ambiguous local evidence, and applies bounded consistency voting before selecting an action.
Fig. 3: Visual reasoning for a Click Order CAPTCHA. From left to right: the web GUI, a text-only CoT that captures object attributes and coarse object locations but may remain verbose or miss fine-grained spatial cues, and the proposed ASCII-box V-IR that retains the object count, spatial layout, key attributes, and relations.
Fig. 4: Visual reasoning for a Place Dot CAPTCHA. From left to right: the web GUI, a text-only CoT that captures object attributes and coarse object locations but may miss critical spatial relations and distances, and the proposed ASCII-box V-IR that retains the essential object count, spatial layout, attributes, and relations.
Family
Action
Acceptance rule
Main challenge
Family
Action
Acceptance rule
Main challenge
Dice Count
Click
Exact total/option
Counting
Object Match
Click
Exact option
Cardinality
Geometry Click
Click
Region or 25-px tol.
Grounding
Select Animal
Mark
Exact set
Recognition
Rotation Match
Drag
Exact angle mod. 360
Orientation
Coordinates
Click
Exact option
Coordinates
Slide Puzzle
Drag
Position, default 10 px
Translation
Path Finder
Click
Exact option
Route tracing
Unusual Detection
Mark
Exact set
Anomaly
Place Dot
Click
Endpoint, default 15 px
Localization
Image Recognition
Mark
Exact set
Recognition
Connect Icon
Click
Exact option
Relations
TABLE I: Action-centric taxonomy and answer checking rules for the 20 Open CaptchaWorld families. “Exact” denotes equality with the stored answer; tolerances are instance-specific when supplied by the benchmark.
Agent
MLLM backbone
Pass@1 (%)
Model-call expenditure ($)
BU-Agent [ 13 ]
OpenAI-o1 [ 63 ]
5.0
94.6
GPT-4o [ 64 ]
5.7
25.8
OpenAI-o3 [ 65 ]
40.0
66.4
Claude-3.5-Haiku [ 66 ]
15.0
9.3
Claude-3.5-Sonnet [ 67 ]
10.0
21.9
DeepSeek-V3 [ 68 ]
20.0
7.3
TABLE II: Open CaptchaWorld results. Expenditure covers recorded model calls only, not detector, browser, network, or local-compute overhead. Legacy BU-Agent rows are reported by Open CaptchaWorld [ 12 ] ; shared-backbone comparisons are author runs.
Fig. 5: Qualitative comparison on misleading click CAPTCHA puzzles: the text-only BU agent [ 13 ] describes the scene but clicks the default region; CaptchaAgent enumerates key layout cues and refines the click direction with localized evidence refinement (LER) and counterfactual consistency verification (CCF) to hit the correct target.
Agent
Backbone
Angular
Gobang
Space
Space Reasoning
Average
Oedipus [ 11 ]
GPT-4V [ 69 ]
37.90
79.70
71.30
65.00
63.50
Gemini [ 70 ]
31.00
56.30
59.50
58.00
51.20
miniGPT-4 [ 19 ]
7.50
16.90
18.00
15.10
14.40
VTT [ 54 ]
✗
✗
55.70
44.50
25.10
Oedipus [ 11 ]
Gemini-2.0 [ 71 ]
35.10
79.10
82.70
62.90
65.00
CaptchaAgent
Gemini-2.0 [ 71 ]
44.00
96.00
100.00
68.00
77.00 (+ 12.0 )
TABLE III: Pass@1 (%) on Oedipus-data [ 11 ] . Oedipus rows are values reported by the released solver evaluation; CaptchaAgent rows are author runs under its offline protocol.
ASCII-based V-IR
Localized Refinement
Counterfactual Consistency
Pass@1 [%]
✓
✓
29.2
✓
✓
36.8
✓
✓
34.8
✓
✓
✓
46.8
TABLE IV: Ablation on Open CaptchaWorld [ 12 ] using Gemini-2.5 Pro [ 61 ] : impact of ASCII-based V-IR, localized refinement, and counterfactual consistency on Pass@1.
Fig. 6: Structured trace for a Connect Icon instance. The reference region and two candidate regions are paired with their ASCII V-IRs, which expose the icon inventories, spatial layouts, and directed connections before action selection.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Action
Success criterion
Main challenge
Dice Count
Click
exact option or total
count top faces correctly
Geometry Click
Click
inside valid region or tolerance
geometric grounding
Rotation Match
Drag
exact angle modulo 360
orientation alignment
Slide Puzzle
Drag
final position within tolerance
translation estimation
Unusual Detection
Mark
exact selected set
anomaly detection
Image Recognition
Mark
exact selected set
category recognition
Appendix
TABLE I: Compact taxonomy of the 20 Open CaptchaWorld CAPTCHA families.
Fig. 1: A full-page screenshot of a Connect Icon puzzle in Open CaptchaWorld. The webpage contains the textual instruction, the left-side reference icon set, and the right-side candidate regions from which CaptchaAgent must identify the matching option.
Fig. 2: Local crops from the same Connect Icon instance. The left crop is the reference icon set to be matched. The two right crops are candidate regions containing both matching icons and distractor icons, making the task depend on explicit icon inventory rather than coarse global similarity alone.