CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs

Zhenhao Zhang; Zhaoyu Fan; Haohan Ying; Jingwen Hu; Hancen Fan; Junhao Zhou; Zitian Chen; Linchao Zhu

cs.CVSep 25, 2026

CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs

Authors: Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu

Organizations: Columbia University · Zhejiang University · University of Rochester · University of Illinois at Urbana-Champaign

Abstract

Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.

Figures & tables

Figure 1: Representative puzzles from the $20$ CAPTCHA types in CaptchaArena , cropped for display. Border colors mark interaction modes and corner icons mark cognitive skills.

Benchmark	Types	CU	Traj.	Reasoning	RL reward	Click Eval.	Scale
Evaluation-only benchmarks
Open CaptchaWorld	$20$	✗	✗	✗	–	–	$463^{\dagger}$
Halligan	$26$	✓	✗	✗	–	tol.	$2{,}600$
Oedipus	$4$	✗	✗	✗	–	–	$400$
HLL	$10$	✓	✗	✗	–	tol.	–
CAPTCHA-X	$7$	✓	✗	✓	–	region	$1{,}839$

Table 1: Comparison of CAPTCHA benchmarks and training datasets, as reported by each paper.

Figure 2: Pipeline from puzzle construction and verification to reasoning annotation, training, and evaluation.

Task Type	Train	Val	Test	Task Type	Train	Val	Test
Geometry Click	$2{,}100$	$200$	$200$	Connect Icon	$2{,}100$	$200$	$200$
Misleading Click	$2{,}100$	$200$	$200$	Coordinates	$2{,}100$	$200$	$200$
Pick Area	$2{,}100$	$200$	$200$	Dart Count	$2{,}100$	$200$	$200$
Place Dot	$2{,}100$	$200$	$200$	Image Matching	$2{,}100$	$200$	$200$
Select Animal	$2{,}100$	$200$	$200$	Object Match	$2{,}100$	$200$	$200$
Bingo	$2{,}100$	$200$	$200$	Path Finder	$2{,}100$	$200$	$200$

Table 2: Dataset statistics for each CAPTCHA type.

	Untrained Base	GUI agents		Closed-source			CaptchaAgent
Type	Qwen3.5-9B	Venus2-9B	GUI-Owl-8B	GPT-5.4	Gemini	Claude	SFT	RL	Human
Geometry Click	0.0	93.5	89.5	92.0	79.5	89.5	68.2	68.2	96.5
Misleading Click	93.5	25.5	82.0	23.0	87.0	83.0	99.1	98.9	100.0
Pick Area	35.5	23.0	33.0	37.5	37.5	36.0	59.8	60.6	97.5
Place Dot	0.0	9.5	22.5	88.5	63.0	15.5	16.1	20.1	100.0
Select Animal	0.0	28.5	99.5	98.5	90.5	100.0	99.3	99.4	98.0

Table 3: Per-type Pass@1 on the full test split, with

200

puzzles per type. Bold marks the best model result in each row; the human reference is excluded from this comparison.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

#	Task	Mode	Interaction	Skills	Action steps	Grading
1	Geometry Click	S	single-click	G	★	mask hit
2	Misleading Click	S	single-click		★	mask miss
3	Pick Area	S	single-click		★	mask hit
4	Place Dot	S	single-click	G	★★	distance tol.
5	Select Animal	S	single-click, $2{\times}3$ grid		★★	target cell
6	Bingo	M	multi-click, $3{\times}3$ grid	L	★★★	any valid swap

Appendix

Table 4: The

20

released CaptchaArena tasks grouped by interaction mode. Mode : S single-click, M multi-click, A arrow-cycle, R real-time, T text-entry. Skills : G grounding, L logic, M math. Action steps gives the mean number of agent actions in the reference trajectory, rounded to one star per action and capped at five.

Table 5: Per-type Pass@

k

breakdown for the SFT and RL policies on the test split.

	Qwen3.5			Open-weight GUI agents
Type	9B	27B	35B	UI-TARS	GUI-Owl	Venus2	Holo-3.1	EvoCUA	Fara1.5
Geometry Click	0.0	0.0	1.0	85.5	89.5	93.5	78.5	92.5	90.0
Misleading Click	93.5	93.5	91.5	74.0	82.0	25.5	58.5	39.5	66.5
Pick Area	35.5	38.5	34.5	33.0	33.0	23.0	27.5	29.5	30.0
Place Dot	0.0	0.0	0.0	8.5	22.5	9.5	5.0	23.0	16.5
Select Animal	0.0	0.0	0.0	93.0	99.5	28.5	80.0	99.5	99.5

Appendix

Table 6: Per-type Pass@1 for open-weight systems on the full test split. The systems are Qwen3.5 ( Qwen Team, 2026 ) , UI-TARS-1.5-7B ( Qin et al., 2025 ; ByteDance Seed, 2025 ) , GUI-Owl-1.5-8B ( Xu et al., 2026 ) , UI-Venus-2-9B ( Venus Team et al., 2026 ) , Holo-3.1-9B ( H Company, 2026 ) , EvoCUA-8B ( Xue et al., 2026 ) and Fara1.5-9B ( Awadallah et al., 2026 ) .

	Qwen3.5			Open-weight GUI agents
Type	9B	27B	35B	UI-TARS	GUI-Owl	Venus2	Holo-3.1	EvoCUA	Fara1.5
Geometry Click	100.0	99.5	99.0	100.0	100.0	100.0	99.5	99.0	100.0
Misleading Click	99.5	100.0	100.0	100.0	100.0	100.0	97.0	100.0	100.0
Pick Area	100.0	100.0	100.0	99.5	100.0	100.0	94.0	100.0	100.0
Place Dot	0.5	0.0	0.0	67.5	100.0	52.5	56.5	100.0	98.0
Select Animal	0.0	0.0	0.0	95.0	99.5	29.0	80.0	100.0	100.0

Appendix

Table 7: Submit rate (%) for open-weight systems. A rollout that never submits is scored as a failure, so the submit rate places an upper bound on Pass@1. Pass@1 for the same systems is in Table 6 .

	Closed-source models			CaptchaAgent
Type	GPT-5.4	Gemini	Claude	SFT	RL
Geometry Click	100.0	100.0	100.0	99.9	99.9
Misleading Click	100.0	99.0	100.0	99.7	99.6
Pick Area	99.0	98.0	100.0	100.0	100.0
Place Dot	100.0	67.5	100.0	95.3	99.8
Select Animal	100.0	93.0	100.0	99.9	99.8

Appendix

Table 8: Submit rate (%) for the closed-source models and CaptchaAgent . Closed-source models use

1

rollout per puzzle, while SFT and RL use

5

		Human		Pass@1		$\Delta$
Mode	Type	Acc.	Time (s)	SFT	RL	SFT	RL
Single-click	Geometry Click	96.5	6.1	68.2	68.2	$-28.3$	$-28.3$
	Misleading Click	100.0	3.3	99.1	98.9	$-0.9$	$-1.1$
	Pick Area	97.5	2.4	59.8	60.6	$-37.7$	$-36.9$
	Place Dot	100.0	4.8	16.1	20.1	$-83.9$	$-79.9$
	Select Animal	98.0	5.7	99.3	99.4	$+1.3$	$+1.4$

Appendix

Table 9: Human reference on

20

types at

200

puzzles each, against both policies.

	Open CaptchaWorld			Halligan
Model	Pass@1	Submit	Steps	Pass@1	Submit	Steps
Qwen3.5-9B	9.8	21.1	9.5	0.9	12.6	12.7
UI-TARS-1.5-7B	14.0	53.8	9.1	0.9	13.2	12.7
SFT (§ 4.2 )	47.2	90.6	4.2	13.6	53.4	8.7
RL (§ 4.3 )	51.0	94.0	3.9	20.0	66.9	7.2

Appendix

Table 10: Generalization to two external evaluation benchmarks. Best results for each metric are shown in bold.

Table 11: Per-type Pass@1 on the two external evaluation benchmarks.

Type	mask/image	rect/image	mask/rect (mean)	mask/rect (min–max)	uniform-in-rect
Geometry Click	0.8%	1.1%	0.78	0.57–0.97	78.3
Pick Area	38.8%	65.1%	0.62	0.37–0.89	61.6

Appendix

Table 12: How much of a target’s bounding rectangle belongs to the target. Mask and rectangle areas are averaged over the

200

test puzzles of each type. The final column gives the accuracy of a click drawn uniformly inside the rectangle under the mask verifier.

System	$n$	Verifier	Rect	$\Delta$	Verifier	Rect	$\Delta$
		Geometry Click			Pick Area
Qwen3.5-9B (untrained)	200	0.0	0.0	$+0.0$	35.5	71.5	$+36.0$
UI-TARS-1.5-7B	200	85.5	90.5	$+5.0$	33.0	77.5	$+44.5$
SFT	1000	68.2	76.2	$+8.0$	59.8	81.6	$+21.8$
RL	1000	68.2	73.4	$+5.2$	60.6	82.3	$+21.7$
uniform click in rect		78.3	100.0		61.6	100.0

Appendix

Table 13: The same clicks evaluated under mask and bounding-rectangle grading.

Panel of Figure 3 — Figure 3: Two UI-TARS clicks accepted by bounding-rectangle grading but rejected by the pixel-mask verifier. CaptchaAgent (RL) click on the same puzzle is shown in blue. Green denotes the verifier mask and orange its tight bounding rectangle.

Puzzle	System	Steps	Actions	Outcome
Hold Button	CaptchaAgent	1	hold(640,408, $\Delta t$ )	correct
Hold Button	UI-TARS	15	$14\times$ click on and around the button; $1\times$ drag	step cap
Slide Puzzle	CaptchaAgent	3	drag(325 $\to$ 394); drag(394 $\to$ 463) at $y{=}666$ ; submit	correct
Slide Puzzle	UI-TARS	12	$3\times$ click on the handle; $9\times$ drag of $6$ – $18$ px	no submission
Rotation Match	CaptchaAgent	6	click(720,847) $\times 5$ (right arrow); submit	correct
Rotation Match	UI-TARS	7	left, left, right, left, right, right, right	no submission

Appendix

Table 14: Action traces behind Figure 4 . Coordinates are viewport pixels;

\times k

denotes repetition of the same action. Step cap means that the episode reaches the step limit without submission.

Configuration	Value
Model
Base model	Qwen3.5-9B
Adapters	rank- $64$ LoRA, $\alpha=128$ , dropout $0.05$
LoRA target modules	$q$ , $k$ , $v$ , $o$ , gate, up, down
Frozen	vision encoder, vision–language aligner
Data

Appendix

Table 15: Supervised fine-tuning configuration.

Configuration	Value
Model
Base model	supervised checkpoint from § 4.2
Trainable	full language-model weights
Frozen	vision encoder, vision–language aligner
Precision	bf16, FSDP with parameter and optimizer offload
Algorithm

Appendix

Table 16: Reinforcement learning configuration.

CAPTCHA Solving

See all 8 papers

May 19, 2026

CaptchaMind: Training CAPTCHA Solvers via Reinforcement Learning with Explicit Reasoning Supervision

CAPTCHAs are widely deployed as human verification mechanisms and frequently block intelligent agents from completing end-to-end automation in real-world web environments. Solving modern CAPTCHAs requires robust multi-step visual reasoning and interaction capabilities, yet training-based approaches have remained absent due to the lack of large-scale training data and process-level annotations. We introduce CaptchaBench, the first CAPTCHA benchmark designed to support large-scale training, comprising 16,000 programmatically generated samples across eight task categories with detailed region and process-level annotations. Systematic evaluation on CaptchaBench reveals that existing methods fail consistently on tasks requiring fine-grained visual detail capture and region-level comparison. We therefore present CaptchaMind, an RL-based solver trained with explicit reasoning process supervision, achieving 82.9% average success rate across eight tasks and 71.0% on real-world instances, substantially outperforming all existing methods without closed-source APIs.

Pengcheng Wang, Haoxiang Liu, Yang Dai +5

1Alibaba Group · 2Southern University of Science and Technology · 3Harbin Institute of Technology (Shenzhen)

CAPTCHA Solving Visual Reasoning

Ask PDF

Oct 9, 2026

CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving

Modern reasoning CAPTCHAs require fine-grained perception, compositional relation reasoning, and grounded browser interaction. End-to-end vision-language model agents remain brittle in this setting because a missed object, incorrect count, or confused relation can propagate directly to the final action. To address this problem, we present CaptchaAgent, a structured visual reasoning framework for interactive CAPTCHA solving. Instead of predicting an action directly from a screenshot, CaptchaAgent first constructs a visual intermediate representation (V-IR) that explicitly records entities, attributes, and spatial or topological relations. A symbol-only ASCII protocol further requires the parser to commit to a concrete object set and count. Building on this global representation, localized evidence refinement revisits ambiguous regions at higher effective resolution, while counterfactual consistency verification tests whether the resulting action remains stable under task-preserving transformations. Across three proprietary backbones on Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over a commercially available browser agent baseline evaluated with the same backbones. On the separate offline Oedipus-data benchmark, it further improves average Pass@1 by 9.2-12.3 percentage points over the Oedipus solver across three matched backbones. The intermediate traces show that exact counting, continuous geometry, and ordered execution remain major sources of automation difficulty for the evaluated agents, providing practical guidance for designing more resilient compositional CAPTCHAs.

Shangquan Sun, Ruoyu Chen, Zhirui Liu +4

Nanyang Technological University, Singapore. · the School of Cyber Security, University of Chinese Academy of Sciences, China. · the School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, Guangdong 518107, P.R. China. +2

CAPTCHA Solving Tool-Using Vision-Language Agents Visually Grounded Reasoning Visual Spatial Reasoning

Ask PDF

Mar 23, 2026

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base agents, while maintaining strong performance on general GUI-agent benchmarks.

Yuxi Chen, Haoyu Zhai, Chenkai Wang +4

University of Illinois Urbana-Champaign.

CAPTCHA Solving Synthetic Data Generation GUI Agents

Ask PDF

Abstract

CAPTCHA Solving

Explore similar work

CaptchaMind: Training CAPTCHA Solvers via Reinforcement Learning with Explicit Reasoning Supervision

CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training