cs.AIOct 23, 2026
SaveCorrect Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier
Organizations: Shanghai Jiao Tong University · The University of Sydney · Shandong University
Abstract
Websites and applications deploy CAPTCHAs and anti-automation mechanisms that use behavioral signals such as interaction trajectories and timing, not outcomes alone, to distinguish humans from automated agents. We call these protected GUI environments, where correctly solving a verification task is insufficient for end-to-end task completion. Agents must turn solver outputs into accepted interactions and adapt execution to runtime feedback under time constraints. We introduce SolveBind, a GUI-agent framework combining hierarchical solving with runtime supervision. A hierarchical solving tree decomposes verification into interdependent Perceive, Ground, Solve, Act, and Verify capabilities. A runtime supervisor grounds action commitments in tool-derived evidence, revises them as evidence arrives, and mediates execution under explicit recovery budgets. Across nine CAPTCHA task families and nine backbone models, SolveBind achieves the highest success rate on eight backbones, outperforming the strongest baseline by 13.3-34.5 percentage points. It achieves the highest success rate on all six AndroidWorld backbones. In a paired GPT-5.5 analysis, MobileAgent produces correct answers but fails dynamic verification on 13 of 90 tasks; SolveBind succeeds on seven of these tasks. With automatic submission, removing binding reduces success from 82.22% to 71.56% and increases latency and token usage. Recovery caps reduce execution cost with a small observed success trade-off: removing them raises success from 82.22% to 83.56%, while increasing latency and token usage to 3.7x and 2.4x their full-supervision values, respectively. These results support evidence-grounded execution control while identifying a success-cost trade-off in recovery and failure boundaries of heuristic submission rules.
Figures & tables
Figure 1: Human verification as a bottleneck in a GUI workflow. Top: a verification gate blocks an invoice-download task. Bottom left: a correct slider position is still rejected. Bottom right: the detector scores trajectory and timing, so human-unlike motion is blocked even at the correct target.
Figure 2: Overview of SolveBind. (1) – (5) illustrate the protected GUI environment, hierarchical solving tree, model, runtime supervisor, and task adapter. The runtime supervisor mediates execution so that only dispatched operations are passed to external tools or the GUI. The bottom panel shows a representative execution cycle for category-guided image selection.
| SolveBind | MobileAgent | UI-TARS Harness | AppAgent | |||||||||
| Backbone | SR | E2E | Tok | SR | E2E | Tok | SR | E2E | Tok | SR | E2E | Tok |
| Policy backbones | ||||||||||||
| GPT-5.5 | 80.0 | 112.0 | 37.80 | 66.7 | 167.3 | 23.82 | 43.3 | 189.3 | 59.30 | 26.7 | 207.8 | 33.66 |
| GPT-5.4 | 46.7 | 40.6 | 74.93 | 11.1 | 162.9 | 37.37 | 15.6 | 91.5 | 73.01 | 6.7 | 195.1 | 56.34 |
| Claude Opus 4.8 | 75.6 | 85.4 | 77.54 | 61.1 | 208.9 | 52.72 | 23.3 | 183.7 | 81.54 | 24.4 | 211.2 | 81.66 |
| Claude Opus 4.6 | 34.4 | 185.2 | 47.76 | 38.9 | 202.2 | 30.75 | 10.0 | 265.7 | 77.38 | 3.3 | 230.6 | 23.43 |
Table 1: Dynamic-verification performance across backbones and agent policies. Each policy reports SR (%, higher is better), E2E (s, lower is better), and Tok (k, lower is better). Bold and underline mark the best and second-best SR per backbone; “–” denotes unavailable results.
| SolveBind | MobileAgent | UI-TARS Harness | AppAgent | |||||||||
| Backbone | SR | E2E | Tok | SR | E2E | Tok | SR | E2E | Tok | SR | E2E | Tok |
| General foundation models | ||||||||||||
| GPT-5.5 | 72.4 | 196.88 | 100.86 | 40.52 | 838.03 | 173.47 | 37.07 | 179.54 | 29.38 | 35.3 | 282.2 | 45.74 |
| Claude Opus 4.8 | 62.1 | 189.43 | 135.59 | 40.52 | 806.0 | 256.18 | 34.48 | 475.13 | 44.58 | 26.7 | 159.1 | 29.17 |
| Gemini 3.1 Pro | 61.2 | 272.96 | 83.67 | 31.03 | 1008.0 | 160.94 | 27.59 | 332.37 | 23.54 | 25.9 | 282.3 | 29.72 |
| GLM-5v-turbo | 54.3 | 274.27 | 134.37 | 6.03 | 1372.2 | 310.90 | 15.52 | 289.05 | 63.44 | 25.0 | 349.8 | 53.48 |
Table 2: AndroidWorld performance across backbones and agent policies. Each policy reports SR (%, higher is better), E2E (s, lower is better), and Tok (k, lower is better). Bold and underline mark the best and second-best SR per backbone.
| Runtime configuration | SR (%) | E2E (s) | Tok (k) | |||
| Full supervision | 82.22 | 112.13 | 41.82 | |||
| No complete mediation | 72.89 | 9.33 | 123.92 | 11.79 | 48.09 | 6.27 |
| No fail-safe defaults | 75.11 | 7.11 | 158.13 | 46.00 | 60.64 | 18.82 |
| No binding | 71.56 | 10.67 | 128.71 | 16.58 | 49.32 | 7.50 |
| Relaxed supervisor | 68.44 | 13.78 | 119.97 | 7.84 | 46.84 | 5.02 |
| Unbounded recovery | 83.56 | 1.33 | 417.45 | 305.32 | 101.40 | 59.58 |
Table 3: Runtime-control ablation over 225 episodes per configuration. Each metric reports its value and, in gray, the change relative to full supervision. SR differences are in percentage points; E2E and token differences use seconds and thousands of tokens, respectively.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Capability | Context & operation | Evidence for advancement |
| Perceive | Task goal and observation access; inspect the current page. | Relevant objects and affordances are identified sufficiently for subsequent reasoning. |
| Ground | Resolve a target or region; localize, associate, or match objects. | Target geometry, identity, correspondence, or ordering is established. |
| Solve | Reason over grounded task content to infer an answer or candidate action. | A supported solution hypothesis or actionable result is available. |
| Act | Execute an applicable operation under the current stage constraints. | The environment returns an execution outcome requiring assessment. |
| Verify | Inspect feedback after an attempted action or unresolved outcome. | Evidence supports progress, completion, failure, or continued uncertainty. |
Table 4: Capability roles and their advancement requirements. Roles denote semantic responsibilities rather than necessarily separate model calls, tools, or modules.
| Current protocol state | Runtime treatment |
| Target evidence is missing | Allow acquisition of relevant evidence. |
| An admissible commitment is pending | Prioritize the committed operation under full control. |
| Supporting conditions have changed | Reconsider the current binding and acquire relevant evidence. |
| Repeated tool failure without new evidence | Redirect selection toward an alternative capability. |
| Applicable recovery budget is exhausted | Stop recovery operations governed by that budget. |
Table 5: State-conditioned treatment of re-localization proposals. The cases illustrate protocol behavior rather than an exhaustive rule-priority list.
| Feedback situation | Default recovery direction |
| Tool not applicable | Reconsider the current affordance and select a relevant grounding or capability branch. |
| Tool error | Seek another evidence source or reacquire the inputs required by the tool. |
| Behavioral rejection | Reconsider the interaction form or sequence while retaining the solution hypothesis when justified. |
| Content rejection | Revise the solution; correct locally when possible, otherwise refresh and ground again. |
| Verification unresolved | Obtain another observation or progress signal before declaring success or repeating a submission. |
| Repeated failure without new evidence | Use failed-attempt history to steer selection away from uninformative repetition. |
Table 6: Default recovery directions conditioned on observable feedback. They guide local recovery decisions rather than define a complete failure diagnosis or cost-optimal policy.
| Method | Decision input | Permitted interaction | Returned feedback |
| SolveBind | Current screenshot; public task instruction; retained interaction evidence; execution progress; enabled-tool outputs. | GUI actions plus enabled perception, localization, cropping, checking, refresh, and recovery capabilities. | Updated screenshot, public page changes, and feedback that does not reveal the answer. |
| MobileAgent / AppAgent | Native public observations and retained interaction history. | Native GUI action space and enabled capabilities. | Updated interface state and public environmental feedback. |
| UI-TARS systems | Task instruction, current screenshot, and limited interaction history. | Click, long press, drag, type, scroll, back, home, wait, screenshot, and termination actions. | Updated screenshot and terminal task outcome. |
Table 7: Agent-visible inputs, permitted interactions, and returned feedback in the CAPTCHA evaluation.
| Environment | Evaluator-only information | Success determination |
| CAPTCHA | Correct answer, hidden verification state, evaluator signals, and complete environment telemetry. | Independent environment verification of the executed interaction. |
| AndroidWorld | Private task parameters, the success oracle, and states or trajectories of other agents. | Official terminal evaluation of the resulting application state. |
Table 8: Evaluator-only information and success determination in the two evaluation environments.
| Limit | Value | Scope |
| Maximum output per model call | 4096 tokens | All CAPTCHA evaluations. |
| Model-request timeout | 180 s | Each model request in the CAPTCHA evaluation. |
| Task-level timeout | 1200 s | All CAPTCHA evaluations. |
| GUI-action budget | 32 | SolveBind/HLL runtime. |
| Read-only-step budget | 4 | SolveBind/HLL runtime; counted separately. |
| Grounding-stage step cap | 3 | SolveBind/HLL runtime. |
Table 9: Execution limits used in the evaluation.
| Backbone | Best baseline | Baseline SR | Our SR | |
| GPT-5.5 | MobileAgent | 66.7 | 80.0 | +13.3 |
| GPT-5.4 | UI-TARS Harness | 15.6 | 46.7 | +31.1 |
| Claude Opus 4.8 | MobileAgent | 61.1 | 75.6 | +14.5 |
| Claude Opus 4.6 | MobileAgent | 38.9 | 34.4 | -4.5 |
| Gemini 3.1 Pro Preview | MobileAgent | 42.2 | 76.7 | +34.5 |
| GLM-5v-turbo | MobileAgent | 38.9 | 57.8 | +18.9 |
Table 10: Verification success relative to the highest-success baseline for each backbone. SR is in percent and is a percentage-point difference calculated from the reported values.
| Backbone | SolveBind | MobileAgent | UI-TARS Harness | AppAgent |
| GPT-5.5 | 37,796 | 23,817 | 59,298 | 33,664 |
| GPT-5.4 | 74,934 | 37,369 | 73,006 | 56,344 |
| Claude Opus 4.8 | 77,539 | 52,721 | 81,543 | 81,663 |
| Claude Opus 4.6 | 47,756 | 30,754 | 77,375 | 23,434 |
| Gemini 3.1 Pro Preview | 59,443 | 63,973 | 92,021 | 26,268 |
| GLM-5v-turbo | 97,493 | 42,802 | 77,924 | 38,366 |
Table 11: Mean total tokens in the dynamic verification evaluation. — denotes an unavailable value.
| Backbone | SolveBind | MobileAgent | UI-TARS Harness | AppAgent |
| GPT-5.5 | 100,855.07 | 173,468.95 | 29,383 | 45,736 |
| Claude Opus 4.8 | 135,585.61 | 256,178.60 | 44,583 | 29,169 |
| Gemini 3.1 Pro Preview | 83,671.19 | 160,938.70 | 23,539 | 29,724 |
| GLM-5v-turbo | 134,372.59 | 310,902.90 | 63,440 | 53,478 |
| MiniMax-M3 | 136,867.59 | 269,488.01 | 30,968 | 15,142 |
| Qwen3.7-plus | 92,594.98 | 228,954.11 | 31,353 | 40,159 |
Table 12: Mean total tokens on AndroidWorld. — denotes an unavailable result.
| Setting | Success / Total | SR (%) | E2E | Tok |
| Dynamic verification | 325/900 | 36.11 | 116.38 | 19,909.43 |
| AndroidWorld | 23/116 | 19.83 | 95.24 | 14,018.81 |
Table 13: Native UI-TARS-1.5-7B with its associated harness. E2E is in seconds; Tok is the mean total-token count per run. Verification and application results use separate scored populations.
| HLL task family | Instances | Successes | E2E (s) | Tok |
| Natural-image sequence selection | 100 | 98 | 33.87 | 4,079.50 |
| Icon sequence selection | 100 | 49 | 124.29 | 17,730.70 |
| Slider alignment | 100 | 22 | 157.94 | 25,756.80 |
| Jigsaw alignment | 100 | 38 | 140.64 | 19,441.90 |
| Missing-patch selection | 100 | 64 | 68.55 | 12,027.70 |
| Tile restoration | 100 | 24 | 123.28 | 22,689.10 |
Table 14: Per-family results for the full native UI-TARS verification evaluation. Each family contains 100 instances; successes sum to the overall total of 325, and the family means of E2E and Tok reproduce the overall values in Table 13 . Tok excludes image tokens.
| Framework | SR (%) | E2E (s) | Tok |
| GPT-5.5 | |||
| SolveBind | 87.4 8.3 | 98.6 19.6 | 40,921 3,438 |
| MobileAgent | 60.0 6.9 | 206.9 6.6 | 25,937 518 |
| UI-TARS Harness | 41.9 1.3 | 204.3 28.4 | 58,621 1,330 |
| AppAgent | 36.30 3.39 | 209.69 26.28 | 30,085.74 3,266.41 |
| Claude Opus 4.8 | |||
Table 15: Repeated-evaluation summaries over three runs, reported as mean standard deviation.
| Quantity | Count | Proportion |
| MobileAgent: static-correct, dynamically rejected, final failure | 13/90 | 14.44% |
| SolveBind success on those same 13 tasks | 7/13 | 53.85% |
| MobileAgent failure, SolveBind success | 13/90 | 14.44% |
| MobileAgent success, SolveBind failure | 1/90 | 1.11% |
Table 16: Paired diagnostic counts for GPT-5.5 on 90 HLL tasks. Conditional success refers to the baseline-defined failure subset, not a causal repair rate.
| GPT-5.5 | Claude Opus 4.8 | Gemini 3.1 Pro | |
| Saved steps | 378 | 585 | 621 |
| Steps with resolution | 376 | 585 | 615 |
| Steps without resolution | 2 | 0 | 6 |
| Overridden steps | 26 | 78 | 79 |
| Override rate | 6.91% | 13.33% | 12.85% |
| Overrides per episode | 0.289 | 0.867 | 0.878 |
Table 17: Per-step decision-log counts for SolveBind, from a 90-episode supplementary batch per backbone ( hard where supported, otherwise normal ; distraction level 2). “Override rate” is the fraction of decision steps with a recorded resolution that were overridden. This batch is not the 90-instance main table or the 225-episode ablation.
| Category | GPT-5.5 | Opus 4.8 | Gemini 3.1 Pro |
| Equivalent name/parameter normalization | 4 | 15 | 18 |
| Commitment-based operation change | 7 | 28 | 26 |
| Readiness-triggered automatic submission | 11 | 16 | 8 |
| Budget-triggered control | 4 | 19 | 27 |
| Total overrides | 26 | 78 | 79 |
Table 18: Primary category of overridden decision steps. Each step is counted once; counts are not episode-level success rates.
| Configuration | Control change | Retained execution paths |
| Full supervision | Evidence-supported pending operations take dispatch precedence. Submission readiness and budget state may trigger submission, checking, refresh, recovery, or termination. | Parsing, validation, execution, state updates, public feedback, budget enforcement, and terminal evaluation. |
| Without complete mediation | Retained state and feedback no longer constrain the dispatched proposal through supervisory replacement. | Parsing and schema checks, executor validation, outer budgets, state recording, public feedback, and terminal evaluation. |
| Without fail-safe defaults | Primary mediation, pending-operation binding, and readiness-based submission remain, while conservative submission restrictions and budget-triggered safeguards are relaxed. | Outer budgets, tool execution, page verification, state recording, and terminal evaluation. |
| No binding | Evidence-supported pending operations no longer have forced dispatch precedence over model proposals. | Model-visible evidence, parsing, legality checks, executor, automatic submission and readiness checks, recovery, and budgets. |
| Relaxed supervision | Pending operations are no longer forcibly dispatched, and readiness alone no longer forces submission. | Recovery state, associated budgets, outer limits, alternative submission paths, and terminal evaluation. |
| Unbounded recovery | The per-category cap on recovery budgets ( in Appendix B.4 ) is removed; recovery can continue until the outer episode timeout, without otherwise changing mediation, defaults, or submission logic. | Mediation, pending-operation binding, fail-safe defaults, readiness-based submission, and terminal evaluation. |
Table 19: Runtime-control configurations used in the ablation. Each variant modifies a specific control mechanism while retaining the execution paths listed in the final column.
| Ablated configuration | SR gain | E2E reduction | Tok reduction |
| Without complete mediation | +9.33 | 9.51% | 13.04% |
| Without fail-safe defaults | +7.11 | 29.09% | 31.03% |
| No binding | +10.67 | 12.88% | 15.21% |
| Relaxed supervision | +13.78 | 6.53% | 10.71% |
| Unbounded recovery | 1.33 | 73.14% | 58.76% |
Table 20: Full control relative to each ablation. Success gains are in percentage points; cost reductions use the ablated condition as the denominator. A negative success gain means the ablated configuration attains higher success than full control.
| Case | Backbone | Termination | Mechanism isolated |
| M-C1 | GPT-5.5 | Verification rejected | Grounding error at one position of an ordered sequence |
| M-C2 | Claude Opus 4.6 | 15 steps, no attempt | Retries vary the wrong parameter; interaction form invalid |
| M-A1 | GPT-5.5 | Step limit, reward 0 | Repeated correction without checking its effect |
| M-A2 | GPT-5.5 | Agent stop, reward 0 | Completion claimed from dispatch, not from evidence |
| T-C1 | GPT-5.5 | Verification rejected | Candidates differing only by orientation not separated |
| T-C2 | GPT-5.5 | Verification rejected | Named target not bound to the region displaying it |
Table 21: Summary of the twelve baseline failure cases. Cases are described from recorded trajectories, with accompanying figures where the displayed material matches the task. Labels name the mechanism the case isolates; they are not population failure frequencies.
Figure 3: M-C1. Icon-sequence selection with GPT-5.5 / MobileAgent. The first two selections match the required order; the third resolves to the wrong icon, and verification is rejected. Three verification attempts produce no success.
Figure 4: M-C2. Slider alignment with Claude Opus 4.6 / MobileAgent. Each drag starts on the image rather than on the handle (circled), so the piece never moves. Retries at steps 3, 12, and 15 repeat the same start point while varying the endpoint; 15 steps yield no verification attempt.
Figure 5: M-A1. Audio file naming with GPT-5.5 / MobileAgent. The agent detects the wrong filename and issues 13 identical backspace taps; the field text is unchanged between steps 6 and 20, and the run terminates at the step limit with zero reward.
Figure 6: M-A2. Brightness setting with GPT-5.5 / MobileAgent. The swipe at step 5 targets a region where no slider is visible; the agent reports the action as successful and stops at step 6, leaving the task unsatisfied.
Figure 7: T-C1. Logic-and-arithmetic interaction with GPT-5.5 and the UI-TARS harness. Two copies of the same uppercase letter differ only by rotation; the agent names the required orientation but its selection does not separate them, and the dynamic check is rejected.
Figure 8: T-C2. Scene character selection with GPT-5.5 and the UI-TARS harness. The required order V, C, W is read correctly, but the selected circles do not contain the named characters; verification reports that the first click was incorrect.
Figure 9: T-A1. Symbol-level text entry with Claude Opus 4.8 and the UI-TARS harness. The full filename is typed in one action, but the field shows only the alphabetic prefix; the underscore and extension are not entered, and the episode ends without the required name.
Figure 10: T-A2. Long-horizon app workflow with Gemini 3.1 Pro Preview and the UI-TARS harness. After the opening steps the agent operates on unrelated applications and returns toward the home screen, and the recipes are never added. The reasoning excerpt shown in the source figure belongs to a different task and is not used in the analysis.
Figure 11: A-C1. Tile restoration with Gemini 3.1 Pro Preview / AppAgent. The agent identifies element 4 as a decoy and announces a correction, then repeats the same tap; the decoy navigates the browser away from the CAPTCHA and the run ends in a false completion.
Figure 12: A-C2. Ordered character selection with Claude Opus 4.8 / AppAgent. The first two selections in the order P, E, Z are correct; the third names the right character but grounds it to the wrong element, and verification fails.
Figure 13: A-A1. Canvas drawing with GLM-5v-turbo / AppAgent. Navigation succeeds through step 13, after which the agent issues tap actions where a swipe is required to draw; the run reaches max-step termination without a stroke.
Figure 14: A-A2. Recipe entry with MiniMax-M3 / AppAgent. The agent selects a valid action but omits the required Summary field from its response; the parser rejects the output and the episode ends in a parse error.
Figure 15: Case study S1. Icon-sequence verification. The three required icons are selected in order, and at step 6 the model’s submit_captcha proposal is dispatched as SubmitCaptcha , executed as a tap at . The step record retains the proposal, supervisor decision, executed operation, tool, outcome, and observation assets; the environment reports successful verification.
Figure 16: Case study S2. Contact creation with a specified phone number. At step 7 the recorded request binds input_text to target index 10 with clear_text: true and an explicit reason referencing the visible phone field, making both the target and the intended field state part of the dispatched operation. The task completes successfully.
More in this topic
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 178 papers