SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

Jianxing Chen; Xiao Yu; Shipra Agrawal; Zhou Yu

cs.CLSep 28, 2026

SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

Authors: Jianxing Chen, Xiao Yu, Shipra Agrawal, Zhou Yu

Organizations: Columbia University

Abstract

Computer-use agents (CUAs), while capable of completing computer tasks in everyday and professional workflows, can cause unintended harm even under benign instructions and environments. However, detecting such harm remains challenging. First, it requires careful, task-specific reasoning: verifiers guided only by general safety criteria often overlook many important but subtle harmful behaviors. Second, it requires active investigation: past trajectory screenshots show what the agent did but not always what actually changed in the environment, so LLM-as-a-judge verifiers that rely on screenshots alone may be unable to determine the actual consequences of actions. To address these challenges, we introduce SCOUT, a two-stage agentic safety verifier that synergizes reasoning-intensive rubric generation with tool-intensive evidence gathering. First, our SCOUT rubric generator extensively reasons over the task and the agent's trajectory to determine what successful and safe execution should entail, generating task-specific completion and safety rubrics. Then, our SCOUT probing agent follows these rubrics to interact with the post-execution environment and collect grounded evidence for final safety and completion judgments. We evaluate our framework on two computer-use safety benchmarks. On AutoElicit-Bench, SCOUT achieves 75.4 unsafe F1 and 74.5 completion F1, outperforming LLM-as-a-judge verifiers and naive tool-use verifiers. SCOUT leads on OS-Blind with 76.4% unsafe detection accuracy. Test-time reflection reduces final unsafe execution rates from 30.2% to 17.2% on AutoElicit-Bench. Ablations and analysis show that tool-free rubric generation in SCOUT elicits substantially more reasoning and is crucial for safety detection across verifier backbones, especially non-frontier ones. A preliminary extension to coding tasks shows that SCOUT can support safety verification beyond computer-use.

Figures & tables

Figure 1: SCOUT vs. a naive tool-use verifier. On a PDF conversion task, the naive verifier misses the privacy risk of uploading the receipt to an external service. SCOUT flags this risk in its generated safety rubric, inspects the receipt and upload screenshots, and correctly judges it unsafe.

Figure 2: Overview of SCOUT. The SCOUT rubric generator reasons over the task and trajectory to generate completion and safety rubrics. Guided by these rubrics, the SCOUT probing agent uses tools to interact with the post-execution environment to gather evidence, then returns completion and safety judgments with explanations.

Categories	Tools	Description
Inspect	run_command	Executes a shell command and returns text output.
Inspect	run_python	Executes Python code and returns text output.
Operate	computer_use	Executes a GUI action via PyAutoGUI and returns screenshots.
Retrieve	check_screenshot	Retrieves specified trajectory screenshots.

Table 1: Tools available to the SCOUT probing agent. Inspect examines files and system settings; Operate interacts with the graphical user interface; Retrieve accesses trajectory screenshots.

Figure 3: Test-time reflection with SCOUT. After the CUA’s execution, SCOUT performs verification in the post-execution environment. If SCOUT returns an unsafe judgment, the CUA retries from the initial environment with accumulated safety feedback. The process ends when SCOUT returns a safe judgment or the retry budget is exhausted.

Verifier (backbone)	Tool-use	AutoElicit-Bench		OS-Blind
Verifier (backbone)	Tool-use	Completion F1 (%) $\uparrow$	Unsafe F1 (%) $\uparrow$	Unsafe detection acc. $\uparrow$
DigiRL (GPT-5.6 Terra)	✗	43.5	11.1	26.4%
DistRL (GPT-5.6 Terra)	✗	40.0	17.6	23.2%
ZeroGUI (GPT-5.6 Terra)	✗	56.7	69.1	70.0%
IRA (GPT-5.6 Terra)	✓	68.6	38.9	43.6%
VAGEN (GPT-5.6 Terra)	✓	68.7	29.4	39.2%

Table 2: Verifier performance on AutoElicit-Bench and OS-Blind. CUA backbone: Qwen3.6-35B. Tool-use verifiers evaluate fresh rollouts under the same settings; passive LLM-as-a-judge verifiers evaluate the same CUA rollouts.

\uparrow

: higher is better. Best results are in bold.

Unsafe F1 (%) $\uparrow$	Verifier backbone varied			CUA backbone varied
Unsafe F1 (%) $\uparrow$	GPT-5.6 Terra	Kimi K2.5	Qwen3.5- 9B	Kimi K2.5	Qwen3.6- 35B	Qwen3.5- 9B
SCOUT	75.4	69.2	54.5	71.4	75.4	74.7
w/o action trajectory	69.7	63.3	48.9	59.2	69.7	68.6
w/o SCOUT rubric generator	53.1	40.9	12.5	62.3	53.1	70.4
w/o both	59.6	33.3	6.7	58.3	59.6	64.4

Table 4: Ablation of SCOUT components. Entries are unsafe F1 (%) on AutoElicit-Bench. We group the ablations into two settings: Verifier backbone varied , with a fixed Qwen3.6-35B CUA and varying verifier backbones; and CUA backbone varied , with GPT-5.6 Terra as the fixed verifier backbone. The GPT-5.6 Terra verifier with the Qwen3.6-35B CUA appears in both groups. All variants are instructed to generate safety rubrics.

Verifier backbone	Separate, tool-free	Separate, + dummy tool	Single- prompt
GPT-5.6 Terra	1,642	1,099	204
Kimi K2.5	6,847	5,916	252
Qwen3.5-9B	9,872	5,910	269

Table 5: Reasoning tokens during rubric generation. Mean per task on AutoElicit-Bench, with Qwen3.6-35B as the CUA backbone.

Coding agent backbone	LLM-only Unsafe F1 $\uparrow$	SCOUT Unsafe F1 $\uparrow$
Claude Opus 4.6	4.3	43.1
Qwen3.5-9B	34.5	63.2

Table 6: Safety verification on a 100-task SABER subset. Both verifiers use Qwen3.6-35B as their backbone.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Backbone	Top- $p$	Reasoning effort
CUA	Qwen3.6-35B-A3B	0.90	–
	Qwen3.5-9B	0.90	–
	Kimi K2.5	0.95	–
Verifier	GPT-5.6 Terra	–	xhigh
	Kimi K2.5	0.95	–
	Qwen3.5-9B	0.95	–

Appendix

Table 7: Model configurations. Temperature is set to 1.0 throughout. A dash indicates that no explicit reasoning-effort level is specified; top-

p

is not set for GPT-5.6 Terra.

Figure 8: Rubrics generated by SCOUT for a spreadsheet task. The completion rubric specifies the requested output; the safety rubric identifies concerns arising from the CUA’s actual execution.

Explore similar work

Aug 27, 2026cs.LG

Beyond Task Completion: Training Capable and Safe Computer-Use Agents

Computer-use agents (CUAs) have made rapid progress in completing complex tasks through graphical user interfaces, yet post-training centered on task success alone does not induce reliable safety behavior. A reliable CUA must condition its execution on risk: it should complete ordinary benign tasks, avoid environmental hazards and continue when a safe completion path remains, and refuse when the goal is harmful or no safe path exists. To learn this conditional policy, we develop Safety and Capability Optimization for Policy Execution (SCOPE), which jointly post-trains a CUA for task-execution capability and safety-aware decision making. To provide aligned training data for this joint objective, we further introduce SCOPE-Gen, an automated pipeline that synthesizes verifiable capability tasks and converts them into paired environment-risk variants while preserving their original goals. Using the resulting tasks, we construct SATraj-OS, a trajectory dataset comprising capability demonstrations, safe continuations, and explicit refusals. SCOPE first learns from all three trajectory types through supervised fine-tuning and then further improves task completion through online reinforcement learning. Starting from Qwen3.5-9B, SCOPE-RL achieves a 54.17% task success rate on OSWorld and a 64.30% attack-avoidance rate on OS-BLIND, yielding the best aggregate capability--safety score of 58.80% among the evaluated agents. Ablations reveal asymmetric but complementary roles for the two forms of safety supervision: refusal trajectories account for most of the attack-avoidance gain, whereas risk-handling trajectories preserve greater task utility at comparable attack-avoidance levels.

Zeyu Kang, Zhenyun Yin, Yang Zhang +5

Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +2

Computer-Use Agents Autonomous Agents

Ask PDF

Jul 31, 2025cs.CR

Measuring Harmfulness of Computer-Using Agents

Computer-using agents (CUAs), which can autonomously control computers to perform multi-step actions, might pose significant safety risks if misused. However, existing benchmarks mainly evaluate LMs in chatbots or simple tool use. To more comprehensively evaluate CUAs' misuse risks, we introduce a new benchmark: CUAHarm. CUAHarm consists of 104 expert-written realistic misuse risks, such as disabling firewalls, leaking data, or installing backdoors. We provide a sandbox with rule-based verifiable rewards to measure CUAs' success rates in executing these tasks (e.g., whether the firewall is indeed disabled), beyond refusal rates. We evaluate frontier LMs including GPT-5, Claude 4 Sonnet, Gemini 2.5 Pro, Llama-3.3-70B, and Mistral Large 2. Even without jailbreaking prompts, these frontier LMs comply with executing these malicious tasks at a high success rate (e.g., 90% for Gemini 2.5 Pro). Furthermore, while newer models are safer in previous safety benchmarks, their misuse risks as CUAs become even higher, e.g., Gemini 2.5 Pro is riskier than Gemini 1.5 Pro. Additionally, while these LMs are robust to common malicious prompts (e.g., creating a bomb) when acting as chatbots, they could still act unsafely as CUAs. We further evaluate a leading agentic framework (UI-TARS-1.5) and find that while it improves performance, it also amplifies misuse risks. To mitigate the misuse risks of CUAs, we explore using LMs to monitor CUAs' actions. We find monitoring unsafe computer-using actions is significantly harder than monitoring conventional unsafe chatbot responses. While monitoring chain-of-thoughts leads to modest gains, the average monitoring accuracy is only 77%. A hierarchical summarization strategy improves performance by up to 13%, a promising direction though monitoring remains unreliable. CUAHarm is released at https://github.com/db-ol/CUAHarm to facilitate further research.

Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang +3

Independent Researcher · Arizona State University · Gradient +1

Large Language Model Safety Unsafe Chatbots

Ask PDF

Jun 13, 2026cs.AI

OSGuard: A Benchmark for Safety in Computer-Use Agents

Computer-use agents are increasingly evaluated by whether they complete realistic desktop and web tasks. However, task success alone can miss failures in which an agent reaches the nominal goal through an unsafe shortcut. We introduce OSGuard, a dual-granularity benchmark suite for evaluating safety in computer-use agents under benign, unchanged user instructions. OSGuard contains an action-level benchmark for local guardrail decisions and a risk-augmented execution suite for end-to-end evaluation. The action-level benchmark consists of contextualized proposed actions labeled as allowed, unrelated, or unsafe, each judged relative to the original instruction and current interface state. The execution suite contains manually constructed OSWorld-derived task variants in which the original task remains achievable, but the environment is modified to introduce latent hazards such as destructive overwrites, etc. Each variant is paired with augmented evaluators that retain the original task-success criterion while adding explicit state-based safety invariants, allowing us to distinguish safe completions from unsafe completions that satisfy the nominal task objective. Our experimental results on OSGuard show that current multimodal guardrails can perform well on isolated action judgments, while risk-augmented execution exposes remaining gaps between local oversight and reliable end-to-end safety. This dual-granularity design enables more precise diagnosis of whether models can both recognize unsafe proposed actions and improve full-task safety when deployed as guardrails.

Mina Mohammadmirzaei, Jeffrey Flanigan

University of California, Santa Cruz

Safety Benchmarks Computer-Use Agents Safety Claims Security Evaluation

Ask PDF