AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

Xing Han Lù; Dheeraj Vattikonda; Sina Hajimiri; Fatemeh Pesaran Zadeh; Parishad BehnamGhader; Ghazwa Darwiche; Amirhossein Kazemnejad; Christopher Pal

cs.AIOct 8, 2026

AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

Authors: Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, +2 more

Organizations: ServiceNow Research · McGill University · Mila – Quebec AI Institute · ÉTS Montréal · Seoul National University · Polytechnique Montréal · Canada CIFAR AI Chair · Université Laval

Abstract

Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement. Despite their flexibility, their reliability on long tasks spanning multiple applications remains unclear. A trajectory, composed of long sequences of screenshots and actions, may appear complete, but in reality violates constraints from the instruction or introduces an unwanted side effect. To identify these errors, a judge needs to examine the trajectory with respect to the user's instruction. To this end, we introduce AgentHorizon, a benchmark of 1,373 computer-use tasks (instruction-trajectory pairs) drawn from 166 hours of human-recorded trajectories spanning three operating systems. By recording trajectories for closely related instructions, we can construct negative tasks by swapping the instructions. This paired design evaluates judges on their ability to distinguish a successful trajectory from one that completed a similar (but incompatible) request. We release the benchmark under three splits: a frontier split, AgentHorizon (AH), a simplified split, AgentHorizon-Simple (AH-S), and a development split, AgentHorizon-Development (AH-D). We further evaluate eleven judges by (1) directly passing the full trajectory (with up to 300 screenshots and actions), and (2) using them as coding agents across five agent harnesses. We find that our best agentic judge, GPT-5.5, achieves 80.9% balanced accuracy on the AH subset. We find that tool-use improves certain models but results in worse performance for open-weight models, and that judges differ drastically in their ability to accept a valid trajectory and reject failed ones. Our findings highlight the need for judges that are capable of locating and verifying often hidden evidence that a task was properly completed inside long interaction histories.

Figures & tables

Figure 1: A plausible output can violate the instruction. In this $115$ -step Inkscape–GIMP recording ( 0cacdac6 , macOS), a pentagon replaces the requested triangular accent. The mismatch survives recoloring and export, giving the crossed item a Critical Mistake label. The outer strips sample $10$ frames from each half of the recording; the enlarged frames show the canvas, color selection, and final composite at steps $38$ , $77$ , and $105$ .

Panel of Figure 2 — Figure 2: One AgentHorizon item: instruction, four illustrative keyframes (the LLM judge sees the full $195$ -step trajectory), judge verdict, and ground truth. Claude Opus 4.7 predicts success=true ; ground truth is a Bad Side Effect (unrequested changelog content published).

Figure 3: Paired construction before label review. Annotators record trajectories for related instructions

A_{\text{instr}}

and

B_{\text{instr}}

. Matched instruction–trajectory combinations are candidate positives; crossed combinations are candidate negatives. Positive retention and negative validity are then reviewed separately.

Mistake types among negatives
	AH-D	AH	AH-S
Judging tasks	162	528	683
Positives	62	227	234
Adversarial negatives	100	301	449
Critical Mistake	32	48	193
Bad Side Effect	26	107	99

Table 1: Composition of the three disjoint benchmark subsets. Mistake-type counts are over negatives; the six negatives without a typed label are shown separately. AH-D is reserved for development and excluded from both evaluation subsets.

Model	Open?	Interface	AH Bal.	AH Pos	AH Neg	AH-S Bal.
Agentic judges
GPT-5.5		Codex	80.9	71.8	90.0	92.6
Gemini 3.1 Pro		Gemini CLI	77.7	65.6	89.7	86.3
Claude Opus 4.7		Claude Code	76.0	76.7	75.4	94.6
Qwen 3.6 27B	✓	OpenCode	70.4	75.3	65.4	93.6
GPT-5.4 mini		Codex	69.2	67.4	71.1	91.2

Table 2: Judge performance on the held-out evaluation subsets. AgentHorizon contains the

528

items that remain difficult under the pooled three-model split; AgentHorizon-Simple is its

683

-item complement. Pos and Neg are class-specific accuracies on AH, and Bal. is their mean. ∗ marks direct rows using the

2{\times}2

image-merge preprocessing described in Appendix B.1 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Hours/task	Total hours
Annotation	$4.00$	$3{,}400.0$
Review L1	$1.70$	$1{,}445.0$
Mistake-category review	$0.25$	$212.5$
Second-stage review	$1.20$	$1{,}020.0$
Total	$\mathbf{7.15}$	$\mathbf{6{,}077.5}$

Appendix

Table 3: Vendor-reported effort over the

850

unique trajectories. The delivered total exceeds the original

1{,}960

-hour estimate because calibration introduced additional review passes.

Stage	Positive	Negative	Total
Paired construction pool	850	850	1,700
Reviewed benchmark	523	850	1,373
AH-D	62	100	162
AH	227	301	528
AH-S	234	449	683

Appendix

Table 4: Stage-by-stage benchmark accounting. The 327 removals affect candidate positives only; AH-D, AH, and AH-S are disjoint and exhaust the reviewed benchmark.

Type	Boundary	Typical recovery
Critical Mistake	The core action or target is not achieved.	Restart or repeat the task; the original outcome cannot satisfy the request.
Bad Side Effect	The goal is achieved, but an unrequested action creates cost, risk, exposure, or cleanup.	Reverse an external action, coordinate with others, or spend substantial effort undoing it.
Misunderstanding	The execution is coherent but misreads a parameter or constraint without external harm.	A local edit or short correction is sufficient.

Appendix

Table 5: Operational boundaries of the annotator-facing failure taxonomy.

Figure 5: Decision-step distributions for the evaluation subsets. AH has a median of

122

steps and 90th percentile of

195

; AH-S has

118

and

179

, respectively.

Model	Interface	Input tokens	Tools	Images	$/trajectory
GPT-5.5	Codex	513,556	18.6	11.9	0.4035
Gemini 3.1 Pro	Gemini CLI	1,393,271	24.2	30.2	0.6172
Claude Opus 4.7	Claude Code	554,387	9.2	7.8	0.6592
Qwen 3.6 27B	OpenCode	243,440	8.1	6.0	0
GPT-5.4 mini	Codex	323,808	20.0	8.4	0.1066
Qwen 3.6 35B-A3B	OpenCode	190,550	8.1	5.6	0

Appendix

Table 6: Resource use for the primary agentic-judge configurations. Means are computed over all

1{,}373

trajectories unless a harness omitted the corresponding telemetry; self-hosted rows have no marginal API charge.

Model	Interface	Critical	Side effect	Misunderstanding
GPT-5.5	Codex	77.7	24.1	72.9
Gemini 3.1 Pro	Gemini CLI	57.5	7.3	87.3
Claude Opus 4.7	Claude Code	75.8	12.1	71.4
Qwen 3.6 27B	OpenCode	60.4	5.6	74.3
GPT-5.4 mini	Codex	60.8	2.6	74.6
Qwen 3.6 35B-A3B	OpenCode	63.0	6.0	53.1

Appendix

Table 7: Exact mistake-type recall on the full benchmark for the primary agentic-judge configurations. Denominators are 273 Critical Mistakes, 232 Bad Side Effects, and 339 Misunderstandings; malformed outputs count as incorrect.

Model	Interface	AH Bal.	AH Pos	AH Neg	AH-S Bal.
GPT-5.5	Codex †	80.9	71.8	90.0	92.6
	OpenCode	77.2	70.9	83.4	92.2
Qwen 3.6 27B	Codex	71.6	70.5	72.8	94.5
	OpenCode †	70.4	75.3	65.4	93.6
	OpenHands	51.2	58.1	44.2	89.0
Gemini 3.1 Flash Lite	Gemini CLI †	52.1	58.6	45.5	75.2

Appendix

Table 8: Models evaluated under multiple agentic interfaces on the final AH and AH-S splits. All runs use prompt P9 and the same benchmark inputs. AH Pos and AH Neg use fixed denominators of 227 and 301; AH-S Bal. uses 234 positives and 449 negatives. Missing or malformed outputs count as incorrect. † marks the primary interface reported in Table 2 .

Instruction	Screenshots	Action log	TP	FN	FP	TN	BA
✗	✓ all	✓	246	5	273	0	49.0%
✓	— final only	✗	104	147	118	155	49.1%
✗	— final only	✗	215	35	245	26	47.8%
✓	✗	✓	205	46	181	92	57.7%
✗	✗	✓	249	2	272	1	49.8%

Appendix

Table 9: Ablation results evaluating which trajectory components are necessary for judging a trajectory.

Explore similar work

Jun 8, 2026cs.AI

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.

Wanli Li, Bowen Zhou, Yunyao Yu +4

Microsoft Research Asia · Tsinghua University

Computer-Use Agent Benchmarks Computer-Use Agents AI Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks Agent Benchmarks

Ask PDF

Sep 15, 2026cs.LG

Locating Hidden Failures Makes Long-Horizon Agents More Reliable

As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed. An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. We study

2518

agent trajectories across software engineering, computer use, and science, close to real deployment, and classify

6967

mistakes into

78

failure types. Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. Yet Scout, a

4

B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.

Salman Rahman, Yubin Kim, Mihir Parmar +15

University of California, Los Angeles · New York University · Google Research +1

Agent Failure Analysis AI Agent Reliability Long-Horizon Agent Evaluation Fault Detection AI Agent Benchmarks

Ask PDF

Jun 28, 2026cs.AI

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 13%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.

Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33

XLANG Lab

Computer-Use Agent Benchmarks Computer-Use Agents Long-Horizon Agent Evaluation Long-Horizon Agent Tasks

Ask PDF