CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Yuxiang Lai; Jike Zhong; Chengzhi Liu; Ming Li; Haonan Wang; Xiaofeng Yang; Liang Zhao; Youbao Tang

cs.MAOct 13, 2026

CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Authors: Yuxiang Lai, Jike Zhong, Chengzhi Liu, Ming Li, Haonan Wang, Xiaofeng Yang, Liang Zhao, Youbao Tang,

Organizations: Google · Emory University · University of Southern California · University of California, Santa Barbara ·

Abstract

Vision-language models (VLMs) are increasingly deployed in multi-agent systems to automate complex computer-use workflows, yet existing benchmarks mostly assume consistent, cooperative environments. In contrast, real-world collaboration rarely runs on consistent information: peer or sub-agents frequently report outdated states, issue contradictory instructions, or compete over shared permissions and files, forcing agents to check others' outputs and verify ground truth in the environment. Evaluating such conflict resolution remains challenging: manual authoring cannot scale across complex systems, while unconstrained LLM generation produces narrative drift, answer leakage, and unsatisfiable rubrics. We introduce CLASHarness (Conflict-Loaded Agent Scenario Harness), a controllable framework for generating verifiable multi-agent conflicts in computer use, with a benchmark of 600 multi-round tasks. By combining hierarchical scenario planning with an eight-step verified micro-pipeline, CLASHarness pre-commits decisive visual facts before rendering and compiles hybrid code and semantic graders directly from executed reference trajectories, guaranteeing every task is solvable by construction. Evaluating 15 frontier models on CLASHarness reveals three core bottlenecks. First, when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information, whereas stronger models inspect visual evidence to verify ground truth. Second, resolving conflicts is much harder than finding them: while agents can spot inconsistencies, actively fixing system state drops task success significantly. Third, coordination protocols remain a universal bottleneck: even frontier models fail over 58% of closing actions, often forgetting to release shared locks or hand off tasks. CLASHarness provides a scalable, verifiable foundation for building reliable multi-agent systems.

Figures & tables

Figure 1: Hierarchical generation pipeline in CLASHarness. Tier 1 initializes a shared multi-agent world from seeds and plugins. Tier 2 refreshes phase milestones every 3–4 rounds and generates each round through eight explicit sub-steps. Ground truth

v^{\star}

is committed before rendering; executed reference effects ground deterministic checks and scoped semantic criteria. The dashed return path carries workspace state to the next round (

r<R

); after

R=10

rounds, generation proceeds to Tier 3 for consistency auditing, visual refinement, and rendering. Visual evidence and checkers share the committed

v^{\star}

, with text-exclusivity gates enforced when visuals are required. Green tags mark deterministic processing; red tags mark bounded correction with diagnostic feedback. Detailed safeguards appear in Appendix Table B1 .

Figure 2: Harness parameter ablations. (a) Step budget: Expanding action horizons unlocks frontier models with high exploration demands while separating efficient planners from thinking loops. (b) Visual modality: Agent performance collapses without image inputs, confirming visual evidence is essential for conflict resolution. (c) Generator model: Relative rankings remain highly consistent across parallel benchmarks synthesized by different generator models.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure B1: Photographic asset scenes in CLASHarness. Shows the four real-world scenes used across all 19 photographic assets in the benchmark. Each photograph contains a decisive ground-truth string (shown below each image) required to resolve the corresponding conflict.

Panel of Figure 4 — Figure B2: Visual design templates across browser-rendered assets. Shows each of the 14 visual rendering styles across packages, ordered by frequency (asset counts in parentheses), spanning diverse UI formats such as metric dashboards, telemetry logs, console tables, and code diffs.

Panel of Figure 5 — Figure B3: Multimodal evidence inspection in a worked investigation round. Illustrates the two visual assets required to satisfy all eight evaluation checks in round access_request_approval_v01_r01 : panel (a) presents the permission matrix containing the decisive conflict verdict ( UNRECONCILED ), while panel (b) contains the authorization passcode ( SEC-PASS-8842-BP ).

#	Effect	Predicate	Criterion
1	assets_accessed	field_contains	id contains iam_console_access_scope_matrix_r01 [gate]
2	messages_sent	field_semantic_match	tell the manager the dashboard reads UNRECONCILED (2/3 approver scopes mismatched) and refuse the premature grant
3	messages_sent	field_semantic_match	relay passcode SEC-PASS-8842-BP to the security reviewer and request WRITE on billing-prod
4	approval_recorded	any_field_equals	billing_prod
5	approval_granted	any_field_equals	billing_prod
6	files_created	field_semantic_match	the report states the finding UNRECONCILED (2/3 approver scopes mismatched)

Appendix

Table B2: Compiled evaluation rubric for round access_request_approval_v01_r01 . Lists the eight scored checks compiled from the reference trajectory. Check 1 acts as a mandatory visual access gate ( role="required" ). Semantic matches are evaluated by the self-consistency ensemble judge.

Panel of Figure 7 — Figure B4: Stale cache vs. authoritative ledger ground truth (round expense_reimbursement_conflict_v03_r03 ). Panel (a) presents the stale cache reporting 850.00 EUR ; panel (b) provides the itemized ground-truth receipt totaling 725.46 EUR ( $678.00\times 1.07$ ). Decisive numerical evidence is exclusive to the rendered image, requiring visual inspection.

Figure D1: Diagnostic check failure rates across failure modes (Ten 20-step API models, High Effort). Compares failure proportions across 3,284 scored checks spanning Content (F1, F2, F4), Visual Grounding (F3), and Structural Protocol (F5, F6) failure modes.

	Bench A	Bench B
Generator model	Gemini 3.6 Flash	Claude Sonnet 5
Step budget	20	30
GT step length, mean / median / max	10.28 / 10 / 18	12.48 / 12 / 24
Budget ratio (budget $\div$ GT), mean	$2.02\times$	$2.51\times$
Rounds with budget ratio $<1.5\times$	8.2% (49)	1.2% (7)
Rounds with unexposed reference strings	18	9

Appendix

Table F1: Structural comparison of parallel generation benchmarks. Contrasts Bench A (Gemini 3.6 Flash) and Bench B (Claude Sonnet 5) across identical 20 scenario seeds and 600 aligned conflict tasks.

(a) Step budget. Same 600 rounds, high effort; only the cap moves.
	Pass Rate (%)				Ended at Cap (%)
Model	20	30	40	$\Delta$	20	40
Claude Opus 5	25.2	25.7	27.0	$+1.8$	0.5	0.0
Claude Sonnet 5	20.8	20.7	21.3	$+0.5$	3.3	0.2
Gemini 3.7 Flash	20.3	26.0	27.2	$+6.8$	40.7	2.5
Gemini 3.6 Flash	18.5	23.5	25.0	$+6.5$	24.7	0.3

Appendix

Table F2: Comprehensive numerical results for harness parameter ablations. (a) Step budget: Pass rates and cap exhaustion when expanding horizon

b=20\to 30\to 40

on Bench A (

\Delta=b_{40}-b_{20}

). (b) Visual modality: Performance of sighted vs. blind agents across full (600), asset-gated (346), and un-gated (254) task groups. (c) Generator difference: Pass rates across seven models evaluated on Bench A vs. Bench B (600 rounds each;

\Delta=\text{B}-\text{A}

Budget ( $b$ )	Pass Rate (%)	95% CI	Cap-Hit (%)	Calls/Task	Reason Tok	$/Solved
$b=20$	12.8 (77/600)	$[10.4,15.7]$	66.8	19.4	6,699	$4.44
$b=30$	19.0 (114/600)	$[16.1,22.3]$	46.8	27.3	8,976	$4.40
$b=40$	29.2 (175/600)	$[25.7,32.9]$	19.2	32.2	11,233	$3.50
$b=60$	30.7 (184/600)	$[27.1,34.5]$	3.3	35.0	11,446	$3.71
Comparator: Gemini 3.7 Flash (same judge, high effort)
$b=20$	20.3 (122/600)	$[17.3,23.7]$	40.7	17.8	4,026	$2.51

Appendix

Table F3: Action horizon scaling for Gemini 3.8 Flash (

n=600

rounds, High effort, Gemini 3.6 Flash judge). Metrics include task pass rate with

95\%

Wilson score confidence intervals, percentage of runs terminating at the step cap, average tool calls, average reasoning tokens, cost per task at introductory pricing, and effective cost per solved task.

Substitute judge	Gemini 3.6 Flash	Claude Sonnet 5	Claude Opus 5
Mean score (primary).
Rank correlation $\rho$	0.987	0.991	0.987
excluding the self-scored row	0.995	0.995	0.989
Maximum rank displacement	2	1	1
Models whose rank moves	3/14	4/14	6/14
Mean shift	$-0.001$	$+0.005$	$+0.021$

Appendix

Table F4: Cross-judge stability ablation. Every semantic check of the high-effort panel is re-evaluated offline by three substitute three-sample ensembles with agent trajectories held fixed: the replay reproduces

24{,}331/24{,}331

deterministic verdicts, so the columns differ only in semantic opinion. The Gemini 3.6 Flash column re-runs the default judge, so it measures sampling noise rather than a judge change. Statistics use the

22{,}938

26{,}100

panel checks (

87.9\%

) all three substitutes scored; the recorded verdict fills the rest in every column and cancels in cross-judge comparisons. Mean score leads because pass rate requires passing all checks, so one flipped check can move a model several places. The substitute judges reorder the models no more than the default judge reorders itself; they primarily shift absolute score levels. Claude Opus 5 raises all fourteen 20-step baseline models, unevenly enough to hand first place to GPT-5.6 Sol—the only change of leader in the table. The self-scored row that the two excluding lines drop is Gemini 3.7 Flash, which issued its own recorded verdicts.

Figure G1: Overall benchmark standings across the 15 evaluated models (High Effort). Displays task pass rate (

\mathtt{pass@budget}

, wide bar) and mean check score (narrow bar, 0–1 scale) across all 600 benchmark tasks. Models are ranked by pass rate;

\text{Rank }\Delta

indicates the difference between mean score rank and pass rank.

Figure G2: Action category glossary and invocation distribution. Partitions all 88 sub-commands in the action DSL into 12 mutually exclusive functional categories (call rates normalized per task across all 600 rounds). The detailed breakdown of domain-specific operations is provided in Figure G7 , and the full command inventory spans Figures G8 – G10 .

Figure G3: Pass rate versus mean check score across models (High Effort). Shows the relationship between all-or-nothing task completion (pass rate) and mean check score across all 15 models.

Figure G4: Pass rate versus cumulative evaluation cost (High Effort). Compares benchmark pass rate against total evaluation cost across models.

Figure G5: Token volume, monetary spend, and execution latency across models (High Effort). Breaks down token usage per task (input prompt, visible generation, internal reasoning), cost per task and per solved task, and execution latency across evaluated models.

Figure G6: Empirical action-category distributions across models (High Effort). Relative proportion of tool calls across the 12 action categories. Each bar sums to 100% of that model’s executed actions. The domain-specific operations grouped under other are detailed in Figure G7 .

Figure G7: Itemized distribution of domain-specific operations ( other category). Breaks down specialized environment operations (including containment locks, deployment approvals, resource pairing, and process scheduling) across models.

Figure G8: Complete command inventory across all evaluated models (Part 1: Rows 1–30 of 89). Displays individual sub-command invocation counts across all 600 tasks per model, mapped to functional categories. Continued in Figure G9 .

Figure G9: Complete command inventory across all evaluated models (Part 2: Rows 31–59 of 89). Continued from Figure G8 ; continued in Figure G10 .

Figure G10: Complete command inventory across all evaluated models (Part 3: Rows 60–89 of 89). Final part of the 88-command execution inventory across all 15 evaluated models.

Figure G11: Visual asset grounding volume versus task pass rate (High Effort). Plots mean asset opens per task against benchmark pass rate, illustrating the decoupling between visual inspection volume and reasoning accuracy.

Figure G12: Episode outcome and termination reason breakdown (High Effort). Visualizes the proportion of episodes ending successfully versus failing from missing actions, incomplete deliverables, or reaching the step limit.

Figure G13: Check failure distribution compared to an independent error baseline (High Effort). Plots the difference between observed check failure frequencies and independent binomial predictions, showing error clustering across models.

Figure G14: Co-failure lift across task difficulty levels (High Effort). Displays empirical co-failure lift recomputed within leave-one-out peer difficulty groups (Easy, Medium, Hard). The final column isolates model-specific clustering after controlling for shared task difficulty across the 595 universally graded rounds (4,000 bootstrap draws).

Figure G15: Per-check accuracy versus failure clustering lift across models (High Effort). Compares average check accuracy against task-level error clustering, showing distinct failure patterns across models with similar average check pass rates.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

May 9, 2026

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture.

Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia +10

University of Utah · University of Maryland, Baltimore County · University of Dhaka +7

AI Agent Reliability Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Communication Topology Multi-Agent Communication LLM Agent Reliability

Ask PDF

Sep 25, 2026

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.

Raphael Shu, Yusen Zhang, Young Min Cho +7

OpenAgents · Columbia University · University of Pennsylvania +2

Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent LLMs Long-Horizon Agent Evaluation

Ask PDF

Oct 1, 2026

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.

Caiqi Zhang, Rujun Han, Zifeng Wang +4

University of Cambridge · Google Cloud AI Research

LLM Agent Verification Long-Horizon LLM Agents LLM Agent Harnesses

Ask PDF

Abstract

Computer-Use Agents

Explore similar work

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks