cs.MAOct 13, 2026
SaveCLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline
Organizations: Google · Emory University · University of Southern California · University of California, Santa Barbara ·
Abstract
Vision-language models (VLMs) are increasingly deployed in multi-agent systems to automate complex computer-use workflows, yet existing benchmarks mostly assume consistent, cooperative environments. In contrast, real-world collaboration rarely runs on consistent information: peer or sub-agents frequently report outdated states, issue contradictory instructions, or compete over shared permissions and files, forcing agents to check others' outputs and verify ground truth in the environment. Evaluating such conflict resolution remains challenging: manual authoring cannot scale across complex systems, while unconstrained LLM generation produces narrative drift, answer leakage, and unsatisfiable rubrics. We introduce CLASHarness (Conflict-Loaded Agent Scenario Harness), a controllable framework for generating verifiable multi-agent conflicts in computer use, with a benchmark of 600 multi-round tasks. By combining hierarchical scenario planning with an eight-step verified micro-pipeline, CLASHarness pre-commits decisive visual facts before rendering and compiles hybrid code and semantic graders directly from executed reference trajectories, guaranteeing every task is solvable by construction. Evaluating 15 frontier models on CLASHarness reveals three core bottlenecks. First, when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information, whereas stronger models inspect visual evidence to verify ground truth. Second, resolving conflicts is much harder than finding them: while agents can spot inconsistencies, actively fixing system state drops task success significantly. Third, coordination protocols remain a universal bottleneck: even frontier models fail over 58% of closing actions, often forgetting to release shared locks or hand off tasks. CLASHarness provides a scalable, verifiable foundation for building reliable multi-agent systems.
Figures & tables
Figure 1: Hierarchical generation pipeline in CLASHarness. Tier 1 initializes a shared multi-agent world from seeds and plugins. Tier 2 refreshes phase milestones every 3–4 rounds and generates each round through eight explicit sub-steps. Ground truth is committed before rendering; executed reference effects ground deterministic checks and scoped semantic criteria. The dashed return path carries workspace state to the next round ( ); after rounds, generation proceeds to Tier 3 for consistency auditing, visual refinement, and rendering. Visual evidence and checkers share the committed , with text-exclusivity gates enforced when visuals are required. Green tags mark deterministic processing; red tags mark bounded correction with diagnostic feedback. Detailed safeguards appear in Appendix Table B1 .
Figure 2: Harness parameter ablations. (a) Step budget: Expanding action horizons unlocks frontier models with high exploration demands while separating efficient planners from thinking loops. (b) Visual modality: Agent performance collapses without image inputs, confirming visual evidence is essential for conflict resolution. (c) Generator model: Relative rankings remain highly consistent across parallel benchmarks synthesized by different generator models.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B1: Photographic asset scenes in CLASHarness. Shows the four real-world scenes used across all 19 photographic assets in the benchmark. Each photograph contains a decisive ground-truth string (shown below each image) required to resolve the corresponding conflict.
Figure B2: Visual design templates across browser-rendered assets. Shows each of the 14 visual rendering styles across packages, ordered by frequency (asset counts in parentheses), spanning diverse UI formats such as metric dashboards, telemetry logs, console tables, and code diffs.
Figure B3: Multimodal evidence inspection in a worked investigation round. Illustrates the two visual assets required to satisfy all eight evaluation checks in round access_request_approval_v01_r01 : panel (a) presents the permission matrix containing the decisive conflict verdict ( UNRECONCILED ), while panel (b) contains the authorization passcode ( SEC-PASS-8842-BP ).
| # | Effect | Predicate | Criterion |
| 1 | assets_accessed | field_contains | id contains iam_console_access_scope_matrix_r01 [gate] |
| 2 | messages_sent | field_semantic_match | tell the manager the dashboard reads UNRECONCILED (2/3 approver scopes mismatched) and refuse the premature grant |
| 3 | messages_sent | field_semantic_match | relay passcode SEC-PASS-8842-BP to the security reviewer and request WRITE on billing-prod |
| 4 | approval_recorded | any_field_equals | billing_prod |
| 5 | approval_granted | any_field_equals | billing_prod |
| 6 | files_created | field_semantic_match | the report states the finding UNRECONCILED (2/3 approver scopes mismatched) |
Table B2: Compiled evaluation rubric for round access_request_approval_v01_r01 . Lists the eight scored checks compiled from the reference trajectory. Check 1 acts as a mandatory visual access gate ( role="required" ). Semantic matches are evaluated by the self-consistency ensemble judge.
Figure B4: Stale cache vs. authoritative ledger ground truth (round expense_reimbursement_conflict_v03_r03 ). Panel (a) presents the stale cache reporting 850.00 EUR ; panel (b) provides the itemized ground-truth receipt totaling 725.46 EUR ( ). Decisive numerical evidence is exclusive to the rendered image, requiring visual inspection.
Figure D1: Diagnostic check failure rates across failure modes (Ten 20-step API models, High Effort). Compares failure proportions across 3,284 scored checks spanning Content (F1, F2, F4), Visual Grounding (F3), and Structural Protocol (F5, F6) failure modes.
| Bench A | Bench B | |
| Generator model | Gemini 3.6 Flash | Claude Sonnet 5 |
| Step budget | 20 | 30 |
| GT step length, mean / median / max | 10.28 / 10 / 18 | 12.48 / 12 / 24 |
| Budget ratio (budget GT), mean | ||
| Rounds with budget ratio | 8.2% (49) | 1.2% (7) |
| Rounds with unexposed reference strings | 18 | 9 |
Table F1: Structural comparison of parallel generation benchmarks. Contrasts Bench A (Gemini 3.6 Flash) and Bench B (Claude Sonnet 5) across identical 20 scenario seeds and 600 aligned conflict tasks.
| (a) Step budget. Same 600 rounds, high effort; only the cap moves. | ||||||
| Pass Rate (%) | Ended at Cap (%) | |||||
| Model | 20 | 30 | 40 | 20 | 40 | |
| Claude Opus 5 | 25.2 | 25.7 | 27.0 | 0.5 | 0.0 | |
| Claude Sonnet 5 | 20.8 | 20.7 | 21.3 | 3.3 | 0.2 | |
| Gemini 3.7 Flash | 20.3 | 26.0 | 27.2 | 40.7 | 2.5 | |
| Gemini 3.6 Flash | 18.5 | 23.5 | 25.0 | 24.7 | 0.3 | |
Table F2: Comprehensive numerical results for harness parameter ablations. (a) Step budget: Pass rates and cap exhaustion when expanding horizon on Bench A ( ). (b) Visual modality: Performance of sighted vs. blind agents across full (600), asset-gated (346), and un-gated (254) task groups. (c) Generator difference: Pass rates across seven models evaluated on Bench A vs. Bench B (600 rounds each; ).
| Budget ( ) | Pass Rate (%) | 95% CI | Cap-Hit (%) | Calls/Task | Reason Tok | $/Solved |
| 12.8 (77/600) | 66.8 | 19.4 | 6,699 | $4.44 | ||
| 19.0 (114/600) | 46.8 | 27.3 | 8,976 | $4.40 | ||
| 29.2 (175/600) | 19.2 | 32.2 | 11,233 | $3.50 | ||
| 30.7 (184/600) | 3.3 | 35.0 | 11,446 | $3.71 | ||
| Comparator: Gemini 3.7 Flash (same judge, high effort) | ||||||
| 20.3 (122/600) | 40.7 | 17.8 | 4,026 | $2.51 | ||
Table F3: Action horizon scaling for Gemini 3.8 Flash ( rounds, High effort, Gemini 3.6 Flash judge). Metrics include task pass rate with Wilson score confidence intervals, percentage of runs terminating at the step cap, average tool calls, average reasoning tokens, cost per task at introductory pricing, and effective cost per solved task.
| Substitute judge | Gemini 3.6 Flash | Claude Sonnet 5 | Claude Opus 5 |
| Mean score (primary). | |||
| Rank correlation | 0.987 | 0.991 | 0.987 |
| excluding the self-scored row | 0.995 | 0.995 | 0.989 |
| Maximum rank displacement | 2 | 1 | 1 |
| Models whose rank moves | 3/14 | 4/14 | 6/14 |
| Mean shift | |||
Table F4: Cross-judge stability ablation. Every semantic check of the high-effort panel is re-evaluated offline by three substitute three-sample ensembles with agent trajectories held fixed: the replay reproduces deterministic verdicts, so the columns differ only in semantic opinion. The Gemini 3.6 Flash column re-runs the default judge, so it measures sampling noise rather than a judge change. Statistics use the of panel checks ( ) all three substitutes scored; the recorded verdict fills the rest in every column and cancels in cross-judge comparisons. Mean score leads because pass rate requires passing all checks, so one flipped check can move a model several places. The substitute judges reorder the models no more than the default judge reorders itself; they primarily shift absolute score levels. Claude Opus 5 raises all fourteen 20-step baseline models, unevenly enough to hand first place to GPT-5.6 Sol—the only change of leader in the table. The self-scored row that the two excluding lines drop is Gemini 3.7 Flash, which issued its own recorded verdicts.
Figure G1: Overall benchmark standings across the 15 evaluated models (High Effort). Displays task pass rate ( , wide bar) and mean check score (narrow bar, 0–1 scale) across all 600 benchmark tasks. Models are ranked by pass rate; indicates the difference between mean score rank and pass rank.
Figure G2: Action category glossary and invocation distribution. Partitions all 88 sub-commands in the action DSL into 12 mutually exclusive functional categories (call rates normalized per task across all 600 rounds). The detailed breakdown of domain-specific operations is provided in Figure G7 , and the full command inventory spans Figures G8 – G10 .
Figure G3: Pass rate versus mean check score across models (High Effort). Shows the relationship between all-or-nothing task completion (pass rate) and mean check score across all 15 models.
Figure G4: Pass rate versus cumulative evaluation cost (High Effort). Compares benchmark pass rate against total evaluation cost across models.
Figure G5: Token volume, monetary spend, and execution latency across models (High Effort). Breaks down token usage per task (input prompt, visible generation, internal reasoning), cost per task and per solved task, and execution latency across evaluated models.
Figure G6: Empirical action-category distributions across models (High Effort). Relative proportion of tool calls across the 12 action categories. Each bar sums to 100% of that model’s executed actions. The domain-specific operations grouped under other are detailed in Figure G7 .
Figure G7: Itemized distribution of domain-specific operations ( other category). Breaks down specialized environment operations (including containment locks, deployment approvals, resource pairing, and process scheduling) across models.
Figure G8: Complete command inventory across all evaluated models (Part 1: Rows 1–30 of 89). Displays individual sub-command invocation counts across all 600 tasks per model, mapped to functional categories. Continued in Figure G9 .
Figure G9: Complete command inventory across all evaluated models (Part 2: Rows 31–59 of 89). Continued from Figure G8 ; continued in Figure G10 .
Figure G10: Complete command inventory across all evaluated models (Part 3: Rows 60–89 of 89). Final part of the 88-command execution inventory across all 15 evaluated models.
Figure G11: Visual asset grounding volume versus task pass rate (High Effort). Plots mean asset opens per task against benchmark pass rate, illustrating the decoupling between visual inspection volume and reasoning accuracy.
Figure G12: Episode outcome and termination reason breakdown (High Effort). Visualizes the proportion of episodes ending successfully versus failing from missing actions, incomplete deliverables, or reaching the step limit.
Figure G13: Check failure distribution compared to an independent error baseline (High Effort). Plots the difference between observed check failure frequencies and independent binomial predictions, showing error clustering across models.
Figure G14: Co-failure lift across task difficulty levels (High Effort). Displays empirical co-failure lift recomputed within leave-one-out peer difficulty groups (Easy, Medium, Hard). The final column isolates model-specific clustering after controlling for shared task difficulty across the 595 universally graded rounds (4,000 bootstrap draws).
Figure G15: Per-check accuracy versus failure clustering lift across models (High Effort). Compares average check accuracy against task-level error clustering, showing distinct failure patterns across models with similar average check pass rates.
More in this topic
Computer-Use Agents
Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.
See all 160 papers