Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

Bowen Ye; Rang Li; Qibin Yang; Yuanxin Liu; Linli Yao; Hanglong Lv; Zhihui Xie; Chenxin An

cs.AIApr 7, 2026

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

Authors: Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An,

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong

Abstract

Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.

Figures & tables

Benchmark	Multi- modal	Multi- turn	Auditable	Safety	Pertur- bation	Sand- boxed
AgentBench [ 17 ]	✗	✓	❖	✗	✗	✓
GAIA [ 21 ]	❖	✗	✗	✗	✗	✓
$\tau$ -bench [ 50 ]	✗	✓	✓	✓	✗	✗
SWE-bench [ 10 ]	✗	✗	❖	✗	✗	✓
WebArena [ 56 ]	❖	✗	❖	✗	✗	✓
VisualWebArena [ 12 ]	✓	✗	❖	✗	✗	✓

Table 1: Comparison of agent evaluation benchmarks across six dimensions: Multimodal (visual input and artifact generation), Multi-turn (simulated user interaction), Auditable (trajectory-level grading), Safety (embedded constraint checking), Perturbation (controlled error injection), and Sandboxed (isolated execution environment). ✓ = full support; ❖ = partial; ✗ = absent.

Figure 1: Overview of the Claw-Eval architecture. Each run proceeds through three phases: Setup, Execution, and Judge, separated by a temporal firewall that prevents grading artifacts from being exposed during execution. Scoring is grounded in three independent evidence channels: execution traces, service-side audit logs, and environment snapshots.

Group	Category	Description	#
General (161)	Easy	Single-service queries, Basic scheduling	71
	Medium	Cross-service coordination, Data retrieval	47
	Hard	Multi-system orchestration, Financial compliance, Ops	43
Multimodal (101)	Video	Simple QA, Video localization	53
	Doc & Image	Chart interpretation, Cross-page reasoning	22
	Code	Webpage generation, SVG animation, Video editing	26

Table 2: Benchmark composition. 300 tasks span three groups testing complementary capabilities: practical workflow execution (General), visual perception and generation (Multimodal), and proactive information acquisition (Multi-turn Dialogue), across 9 fine-grained categories. The tasks are human-curated to cover diverse modalities, interaction patterns, and professionally grounded scenarios representative of realistic agent deployments, in the spirit of recent benchmarks [ 27 , 19 , 48 , 52 ] .

Model	General			Multi-turn			Overall
Model	Score	Pass@3	Pass^3	Score	Pass@3	Pass^3	Score	Pass@3	Pass^3
Claude Opus 4.6	80.6	80.8	70.8	79.6	89.5	68.4	80.4	82.4	70.4
Claude Sonnet 4.6	81.3	81.4	68.3	81.9	89.5	65.8	81.4	82.9	67.8
GPT 5.4	78.3	75.8	60.2	79.0	89.5	60.5	78.4	78.4	60.3
Gemini 3.1 Pro	76.6	80.8	55.9	80.2	92.1	65.8	77.3	82.9	57.8
MiMo V2 Pro	76.0	72.7	57.1	81.0	92.1	60.5	77.0	76.4	57.8

Table 3: Main evaluation results. Score = average task score (%), Pass@3 = pass@3 rate (%, any trial passes), Pass^3 = pass^3 rate (%, all trials pass). Best results per column are bolded ; second best are underlined . Models are sorted by Pass^3, then by Pass@3 for tied cases.

Panel of Figure 3 — Figure 3: Comparison of a vanilla LLM judge against the hybrid grading pipeline across five models. The vanilla judge receives the full conversation transcript and complete grader source code; only server-side audit logs and post-execution environment snapshots are withheld.

Panel of Figure 4 — Figure 4: Effect of error injection on agent performance. Three models are evaluated on General tasks at error rates 0.0–0.6. Pass@3 remains largely stable while Pass^3 drops sharply.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Functional Group	Tools	Purpose
System Layer
Code Execution	Bash	Execute shell commands
File Operations	Read , Write , Edit	Read, create, and modify files
Codebase Search	Glob , Grep	Find files by pattern; Search content by regex
Web Interaction	BrowserScreenshot , WebSearch , WebFetch	Capture screenshots; Search and fetch web pages
Multimodal Media	ReadMedia , Download	Process video/image/PDF; Download files

Appendix

Table 6: Agent capability layers. The system layer provides 11 built-in tools covering the core action space of real-world agent tasks; the service layer exposes task-specific mock APIs that simulate production services.

Subset	# Cases	Hybrid agreement (%)	Vanilla agreement (%)
Safety disagreements	12	100.0	0.0
Robustness disagreements	15	86.7	6.7
Jointly positive controls	50	94.0	94.0
Jointly negative controls	50	96.0	96.0

Appendix

Table 7: Blinded human adjudication on hybrid–vanilla disagreement cases, augmented with a matched control set. For jointly labeled controls, the two systems necessarily have identical agreement with the adjudicated human outcome.

Task group	# Tasks	Aligned (%)	Coverage adequate (%)	Evidence-grounded (%)
General	100	95.0	98.0	98.0
Multimodal	50	92.0	92.0	94.0
Multi-turn	30	96.7	96.7	100.0
Overall	180	94.4	96.1	97.2

Appendix

Table 8: Human audit of rubric adequacy and coverage for tasks containing judgment-based rubric components. Values denote the percentage of audited tasks satisfying each criterion.

Task group	Deployed judge	# Items	Exact agreement (%)
General	Gemini-3-Flash	100	96.0
Multimodal	Gemini-3-Flash	100	95.0
Multi-turn	Claude-Opus-4.6	100	94.0
Overall	—	300	95.0

Appendix

Table 9: Alignment between the deployed LLM judges and human reference scoring on sampled judgment-based rubric items. Exact agreement denotes exact score match.

Model	General	Multi-turn	Multimodal
Claude Opus 4.6	80.6 $\pm$ 1.2	79.6 $\pm$ 2.0	54.7 $\pm$ 0.6
Claude Sonnet 4.6	81.3 $\pm$ 0.6	81.9 $\pm$ 0.3	50.9 $\pm$ 1.7
GPT 5.4	78.3 $\pm$ 1.6	79.0 $\pm$ 2.5	54.4 $\pm$ 1.9
Gemini 3.1 Pro	76.6 $\pm$ 0.5	80.2 $\pm$ 0.6	45.7 $\pm$ 0.8
MiMo V2 Pro	76.0 $\pm$ 0.5	81.0 $\pm$ 0.8	–
Qwen 3.5 397A17B	73.8 $\pm$ 1.2	75.6 $\pm$ 1.4	50.2 $\pm$ 5.0

Appendix

Table 10: Average score

\pm

standard deviation (

\%

) across three independent trials, grouped by task type. Row order mirrors Table 3 .

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

Jul 30, 2026

ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.

Xingjian Wu, Xuhang Zhu, Xingchen Liu +6

Meituan

AI Agent Reliability LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Agent Training Process-Based Agent Evaluation

Ask PDF

Apr 6, 2026

ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification. We release the trajectories and future dataset at https://clawsbench.com.

Xiangyi Li, Kyoung Whan Choe, Yimin Liu +12

BenchFlow · RLWRLD · Ohio State University +9

LLM Agent Evaluation LLM Agents AI Agent Safety Benchmarks

Ask PDF

May 11, 2026

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

Shuangrui Ding, Xuanlang Dai, Long Xing +14

1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8

LLM Agent Harnesses AI Agent Benchmarks Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents Terminal Agents LLM Agent Evaluation+4

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation