Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.
Figures & tables
Benchmark
Multi- modal
Multi- turn
Auditable
Safety
Pertur- bation
Sand- boxed
AgentBench [ 17 ]
✗
✓
❖
✗
✗
✓
GAIA [ 21 ]
❖
✗
✗
✗
✗
✓
τ -bench [ 50 ]
✗
✓
✓
✓
✗
✗
SWE-bench [ 10 ]
✗
✗
❖
✗
✗
✓
WebArena [ 56 ]
❖
✗
❖
✗
✗
✓
VisualWebArena [ 12 ]
✓
✗
❖
✗
✗
✓
Table 1: Comparison of agent evaluation benchmarks across six dimensions: Multimodal (visual input and artifact generation), Multi-turn (simulated user interaction), Auditable (trajectory-level grading), Safety (embedded constraint checking), Perturbation (controlled error injection), and Sandboxed (isolated execution environment). ✓ = full support; ❖ = partial; ✗ = absent.
Figure 1: Overview of the Claw-Eval architecture. Each run proceeds through three phases: Setup, Execution, and Judge, separated by a temporal firewall that prevents grading artifacts from being exposed during execution. Scoring is grounded in three independent evidence channels: execution traces, service-side audit logs, and environment snapshots.
Table 2: Benchmark composition. 300 tasks span three groups testing complementary capabilities: practical workflow execution (General), visual perception and generation (Multimodal), and proactive information acquisition (Multi-turn Dialogue), across 9 fine-grained categories. The tasks are human-curated to cover diverse modalities, interaction patterns, and professionally grounded scenarios representative of realistic agent deployments, in the spirit of recent benchmarks [ 27 , 19 , 48 , 52 ] .
Model
General
Multi-turn
Overall
Score
Pass@3
Pass^3
Score
Pass@3
Pass^3
Score
Pass@3
Pass^3
Claude Opus 4.6
80.6
80.8
70.8
79.6
89.5
68.4
80.4
82.4
70.4
Claude Sonnet 4.6
81.3
81.4
68.3
81.9
89.5
65.8
81.4
82.9
67.8
GPT 5.4
78.3
75.8
60.2
79.0
89.5
60.5
78.4
78.4
60.3
Gemini 3.1 Pro
76.6
80.8
55.9
80.2
92.1
65.8
77.3
82.9
57.8
MiMo V2 Pro
76.0
72.7
57.1
81.0
92.1
60.5
77.0
76.4
57.8
Table 3: Main evaluation results. Score = average task score (%), Pass@3 = pass@3 rate (%, any trial passes), Pass^3 = pass^3 rate (%, all trials pass). Best results per column are bolded ; second best are underlined . Models are sorted by Pass^3, then by Pass@3 for tied cases.
Figure 5
Figure 3: Comparison of a vanilla LLM judge against the hybrid grading pipeline across five models. The vanilla judge receives the full conversation transcript and complete grader source code; only server-side audit logs and post-execution environment snapshots are withheld.
Figure 4: Effect of error injection on agent performance. Three models are evaluated on General tasks at error rates 0.0–0.6. Pass@3 remains largely stable while Pass^3 drops sharply.
Figure 5: Multi-turn dialogue analysis across 13 models. A simulated user progressively reveals information over up to 8 rounds; the agent must ask clarifying questions before composing its answer.
Figure 9
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Functional Group
Tools
Purpose
System Layer
Code Execution
Bash
Execute shell commands
File Operations
Read , Write , Edit
Read, create, and modify files
Codebase Search
Glob , Grep
Find files by pattern; Search content by regex
Web Interaction
BrowserScreenshot , WebSearch , WebFetch
Capture screenshots; Search and fetch web pages
Multimodal Media
ReadMedia , Download
Process video/image/PDF; Download files
Appendix
Table 6: Agent capability layers. The system layer provides 11 built-in tools covering the core action space of real-world agent tasks; the service layer exposes task-specific mock APIs that simulate production services.
Subset
# Cases
Hybrid agreement (%)
Vanilla agreement (%)
Safety disagreements
12
100.0
0.0
Robustness disagreements
15
86.7
6.7
Jointly positive controls
50
94.0
94.0
Jointly negative controls
50
96.0
96.0
Appendix
Table 7: Blinded human adjudication on hybrid–vanilla disagreement cases, augmented with a matched control set. For jointly labeled controls, the two systems necessarily have identical agreement with the adjudicated human outcome.
Task group
# Tasks
Aligned (%)
Coverage adequate (%)
Evidence-grounded (%)
General
100
95.0
98.0
98.0
Multimodal
50
92.0
92.0
94.0
Multi-turn
30
96.7
96.7
100.0
Overall
180
94.4
96.1
97.2
Appendix
Table 8: Human audit of rubric adequacy and coverage for tasks containing judgment-based rubric components. Values denote the percentage of audited tasks satisfying each criterion.
Task group
Deployed judge
# Items
Exact agreement (%)
General
Gemini-3-Flash
100
96.0
Multimodal
Gemini-3-Flash
100
95.0
Multi-turn
Claude-Opus-4.6
100
94.0
Overall
—
300
95.0
Appendix
Table 9: Alignment between the deployed LLM judges and human reference scoring on sampled judgment-based rubric items. Exact agreement denotes exact score match.
Model
General
Multi-turn
Multimodal
Claude Opus 4.6
80.6 ± 1.2
79.6 ± 2.0
54.7 ± 0.6
Claude Sonnet 4.6
81.3 ± 0.6
81.9 ± 0.3
50.9 ± 1.7
GPT 5.4
78.3 ± 1.6
79.0 ± 2.5
54.4 ± 1.9
Gemini 3.1 Pro
76.6 ± 0.5
80.2 ± 0.6
45.7 ± 0.8
MiMo V2 Pro
76.0 ± 0.5
81.0 ± 0.8
–
Qwen 3.5 397A17B
73.8 ± 1.2
75.6 ± 1.4
50.2 ± 5.0
Appendix
Table 10: Average score ± standard deviation ( % ) across three independent trials, grouped by task type. Row order mirrors Table 3 .
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8