cs.AIAug 23, 2026

ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

Authors: YuanHang Xiao

Organizations: The Chinese University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.

Figures & tables

Figure 1: ClawProBench overview. A 102-scenario runtime-coverage profile and a frozen 68-scenario realistic holdout feed the same live execution, trace capture, status reporting, and safety-gated scoring pipeline.
Figure 2: Full-profile scenario composition. The 102-scenario profile covers six capability dimensions, is deliberately skewed toward hard/expert tasks (91/102), and separates 66 workspace-live tasks from 36 OpenClaw-native tasks. Native-surface counts are non-mutually exclusive because one scenario can exercise multiple surfaces.
QuestionResultInterpretation
SaturationTop full-profile score = 0.7671; top-5 spread = 0.0661Strong models remain separated below the ceiling.
Native surfacesNative mean 0.5238 vs. workspace mean 0.6415; stratified gap remains >0.10>0.10The native slice supplies complementary, non-causal diagnostic signal beyond file/evidence tasks.
Status semanticsClean-base top-15 overlap = 11/15; execution-failure entries contribute 1/15Hiding run status changes the interpretation of leaderboard comparability.
Holdout reliabilityHoldout pass@k-any 0.6638 vs. strict pass 0.2890Solving once is much easier than solving reliably.
Rank robustnessFull vs. holdout: ρ=0.1754\rho=0.1754 , tie-aware 95% CI [−0.23,0.54][-0.23,0.54] ; full vs. native: ρ=0.7612\rho=0.7612The holdout adds a high-uncertainty ranking view rather than a redundant smaller leaderboard.
Scoring sensitivityUnweighted correctness mean vs. source composite: ρ=0.8060\rho=0.8060 ; max rank shift = 46 ( N=66N=66 )The views are not interchangeable; the controlled C/PC/P weight sweep isolates weight sensitivity.
Table 6: Diagnostic evidence beyond the leaderboard. Primary expanded views use the 29-model cross-profile denominator and all 66 component-resolved scoring rows; positive-score filtered views remain available in Appendix E .
Figure 3: Frozen-holdout reliability and rank robustness. Over 37 evaluated 68-scenario holdout entries, pass@k-any exceeds strict three-trial success (0.6638 vs. 0.2890). The plotted rank view is the 27-model positive-score comparison (Spearman 0.1300) with a wide 95% bootstrap interval; the denominator-complete 29-model estimate is reported in the text and Table 27 .
Figure 4: Scoring views are not interchangeable. The plotted fixed-denominator view covers 60 resolved positive-score full-profile reports: correctness-only scores correlate with the manifest ranking (Spearman 0.8334) but move one model by as many as 42 positions. The denominator-complete 66-row correctness view is reported in the text and Table 42 ; it increases the maximum shift to 46.
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

Explore similar work

CardsList
Jun 2, 2026

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introduce RealClawBench, a live benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use. Real user requests are challenging to benchmark because they often depend on local execution environments, involve implicit or underspecified intent, and require nontrivial verification. RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers, which together convert real sessions into reproducible, automatically scored tasks. The resulting release contains 281 executable tasks sampled from a much larger real-session pool while preserving the source distribution, with maximum final-vs-source Jensen-Shannon divergence of 0.0448. Evaluating 14 contemporary models shows that the best system solves only 65.8% of tasks, revealing substantial headroom on realistic developer-agent workloads. By turning real deployed sessions into controlled evaluation instances, RealClawBench provides a practical path toward benchmarks that better measure agent capability in actual use. Code is available at:https://anonymous.4open.science/r/real-claw-bench-582B.

Zongwei Lv, Zhewen Tan, Yaoming Li +7

1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*

Agent BenchmarksAI Coding AgentsSoftware Engineering BenchmarksAI Agent BenchmarksAutomated EvaluationComputer-Use Agent Benchmarks+3
AskPDF
Apr 7, 2026

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.

Bowen Ye, Rang Li, Qibin Yang +10

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong

AI Agent EvaluationAI Agent BenchmarksAI Agent ReliabilityAI Agent Safety Benchmarks
AskPDF
Jul 30, 2026

ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.

Xingjian Wu, Xuhang Zhu, Xingchen Liu +6

Meituan

AI Agent ReliabilityLLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent BenchmarksLLM Agent TrainingProcess-Based Agent Evaluation
AskPDF