cs.LGApr 5, 2026
SaveClawArena: Benchmarking AI Agents in Evolving Information Environments
Organizations: UNC-Chapel Hill · University of California, Berkeley · University of California, Santa Cruz
Abstract
AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.
Figures & tables
Figure 1: Overview of ClawArena . An evolving information environment with conflicting and progressively updated evidence is structured along three coupled evaluation dimensions, instantiated through two complementary question formats, and summarized by a composite reliability score that jointly reflects accuracy and behavioural consistency.
| Benchmark | Task sourcing | Exec. mode | MSC | DU | MU | Pref. | Verification | Frmw. | Scale (Q / Scen.) |
|---|---|---|---|---|---|---|---|---|---|
| ClawBench | Manual pool | Live web | ✗ | ✗ | ✗ | ✗ | rule+llm | 8 | 283 / 144 sites |
| Claw-Eval | Curated from upstream | Sandbox + mock | ✗ | ✗ | ✓ | ✗ | rule+llm | 1 | 300 / 9 cats |
| Claw-Eval-Live | Live signals (quarterly) | Mock services | ✗ | ✗ | ✗ | ✗ | rule+llm | 1 | 105 / 17 fam. |
| ClawMark | Manual + AI synthesis | Sandboxed services | ✓ | ✓ | ✓ | ✗ | rule-based | 1 | 100 / 13 scen. |
| ClawsBench | Expert-designed | Mock services | ✓ | ✗ | ✗ | ✗ | rule-based | 4 | 44 / 5 svc. |
| MetaClawBench | Synthetic | Workspace sim | ◐ | ✓ | ✓ | ◐ | rule-based | 1 | 346 / 30 days |
Table 1: Comparison with harness-native agent benchmarks. ✓ = supported, ✗ = unsupported, ◐ = partial (mechanism present but weakened, e.g., third-party messages exposed only as plain files rather than channel-tagged sessions, or preferences encoded as static persona rather than learned in silent rounds). MSC: agent must analyse user–third-party conversations distinguished by channel. DU: environment is overwritten between user turns (intra-loop tool-return dynamics do not count). MU: user re-engages with new queries across rounds. Pref.: implicit user preferences applied in silent-exam rounds.
| Tag | Dimensions | Representative question |
| Single dimension | ||
| MS | Multi-Source | Identify which statements are supported across a direct message, a group chat, and a log export. |
| DU | Dynamic Update | Update the conclusion after a new audit file or appended message arrives. |
| P | Personalization | Recall or apply the learned output style without an explicit reminder. |
| Pairwise interaction | ||
| MS DU | MS DU | Reassess a cross-source claim after the evidence state changes. |
Table 2: Question taxonomy of ClawArena . Each row shows a non-empty subset of {MS, DU, P} and a representative task.
Figure 2: ClawArena construction pipeline. Real-world distributions and character profiles feed a three-stage bootstrap, producing the 12-scenario benchmark organized into six layers and refined through three validation passes.
| Model | TCR | Robustness | CRS | ||||
|---|---|---|---|---|---|---|---|
| Avg | MC | EC | Overall | SC | FD | ||
| Proprietary | |||||||
| GPT-5.5 | 78.34 | 75.79 | 79.34 | 58.22 | 61.24 | 95.06 | 68.28 |
| GPT-5.1 | 70.33 | 75.79 | 68.18 | 56.24 | 58.96 | 95.37 | 63.28 |
| GPT-5.4 | 71.22 | 71.58 | 71.07 | 46.77 | 51.51 | 90.78 | 58.99 |
| Gemini-3.1-Pro | 69.57 | 66.32 | 71.07 | 45.62 | 50.54 | 90.23 | 57.59 |
Table 3: Cross-model comparison on 12 scenarios under OpenClaw. Models are partitioned into proprietary (top) and open-weight (middle), each sorted by CRS. The bottom block reports three Anthropic models under their provider-native Claude Code harness (incompatible with OpenClaw); their numbers are not directly comparable with the OpenClaw rows above. Bold marks the best per column within each of the first two groups; underline marks the runner-up. † denotes self-hosted vLLM deployment; all others use provider APIs.
| Model | Framework | TCR | Robustness | CRS | ||||
|---|---|---|---|---|---|---|---|---|
| Avg | MC | EC | Overall | SC | FD | |||
| GPT-5.5 | OpenClaw | 78.34 | 75.79 | 79.34 | 58.22 | 61.24 | 95.06 | 68.28 |
| Claude Code | 59.31 | 67.37 | 55.79 | 30.30 | 36.27 | 81.84 | 44.50 | |
| PicoClaw | 77.15 | 74.74 | 78.10 | 56.66 | 59.49 | 94.74 | 66.87 | |
| NanoBot | 73.00 | 72.63 | 73.14 | 50.98 | 55.24 | 90.76 | 61.88 | |
| GPT-5.1 | OpenClaw | 70.33 | 75.79 | 68.18 | 56.24 | 58.96 | 95.37 | 63.28 |
Table 4: Cross-framework comparison. Three models are evaluated across all four frameworks. Bold marks the best per column within each model block; underline marks the runner-up.
| Model | Config | TCR | Robustness | CRS | |||
|---|---|---|---|---|---|---|---|
| Avg | MC | EC | SC | FD | |||
| GPT-5.5 | Baseline | 78.34 | 75.79 | 79.34 | 61.24 | 95.06 | 68.28 |
| +MetaClaw skills | 78.86 | 75.79 | 80.58 | 61.90 | 95.19 | 68.89 (+0.61) | |
| GPT-5.1 | Baseline | 70.33 | 75.79 | 68.18 | 58.96 | 95.37 | 63.28 |
| +MetaClaw skills | 70.62 | 74.74 | 69.01 | 59.05 | 96.22 | 63.72 (+0.44) | |
| GLM-5.1 | Baseline | 72.70 | 72.63 | 72.73 | 52.74 | 92.07 | 60.63 |
Table 5: Skill-driven self-evolution ablation. Each pair shares the same model and base OpenClaw configuration; the +MetaClaw variant adds skill injection. CRS is the change relative to the baseline.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Cross-domain data sample gallery from ClawArena . Each tile presents one scenario in a distinct professional context, highlighting the workspace, session sources, evaluation question, and an evidence chain that traverses conflicting and time-evolving observations. The center logo reflects the benchmark’s adversarial spirit: agents must “claw” through conflicting evidence to reach the ground truth.
| Provider | Access | Models |
|---|---|---|
| OpenAI | Codex API | GPT-5.5, GPT-5.4, GPT-5.2 |
| Azure OpenAI | Azure endpoint | GPT-5.1, Kimi-K2.5 |
| Anthropic | Claude OAuth | Claude Opus-4.7, Claude Sonnet-4.6, Claude Haiku-4.5 |
| OpenRouter | OpenRouter API | GLM-5.1 ( z-ai ), GLM-4.7-Flash ( z-ai ), Qwen3.6-Plus |
| ( qwen ), DeepSeek-V4-Pro ( deepseek ), Gemini-3.1-Pro | ||
| ( google ), Mimo-V2.5-Pro ( xiaomi ), Ling-2.6 |
Table 6: Provider routing for all evaluated models.
| Model | TP | max-model-len | tool-call-parser | reasoning-parser | KV cache |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 4 | 204,800 | qwen3_coder | qwen3 | bf16 |
| Gemma-4-31B-it | 4 | 204,800 | gemma4 | gemma4 | fp8 |
Table 7: vLLM serving configuration for self-hosted models. TP denotes tensor parallelism; max-model-len is the served context length; tool-call-parser and reasoning-parser select the vLLM output-format adapters.
| Source | OpenClaw | Claude Code | NanoBot | PicoClaw | ||||
|---|---|---|---|---|---|---|---|---|
| Tokens | % | Tokens | % | Tokens | % | Tokens | % | |
| Main session | 10,457 | 0.221 | 4,170 | 0.088 | 7,976 | 0.168 | 7,989 | 0.168 |
| History sessions | 158,090 | 3.335 | 167,139 | 3.531 | 167,139 | 3.519 | 159,359 | 3.361 |
| Workspace (base) | 1,483,834 | 31.307 | 1,472,224 | 31.103 | 1,484,686 | 31.260 | 1,484,566 | 31.305 |
| Workspace update | 2,797,037 | 59.015 | 2,797,133 | 59.095 | 2,797,037 | 58.891 | 2,797,037 | 58.983 |
| Session update | 122,271 | 2.580 | 124,800 | 2.637 | 124,800 | 2.628 | 125,343 | 2.643 |
Table 8: Token distribution across input sources in ClawArena , reported independently for each of the four compatible frameworks. All counts use the cl100k_base tokenizer; jsonl session files are parsed into a normalized message sequence and rendered into the ChatML template before counting, while markdown transcripts and all non-session artefacts are counted verbatim (see Appendix E narrative).
| Dimension | Metric | Mean | Min | Max |
| Multi-choice | Options per round | 6.38 | 5 | 10 |
| Correct answers per round | 4.46 | 2 | 8 | |
| Executable-check | Sandbox timeout (s) | 40.2 | 30 | 60 |
| expect_exit coverage (%) | 100.0 | — | — | |
| expect_stdout coverage (%) | 0.4 | — | — |
Table 9: Multi-choice prompt shape and executable-check feature coverage in ClawArena . MC statistics are computed over all 95 multi-choice rounds; EC statistics are computed over all 242 executable-check rounds.
| Facet | Category | Count | % |
|---|---|---|---|
| Target (events) | Workspace-targeted | 42 | 50.6 |
| Session-targeted | 37 | 44.6 | |
| Group-targeted | 4 | 4.8 | |
| Action (files) | append | 14 | 56.0 |
| new | 11 | 44.0 |
Table 10: Staged-update distribution. Target reports the 83 update events; Action reports the 25 file-level update operations triggered by those events.
| Scenario | Rounds | MC | EC | w/Pref | w/Upd | Upd | UpdF | WSF | Tokens |
|---|---|---|---|---|---|---|---|---|---|
| hil_s1 | 24 | 8 | 16 | 3 | 4 | 8 | 25 | 24 | 4,344,165 |
| hil_c7 | 28 | 8 | 20 | 2 | 3 | 6 | 9 | 12 | 44,210 |
| hil_d3 | 30 | 8 | 22 | 2 | 4 | 8 | 10 | 12 | 44,137 |
| hil_e4 | 24 | 7 | 17 | 2 | 3 | 6 | 6 | 11 | 40,112 |
| hil_g4 | 27 | 8 | 19 | 2 | 3 | 7 | 8 | 10 | 37,429 |
| hil_g1 | 30 | 8 | 22 | 3 | 4 | 7 | 8 | 10 | 36,077 |
Table 11: Per-scenario breakdown under the OpenClaw layout. Columns: rounds, multi-choice rounds (MC), executable-check rounds (EC), rounds with a preference (w/Pref), rounds exposing updates (w/Upd), total update events (Upd), files touched by updates (UpdF), baseline workspace files (WSF, excluding framework-injected session transcripts), and total tokens.
Figure 4: Case 1 (MS-R, NexaFlow API breach, q7 + q25): both GPT-5.1 configurations omit the top-level 2,340-record confirmation in q7 despite correctly absorbing five other log-derived facts; q25 exhibits five distinct failure modes across models, with only GPT-5.1 (OC), Mimo V2.5, and Ling-2.6 achieving a perfect reliability ranking. Case 2 (exec_check, hil_g4/q3 and hil_i2/q4 ): GPT-5.1 under both OpenClaw and Claude Code fails both tasks with identical schema-compliance errors, while six OpenClaw-hosted models pass hil_i2/q4 at a 6/6 rate—isolating the failure to the model rather than the framework.
Figure 5: Case 3 (MS-R, research integrity, q1 + q2): q1 is the only round in the benchmark with a 0% exact-match across all eight models, because the correct options are compound relational statements (e.g., IRB-precedes-extraction). Case 4 (DU-R, authority-influenced revision, q5 + q17): a single IRB pre-registration option is missed by the majority in both rounds, while all eight configurations correctly decline the overtly false “endorsed complaint” and “recommended retraction” options—failure here is one of omission, not capitulation to authority.
Figure 6: Case 5 (P-R + exec_check, wrongful-termination report, q18 + q26): GPT-5.1 under both frameworks fails both rounds by omitting the YYYY-MM-DD_ filename prefix; Mimo V2.5 and Gemini-3.1 pass the lenient midterm check but fall off once q26 layers three literal ISO dates and 5 section headings. Case 6 (exec_check, NexaFlow GDPR bundle, q15 + q23 + q26): all three questions achieve 0/8 pass rates; the shared failure mode is compound output-format compliance (timezone-aware timestamps, typed JSON fields with provenance citation, enum-constrained priority labels), defining a current structural-output ceiling.
Figure 7: Case 7 (DU-R, 618 GPU fraud, q7 + q13 + q25): failure clusters are update-specific rather than cumulative—four models conflate the CNY 32,000 partial refund with full-price restitution on q13, while Ling-2.6 and DeepSeek-V4 show isolated answer-extraction failures when the response format drifts from their usual \bbox{} emission. Case 8 (exec_check, JSON schema adherence, hil_g4/q3 + hil_f7/q21 + hil_f7/q22 ): pass rates track schema documentation style—only 2/8 pass hil_g4/q3 whose key is described in natural language, versus 6/8 on hil_f7/q21 where the four required keys are listed verbatim.
Figure 8: Case 9 (MS-R + DU-R, wrongful termination, q13 + q27): four models miss the option where Sun Wei admits the termination conversation lacked an email record—a claim that requires conjoining two elements of the same reply; in q27 seven of eight achieve perfect synthesis and only DeepSeek-V4 drops the two-part PIP-violation statement. Case 10 (exec_check + MS-R, pipeline authorship and final synthesis, hil_i2/q8 + hil_c7/q28 ): only GPT-5.1 (OC) passes the M2-contrast write-up (1/8); on the final synthesis GPT-5.1 (CC), Mimo V2.5, and GLM-5.1 miss the same forward-looking remediation option, while DeepSeek-V4 produces no extractable answer.
More in this topic
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 519 papers