AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.
Figures & tables
Figure 1: Overview of ClawArena . An evolving information environment with conflicting and progressively updated evidence is structured along three coupled evaluation dimensions, instantiated through two complementary question formats, and summarized by a composite reliability score that jointly reflects accuracy and behavioural consistency.
Benchmark
Task sourcing
Exec. mode
MSC
DU
MU
Pref.
Verification
Frmw.
Scale (Q / Scen.)
ClawBench
Manual pool
Live web
✗
✗
✗
✗
rule+llm
8
283 / 144 sites
Claw-Eval
Curated from upstream
Sandbox + mock
✗
✗
✓
✗
rule+llm
1
300 / 9 cats
Claw-Eval-Live
Live signals (quarterly)
Mock services
✗
✗
✗
✗
rule+llm
1
105 / 17 fam.
ClawMark
Manual + AI synthesis
Sandboxed services
✓
✓
✓
✗
rule-based
1
100 / 13 scen.
ClawsBench
Expert-designed
Mock services
✓
✗
✗
✗
rule-based
4
44 / 5 svc.
MetaClawBench
Synthetic
Workspace sim
◐
✓
✓
◐
rule-based
1
346 / 30 days
Table 1: Comparison with harness-native agent benchmarks. ✓ = supported, ✗ = unsupported, ◐ = partial (mechanism present but weakened, e.g., third-party messages exposed only as plain files rather than channel-tagged sessions, or preferences encoded as static persona rather than learned in silent rounds). MSC: agent must analyse user–third-party conversations distinguished by channel. DU: environment is overwritten between user turns (intra-loop tool-return dynamics do not count). MU: user re-engages with new queries across rounds. Pref.: implicit user preferences applied in silent-exam rounds.
Tag
Dimensions
Representative question
Single dimension
MS
Multi-Source
Identify which statements are supported across a direct message, a group chat, and a log export.
DU
Dynamic Update
Update the conclusion after a new audit file or appended message arrives.
P
Personalization
Recall or apply the learned output style without an explicit reminder.
Pairwise interaction
MS × DU
MS × DU
Reassess a cross-source claim after the evidence state changes.
Table 2: Question taxonomy of ClawArena . Each row shows a non-empty subset of {MS, DU, P} and a representative task.
Figure 2: ClawArena construction pipeline. Real-world distributions and character profiles feed a three-stage bootstrap, producing the 12-scenario benchmark organized into six layers and refined through three validation passes.
Model
TCR
Robustness
CRS
Avg
MC
EC
Overall
SC
FD
Proprietary
GPT-5.5
78.34
75.79
79.34
58.22
61.24
95.06
68.28
GPT-5.1
70.33
75.79
68.18
56.24
58.96
95.37
63.28
GPT-5.4
71.22
71.58
71.07
46.77
51.51
90.78
58.99
Gemini-3.1-Pro
69.57
66.32
71.07
45.62
50.54
90.23
57.59
Table 3: Cross-model comparison on 12 scenarios under OpenClaw. Models are partitioned into proprietary (top) and open-weight (middle), each sorted by CRS. The bottom block reports three Anthropic models under their provider-native Claude Code harness (incompatible with OpenClaw); their numbers are not directly comparable with the OpenClaw rows above. Bold marks the best per column within each of the first two groups; underline marks the runner-up. † denotes self-hosted vLLM deployment; all others use provider APIs.
Model
Framework
TCR
Robustness
CRS
Avg
MC
EC
Overall
SC
FD
GPT-5.5
OpenClaw
78.34
75.79
79.34
58.22
61.24
95.06
68.28
Claude Code
59.31
67.37
55.79
30.30
36.27
81.84
44.50
PicoClaw
77.15
74.74
78.10
56.66
59.49
94.74
66.87
NanoBot
73.00
72.63
73.14
50.98
55.24
90.76
61.88
GPT-5.1
OpenClaw
70.33
75.79
68.18
56.24
58.96
95.37
63.28
Table 4: Cross-framework comparison. Three models are evaluated across all four frameworks. Bold marks the best per column within each model block; underline marks the runner-up.
Model
Config
TCR
Robustness
CRS
Avg
MC
EC
SC
FD
GPT-5.5
Baseline
78.34
75.79
79.34
61.24
95.06
68.28
+MetaClaw skills
78.86
75.79
80.58
61.90
95.19
68.89 (+0.61)
GPT-5.1
Baseline
70.33
75.79
68.18
58.96
95.37
63.28
+MetaClaw skills
70.62
74.74
69.01
59.05
96.22
63.72 (+0.44)
GLM-5.1
Baseline
72.70
72.63
72.73
52.74
92.07
60.63
Table 5: Skill-driven self-evolution ablation. Each pair shares the same model and base OpenClaw configuration; the +MetaClaw variant adds skill injection. Δ CRS is the change relative to the baseline.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Cross-domain data sample gallery from ClawArena . Each tile presents one scenario in a distinct professional context, highlighting the workspace, session sources, evaluation question, and an evidence chain that traverses conflicting and time-evolving observations. The center logo reflects the benchmark’s adversarial spirit: agents must “claw” through conflicting evidence to reach the ground truth.
Provider
Access
Models
OpenAI
Codex API
GPT-5.5, GPT-5.4, GPT-5.2
Azure OpenAI
Azure endpoint
GPT-5.1, Kimi-K2.5
Anthropic
Claude OAuth
Claude Opus-4.7, Claude Sonnet-4.6, Claude Haiku-4.5
Table 6: Provider routing for all evaluated models.
Model
TP
max-model-len
tool-call-parser
reasoning-parser
KV cache
Qwen3.6-35B-A3B
4
204,800
qwen3_coder
qwen3
bf16
Gemma-4-31B-it
4
204,800
gemma4
gemma4
fp8
Appendix
Table 7: vLLM serving configuration for self-hosted models. TP denotes tensor parallelism; max-model-len is the served context length; tool-call-parser and reasoning-parser select the vLLM output-format adapters.
Source
OpenClaw
Claude Code
NanoBot
PicoClaw
Tokens
%
Tokens
%
Tokens
%
Tokens
%
Main session
10,457
0.221
4,170
0.088
7,976
0.168
7,989
0.168
History sessions
158,090
3.335
167,139
3.531
167,139
3.519
159,359
3.361
Workspace (base)
1,483,834
31.307
1,472,224
31.103
1,484,686
31.260
1,484,566
31.305
Workspace update
2,797,037
59.015
2,797,133
59.095
2,797,037
58.891
2,797,037
58.983
Session update
122,271
2.580
124,800
2.637
124,800
2.628
125,343
2.643
Appendix
Table 8: Token distribution across input sources in ClawArena , reported independently for each of the four compatible frameworks. All counts use the cl100k_base tokenizer; jsonl session files are parsed into a normalized message sequence and rendered into the ChatML template before counting, while markdown transcripts and all non-session artefacts are counted verbatim (see Appendix E narrative).
Dimension
Metric
Mean
Min
Max
Multi-choice
Options per round
6.38
5
10
Correct answers per round
4.46
2
8
Executable-check
Sandbox timeout (s)
40.2
30
60
expect_exit coverage (%)
100.0
—
—
expect_stdout coverage (%)
0.4
—
—
Appendix
Table 9: Multi-choice prompt shape and executable-check feature coverage in ClawArena . MC statistics are computed over all 95 multi-choice rounds; EC statistics are computed over all 242 executable-check rounds.
Facet
Category
Count
%
Target (events)
Workspace-targeted
42
50.6
Session-targeted
37
44.6
Group-targeted
4
4.8
Action (files)
append
14
56.0
new
11
44.0
Appendix
Table 10: Staged-update distribution. Target reports the 83 update events; Action reports the 25 file-level update operations triggered by those events.
Scenario
Rounds
MC
EC
w/Pref
w/Upd
Upd
UpdF
WSF
Tokens
hil_s1
24
8
16
3
4
8
25
24
4,344,165
hil_c7
28
8
20
2
3
6
9
12
44,210
hil_d3
30
8
22
2
4
8
10
12
44,137
hil_e4
24
7
17
2
3
6
6
11
40,112
hil_g4
27
8
19
2
3
7
8
10
37,429
hil_g1
30
8
22
3
4
7
8
10
36,077
Appendix
Table 11: Per-scenario breakdown under the OpenClaw layout. Columns: rounds, multi-choice rounds (MC), executable-check rounds (EC), rounds with a preference (w/Pref), rounds exposing updates (w/Upd), total update events (Upd), files touched by updates (UpdF), baseline workspace files (WSF, excluding framework-injected session transcripts), and total tokens.
Figure 4: Case 1 (MS-R, NexaFlow API breach, q7 + q25): both GPT-5.1 configurations omit the top-level 2,340-record confirmation in q7 despite correctly absorbing five other log-derived facts; q25 exhibits five distinct failure modes across models, with only GPT-5.1 (OC), Mimo V2.5, and Ling-2.6 achieving a perfect reliability ranking. Case 2 (exec_check, hil_g4/q3 and hil_i2/q4 ): GPT-5.1 under both OpenClaw and Claude Code fails both tasks with identical schema-compliance errors, while six OpenClaw-hosted models pass hil_i2/q4 at a 6/6 rate—isolating the failure to the model rather than the framework.
Figure 5: Case 3 (MS-R, research integrity, q1 + q2): q1 is the only round in the benchmark with a 0% exact-match across all eight models, because the correct options are compound relational statements (e.g., IRB-precedes-extraction). Case 4 (DU-R, authority-influenced revision, q5 + q17): a single IRB pre-registration option is missed by the majority in both rounds, while all eight configurations correctly decline the overtly false “endorsed complaint” and “recommended retraction” options—failure here is one of omission, not capitulation to authority.
Figure 6: Case 5 (P-R + exec_check, wrongful-termination report, q18 + q26): GPT-5.1 under both frameworks fails both rounds by omitting the YYYY-MM-DD_ filename prefix; Mimo V2.5 and Gemini-3.1 pass the lenient midterm check but fall off once q26 layers three literal ISO dates and ≥ 5 section headings. Case 6 (exec_check, NexaFlow GDPR bundle, q15 + q23 + q26): all three questions achieve 0/8 pass rates; the shared failure mode is compound output-format compliance (timezone-aware timestamps, typed JSON fields with provenance citation, enum-constrained priority labels), defining a current structural-output ceiling.
Figure 7: Case 7 (DU-R, 618 GPU fraud, q7 + q13 + q25): failure clusters are update-specific rather than cumulative—four models conflate the CNY 32,000 partial refund with full-price restitution on q13, while Ling-2.6 and DeepSeek-V4 show isolated answer-extraction failures when the response format drifts from their usual \bbox{} emission. Case 8 (exec_check, JSON schema adherence, hil_g4/q3 + hil_f7/q21 + hil_f7/q22 ): pass rates track schema documentation style—only 2/8 pass hil_g4/q3 whose key is described in natural language, versus 6/8 on hil_f7/q21 where the four required keys are listed verbatim.
Figure 8: Case 9 (MS-R + DU-R, wrongful termination, q13 + q27): four models miss the option where Sun Wei admits the termination conversation lacked an email record—a claim that requires conjoining two elements of the same reply; in q27 seven of eight achieve perfect synthesis and only DeepSeek-V4 drops the two-part PIP-violation statement. Case 10 (exec_check + MS-R, pipeline authorship and final synthesis, hil_i2/q8 + hil_c7/q28 ): only GPT-5.1 (OC) passes the M2-contrast write-up (1/8); on the final synthesis GPT-5.1 (CC), Mimo V2.5, and GLM-5.1 miss the same forward-looking remediation option, while DeepSeek-V4 produces no extractable answer.
Large language model (LLM) agents have achieved strong performance on a wide range of benchmarks, yet most evaluations assume static environments. In contrast, real-world deployment is inherently dynamic, requiring agents to continually align their knowledge, skills, and behavior with changing environments and updated task conditions. To address this gap, we introduce EvoArena, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains. We further propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory. Experiments show that current agents struggle on EvoArena, achieving an average accuracy of 39.6% across evolving terminal, software, and social-preference domains. EvoMem consistently improves performance, yielding an average gain of 1.5% on EvoArena and also improving standard benchmarks such as GAIA and LoCoMo by 6.1% and 4.8%. Beyond individual tasks, EvoMem further improves chain-level accuracy by 3.7% on EvoArena, where success requires completing a consecutive sequence of related evolutionary subtasks. Mechanistic analysis shows that EvoMem improves evidence capture in the memory, indicating better preservation of complete evolving environment states. Our results highlight the importance of modeling evolution in both evaluation and memory for reliable agent deployment.
Jundong Xu, Qingchuan Li, Jiaying Wu +11
National University of Singapore · Singapore Management University · University of Washington +5
OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments. Evaluating these assistants is fundamentally a fidelity problem: benchmarks must be faithful both to the distribution of real assistant tasks and to the execution semantics of the environments in which those tasks unfold. Existing benchmarks often lose fidelity in one dimension or the other. Their task distributions are shaped by what is easy to isolate, mock, and verify, underrepresenting real-world difficulties such as cross-service dependency, contaminated state, implicit intent, and runtime change. Their environments are either live but hard to reproduce, or reproducible but reduced to endpoint-level stubs that remove sessions, artifacts, state transitions, and downstream side effects. We introduce LiveClawBench, a benchmark designed around this dual-fidelity requirement. LiveClawBench combines a Triple-Axis Complexity Framework for difficulty-driven task construction with reproducible full-stack mock applications that preserve stateful execution semantics. With 134 executable cases across 10 domains with 22 mocked services, LiveClawBench supports controlled, extensible, and factor-level diagnostic evaluation of realistic agentic tasks. We release the benchmark resources: (1) Benchmark: https://github.com/Mosi-AI/LiveClawBench; (2) Leaderboard: https://mosi-ai.github.io/LiveClawBench/leaderboard/; (3) Trajectories: https://huggingface.co/datasets/Mosi-AI/LiveClawbench-trajectories
Xiang Long, Li Du, Yilong Xu +11
Samsung Research, Beijing, China · HKUST (Guangzhou) · City University of Hong Kong +1
Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introduce RealClawBench, a live benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use. Real user requests are challenging to benchmark because they often depend on local execution environments, involve implicit or underspecified intent, and require nontrivial verification. RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers, which together convert real sessions into reproducible, automatically scored tasks. The resulting release contains 281 executable tasks sampled from a much larger real-session pool while preserving the source distribution, with maximum final-vs-source Jensen-Shannon divergence of 0.0448. Evaluating 14 contemporary models shows that the best system solves only 65.8% of tasks, revealing substantial headroom on realistic developer-agent workloads. By turning real deployed sessions into controlled evaluation instances, RealClawBench provides a practical path toward benchmarks that better measure agent capability in actual use. Code is available at:https://anonymous.4open.science/r/real-claw-bench-582B.