Organizations: School of Mathematics and Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China · Center for Applied Mathematics & Shanghai Key Laboratory of Contemporary Applied Mathematics, Fudan University, Shanghai, China
The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI
Figures & tables
Figure 1: The construction of CirrusBench. The process comprises four core phases: Phase 1: Data Collection ; Phase 2: Preprocessing (Steps 1–5); Phase 3: Knowledge-Grounded Annotation (Step 6); Phase 4: Quality Assurance (Step 7). The final benchmark supports both single-turn subtask evaluation and multi-turn task evaluation.
Category
Task Count
Frequency (%)
Server
280
18.7%
Registration
220
14.7%
SMS Services
200
13.3%
Website
120
8.0%
Email
80
5.3%
Others (15+ )
600
40.0%
Table 1: Distribution of Tasks Across Service Categories. The dataset ( N=1,500 ) is sampled from the top 20 service categories with highest ticket frequency in real-world service operations.
Figure 3
Reply Type
Count
Proportion (%)
Solution
2,095
63.35%
Standard
529
16.00%
Inquire
483
14.61%
Clarify
130
3.93%
Appease
70
2.12%
Total
3,307
100.00%
Table 2: Distribution of reply types of sub-tasks.
Figure 4: The evaluation framework of CirrusBench. The evaluation of a multi-turn task proceeds as a sequence of single-turn subtask sessions.
Models
Pass@1 SR (%) ↑
Pass@2 SR (%) ↑
ATPR (%) ↑
ALJ ↑
ANEI (%) ↑
Avg. All Tokens
Avg. Input Tokens
AMTL (tokens) ↓
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
w tool
w/o tool
Qwen3-235B-A22B-Instruct-2507
11.0
19.0
17.4
26.2
25.6
34.2
0.101
0.156
12.3
31.5
11900.1
11714.3
11710.0
11120.6
190.1
593.7
Qwen3-235B-A22B-Thinking-2507
11.0
13.3
18.8
18.5
27.8
24.8
0.106
0.090
12.5
23.7
13659.2
12013.9
12596.1
10973.3
1063.1
1040.5
Qwen3-Max
14.3
17.1
19.4
23.7
28.6
32.2
0.108
0.137
12.5
31.8
12643.3
11666.3
12409.7
11194.5
233.6
471.8
DeepSeek-V3.2
9.6
15.2
15.5
23.5
21.9
31.4
0.104
0.138
12.5
31.5
12006.2
11577.0
11813.4
11060.5
192.8
516.5
DeepSeek-R1
7.9
15.2
10.1
21.1
21.1
29.3
0.082
0.118
11.1
29.0
16039.5
11748.2
14565.3
10881.1
1474.2
867.2
Table 3: Evaluation of various models across dataset with tool-call requirement and dataset without tool-call requirement of CirrusBench. Red color indicates models with explicit thinking capabilities. Bold values indicate the best performance in each column, and underlined values indicate the worst performance in each column.
Figure 5: Overall Success Rate results on CirrusBench.
Token Range
pass@1 SR (%) ↑
pass@2 SR (%) ↑
TIA ↑
TSA ↑
TEV ↑
0 – 8,000
40.4
60.7
0.531
0.477
0.261
8,000 – 16,000
43.2
63.4
0.535
0.441
0.244
Over 16,000
44.8
64.8
0.536
0.451
0.267
Table 4: Model performance across input token ranges. (on GPT-5)
Count of Checkpoints
Pass@1 SR(%) ↑
Pass@2 SR(%) ↑
TPR (%) ↑
ALJ ↑
ANEI ↑
AMTL (tokens) ↓
1
45.8
58.5
58.5
0
0.58
2410
2
28.1
39.6
47.1
0.19
0.33
2270
3
17.4
29.5
42.0
0.28
0.30
2237
4
9.2
19.2
32.9
0.40
0.31
2190
Others (5+ )
5.3
14.6
15.7
0.59
0.35
2151
Table 5: Model performance across different counts of checkpoints. (on GPT-5)
Category
Pass@1 SR(%) ↑
Pass@2 SR(%) ↑
TPR (%) ↑
ALJ ↑
ANEI ↑
AMTL (tokens) ↓
Server
21.6
31.4
41.0
0.319
0.364
2304
Registration
19.4
32.0
40.6
0.286
0.322
2099
SMS Services
20.6
34.0
41.8
0.248
0.358
2393
Website
36.8
49.1
54.9
0.281
0.465
2218
Email
18.9
27.5
36.7
0.172
0.347
2190
Others (15+ )
26.7
37.8
47.2
0.230
0.409
2228
Table 6: Model performance across service categories. (on GPT-5)
Figure 6: Pass@1, pass@2 success rate and AMTL by different reply types .
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.
Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification. We release the trajectories and future dataset at https://clawsbench.com.
LLM agents are widely adopted in service chatbots for rule-guided support and information access, yet emotionally demanding complaint handling still relies heavily on human agents. We introduce ServiceBench, to our knowledge the first benchmark dedicated to complaint handling, with 2,500 scenarios spanning 184 complaint categories and five service industries. We propose Asymmetric Progressive Complaint Simulation (APCS) to simulate realistic complaint handling, together with a justice-based evaluation framework. Our experiments reveal that even capable LLM agents struggle to handle complaints effectively. They lack the ability to proactively gather information and strategically guide conversations toward resolution, often resorting to premature compensation. We further show that targeted post-training with SFT and DPO substantially improves these capabilities, enabling a smaller model to match or surpass much larger models on several outcomes.