cs.HCOct 23, 2026

CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments

Authors: Monojit Banerjee, Zhujun Lan, Defu Cao

Organizations: Salesforce, Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. We evaluate the complete agent system through these contracts and report scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve. Github repository: https://github.com/salesforce-misc/crm-agent-bench

Figures & tables

Figure 1: A duplicated $15.50 checkout: the response, action, and order checks all pass, yet the state check catches the doubled loyalty balance (310 instead of 155) and the strict contract fails (Appendix D.2 ).
Figure 2: CRM-Agent-Bench setup and a multi-turn demo. (a) An on-policy user interacts with the full routed agent; each turn exposes one scoped capability over a freshly reset CRM backend. Recorded dialogue, routes, tool calls, and terminal state are checked conjunctively against the task contract. (b) In an observed synthetic benchmark trajectory, the agent rejects an invalid transfer without changing state, then reselects the transfer node, validates, and executes the user’s corrected request. All declared action, argument, order, state, and guardrail checks pass.
Figure 3: Multi-capability coverage and execution requirements in the 555-task evaluation set. Each bar counts tasks with the indicated property; percentages use all 555 tasks. The categories overlap and describe declared task requirements.
Figure 4: Sixteen full-suite runs on the audited 555-task canonical set, ranked by strict accuracy, colored by family and marked with their developer logos; strict-accuracy percentages and dollar-prefixed estimated tested-agent costs are annotated per bar.
Figure 5: Domain leadership varies across the sixteen systems. Rows follow first-round overall strict-accuracy rank; each cell reports strict success (%) within a domain, whose task count is shown in the column header. All models use the same per-domain denominators.
Figure 6: Success retained as execution checks are added to a response-only filter. Each panel uses all 555 tasks at every stage. Bars add action, argument, ordering, and state checks; the strict endpoint adds the remaining declared checks and model-attributable error handling. Undeclared or unavailable component verdicts are neutral at partial stages. Annotated drops are differences between the displayed percentages.
Figure 7: Efficiency and accuracy over the 555 scored tasks. Each panel plots strict success against a mean tested-agent efficiency measure: LLM latency, tokens, and tool calls (simulator and judge usage excluded). Dark segments trace each panel’s Pareto frontier (lower resource use with higher success preferred); filled markers lie on it, hollow markers are dominated. Marker IDs follow the Figure 4 rank order, and color and shape encode model family.
Figure 8: Compliance and completion on the task set, in Table 6 rank order. Filled circles: strict success on the 390 tasks without negative assertions; open circles: strict success on the 165 policy-constrained tasks; bars: their difference. Carets mark the share of policy-constrained tasks passing all applicable negative checks, and sit right of the open circles, so most failures on those tasks come from the accompanying required components rather than violations.
Table 1: Comparison of representative agent benchmarks by evaluation contract and deployment setting. marks a native mechanism scored when applicable, limited or indirect support, and – no primary support. The process-outcome gate distinguishes reaching the correct outcome from following the required procedure by requiring both declared checks to pass. Self-contained means that no live vendor backend is required.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Detailed expansion of the audited evaluation setup in Figure 2 (a). The task contract, on-policy user, graph-routed agent, freshly reset CRM environment, and conjunctive grader form the shared backbone. This appendix view opens those blocks to show the frozen domain specification, one-node-per-turn access policy, evidence channels, task-declared checks, the capability-exposure audit, and operational readouts.
Figure 10: Anatomy of a task contract, shown for the real transfer task payments_transfer_frozen_source_01 . Its required actions, arguments, ordered subsequence, semantic explanation, unchanged balances, and forbidden transfer attempt must all pass; component results remain visible to localize a failure.
Figure 11: Composition of the audited 555-task evaluation set: domain (inner) →\rightarrow mutually exclusive first capability (middle) →\rightarrow easy/medium/hard within capability (outer). Each keyed row reports its total and E/M/H counts; N is the overlapping count of tasks with a forbidden action, argument, or protected-state check, not a fourth partition.
Figure 12: Graph-routed reference scaffold for the payments domain. A per-turn classifier selects exactly one capability node (exposing only that node’s scoped tools) or returns a deterministic unsafe / out_of_scope refusal, with no node-initiated handoffs. The highlighted path (turn 1 →\rightarrow transfer, turn 2 →\rightarrow checkout) shows how a trial traverses several capabilities across turns without exposing the full tool inventory at once.
Figure 13: Composition of all scored tasks that did not become strict successes for each of the 16 runs (of 555 scored). Each pie corresponds to one model, and its subtitle gives the total number of non-successes. To make the slices mutually exclusive, each non-success receives one primary label using the harness status precedence: action, forbidden, state, arguments, semantic response, ordering, then collateral state. “Negative checks” combines the forbidden-action/argument and collateral-state labels. The hatched “Tool-call error” segment contains two tasks that errored (e.g., on a hallucinated tool call) without a failed component verdict available for primary attribution. Both remain model-attributable strict failures, and there are zero timeouts or infrastructure exclusions. In-slice labels are task counts, not percentages, and are shown only when space permits. Models are ordered by total non-success count.
Figure 14: Component-grader pass rates across the 16 runs, each computed over that component’s recorded verdicts. Denominators are 529 (action), 493 (arguments), 164 (ordering), 180 (state), 136 (forbidden), and 45 (collateral). Semantic response has 402 recorded verdicts per run, except Opus (401), Nemotron Super (396), and Nemotron 3.5 (392). Undeclared checks and missing component verdicts are omitted from these rates. Columns therefore describe different task subsets. “Forbidden” aggregates forbidden-action and forbidden-argument checks; “Collateral” evaluates protected state.
Figure 15: Per-metric standing of the 16 completed runs. Each row is an independent number line placing every run at its real value on that metric’s own scale. Marker IDs follow the Figure 4 rank order and map to the key below; small vertical offsets only separate colliding markers and do not encode value. Runs that share a headline strict-success number (three tie at 81.26%, with two further ties at 79.10% and 64.32%) nonetheless separate on one or more additional axes: domain macro, easy- and hard-task success, and a 69.4 ×\times range in cache-neutral cost per success (log axis; shown for the 13 priced runs, from 0.00599forGPT−4.1Nanoto0.00599 for GPT-4.1 Nano to 0.416 for Claude Opus 5). This per-metric view is the complement to the rank crossings in Figure 16 .
Figure 16: Rank of each completed run (1 = best of 16) across six higher-is-better metrics: scored success, domain macro, easy-, medium-, and hard-task success, and customer-support success. Shared competition ranks are used: equal values receive the same rank, and subsequent ranks advance by the number of tied systems. Endpoint labels are offset only for legibility; data points remain at their shared ranks. No system dominates every axis. Gemini 3.8 Flash leads five of the six metrics but falls to third on customer support, where Muse Glimmer 30B—twelfth on scored success— ranks first and Claude Opus 5 (second overall) places second; Claude Sonnet 5 climbs from joint-fifth to second on hard tasks. Ranks are computed from the same one-trial values used in Figure 4 and Table 7 . åCost per success is omitted here because three open-weight runs are unpriced; it is shown on its own log axis in Figure 15 .
Figure 17: How to read a trace demo. Each card links the declared contract to runtime evidence and the individual grader decisions that produce the reported outcome.
Figure 18: Trace-demo provenance. Each narrative joins the complete run artifact to its task contract using the model, seed, and task ID.
Figure 19: System flow for the Retail domain (representative commerce/returns flow). Audited evaluation-set lineage: hybrid (95 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 20: System flow for the Customer Support domain. Audited evaluation-set lineage: hybrid (125 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 21: System flow for the Field Service domain. Audited evaluation-set lineage: seed workflow (23 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 22: System flow for the IT Service Management domain. Audited evaluation-set lineage: recipe-derived (66 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 23: System flow for the Payments domain. Audited evaluation-set lineage: recipe-derived (73 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 24: System flow for the Travel domain. Audited evaluation-set lineage: recipe-derived (93 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 25: System flow for the Onboarding domain (representative customer-onboarding flow). Audited evaluation-set lineage: recipe-derived (41 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 26: System flow for the Sales domain (representative sales-development flow). Audited evaluation-set lineage: seed workflow (18 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
Figure 27: System flow for the Marketing domain (representative campaign flow). Audited evaluation-set lineage: seed workflow (21 tasks). Key: R / W / C denote read, write, and computation tools; blue marks intermediate execution, green the accepted completion path, amber a policy gate, and red rejection without business-state mutation. Evaluation checks apply only when declared in the task contract.
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Explore similar work

CardsList
Oct 19, 2026

CXBench: A Benchmark for Process-Compliant LLM Agents in Regulated Customer-Experience Workflows

Tool-using language agents increasingly perform regulated customer-service workflows, where a correct final state is insufficient: agents must also provide disclosures, obtain consent, and follow required procedures. Existing tool-agent benchmarks largely score final business states, accepting trajectories that reach correct outcomes through non-compliant processes. We introduce cxbench, a benchmark that evaluates outcome and process compliance separately across six regulated domains. Each task combines a customer request, a tool-based environment, and explicit process requirements. Rule cards link cited regulatory obligations to observable trace evidence. A hybrid grader combines deterministic checks for ordering and authority with an LLM judge for semantic requirements. We define a false-pass as an outcome-pass run that fails process checks. Across seven models and 120 tasks, 55.5% of the 840 retained executions are false-passes, representing 80.1% of outcome-passing runs; only 13.8% pass both checks. Process-inclusive evaluation changes the observed model ordering and exposes differences hidden by outcome-only scoring. cxbench makes these failures inspectable through trace-linked criteria.

Boxi Yu, Yuzhong Zhang, Zhenfeng Su +7

Lero, the Research Ireland Centre for Software, University of Limerick, Ireland. · The Chinese University of Hong Kong, Shenzhen, China. · Xi’an Jiaotong University, China. +3

LLM Agent EvaluationProcess-Based Agent EvaluationAI Regulatory ComplianceCustomer Support Automation
AskPDF
Mar 9, 2026

$OneMillion-Bench: How Far are Language Agents from Human Experts?

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench ($OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, $OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.

Yang Liu, Jiaqi Li, Jun Bai +20

BIGAI · Humanlaya · xbench +1

LLM Agent EvaluationAI Agent BenchmarksAI Agent Reliability
AskPDF
Jun 22, 2026

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench

Jincheng Zhong, Weizhi Wang, Che Jiang +5

Horizon Research, Frontis.AI

LLM Agent EvaluationTool-Augmented Language Model AgentsComputer-Use Agent BenchmarksAI Agent BenchmarksComputer-Use Agent EvaluationEnterprise AI Agents+1
AskPDF