CXBench: A Benchmark for Process-Compliant LLM Agents in Regulated Customer-Experience Workflows

Boxi Yu; Yuzhong Zhang; Zhenfeng Su; Linfu Xi; Yinghan Yao; Shitao Chen; Yang Cao; Liting Lin

cs.AIOct 19, 2026

CXBench: A Benchmark for Process-Compliant LLM Agents in Regulated Customer-Experience Workflows

Authors: Boxi Yu, Yuzhong Zhang, Zhenfeng Su, Linfu Xi, Yinghan Yao, Shitao Chen, Yang Cao, Liting Lin,

Organizations: Lero, the Research Ireland Centre for Software, University of Limerick, Ireland. · The Chinese University of Hong Kong, Shenzhen, China. · Xi’an Jiaotong University, China. · Peking University, China. ·

Abstract

Tool-using language agents increasingly perform regulated customer-service workflows, where a correct final state is insufficient: agents must also provide disclosures, obtain consent, and follow required procedures. Existing tool-agent benchmarks largely score final business states, accepting trajectories that reach correct outcomes through non-compliant processes. We introduce cxbench, a benchmark that evaluates outcome and process compliance separately across six regulated domains. Each task combines a customer request, a tool-based environment, and explicit process requirements. Rule cards link cited regulatory obligations to observable trace evidence. A hybrid grader combines deterministic checks for ordering and authority with an LLM judge for semantic requirements. We define a false-pass as an outcome-pass run that fails process checks. Across seven models and 120 tasks, 55.5% of the 840 retained executions are false-passes, representing 80.1% of outcome-passing runs; only 13.8% pass both checks. Process-inclusive evaluation changes the observed model ordering and exposes differences hidden by outcome-only scoring. cxbench makes these failures inspectable through trace-linked criteria.

Figures & tables

Figure 1: Two traces with the same final state. An outcome-only oracle accepts both, whereas cxbench identifies Trace B as a false-pass because it omits the written-confirmation notice required in this case under Regulation E (see footnote 1 ). Draft confirmation is a separate cxbench runtime control rather than a statutory requirement.

Benchmark	Process	Statute	False-pass	Runtime submit
AgentBench / OSWorld ( Liu et al., 2024 ; Xie et al., 2024 )	$\times$	$\times$	$\times$	$\times$
tau-bench / tau2-bench ( Yao et al., 2024 ; Barres et al., 2025 )	$\times$	$\times$	$\times$	$\times$
TheAgentCompany ( Xu et al., 2024 )	$\times$	$\times$	$\times$	$\times$
LegalBench ( Guha et al., 2023 )	$\times$	$\times$	$\times$	$\times$
SOP-Bench ( Nandi et al., 2025 )	$\times$	$\times$	$\times$	$\times$
SOPBench / SOP-Maze ( Li et al., 2025 ; Wang et al., 2025b )	✓	$\times$	$\times$	$\times$

Table 1: Comparison with representative benchmarks on process evaluation, regulatory grounding, false-pass reporting, and runtime-controlled submission. A checkmark indicates support for the listed feature; a cross indicates its absence, rather than the absence of all policy-related evaluation.

Figure 2: The cxbench evaluation pipeline. The agent interacts with a simulated customer and business tools, producing a final state and an interaction trace. Outcome evaluation checks the final state; process evaluation checks the trace using deterministic rules and an LLM judge. In the default mode, the runtime submits drafts after customer confirmation.

Produce	Role	Responsibility
Execution	Agent	Interacts with the customer and uses available tools to complete the task while following the applicable requirements.
Execution	Customer simulator	Provides information based on private customer facts and confirms drafts whose required fields match those facts.
Execution	Runtime	Executes tool calls, records the interaction, and controls submission in mediated-submit mode.
Evaluation	Rule layer	Checks deterministic conditions, including event ordering, tool permissions, and agreement between confirmed and submitted fields.
Evaluation	LLM judge	Assesses semantic requirements using a fixed rubric and task-specific checklist.
Validation	Expert reviewers	Reassesses selected traces to examine the validity of the automated evaluation.

Table 2: Roles in cxbench . The agent, customer simulator, and runtime produce the interaction; the rule layer and LLM judge evaluate it. Expert reviewers independently reassess selected traces for validation and do not determine the reported process verdict.

Figure 3: From regulatory requirements to process evaluation. A rule card connects a cited requirement to observable trace evidence and an evaluation criterion (top). Deterministic checks and an LLM judge assess the requirements applicable to each task and jointly determine the process verdict (bottom). A correct outcome accompanied by a failed process verdict is a false-pass.

Model	Evaluated	Outcome-pass	Both-pass			False-pass
	records	Rate (%)	Count	Rate (%)	95% CI	Rate (%)	Count
glm-5.1	120	77.50	26	21.67	[15.24, 29.85]	55.83	67
gpt-5.4	120	64.17	25	20.83	[14.53, 28.95]	43.33	52
gpt-5.4-mini	120	61.67	18	15.00	[9.70, 22.47]	46.67	56
qwen3.6-plus	120	70.83	18	15.00	[9.70, 22.47]	55.83	67
deepseek-v4-flash	120	77.50	16	13.33	[8.38, 20.56]	64.17	77

Table 3: Per-model results over all 120 tasks. Evaluated records include task failures. Outcome-pass measures outcome correctness; both-pass requires passing both outcome and process checks; false-pass denotes outcome-pass with process-fail. Rates and Wilson 95% confidence intervals (CI) are percentages; CIs apply to the both-pass rate.

Mode	Outcome pass	Process pass	Both pass	False pass
Mediated-submit	64.17	21.67	20.83	43.33
Direct-submit	67.50	0.83	0.83	66.67
Direct-submit (checks excluded)	67.50	20.00	18.33	49.17

Table 4: Submission-control comparison for gpt-5.4 . All rows use the same 120 recorded traces. The third row excludes submission-mechanism checks from both the hard-rule and LLM-judge components and is a rescoring condition rather than an additional execution mode. All entries are percentages.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Element	Meaning	Schema or runtime representation
$g$	Customer goal	customer_goal
$m_{0}$	Opening message	seed_message
$s_{0}$	Initial execution state	initial_state , hidden_facts , and runtime-initialized history and drafts
$\mathcal{U}$	Domain tool interface	tools , resolved against the domain’s registered tool implementations
$\Lambda$	Policy context and process requirements	policy , including source references, process requirements, and associated grading specifications; referenced rules
$\mathcal{K}$	Fields requiring confirmation	confirmation_contract

Appendix

Table 5: Correspondence between the task representation and the task schema. Registered domain implementations provide the tool behavior and state initialization used with these fields.

Figure 4: Selected components of a banking task specification for an unauthorized ACH debit: (a) the customer goal, opening message, and initial business state; (b) domain-specific tool interfaces; (c) statute-linked policy requirements; and (d) the associated grading configuration. Private customer facts, confirmation fields, and other details are omitted for brevity.

Component	Information used
Agent	Opening message, agent-facing policy instructions, permitted tool definitions, customer replies, and tool results.
Customer simulator	Private customer facts, customer-facing messages, and presented draft fields used to generate replies and confirmation decisions.
Runtime	Task initialization data, registered tools, runtime mode, confirmation fields, drafts, and recorded interaction events.
Outcome checker	Customer goal, relevant reference facts, and final business state.
Process rule layer	Applicable deterministic rules and the recorded evidence needed to check ordering, permissions, and confirmation consistency.
LLM judge	The interpretation rubric, task-specific checklist, relevant task context, and recorded execution evidence.

Appendix

Table 6: Information used by the execution and evaluation components. Access is described by function; it does not imply that each component receives the complete task file.

Component	Description
Source provision	Regulation E, 12 CFR § 1005.11(b)(2), concerning written confirmation of an oral error notice.
Task conditions	The notice is oral, and the institution requires written confirmation.
Required information	The written-confirmation requirement and the address to which it should be submitted.
Trace evidence	Communication of the required information during the recorded interaction.
Benchmark criterion	The disclosure precedes the request for customer confirmation of the filing draft.

Appendix

Table 7: Regulation E disclosure example: the source obligation, applicable task conditions, required information, and benchmark evaluation criterion.

Category	Scope
Pipeline gates	The draft–confirm–submit ordering requirements shared across domains.
Domain ordering	Precedence between domain-specific steps, such as reading a policy summary before checking reporter authority.
Disclosure gates	Required disclosures before designated customer-confirmation events.
Semantic prohibitions	Restrictions such as unsupported outcome promises, prohibited collection of secrets, and unlawful disconnection threats.

Appendix

Table 8: Categories in the process-rule catalog. Definitions are presented here; observed rule failures are reported in Appendix G.2 .

Domain	Read	Check	Draft	Notice	Submit
Banking	6	5	3	4	2
Telecommunications	5	4	2	3	2
Insurance	6	5	3	3	2
Healthcare	5	3	1	2	1
Utilities	5	5	4	3	1
Government benefits	5	4	2	2	1

Appendix

Table 9: Registered tools by domain and tool category. Counts include aliases.

Figure 5: Outcome, hybrid process, and both-pass rates across models and domains. Each cell uses all 20 tasks for one model–domain pair. All panels share a 0–100% color scale.

Domain	Outcome pass	Both pass	False pass
Banking	94.29	5.71	88.57
Government benefits	84.29	4.29	80.00
Healthcare	13.57	2.14	11.43
Insurance	80.71	17.86	62.86
Telecommunications	67.14	26.43	40.71
Utilities	75.71	26.43	49.29

Appendix

Table 10: Pooled results by domain. Each domain contains 140 evaluations. All entries are percentages.

Figure 6: Failure rates for the 16 most frequently failing hard rules across the seven models. Each rate is normalized by the number of records whose task configures that rule for the corresponding model, so denominators vary by rule. Failures include missing required events as well as incorrect event ordering.

Mode	Outcome pass	Process pass	Both pass	False pass
Mediated-submit	64.17	21.67	20.83	43.33
Direct-submit	67.50	0.83	0.83	66.67
Direct-submit (checks excluded)	67.50	20.00	18.33	49.17

Appendix

Table 11: Submission-control ablation for gpt-5.4 . All rows use the same 120 task specifications and recorded traces. The third row excludes submission-mechanism checks from both the hard-rule and LLM-judge components; it is a rescoring condition rather than an additional execution mode. All entries are percentages.

Agent	gpt-5.6-sol	gpt-6-astra	$\Delta$ (pp)
gpt-5.4	20.83	20.00	$-0.83$
gpt-5.4-mini	15.00	17.50	$+2.50$
gpt-4o-mini	2.50	2.50	$0.00$
glm-5.1	21.67	28.33	$+6.67$
qwen3.6-plus	15.00	14.17	$-0.83$
qwen3.5-plus	8.33	5.83	$-2.50$

Appendix

Table 12: Both-pass rates under two judge configurations. Each model uses the same 120 traces for both judges. Rates are percentages;

\Delta

is the change from gpt-5.6-sol to gpt-6-astra in percentage points.

	gpt-5.6-sol : Pass	gpt-5.6-sol : Fail
Expert: Pass	23 (TP)	4 (FN)
Expert: Fail	10 (FP)	83 (TN)

Appendix

Table 13: Comparison of gpt-5.6-sol verdicts with expert labels on 120 traces generated by gpt-5.4-mini . These traces were used during judge calibration. Process pass is the positive class.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents
May 7, 2026Ashwani Anand, Ivi Chatzi, Ritam Raha +1Computer-Use Agent Benchmarks AI Agent Reliability
Max Planck Institute for Software Systems, Kaiserslautern, Germany
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
Mar 15, 2026Shengda Fan, Xuyan Ye, Yupeng Huo +9LLM Agent Evaluation Process-Based Agent Evaluation
Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China +1
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley

May 7, 2026J/K move · Enter open · S save

MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents

Ashwani Anand, Ivi Chatzi, Ritam Raha, Anne-Kathrin Schmuck

Max Planck Institute for Software Systems, Kaiserslautern, Germany

Computer-Use Agent Benchmarks AI Agent Reliability LLM Agent Verification AI Agent Benchmarks Synthetic Benchmark Generation Tool-Using Agents+1

Open paper AskPDF arXiv

Tool-using large language model (LLM) agents are increasingly deployed in settings where their reliable behavior is governed by strict procedural manuals. Ensuring that such agents comply with the rules from these manuals is challenging, as they are typically written for humans in natural language while agent behavior manifests as an execution trace of tool calls. Existing evaluations of LLM agents rely on manually constructed benchmarks or LLM-based judges, which either do not scale or lack reliability for complex, long-horizon manuals. To overcome these limitations, we present MANTRA, a framework for automatically synthesizing machine-checkable compliance benchmarks from natural-language manuals and tool schemas. MANTRA independently generates (i) a symbolic world model capturing procedural dependencies, and (ii) a set of trace-level compliance checks for a given task, and validates their consistency using SMT solving. A structured repair loop resolves inconsistencies, requiring human intervention only as a fallback. %This yields benchmarks that are formally validated. Importantly, MANTRA supports arbitrary domains and long procedural manuals, and provides a tunable notion of task complexity which is utilized to automatically derive challenging tasks accompanying compliance checks. Using MANTRA, we build a new benchmark suite with 285 tasks across 6 domains scaling to 50+ page manuals with minimal human effort. Empirically, we show that the compliance checks are richer with stronger constraint enforcement compared to existing benchmarks. Additionally, the granularity of the checks can be used for debugging the agents' failure modes. These results demonstrate that combining automated benchmark generation with formally grounded validation methods enables scalable and reliable benchmarking of tool-using agents.

Abstract

LLM Agent Evaluation

Explore similar work

MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents