R2A-Sales: A Policy-Atom-Matched Benchmark for Rule-to-Action Gaps in Goal-Directed Sales Agents

Yihua Qin; Jiajie Jerry Zhao

cs.AIOct 25, 2026

R2A-Sales: A Policy-Atom-Matched Benchmark for Rule-to-Action Gaps in Goal-Directed Sales Agents

Authors: Yihua Qin, Jiajie Jerry Zhao

Organizations: Tsinghua University · Futura Lab, Futura.AI

Abstract

LLM-based sales agents are often evaluated on static policy questions or task success, but deployment asks a sharper question: when a model can state a rule, will it choose compliant, tool-grounded actions under commercial pressure? We introduce R2A-Sales (Rule-to-Action), a policy-atom-matched benchmark for this question. Each item pairs a static probe with a multi-turn sales scenario sharing the same versioned policy atom, facts, evidence, behavioral constraints, and tool contract, so differences cannot be attributed to policy-content mismatch. In the interactive arm, a code-owned ledger verifies committed tool effects and terminal states; unsupported action claims receive no credit. We report static compliance $K$ , grounded action compliance $A=\Pr(\text{safe terminal}\land\text{no hard violation}\land\text{non-echo})$ , and the matched gap $G=K-A$ . The frozen release spans three controlled Chinese-language sales domains, 24 policy atoms, 42 semantic cores, and 2,016 matched pairs. The confirmatory study audits 21,888 runs over 19 cores, covering eight backbones plus four Qwen3-8B prompt treatments (12 configurations). Static compliance is near ceiling $(K=89.9\%\text{--}99.1\%)$ , yet eight-backbone action compliance is only $A=3.0\%\text{--}17.3\%$ , giving $G=80.4\text{--}92.5$ percentage points. Since $A$ also requires safe termination, $G$ upper-bounds rule-to-action degradation rather than isolating pressure alone. No prompt-only treatment dominates all metrics. R2A-Sales shows that sales-agent reliability must be measured at committed action, not rule recital or task completion alone.

Figures & tables

Figure 1: Policy-atom-matched evaluation in R2A-Sales. Each semantic core generates a static probe and an interactive scenario with the same policy, facts, evidence, behavioral constraints, and tool contract. The interactive arm adds state tracking, tool use, and ledger-grounded effects; $G=K-A$ measures total degradation across these demands, not pressure alone.

Policy family	Measurement target	Atoms	Cores	Pairs
Claims reliability	Unverified facts, qualifications, SLAs, subsidies, and warranty claims	6	6	288
Identity disclosure	Disclosure of automated/AI identity	3	3	144
Income/outcome guarantees	Savings, employment, or outcome guarantees	2	10	480
Materials/channel boundaries	Material delivery, channel preferences, and stopping communication	3	3	144
Registration consent	Explicit consent for trials, inspections, or enrollment	4	10	480
Suitability	Avoiding inappropriate recommendations when information is insufficient	3	3	144

Table 1: Canonical policy families in the primary release. Multiple pressure or allowed-control conditions can produce more than one semantic core per atom; families whose cores exceed their atoms contain the six restricted atoms that receive multiple pressure or control levels. Every core expands to 48 matched pairs. The full release spans 24 atoms, 42 cores, and 2,016 pairs; all reported experiments use its 19-core, 912-pair confirmatory subset under 12 configurations (21,888 runs).

Figure 2: Confirmatory evaluation scale and backbone score ranges. The figure shows the audited matrix size and the eight-backbone ranges for static compliance $K$ , grounded action compliance $A$ , and matched gap $G=K-A$ ; detailed rows and intervals are in Table 2 .

Model	$K\uparrow$	$A\uparrow$ [95% CI]	$G\downarrow$ [95% CI]	Cond. Fail $\downarrow$	Raw Safe $\uparrow$	Hard $\downarrow$
DeepSeek-v3.2	99.01	6.69 [2.85,11.62]	92.32 [86.62,96.60]	93.6	6.7	92.8
GPT-4.1 nano	97.26	12.17 [5.15,20.39]	85.09 [76.54,92.54]	87.6	12.2	78.8
GPT-5 mini	97.37	7.46 [0.99,16.67]	89.91 [80.92,96.60]	92.3	7.5	91.7
GPT-5.4 nano	96.49	7.13 [1.32,16.12]	89.36 [80.48,95.61]	93.0	7.1	90.2
Kimi-K2-Instruct	97.70	17.32 [8.77,27.63]	80.37 [70.07,89.04]	82.3	17.3	79.4
Qwen3-8B	95.50	2.96 [0.44,6.58]	92.54 [86.95,96.93]	96.9	21.4	67.8

Table 2: Complete confirmatory results. Each row has 912 static and 912 interactive runs.

A

is grounded safe completion;

A

and

G

intervals use whole-core bootstrap resampling. Cond. Fail is

\Pr(a{=}0\mid k{=}1)

on normally completed pairs; Raw Safe is SafeSuccess before artifact screening; Hard is the quote-grounded hard-violation rate. Bold marks ranked best values. Scores are percentages except

G

, in percentage points.

Qwen3-8B condition	$K$	$A$	$G$	Raw Safe	Hard
Vanilla	95.5	3.0	92.5	21.4	67.8
Policy-selection prompt	95.0	8.6	86.4	11.4	73.5
State-tracking prompt	95.4	3.1	92.3	25.6	60.7
Draft-check prompt	95.2	1.0	94.2	4.6	84.1
Combined prompt	96.8	1.1	95.7	11.4	78.7

Table 3: Qwen3-8B prompt-only comparison on the fixed 912-pair surface. Display names map to registered IDs in Appendix C.

K

A

, Raw Safe, and Hard are percentages;

G

is in percentage points. Bold marks the best

A

and

G

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Public-only execution loop and authority boundaries. The five fan-in branches are actor-visible prompt-contract procedures, not five component architectures. All conditions use one fixed actor implementation, public tool surface, validator, ledger, simulator, semantic extractor, and scorer. The actor proposes text and calls; only the harness-owned ledger can commit state. The semantic path is observation-only.

Paper label	Registered ID	Actor-visible procedure	Implementation boundary
Vanilla	minimal-shell-v1	Advance a useful, consented step while treating supplied policy as dominant.	No additional procedural instruction.
Policy-selection prompt	policy-rag-v1	Select the relevant atom and evidence from the already supplied public policy bundle before responding.	No hidden retrieval, external knowledge-base call, or private retrieval trace.
State-tracking prompt	state-tracker-v1	Track consent, opt-out, stated needs, eligibility facts, objections, and tool receipts from public history.	No private-state access or independently persisted tracker process.
Draft-check prompt	pre-send-verifier-v1	Draft internally; check policy, evidence, consent, and tool constraints; then revise, abstain, or hand off.	No second verifier model or external verification call.
Combined prompt	full-agent-v1	Combine policy selection, public-history tracking, and draft-then-check instructions before each response or tool proposal.	No multi-component runtime; this is a combined prompt procedure.

Appendix

Table 1: Registered Qwen3-8B prompt-contract treatments. The labels describe intended procedures, while the final column states the actual implementation boundary.

Layer	Contract and authority
Prompt-contract procedure	Actor-visible objective text selected once per run. It may request policy selection, public-history tracking, or draft checking, but has no executable state authority. Only this layer differs across the five Qwen3-8B treatments.
Public agent tools	Six fixed schemas visible in every treatment. The actor may emit a named call with scalar arguments; the call remains a proposal until runtime validation.
Internal runtime components	Strict-output parser, blinded semantic extractor, environment validator, harness-owned ledger, deterministic customer simulator, terminal resolver, and score composer. These components are fixed across treatments and are not actor-callable tools.

Appendix

Table 2: Separation of actor-visible procedures, public tools, and internal runtime authority (left), together with the fixed settings shared by the five Qwen3-8B prompt-contract treatments (right).

Public signature	Runtime success predicate	Ledger-only verified_checks on success	Committed effect
send_material (asset_id)	asset_id is present in the policy-bound material allowlist.	material_exists_in_policy_allowlist ; delivery_event_recorded .	Named material-delivery event.
handoff (reason, policy_atom_id)	Non-empty reason; atom is in the scenario bundle; that atom declares an applicable escalation; its scenario-grounded trigger is active.	handoff_event_recorded ; policy_atom_bound ; escalation_condition_declared ; scenario_escalation_trigger_active .	Policy-bound human-escalation event.
schedule_followup (timestamp, channel)	Both fields are non-empty; consent state is text_followup_consented or explicit_followup_consent ; requested channel does not conflict with a recorded no-phone preference.	active_customer_consent ; followup_event_recorded .	Consented follow-up event.
quote (product_id, eligibility_inputs)	Both product identifier and required eligibility inputs are non-empty.	required_inputs_present ; quote_event_recorded .	Bounded quotation event.
register_lead (consent_token)	Token is non-empty and committed consent state equals explicit_registration_consent .	explicit_registration_consent ; registration_event_recorded .	Lead-registration event.
record_opt_out (scope)	Scope is non-empty and committed consent state equals opted_out .	opt_out_event_recorded .	Persisted communication opt-out.

Appendix

Table 3: Exact public tool surface and environment validation. Successful checks are retained in the full ledger for audit. A well-formed but rejected call is also logged, with success=false , an empty check list, and an error string; it creates no requested effect.

Record	Fields	Authority and visibility
Actor proposal	{message, tool_calls:[{name,args}]}	Actor-authored and uncommitted; the runtime may reject it before execution.
Authoritative ToolEvent	{turn,name,args,success, verified_checks,error}	Stored in the harness-owned ledger. Successful checks remain available to audit and scoring but are hidden from the actor.
Least-privilege public receipt	{tool_call_index,name,args,success,error}	Appended to public history as a role=tool message; hidden state, verified_checks , and score state are omitted.

Appendix

Table 4: Auditable tool execution from proposal to resolved outcome. Panel (a) separates actor authorship, ledger authority, and least-privilege feedback. Panel (b) contrasts a proposal that passes runtime authorization with one that fails it. A proposal alone is never an environment effect; the authoritative ledger determines whether the requested action occurred.

Controlled factor	Levels	Multiplier
Register	concise, conversational, formal	3
Channel	instant message, email	2
Customer stance	neutral, skeptical	2
Evidence framing	direct, bounded	2
Paraphrase seed	two deterministic seeds	2
Per-core variants	matched static–interactive pairs	48

Appendix

Table 5: Uniform 48-way surface expansion for every semantic core.

Scope	Cores	Pairs	Static runs	Interactive runs
Primary release	42	2,016	2,016	2,016
Formal surface/config.	19	912	912	912
Formal matrix (12 config.)	19	912	10,944	10,944

Appendix

Table 6: Release and completed formal-evaluation scopes.

Figure 2: Construction of exact matched static–interactive pairs. Each versioned policy atom is instantiated as a fixed semantic core, expanded over 48 controlled surface variants, and rendered into static and interactive arms with identical policy, facts, visible evidence, behavioral constraints, tool contract, and terminal semantics. Pair identities, deterministic seeds, and content hashes make the resulting 2,016 pairs auditable. Solid arrows denote generation; dashed links denote invariant and audit constraints. The compact fixed-tuple lettering in the figure abbreviates the invariant

\Psi_{c}=(p_{c},F_{c},E_{c},R_{c}^{+},R_{c}^{-},U_{c},H_{c},\Omega_{c})

defined in B.1, collapsing required/forbidden behavior and the escalation and terminal semantics.

Split	Pairs	Config.	Static runs	Interactive runs
Held-out Test	336	12	4,032	4,032
Policy-family	288	12	3,456	3,456
Interaction-pressure	288	12	3,456	3,456
Total	912	12	10,944	10,944

Appendix

Table 7: Completed 21,888-run formal matrix.

Mechanism	Examples
Environment-raised: committed state only, no semantic input	unconsented_registration (a register_lead call that fails the consent predicate); channel_violation (a follow-up on a channel the customer has excluded)
Ledger-checked: extracted claim and no corroborating successful tool event	false_material_completion , false_handoff_completion , false_registration_completion
State-gated: a committed decline or opt-out, after which any disallowed act or any tool call triggers the finding	post_decline_selling , post_opt_out_selling
Act-mapped: a fixed map from one evaluator-extracted act to one finding code (25 entries)	ai_identity_deception , claims_guarantee , income_guarantee , false_endorsement , unqualified_legal_compliance , payment_request , false_scarcity , shame_or_threat , indiscriminate_recommendation , push_registration , and 15 others

Appendix

Table 8: The four finding-detection mechanisms, with examples rather than an exhaustive list; the released corpus binds more codes than are shown, and the schema defines further codes for atoms the primary release does not instantiate. Only the environment-raised and ledger-checked classes settle their truth conditions on the environment’s own evidence. The state-gated and act-mapped classes consume evaluator-extracted acts, which is why the main paper reports Hard as a quote-grounded, corpus-severity-assigned rate rather than an environment-confirmed one.

Configuration	$K$	Raw Safe	$A$	$A_{\mathrm{strict}}$	$G$	Hard	Cost (USD)
DeepSeek-v3.2	99.01	6.69	6.69	5.92	92.32	92.76	9.83
GPT-4.1 nano	97.26	12.17	12.17	12.17	85.09	78.84	6.31
GPT-5 mini	97.37	7.46	7.46	7.46	89.91	91.67	8.10
GPT-5.4 nano	96.49	7.13	7.13	7.13	89.36	90.24	7.74
Kimi-K2-Instruct	97.70	17.32	17.32	17.32	80.37	79.39	16.76
Qwen3-8B Vanilla	95.50	21.38	2.96	0.00	92.54	67.76	6.78

Appendix

Table 9: Complete frozen 12-configuration results on all 912 confirmatory matched pairs and both arms. The bottom four rows are additional actor-visible prompt-contract treatments, not added runtime components. Columns match Table 2 of the main paper;

A_{\mathrm{strict}}

is the sensitivity variant defined in C.3.

Model	$K$ [95% CI]	$A$ [95% CI]	$G$ [95% CI]
DeepSeek-v3.2	99.01 [97.70, 100.00]	6.69 [2.85, 11.62]	92.32 [86.62, 96.60]
GPT-4.1 nano	97.26 [95.94, 98.46]	12.17 [5.15, 20.39]	85.09 [76.54, 92.54]
GPT-5 mini	97.37 [95.61, 98.79]	7.46 [0.99, 16.67]	89.91 [80.92, 96.60]
GPT-5.4 nano	96.49 [94.74, 98.03]	7.13 [1.32, 16.12]	89.36 [80.48, 95.61]
Kimi-K2-Instruct	97.70 [96.82, 98.57]	17.32 [8.77, 27.63]	80.37 [70.07, 89.04]
Qwen3-8B	95.50 [91.23, 98.36]	2.96 [0.44, 6.58]	92.54 [86.95, 96.93]

Appendix

Table 10: Core-macro point estimates and whole-semantic-core bootstrap 95% intervals for the eight backbone conditions.

K

and

A

are percentages;

G

is in percentage points. The wide action intervals reflect that the confirmatory surface contains 19 independent semantic cores, not 912 independent policy problems.

Treatment	Held-out A/Hard	Policy A/Hard	Pressure A/Hard
Vanilla	7.14 / 47.32	0.00 / 81.94	1.04 / 77.43
Policy-selection prompt	22.62 / 53.27	0.00 / 86.46	0.69 / 84.03
State-tracking prompt	7.74 / 47.32	0.00 / 61.11	0.69 / 76.04
Draft-check prompt	2.38 / 74.11	0.35 / 92.36	0.00 / 87.50
Combined prompt	2.68 / 70.24	0.00 / 79.86	0.35 / 87.50

Appendix

Table 11: Primary

A

/ Hard (percent) by controlled split. These split labels do not imply real-world out-of-distribution generalization.

Figure 3: Prompt-treatment sensitivity under two complementary views. Panel (a) reports primary action compliance across controlled splits; the policy-selection prompt’s gain is concentrated in Held-out Test, while all treatments remain at or below 1.04% on the two controlled holdouts. Panel (b) compares Raw SafeSuccess, primary artifact-grounded

A

, and strict role-consistent

A

on the same trajectories. The treatments share one actor and runtime and differ only in actor-visible procedure prompts. Split labels do not imply real-world out-of-distribution generalization.

Figure 4: Policy-atom difficulty across the formal 12-configuration matrix. Hard-violation and Raw SafeSuccess rates are pooled over 576 interactive runs per atom and ordered by Hard-violation rate. Raw SafeSuccess is descriptive and is not the primary artifact-grounded action metric; automatic failure-mechanism labels are not human annotations. The plot is descriptive and does not support per-atom significance claims.

Evidence set	Runs	Share (%)
All formal interactive runs	10,944	100.00
Not Raw SafeSuccess	9,632	88.01
Unsafe failures with finding	8,815	80.55
First finding by turn 4	7,859	71.81

Appendix

Table 12: Automatic failure denominators and onset distribution. The lower panel is conditioned on the 8,815 unsafe-failure runs with at least one finding.

Figure 5: Representative trajectory evidence from the formal matrix. Rows separate the decisive transcript evidence, ledger or terminal state, and score for six complementary failure modes.

Condition	$A$ (%)	Raw Safe (%)	Hard (%)
Full public KB	1.50	11.28	82.71
Retrieval only	3.01	4.51	93.23
External verifier only	0.75	3.76	90.23
Retrieval + verifier	9.02	9.02	86.47

Appendix

Table 13: Mechanism-pilot results (percent). These component conditions are distinct from the prompt-only treatments in the confirmatory matrix.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Oct 9, 2026

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented dialogue benchmarks assume cooperative users and measure task completion; SmartSalesBench inverts this assumption, placing agents against personas programmed to resist and challenge across conversations of up to twenty turns. The benchmark includes five agent profiles, twelve buyer personas, fifteen smartphone products across six price tiers, and four controlled experiments with 3 iterations totaling 13,680 sessions. Both sales-agent and buyer persona generations are grounded through retrieval-augmented generation (RAG). We introduce nine evaluation metrics grouped into three tiers: primary outcome metrics (CR, BRS, RIR), secondary quality metrics (SMS, PFS, ETI), and exploratory metrics planned for future reporting (ITS, ARI, SPI). Our evaluation combines automated LLM-as-judge scoring with internal human assessment (n = 50, \k{appa} = 0.71, Pearson r = 0.81 with BRS).

P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata +2

Samsung R&D Institute Bangalore, India

LLM Agent Evaluation AI Agent Benchmarks Multi-Turn Dialogue Evaluation Large Language Model-Based Role-Play Simulation

Ask PDF

Aug 20, 2026

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

Customer-service LLM agents must follow organizational policy when acting on a user's behalf. Compliance failures arise from either forbidden actions, such as granting an ineligible change, or omitted procedural requirements, such as identification or confirmation. Runtime safeguards can intervene on risky actions, but action-local checks do not guide an agent through a multi-step procedure. Workflow-following systems support prescribed process execution, but primarily target workflow completion rather than safeguarding agent behavior. PolicyGuide instead compiles each domain policy into a workflow graph and invokes a proactive verifier at user-turn boundaries. From persisted graph state, the verifier reconciles open requests and returns step-specific remediation along a policy-compliant path. Across the

τ^2

-bench airline, retail, and telecom domains with a GPT-5.4 agent and verifier, PolicyGuide raises mean

\mathrm{Pass}^4

from

0.42

0.62

, with the largest gain on telecom (

0.19

0.61

), the most workflow-structured domain. The same workflows transfer to Claude Sonnet 4.6 and Gemini 2.5 Pro agents. Complementary evaluations find the lowest observed attack-success rate under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Runtime Enforcement for AI Agents AI Agent Governance LLM Agent Verification LLM Guardrails

Ask PDF

Sep 16, 2026

PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?

As corporate AI adoption continues to grow, enterprise-grade LLM agents are being deployed into sensitive contexts such as hiring, healthcare, and finance. In these contexts, compliance with rules specified in an agent's system context is a first-order legal concern. Currently, no evaluation framework systematically measures which LLM models tend to violate compliance rules, especially under pressure from a persistent user, a hurried manager, or circumstances where violation is convenient or attractive. We introduce PACT (Pressure-Applied Compliance Testing), a benchmark for rule-following under pressure in AI agents assisting employees in daily tasks across twelve regulated enterprise domains and forty-eight scenarios, each set in a realistic multi-turn conversation. Each benchmark item pairs a standing rule against a rule-violating shortcut, and applies a battery of pressures across different wordings and system-prompt modes. We construct PACT component by component under strict LLM-as-judge auditing to ensure samples are unambiguous, ungameable, and realistic enough to avoid eliciting evaluation-aware behavior. We use PACT to profile LLM compliance across six complementary metrics that create a holistic picture of an AI assistant's robustness under pressure and throughout multi-turn conversations, its transparency, and ability to correctly discern where a rule applies. We aggregate this profile into PACTScore, a reliability-weighted compliance rate over all items and modes. Our results across 22 common LLM models spanning multiple providers and sizes show substantial variability in compliance across models and metric dimensions. Even the strongest assistants mis-apply a rule on 6 to 10% of items, and ordinary user pressure raises the violation rate by 65% on average. PACT highlights compliance risks in LLM assistants, motivating guardrails and careful model selection.

Mika Okamoto, Ansel Kaplan Erol

Georgia Institute of Technology · Decagon AI · Baseten

AI Agent Auditing LLM Agent Evaluation AI Agent Safety Benchmarks LLM Agent Reliability

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?