SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

P N Bhargav Teja; Velpula Lokesh Reddy; Madhu Basumata; Susen Sarkar; Rajasekhara Reddy Duvvuru Muni

cs.AIOct 9, 2026

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

Authors: P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata, Susen Sarkar, Rajasekhara Reddy Duvvuru Muni

Organizations: Samsung R&D Institute Bangalore, India

Abstract

We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented dialogue benchmarks assume cooperative users and measure task completion; SmartSalesBench inverts this assumption, placing agents against personas programmed to resist and challenge across conversations of up to twenty turns. The benchmark includes five agent profiles, twelve buyer personas, fifteen smartphone products across six price tiers, and four controlled experiments with 3 iterations totaling 13,680 sessions. Both sales-agent and buyer persona generations are grounded through retrieval-augmented generation (RAG). We introduce nine evaluation metrics grouped into three tiers: primary outcome metrics (CR, BRS, RIR), secondary quality metrics (SMS, PFS, ETI), and exploratory metrics planned for future reporting (ITS, ARI, SPI). Our evaluation combines automated LLM-as-judge scoring with internal human assessment (n = 50, \k{appa} = 0.71, Pearson r = 0.81 with BRS).

Figures & tables

Fig. 1: SmartSalesBench architecture. Five conversational stages are orchestrated in LangGraph; RAG grounding is applied to both sales-agent and buyer-persona turns. The evaluation layer includes three local Ollama judges (Llama 3.2:3B, Qwen2.5:7B, and Mistral:7B), an independent human-evaluation check, session logging, and statistical analysis.

Agent	Role	Primary Tactic	Max Discount	SPI Profile
Vikram	Store Executive	Target Expansion (discovery-first)	$\mathrm{Rs.\,}$ 2,000	Low–Medium
Rajesh	Store Manager	Authority + Urgency (escalation lead)	$\mathrm{Rs.\,}$ 5,000	High
Divya	Call Center	Script-driven + exclusive offers	$\mathrm{Rs.\,}$ 1,500	Medium
Neha	Brand Ambassador	Lifestyle identity framing †	$\mathrm{Rs.\,}$ 1,000	Low
Sanjay	Technical Advisor	Value proposition via spec translation	$\mathrm{Rs.\,}$ 2,500	Low

TABLE I: Agent Profiles

Persona	Resistance Type	Budget ( $\mathrm{Rs.\,}$ )	Core Fear	Persuasion Arc Trigger
Ananya	Eco Conscious	40,000	Contributing to e-waste	Trade-in and recycling programme
Arjun	Samsung Upgrader	60,000	Insufficient upgrade value	Camera and display differential
Dinesh	Business Owner	1,20,000	Productivity downtime	Enterprise battery and uptime specs
Karthik	OnePlus Fan	35,000	Losing clean stock Android	Direct One UI vs OxygenOS comparison
Krishnamurthy	Senior Buyer	25,000	Phone too complex	Accessibility feature demonstration
Lakshmi	First-Time Buyer	20,000	Wrong first-phone choice	Simplicity and senior-friendly features

TABLE II: Buyer Persona Set

Tier	Metrics	Role	Reported in
Primary	CR, BRS, RIR	Main benchmark comparison	All tables, all experiments
Secondary	SMS, PFS, ETI	Interpretability of primary	Selected tables and Discussion
Exploratory	ITS, ARI, SPI	Future reporting	Logged; not primary results here

TABLE III: Three-Tier Metric Organisation

Weighting scheme	PDF weight	RES weight	TPD weight	BRS weight	r with human judgments
Current (proposed)	0.35	0.25	0.20	0.20	r = 0.81
Equal weight	0.25	0.25	0.25	0.25	r = 0.74
Outcome-heavy	0.50	0.20	0.15	0.15	r = 0.69

TABLE IV: Sms Weight Sensitivity Analysis

ID	Name	Manipulation	max_turns	Sessions	Research Question
EXP-1	Baseline	None (5 agents, standard)	10	3,600	Which agent-persona pairings perform best?
EXP-2	Escalation	Rajesh at turn $\geq$ 5 if intent $\leq$ 4.0	10	2,880*	Does escalation improve CR without degrading BRS?
EXP-3	Extended	max_turns = 20	20	3,600	Does persistence improve CR or raise RIR?
EXP-4	Short	max_turns = 5	5	3,600	Can agents close faster; how does BRS change?

TABLE V: Controlled Experiment Design

Fig. 2: Customer intent trajectory by purchase outcome for EXP-1 ( $n=3{,}600$ ). Purchased sessions show a steady-rise pattern, whereas non-purchased sessions plateau after approximately turn 3. Shaded regions denote $\pm 2$ SD.

Metric	Target	Aggregate	Llama3.1-8B	Qwen2.5-3B	Phi3.5-mini	Gemma2-2B	Interpretation
CR (95% CI)	$\geq$ 25%	44.4% [42.8, 46.0]	24.2% [21.4, 27.0]	37.1% [34.0, 40.3]	72.6% [69.6, 75.5]	43.7% [40.4, 46.9]	Above target; consistent with cold-contact retail norms
Avg BRS	$\geq$ 6.5	7.11	7.49	7.20	6.79	6.95	Strong resistance; benchmark data poses realistic challenge
Avg SMS	$\geq$ 5.5	5.68	5.58	5.67	5.02	6.44	Above target; persuasion arc quality is adequate
Avg Turns	7–10	8.15	8.50	8.33	7.41	8.35	Within target; conversation length is realistic
RIR	<8%	0.53%	1.15%	0.51%	0.34%	0.12%	Well within target; personas hold character
Fidelity (PFS)	$\geq$ 6.0	7.09	6.82	7.03	8.13	6.40	Above target; persona identity is stable

TABLE VI: Exp-1 Baseline Results (Aggregate and Per-Model, n=900 each)

Fig. 3: Key performance indicators across four controlled experiments with Llama3.1-8B model. EXP-2 (Escalation) has the highest CR (25.4%), while EXP-4 (5 turns) has the lowest CR (10.1%).

Fig. 4: Key performance indicators across four controlled experiments with Phi 3.5-mini-3.8B model. EXP-1 has the highest CR (72.6%), while EXP-4 (5 turns) has the lowest CR (16.4%).

Metric	EXP-1	EXP-2	EXP-3	EXP-4	Key Finding
CR (%)	24.2	25.4	22.9	10.1	$\chi^{2}$ =68.74, V=0.18, p<0.001
BRS	7.49	7.51	7.49	6.95	Stable across all conditions
SMS	5.58	5.56	5.63	6.26	Appears inversely related to turn count
Turns (avg)	8.43	8.4	8.34	3.9	As designed; confirms manipulation validity
RIR (%)	1.28	0.94	0.99	1.10	$\chi^{2}$ =0.020, p=0.999 — flat across all conditions
Fidelity	7.53	7.51	7.57	7.51	Slight decrease under extended turns

TABLE VII: Cross-Experiment Comparison-Llama3.1:8b

Fig. 5: Generator-model screening across CR, BRS, and persona fidelity. Table VIII reports only the validated models.

Model (full identifier)	CR (%)	BRS	RIR (%)	Fidelity	Conv. n*	Failure Mode
Llama3.1-8B (Meta-Llama-3.1-8B-Instruct)	26.2	7.47	7.03	6.95	236	Selected; strongest verified balance
Qwen2.5-3B (Qwen2.5-3B-Instruct)	35.4	7.18	12.71	7.62	319	Resistance erosion
Gemma2-2B (Gemma-2-2b-it)	39.2	6.92	8.51	7.06	353	Persona collapse
Phi3.5-mini (Phi-3.5-mini-instruct)	70.7	6.70	13.39	8.17	636	Over-conversion; low resistance

TABLE VIII: Validated Generator-Model Comparison

Tier(Sample Size)	Model	n	CR (%)	BRS	SMS	Fidelity
900 sessions	Llama3.1-8B	900	26.2	7.47	6.02	6.95
	Qwen2.5-3B	900	35.4	7.18	6.39	7.62
	Gemma2-2B	900	39.2	6.92	7.14	7.06
	Phi3.5-mini	900	70.7	6.70	5.88	8.17
600 sessions	Llama3.1-8B	600	25.8	7.48	6.01	6.90
	Qwen2.5-3B	600	36.2	7.19	6.41	7.60

TABLE IX: Generator-Model Stability Across Session-Volume Tiers

Rank	Model	CR Var.	BRS Var.	Fidelity Var.	RIR Var.	Overall
1	Llama3.1-8B	0.4 pp	0.05	0.05	0.55%	MOST STABLE
2	Gemma2-2B	0.9 pp	0.03	0.05	0.10%	Stable
3	Phi3.5-mini	1.3 pp	0.06	0.03	0.63%	Moderate
4	Qwen2.5-3B	3.4 pp	0.02	0.02	0.88%	LEAST STABLE

TABLE X: Final Stability Ranking (Lower Variation = More Stable)

Fig. 6: Conversion rate across all 60 agent–persona combinations in EXP-1 ( $n=900$ ). Personas are ordered alphabetically; the highest pairing is Sanjay (Tech Adv.) $\times$ Priya (Apple Loyal).

Agent (Role)	Persona (Type)	CR (%)	n	Persona-Aligned Interpretation
Sanjay (Tech Adv.)	Priya (Apple Loyal)	51.8	720	Spec translation can address iOS feature-parity concerns
Neha (Brand Amb.)	Priya (Apple Loyal)	51.3	733	Brand framing accompanies emphasis on iOS feature parity
Vikram (Store Exec.)	Priya (Apple Loyal)	50.6	720	Discovery-first interaction can surface ecosystem-switching concerns
Divya (Call Ctr.)	Priya (Apple Loyal)	50.1	719	iOS feature-parity information addresses ecosystem-disruption concerns
Neha (Brand Amb.)	Karthik (OnePlus Fan)	49.7	690	One UI–OxygenOS comparison addresses clean-Android concerns
Bottom Five Pairings

TABLE XI: Top and Bottom Agent–Persona Pairings

Test	Hypothesis	Statistic	p-value	Effect Size
Chi-square	CR across 4 experiments	$\chi^{2}$ =867.48, df=3	<0.001	CramÃ©r’s V = 0.25 (medium)
One-way ANOVA	BRS across 4 experiments	F=121.97	<0.001	$\eta^{2}$ = 0.026 (small)
Point-biserial r	SMS predicts purchase	r=-0.227	<0.001	Small effect; negative
Chi-square	RIR across 4 experiments	$\chi^{2}$ =3.21, df=3	0.360	Slight variation in RIR (0.44–0.59%)

TABLE XII: Statistical Tests And Effect Sizes

Fig. 7: Sample-size convergence using rolling CR averages. Dashed vertical lines denote empirical stabilization points; Phi3.5-mini converges later than the other validated generator models.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill
Apr 8, 2026Xuanbo Su, Wenhao Hu, Le Zhan +8LLM Agent Evaluation Persuasion in Language Models
Bairong Inc. · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +4
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Aug 6, 2026Noam Koren, Roy Bar-Haim, Abigail GoldsteenAI Agent Benchmarks Task-Oriented Dialogue Systems
IBM Research
SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
May 8, 2026Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2RL for Language Models Large Language Model-Based Role-Play Simulation
Salesforce Research · University of California Los Angeles

Apr 8, 2026J/K move · Enter open · S save

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

Xuanbo Su, Wenhao Hu, Le Zhan, Yuting Xie+7

Bairong Inc. · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy+4

LLM Agent Evaluation Persuasion in Language Models Multi-Turn Dialogue Evaluation Large Language Model-Based Role-Play Simulation

Open paper AskPDF arXiv

Sales dialogues require multi-turn, goal-directed persuasion under asymmetric incentives, which makes them a challenging setting for large language models (LLMs). Yet existing dialogue benchmarks rarely measure deal progression and outcomes. We introduce SalesLLM benchmark, a bilingual (ZH/EN) benchmark derived from realistic applications covering Financial Services and Consumer Goods, built from 30,074 scripted configurations and 1,805 curated multi-turn scenarios with controllable difficulty and personas. We propose a fully automatic evaluation pipeline that combines (i) an LLM judge for sales-process progress, and (ii) fine-tuned BERT classifiers for end-of-dialogue buying intent. To improve simulation fidelity, we train a user model, CustomerLM, with SFT and DPO on 8,000+ crowdworker-involved sales conversations, reducing role inversion from 17.44% (GPT-4o) to 8.8%. SalesLLM benchmark scores correlate strongly with human ratings (average Pearson r=0.86; inter-annotator agreement Krippendorff's alpha=0.86, 500 annotations per model). Across 15 mainstream LLMs, in Chinese the strongest models are competitive with typical (junior-to-intermediate) human salespeople -- not with sales experts -- while weaker ones fall below this baseline, and cross-lingual consistency remains poor. SalesLLM benchmark serves as a scalable benchmark for outcome-oriented sales agents. Our code and data are released at https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill

Abstract

LLM Agent Evaluation

Explore similar work

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators