Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

Xuanbo Su; Wenhao Hu; Le Zhan; Yuting Xie; Kailin Lyu; Kaijie Chen; Ziwei Li; Yeqiang Wang

cs.CLApr 8, 2026

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

Authors: Xuanbo Su, Wenhao Hu, Le Zhan, Yuting Xie, Kailin Lyu, Kaijie Chen, Ziwei Li, Yeqiang Wang,

Organizations: Bairong Inc. · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy · Tongji University ·

Abstract

Sales dialogues require multi-turn, goal-directed persuasion under asymmetric incentives, which makes them a challenging setting for large language models (LLMs). Yet existing dialogue benchmarks rarely measure deal progression and outcomes. We introduce SalesLLM benchmark, a bilingual (ZH/EN) benchmark derived from realistic applications covering Financial Services and Consumer Goods, built from 30,074 scripted configurations and 1,805 curated multi-turn scenarios with controllable difficulty and personas. We propose a fully automatic evaluation pipeline that combines (i) an LLM judge for sales-process progress, and (ii) fine-tuned BERT classifiers for end-of-dialogue buying intent. To improve simulation fidelity, we train a user model, CustomerLM, with SFT and DPO on 8,000+ crowdworker-involved sales conversations, reducing role inversion from 17.44% (GPT-4o) to 8.8%. SalesLLM benchmark scores correlate strongly with human ratings (average Pearson r=0.86; inter-annotator agreement Krippendorff's alpha=0.86, 500 annotations per model). Across 15 mainstream LLMs, in Chinese the strongest models are competitive with typical (junior-to-intermediate) human salespeople -- not with sales experts -- while weaker ones fall below this baseline, and cross-lingual consistency remains poor. SalesLLM benchmark serves as a scalable benchmark for outcome-oriented sales agents. Our code and data are released at https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill

Figures & tables

Figure 1: Overview of the SalesLLM benchmark: (1) Standardized Script Generation: Synthesizing role-play scripts by integrating product inventories, persona pools, and difficulty controls. (2) CustomerLM Development & Simulation: Training a specialized simulator via SFT and DPO to conduct simulated dialogues with evaluated sales LLMs. (3) Evaluation Pipeline: Evaluating sales performance through a dual-metric evaluation framework combining BERT-based intent classification for outcomes and LLM-as-a-judge scoring for process quality.

Figure 2: The Workflow of the Script Generation Pipeline. The construction process follows a logical transition from raw resources to synthesized scripts: (1) Raw Component Generation: The pipeline establishes three pillars: Product Information, where raw items are expanded into detailed profiles and deduplicated via MinHash; Persona Generation, involving the sampling of demographic attributes (Age, City, and Career); and the specification of Difficulty Profiles. (2) Persona-Product Pairing: The processed product-service pool is mapped to the sampled personas to form unique Persona-Product Pairs as the contextual foundation for the scenario. (3) LLM-based Information Synthesis: Based on these pairs and the assigned difficulty, LLMs generate the final scripts, including a User Script with product-specific motivations and pain points, alongside a Sales Script containing structured product features and selling points.

Difficulty	Avg. Score	# Scenarios
easy	6.79	2,779
medium	6.22	2,998
hard	5.45	1,219
very hard	5.33	3,320
adversarial	4.98	2,157

Table 1: Average SalesLLM benchmark scores by difficulty level, aggregated over Gemini-3-Flash, Qwen2.5-72B, Doubao-1.5-pro, DeepSeek-Chat, GLM-4.6, and Qwen3-Max. Difficulty is controlled solely through the customer’s system prompt.

Figure 3: The training pipeline of CustomerLM: (1) SFT on 8,000+ human-annotated sales dialogues to learn fundamental customer behaviors; (2) DPO using 268 contrastive pairs—refined through LLM judging and human correction—to suppress assistant-style bias and role inversion for enhanced persona consistency.

Intent classifier	ZH	EN
GPT-4o, zero-shot	69.60	68.85
GPT-4o, 10-shot	78.42	81.92
Fine-tuned BERT (ours)	93.51	92.94

Table 2: Prompt-controlled buying-intent classification (accuracy, %). Few-shot prompting helps GPT-4o substantially, yet a sizeable gap remains.

	GPT-4o User		CustomerLM
Assistant Model	ZH	EN	ZH	EN
Doubao-1.5-pro-32k	6.50	6.31	6.89 †	5.48
Qwen3-Max	6.48	5.97	5.55	5.56
DeepSeek-Chat (V3.1)	6.74	6.10	7.03	5.80
GPT-4o	6.16	5.53	6.15	5.19
GLM-4-9B	6.01	5.92	7.14	5.55

Table 3: Overall performance (SalesLLM benchmark score) of different Assistant Models when evaluated against two user simulators: GPT-4o and our CustomerLM Model. Results are reported for both Chinese (ZH) and English (EN) scenarios. The best result is bolded, the second best is underlined, and the third best is marked with † .

Panel of Figure 4 — Figure 4: Distribution of SalesLLM benchmark scores vs. dialogue turns across 1,805 scenarios (Chinese and English).

Figure 5: Correlation analysis between Human evaluation scores and SalesLLM benchmark scores. We evaluated four representative models, with 500 dialogues per model annotated by human experts. The overall average Pearson correlation coefficient is $r=0.86$ , demonstrating strong alignment between automated and human judgments.

	GPT-4o User		CustomerLM
Assistant Model	ZH	EN	ZH	EN
Doubao-1.5-pro-32k	7.46	6.62	6.16	5.93
DeepSeek-Chat (V3.1)	6.47	6.96	6.24	5.67
Gemini-3-pro	7.17	7.03	6.56	5.95

Table 4: Performance of multi-product selling of different Assistant Models when evaluated against two user simulators: GPT-4o and our CustomerLM Model. Results are reported for both Chinese (ZH) and English (EN) scenarios.

	GPT-4o User		CustomerLM
Assistant Model	ZH	EN	ZH	EN
Doubao-1.5-pro-32k	4.91	4.62	4.58	4.4
DeepSeek-Chat (V3.1)	4.58	4.48	5.92	5.82
Gemini-3-pro	4.43	4.58	6.20	4.88

Table 5: Performance of long horizon selling of different Assistant Models when evaluated against two user simulators: GPT-4o and our CustomerLM Model. Results are reported for both Chinese (ZH) and English (EN) scenarios.

Model	BLEU	R-L	Sem.	RI (%)
GPT-4o	0.10	0.07	0.57	17.44
UserLM	0.06	0.06	0.50	21.55
USP	0.08	0.08	0.52	18.76
CustomerLM	0.12	0.10	0.59	8.8

Table 6: Ablation study comparing CustomerLM with GPT-4o, UserLM, and USP as user models. R-L denotes ROUGE-L and RI denotes role inversion; higher is better except for RI.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Product	# Dialogues	Avg. Turns	Avg. Msg Len
Security	189	3.32	14.07
Banking	50	4.79	10.2
Fund	29	2.61	16.23

Appendix

Table B.1: The statistical information of CustomerLM test data. # Dialogues is the number of dialogues, Avg. Turns is the average turns of the dialogues, Avg. Msg Len is the average user message lengths.

Product	# Dialogues	Avg. Turns	Avg. Msg Len
Security	4748	2.3	16.25
Banking	1742	5.98	12.03
Fund	1794	5.42	11.97

Appendix

Table B.2: The statistical information of CustomerLM training data. # Dialogues is the number of dialogues, Avg. Turns is the average turns of the dialogues, Avg. Msg Len is the average number of words per user message.

Lang	#Dia	Avg msgs/conv	Avg user msgs
ZH (118_zh)	118	7.98	3.85
EN (150_en)	150	8.13	3.83

Appendix

Table B.3: Held-out ablation test set statistics (Sec. 4.5 ). Averages exclude the system prompt. Lang is language, Dia is dialogues.

Metric	GPT-4o	CustomerLM
N evaluated turns	750	751
History length	5.33	5.33
BLEU-4	0.1000 $\pm$ 0.1635	0.1218 $\pm$ 0.1749
ROUGE-1	0.0846 $\pm$ 0.1207	0.1062 $\pm$ 0.1499
ROUGE-2	0.0176 $\pm$ 0.0554	0.0250 $\pm$ 0.0790
ROUGE-L	0.0725 $\pm$ 0.1064	0.0985 $\pm$ 0.1414

Appendix

Table B.4: Turn-level user-likeness statistics on the held-out set (mean

\pm

std; without “±”: mean). Best results are bolded.

Parameter	Value
Method	SFT / DPO
Training type	Full fine-tuning
Hardware	5 $\times$ A100 (80 GB)
Distributed training	DeepSpeed ZeRO-3
dtype	bfloat16
Epochs	2 / 1

Appendix

Table B.5: SFT/DPO training configuration for CustomerLM. Where two values are given, they correspond to SFT/DPO respectively.

Component	# Seeds	# Instances
E-commerce products	33	10,074
Financial services	300	20,000
User personas	–	19,138

Appendix

Table C.1: Statistics of the SalesLLM benchmark scenario space.

Category	#instances	Share (%)
Pet Supplies	392	3.89
Magazine Subscriptions	367	3.64
Patio Lawn and Garden	364	3.61
Office Products	361	3.58
Health and Personal Care	358	3.55
Sports and Outdoors	351	3.48

Appendix

Table C.2: Category distribution of the E-commerce scenario space in SalesLLM benchmark.

Category	# Instances	Share (%)
bank_deposits	10,521	52.61
stocks	6,109	30.55
insurance	2,858	14.29
bund_fund	512	2.56
Total	20,000	100.00

Appendix

Table C.3: Distribution of Financial Services categories.

Profile	Buy prop. $p_{k}$	Buyer Prompt
easy	0.80	Open-minded, motivated buyer with clear pain points and flexible budget; decides quickly if the product is a plausible fit.
medium	0.50	Balanced buyer with concrete but resolvable concerns (e.g., price or fit); requires reasonable evidence and engages in moderate objection handling.
hard	0.20	Skeptical, price-sensitive, and risk-averse buyer; defaults to negative purchase intent unless strong, specific evidence and clear ROI are demonstrated.
very hard	0.10	Highly skeptical enterprise buyer with strict compliance and procurement constraints; requires detailed proof, references, and process alignment, typically postponing purchase.
adversarial	0.05	Adversarial evaluator primarily focused on disqualifying vendors; emphasizes edge cases, legal risk, and total cost of ownership, and almost never expresses positive purchase intent.

Appendix

Table C.4: Difficulty profiles used to control buy propensity and buyer style for the simulated customer.

Model	Lang	Acc	F1
GPT-4o (zero-shot)	ZH	69.60%	0.73
GPT-4o (10-shot)	ZH	78.42%	–
Trained BERT	ZH	93.51%	0.93
GPT-4o (zero-shot)	EN	68.85%	0.72
GPT-4o (10-shot)	EN	81.92%	–
Trained BERT	EN	92.94%	0.93

Appendix

Table E.1: Prompt-controlled accuracy comparison: fine-tuned BERT vs. GPT-4o under zero-shot and 10-shot prompting (Appendix L ). Lang is language (ZH is Chinese, EN is English), Acc is accuracy and F1 is F1 score.

Figure J.1: Decision Factor Distribution by Age Group. Stacked bars show the total number of decision-factor mentions ( Count ) for each age group in the SalesLLM benchmark persona set. Colors denote decision-factor categories (Return & Cost, Convenience & Digital, Fit & Flexibility, Safety & Protection, Support & Aftercare, Transparency & Terms, Social Proof, Brand Trust, Core Quality, Availability & Delivery), and the percentages inside each segment indicate that category’s share of mentions within the corresponding age group.

Panel of Figure 23 — Figure K.1: Qualitative comparison: DeepSeek (Proactive) vs. GPT-4o (Passive).

Metric	Value
Mean pairwise Pearson $r$	0.82
$\mathrm{ICC}(2,1)$	0.82
Krippendorff’s $\alpha$ (interval)	0.86

Appendix

Table M.1: Inter-annotator agreement (2,000 dialogues, eight annotators).

	easy	med.	hard	v. hard	adv.	Mean
$r$	0.79	0.88	0.75	0.85	0.87	0.83

Appendix

Table M.2: Human–auto Pearson

r

within each difficulty tier.

Sub-dimension	$r$
Verbal purchase commitment	0.86
Concrete next-step alignment	0.79
Key information elicitation	0.80
Objection resolution effectiveness	0.82

Appendix

Table M.3: Auto-vs-human Pearson

r

per Selling Performance sub-dimension (definitions: Appendix P ).

Model	$\alpha{=}0.1$	$\alpha{=}0.3$	$\alpha{=}0.5$	$\alpha{=}0.7$
Doubao-1.5-pro-32k	0.748	0.749	0.746	0.678
DeepSeek-Chat	0.802	0.812	0.906	0.765
GPT-4o	0.831	0.829	0.910	0.731
Qwen3-Max	0.829	0.824	0.884	0.733
Average	0.802	0.804	0.862	0.727

Appendix

Table N.1: Pearson correlation (

r

) between automated SalesLLM benchmark scores and human judgments at different values of the weighting parameter

\alpha

. Higher is better.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents
Oct 9, 2026P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata +2LLM Agent Evaluation AI Agent Benchmarks
Samsung R&D Institute Bangalore, India
SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
May 8, 2026Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2RL for Language Models Large Language Model-Based Role-Play Simulation
Salesforce Research · University of California Los Angeles
Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
Jul 11, 2026Jinglan Gong, Jiefan Lu, Hewei Guo +5Language Model Generation Evaluation Persona Consistency
SenseTime Research · University of Science and Technology of China · Tsinghua University

Oct 9, 2026J/K move · Enter open · S save

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata, Susen Sarkar+1

Samsung R&D Institute Bangalore, India

LLM Agent Evaluation AI Agent Benchmarks Multi-Turn Dialogue Evaluation Large Language Model-Based Role-Play Simulation

Open paper AskPDF arXiv

We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented dialogue benchmarks assume cooperative users and measure task completion; SmartSalesBench inverts this assumption, placing agents against personas programmed to resist and challenge across conversations of up to twenty turns. The benchmark includes five agent profiles, twelve buyer personas, fifteen smartphone products across six price tiers, and four controlled experiments with 3 iterations totaling 13,680 sessions. Both sales-agent and buyer persona generations are grounded through retrieval-augmented generation (RAG). We introduce nine evaluation metrics grouped into three tiers: primary outcome metrics (CR, BRS, RIR), secondary quality metrics (SMS, PFS, ETI), and exploratory metrics planned for future reporting (ITS, ARI, SPI). Our evaluation combines automated LLM-as-judge scoring with internal human assessment (n = 50, \k{appa} = 0.71, Pearson r = 0.81 with BRS).

Abstract

LLM Agent Evaluation

Explore similar work

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging