Benchmarking the Personalization Capabilities of Large Language Models

Ashutosh Srivastava; Siddharth Yedlapati; Vinay Aggarwal; Yaman Kumar Singla; Shashwat Dixit; Jitendra Ajmera; Balaji Krishnamurthy

cs.AIMay 23, 2026

Benchmarking the Personalization Capabilities of Large Language Models

Authors: Ashutosh Srivastava, Siddharth Yedlapati, Vinay Aggarwal, Yaman Kumar Singla, Shashwat Dixit, Jitendra Ajmera, Balaji Krishnamurthy

Organizations: Adobe Media & Data Science Research

Abstract

Personalization is classically a two-party problem: a sender chooses what to say, and a receiver with independent objectives decides whether to act. A salesperson pitching the same analytics product leads with HIPAA compliance for a hospital and real-time reporting for a retailer, expecting a different argument to work on each. Existing LLM personalization benchmarks measure a narrower, one-party property: whether output matches the preferences of the same user it serves-sender and receiver being the same, as when RLHF aligns an assistant to its own user. The two-party case is harder to study automatically, since it needs ground truth linking specific content to an observed receiver action. Sales outreach provides this: a message written for one prospect, recorded against whether it produced a reply, a call, or a closed deal. We introduce SDR-Arena, a framework for benchmarking two-party generative personalization at scale, and SDR-Bench, a public corpus of 50,000 customer success stories across 22 industries and 3500 enterprises. Given only pre-outcome information, an agent must reconstruct the arguments that won the deal, scored by a weighted nugget-recall metric (WCS). The best model, Claude Sonnet 4.6, reaches 55.8% WCS indicating it recovers only half the winning content-a plateau we observe across model families that costly deep-research pipelines do not close. An ablation shows the cause is retrieval, not reasoning: models improve substantially given the facts a human researcher would gather, but rarely find them through web search alone. Two studies with professional SDRs support the metric: only 48% of generated pitches were rated usable without editing, and WCS yields model rankings consistent with evaluation against expert strategies authored independently of any model output.

Figures & tables

Figure 1: Overview of SDR-Arena showcasing how LLM generated output is compared with artifacts like Sales Emails, Transcripts & Success Stories to benchmark their personalization capability

Figure 2: Sales Journey: From Prospecting to Outreach to Call and eventual Deal Closure leading to Success Story publication

Model	Tech.	Mfg.	Energy	IT	Agg.
Search
Claude Sonnet 4.6	52.6	60.5	54.0	54.9	55.8
GPT-5.4-mini	40.99	46.02	41.18	45.39	44.63
GPT-5.4	38.21	45.46	42.90	46.67	44.32
STORM-QWEN-2.5	43.40	41.59	39.24	44.60	42.51
GPT-4o-mini	33.30	38.54	37.07	38.05	37.46

Table 1: SDR-Bench. WCS on success stories across four industries and on the whole set.

Model	Health. ( $<$ $1B)	Tech. ( $>$ $10B)	Calls
Search
Claude Sonnet 4.6	60.89	61.33	42.85
GPT-5.4-mini	43.67	52.66	35.72
GPT-5.4	48.71	53.02	33.61
STORM-QWEN-2.5	32.27	39.24	33.80
GPT-4o-mini	39.14	45.63	29.68

Table 2: Private Enterprise Dataset. WCS on successful outreach emails and sales call transcripts.

	QWEN-2.5-72B		Claude Sonnet 4.6
Report supplied	Direct	$+$ Search	Direct	$+$ Search
$+$ Customer	$-2.5$	$-3.6$	$+0.9$	$-2.4$
$+$ Seller	$+5.2$	$+1.8$	$+5.9$	$+1.1$
$+$ Product	$\mathbf{+5.5}$	$+2.8$	$+5.7$	$+1.7$
$+$ All	$+5.3$	$\mathbf{+3.3}$	$\mathbf{+7.0}$	$\mathbf{+6.6}$

Table 3: Expert-guided research reports.

\Delta

WCS from expert-curated reports covering the customer, seller, product, or all three. Results are relative to each configuration’s no-report baseline. All results reported as averages over 3 runs.

Model	$N$ -way Acc. (%)	Email-Succ. WCS	Story-Agg. WCS
Claude Sonnet 4.6	45.8	61.11	55.80
GPT-5.4	40.5	50.87	44.32
GPT-5.4-mini	39.0	48.17	44.63
GPT-4o	37.0	42.90	35.42
GPT-4o-mini	37.0	42.39	37.46
STORM	36.2	35.76	42.51

Table 4: Discriminative

N

-way accuracy (

N=5

: one successful email vs. four same-industry unsuccessful cold outbounds), for the web-search LLMs and the deep-research agents, alongside their WCS on enterprise emails and success stories.

Figure 5: Token usage vs. performance on SDR-Bench. Each point is a configuration; the

x

-axis is total tokens per query (log scale), the

y

-axis is WCS, and bubble area encodes inference cost per query.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Filtration Criteria	Count
Domains for companies with $>$ $1B revenue	$\sim$ 30k
B2B companies with $>$ $1B revenue	12,080
Companies whose URLs were discovered	3,500
Total webpages found	$\sim$ 7M
Candidate success story URLs	$\sim$ 400k
After Qwen content filter (listicle/parent/ generic/no-date pages removed)	$\sim$ 60k

Appendix

Table 5: Filtration criteria and surviving counts for the public data collected to build SDR-Bench.

Metric / Artifact Category	Healthcare	Tech
Number of SDRs	3	124
Total Emails Collected	48,150	609,191
Deduplication	31,034	186,379
Sales Outreach Emails	24,506	90,809
Successful Outreach (Call Scheduled)	354	13,236

Appendix

Table 6: Processing of Enterprise Sales Email Data

Figure 6: Personalization in actual sales emails: SDRs adapt the same product positioning across recipients with distinct needs and pain points.

TP	FP	FN	Precision	Recall	F1
138	11	3	0.92	0.97	0.95

Appendix

Table 7: Human evaluation of LLM-extracted pitch points on 30 customer success stories, assessing precision (factual consistency, hallucination flagging) and recall (missed pitch points).

Model	Mode	Prompt	Compl.	Cost	WCS
		Tok.	Tok.	($)
STORM	Search	29k	6.3k	0.135	42.51
ODR	Search	66k	8.6k	0.250	33.53
Claude Sonnet 4.6	Search	79k	2.2k	0.270	55.80
Claude Sonnet 4.6	Direct	0.3k	1.2k	0.019	49.88
GPT-5.4-mini	Search	7.6k	0.7k	0.009	44.63

Appendix

Table 8: Per-query token usage, inference cost, and WCS on the SDR-Bench evaluation set. Search = temporally-constrained web search; Direct = no tool use. Deep-research agents are search-only. Token counts are mean prompt/completion tokens per query; costs use public API list prices at evaluation time. The deep-research pipelines (STORM, ODR) consume one to two orders of magnitude more tokens than the LLM

{+}

search baselines for comparable or lower WCS.

Figure 7: Qualitative example of the Coverage Judge: ground-truth pitch points matched against the agent’s generated pitch points and scored on the $0$ – $5$ rubric of Appendix A.4 , illustrating how partial topic matches are separated from strategic bullseyes.

Model	Mean WCS	95% CI
Claude Sonnet 4.6	0.5580	[0.5238, 0.5906]
GPT-5.4-mini	0.4488	[0.4265, 0.4700]
GPT-5.4	0.4457	[0.4216, 0.4683]
STORM	0.4251	[0.4003, 0.4517]
GPT-4o-mini	0.3767	[0.3567, 0.3977]
Qwen-2.5-72B	0.3684	[0.3478, 0.3878]

Appendix

Table 9: Bootstrap estimates of WCS on the SDR-Bench evaluation set (

1{,}000

resamples), with per-configuration means and 95% confidence intervals. The weaker GPT-family and Qwen intervals overlap (the personalization plateau), while Claude Sonnet 4.6 separates clearly above the rest.

Figure 8: SDR-Bench: Distribution of the count of success stories across different industries, complementing the company-level distribution in Figure 4 . The corpus spans a broad range of sectors, with technology, manufacturing, and energy the most represented.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Re-Centering Humans in LLM Personalization
Jun 4, 2026Lechen Zhang, Jiarui Liu, Tal AugustHuman Preference Evaluation LLM Evaluation
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM Evaluation LLM Personalization
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1

Jun 4, 2026J/K move · Enter open · S save

Re-Centering Humans in LLM Personalization

Lechen Zhang, Jiarui Liu, Tal August

†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University

Human Preference Evaluation LLM Evaluation LLM Personalization Personalized Language Models User Modeling Personalized Text Generation+1

Open paper AskPDF arXiv

Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

Abstract

Language Model Generation Evaluation

Explore similar work

Re-Centering Humans in LLM Personalization

ΨΨΨ-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues