Personalization is classically a two-party problem: a sender chooses what to say, and a receiver with independent objectives decides whether to act. A salesperson pitching the same analytics product leads with HIPAA compliance for a hospital and real-time reporting for a retailer, expecting a different argument to work on each. Existing LLM personalization benchmarks measure a narrower, one-party property: whether output matches the preferences of the same user it serves-sender and receiver being the same, as when RLHF aligns an assistant to its own user. The two-party case is harder to study automatically, since it needs ground truth linking specific content to an observed receiver action. Sales outreach provides this: a message written for one prospect, recorded against whether it produced a reply, a call, or a closed deal. We introduce SDR-Arena, a framework for benchmarking two-party generative personalization at scale, and SDR-Bench, a public corpus of 50,000 customer success stories across 22 industries and 3500 enterprises. Given only pre-outcome information, an agent must reconstruct the arguments that won the deal, scored by a weighted nugget-recall metric (WCS). The best model, Claude Sonnet 4.6, reaches 55.8% WCS indicating it recovers only half the winning content-a plateau we observe across model families that costly deep-research pipelines do not close. An ablation shows the cause is retrieval, not reasoning: models improve substantially given the facts a human researcher would gather, but rarely find them through web search alone. Two studies with professional SDRs support the metric: only 48% of generated pitches were rated usable without editing, and WCS yields model rankings consistent with evaluation against expert strategies authored independently of any model output.
Figures & tables
Figure 1: Overview of SDR-Arena showcasing how LLM generated output is compared with artifacts like Sales Emails, Transcripts & Success Stories to benchmark their personalization capability
Figure 2: Sales Journey: From Prospecting to Outreach to Call and eventual Deal Closure leading to Success Story publication
Figure 3
Model
Tech.
Mfg.
Energy
IT
Agg.
Search
Claude Sonnet 4.6
52.6
60.5
54.0
54.9
55.8
GPT-5.4-mini
40.99
46.02
41.18
45.39
44.63
GPT-5.4
38.21
45.46
42.90
46.67
44.32
STORM-QWEN-2.5
43.40
41.59
39.24
44.60
42.51
GPT-4o-mini
33.30
38.54
37.07
38.05
37.46
Table 1: SDR-Bench. WCS on success stories across four industries and on the whole set.
Model
Health. ( < $1B)
Tech. ( > $10B)
Calls
Search
Claude Sonnet 4.6
60.89
61.33
42.85
GPT-5.4-mini
43.67
52.66
35.72
GPT-5.4
48.71
53.02
33.61
STORM-QWEN-2.5
32.27
39.24
33.80
GPT-4o-mini
39.14
45.63
29.68
Table 2: Private Enterprise Dataset. WCS on successful outreach emails and sales call transcripts.
QWEN-2.5-72B
Claude Sonnet 4.6
Report supplied
Direct
+ Search
Direct
+ Search
+ Customer
−2.5
−3.6
+0.9
−2.4
+ Seller
+5.2
+1.8
+5.9
+1.1
+ Product
+5.5
+2.8
+5.7
+1.7
+ All
+5.3
+3.3
+7.0
+6.6
Table 3: Expert-guided research reports. Δ WCS from expert-curated reports covering the customer, seller, product, or all three. Results are relative to each configuration’s no-report baseline. All results reported as averages over 3 runs.
Model
N -way Acc. (%)
Email-Succ. WCS
Story-Agg. WCS
Claude Sonnet 4.6
45.8
61.11
55.80
GPT-5.4
40.5
50.87
44.32
GPT-5.4-mini
39.0
48.17
44.63
GPT-4o
37.0
42.90
35.42
GPT-4o-mini
37.0
42.39
37.46
STORM
36.2
35.76
42.51
Table 4: Discriminative N -way accuracy ( N=5 : one successful email vs. four same-industry unsuccessful cold outbounds), for the web-search LLMs and the deep-research agents, alongside their WCS on enterprise emails and success stories.
Figure 5: Token usage vs. performance on SDR-Bench. Each point is a configuration; the x -axis is total tokens per query (log scale), the y -axis is WCS, and bubble area encodes inference cost per query.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Filtration Criteria
Count
Domains for companies with > $1B revenue
∼ 30k
B2B companies with > $1B revenue
12,080
Companies whose URLs were discovered
3,500
Total webpages found
∼ 7M
Candidate success story URLs
∼ 400k
After Qwen content filter (listicle/parent/ generic/no-date pages removed)
∼ 60k
Appendix
Table 5: Filtration criteria and surviving counts for the public data collected to build SDR-Bench.
Metric / Artifact Category
Healthcare
Tech
Number of SDRs
3
124
Total Emails Collected
48,150
609,191
Deduplication
31,034
186,379
Sales Outreach Emails
24,506
90,809
Successful Outreach (Call Scheduled)
354
13,236
Appendix
Table 6: Processing of Enterprise Sales Email Data
Figure 6: Personalization in actual sales emails: SDRs adapt the same product positioning across recipients with distinct needs and pain points.
TP
FP
FN
Precision
Recall
F1
138
11
3
0.92
0.97
0.95
Appendix
Table 7: Human evaluation of LLM-extracted pitch points on 30 customer success stories, assessing precision (factual consistency, hallucination flagging) and recall (missed pitch points).
Model
Mode
Prompt
Compl.
Cost
WCS
Tok.
Tok.
($)
STORM
Search
29k
6.3k
0.135
42.51
ODR
Search
66k
8.6k
0.250
33.53
Claude Sonnet 4.6
Search
79k
2.2k
0.270
55.80
Direct
0.3k
1.2k
0.019
49.88
GPT-5.4-mini
Search
7.6k
0.7k
0.009
44.63
Appendix
Table 8: Per-query token usage, inference cost, and WCS on the SDR-Bench evaluation set. Search = temporally-constrained web search; Direct = no tool use. Deep-research agents are search-only. Token counts are mean prompt/completion tokens per query; costs use public API list prices at evaluation time. The deep-research pipelines (STORM, ODR) consume one to two orders of magnitude more tokens than the LLM + search baselines for comparable or lower WCS.
Figure 7: Qualitative example of the Coverage Judge: ground-truth pitch points matched against the agent’s generated pitch points and scored on the 0 – 5 rubric of Appendix A.4 , illustrating how partial topic matches are separated from strategic bullseyes.
Model
Mean WCS
95% CI
Claude Sonnet 4.6
0.5580
[0.5238, 0.5906]
GPT-5.4-mini
0.4488
[0.4265, 0.4700]
GPT-5.4
0.4457
[0.4216, 0.4683]
STORM
0.4251
[0.4003, 0.4517]
GPT-4o-mini
0.3767
[0.3567, 0.3977]
Qwen-2.5-72B
0.3684
[0.3478, 0.3878]
Appendix
Table 9: Bootstrap estimates of WCS on the SDR-Bench evaluation set ( 1,000 resamples), with per-configuration means and 95% confidence intervals. The weaker GPT-family and Qwen intervals overlap (the personalization plateau), while Claude Sonnet 4.6 separates clearly above the rest.
Figure 8: SDR-Bench: Distribution of the count of success stories across different industries, complementing the company-level distribution in Figure 4 . The corpus spans a broad range of sectors, with technology, manufacturing, and energy the most represented.
Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.
Lechen Zhang, Jiarui Liu, Tal August
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose Ψ-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in Ψ-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on Ψ-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.
Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities. To address this gap, we introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains, including clothing, food, housing, and mobility. To support scalable benchmark construction while mitigating data sparsity and privacy concerns, LUNAR uses a multi-stage coarse-to-fine synthesis pipeline grounded in real-world behavioral patterns. Fidelity analyses show closer alignment with real behavioral distributions than other synthetic benchmarks. Experiments on 19 mainstream LLMs show that access to behavioral logs is necessary but not sufficient for deep personalization: neither more context nor larger models guarantees better performance; effective personalization depends on selecting and integrating relevant evidence across domains. Direct retrieval of fine-grained behavioral records consistently outperforms compressed memory, while stronger personalization can come at the cost of privacy protection. These findings identify evidence selection, cross-domain integration, and privacy control as key challenges for personalized LLMs.
Jiahao Zhang, Yongzhi Tong, Zelin Fu +4
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1