Estimating advertising performance in cold-start settings is challenging because historical logs are unavailable before launch. Large language models (LLMs) offer a potential alternative by simulating persona-conditioned user responses, but it remains unclear which design choices actually drive LLM-based persona simulation. Using real Google Ads e-commerce campaign data across five demographic segments, we identify three signal-form dimensions that affect simulation quality: (i) a demographic anchor without noisy rich attributes, (ii) a relative qualitative banner-blindness signal rather than quantitative anchors, and (iii) prompt phrasing. On this campaign, our method attains an MAE of 24.9 and a MAPE of 26.1%, a 51.3% lower MAE than the best tested non-LLM baseline and 77.3% lower than an LLM configuration using explicit CTR anchors. Systematic ablations show that minimal demographic descriptions achieve lower mean error than richer persona configurations, while relative qualitative cues substantially outperform the tested quantitative anchors. We additionally compare these configurations with direct-prior and rule-based baselines and with alternative LLM configurations.
Figures & tables
Figure 1: Overview of our LLM persona simulation framework. We examine three key dimensions of signal form that affect simulation quality: (i) a demographic anchor, (ii) a relative qualitative banner-blindness signal, and (iii) careful prompt phrasing. The LLM simulator generates repeated binary click decisions, which are aggregated into predicted CTR and predicted clicks.
Persona
Age
Gender
Parent status
Actual CTR (%)
A
45–54
Female
Parent
0.680
B
55–64
Male
Parent
0.607
C
25–34
Female
Non-parent
0.174
D
25–34
Male
Non-parent
0.200
E
35–44
Female
Parent
0.266
Table 1: Audience segments used as personas and their observed CTR over the 7-day evaluation period. All segments additionally share the same interests setting (“Shoppers”); income range is part of the base persona and is included in the prompt.
Component
Persona ( ps ): You are a user with the following demographic attributes: age range, gender, parental status, income range, and interests.
Ad creative ( a ): You are shown the following display ad, including text metadata and image.
Generic banner-blindness instruction (GBI): The “No-Click” Default: Acknowledge “Banner Blindness.” Most users ignore ads. Do not default to a positive action.
Relative banner-blindness cue: Younger adults tend to show much stronger banner blindness than older adults.
Query ( q ): Would this user click the ad? Return a binary click decision.
Table 2: Abbreviated prompt structure of our method.
Method
MAE ↓
MAPE (%) ↓
RMSE ↓
Non-LLM Baselines
Direct Prior 0.2%
51.1
37.1
73.0
Direct Prior 0.5%
66.4
93.5
75.5
Direct Prior 1.0%
176.2
252.0
198.0
Rule-based (avg)
54.5
81.7
63.8
Rule-based (weighted avg)
57.5
85.9
58.3
Table 3: Main results on 7-day evaluation (5 personas, 974,423 impressions, 3,578 clicks). Our method attains the lowest error among all tested configurations on this campaign. LLM methods report mean ± SD over 3 independent runs; non-LLM baselines are deterministic. Best values on aggregate metrics in bold . ↓ : lower is better.
Prompt
Actual clicks
Pred. clicks
Ratio
Demographics
3,578
947,516
264.82
Demographics + datetime
3,578
937,482
262.01
Rich persona
3,578
924,119
258.28
Rich persona + datetime
3,578
910,634
254.51
Table 4: Click inflation after removing the generic banner-blindness instruction. The model, persona, and simulation configurations are otherwise unchanged. Ratio =C^/C .
Persona configuration & signal
v1 MAE
v2 MAE
Relative qualitative banner-blindness
Base persona only
35.2±5.9
24.9±1.9
Base + datetime + personality
37.4±1.3
26.1±3.4
Base + datetime + personality + occupation
41.1±2.9
37.7±3.0
Explicit CTR prior (quantitative)
Base + datetime + personality + occupation
109.9±1.6
Table 5: Ablation over the three signal-form components on the 7-day evaluation. Results report mean ± SD over three independent runs. Persona granularity varies across rows and signal type across the two blocks, while the two columns report the v1 and v2 phrasings of the banner-blindness cue.
Explicit CTR Prior
Ours
Persona
Actual CTR
Pred.
Ratio
Pred.
Ratio
A (45–54 F Parent)
0.680
0.128
0.19
0.497
0.73
B (55–64 M Parent)
0.607
0.062
0.10
0.703
1.16
C (25–34 F Non-parent)
0.174
0.447
2.56
0.260
1.50
D (25–34 M Non-parent)
0.200
0.313
1.56
0.171
0.85
E (35–44 F Parent)
0.266
0.194
0.73
0.223
0.84
Table 6: Per-persona CTR calibration for our method and the Explicit CTR Prior configuration. Predicted and actual CTR are given as percentages, and the ratio CTR/CTR measures over- or under-estimation, with values closer to 1.0 indicating better calibration.
Method
CTR prior
MAE
MAPE (%)
RMSE
Direct prior
0.2%
51.1
37.1
73.0
Direct prior
0.5%
66.4
93.5
75.5
Direct prior
1.0%
176.2
252.0
198.0
Explicit CTR Prior
0.2%
82.3
76.0
99.4
Explicit CTR Prior
0.5%
109.9
127.7
112.0
Explicit CTR Prior
1.0%
129.2
169.3
153.4
Table 7: Sensitivity to CTR prior. Direct prior predicts clicks as impressions multiplied by a fixed CTR. Lower is better.
Online advertising studies how digital ads are targeted, ranked, priced, measured, and delivered while balancing budgets, business goals, and user interests.
On online advertising platforms, newly introduced promotional ads face the cold-start problem, as they lack sufficient user feedback for model training. In this work, we propose LLM-HYPER, a novel framework that treats large language models (LLMs) as hypernetworks to directly generate the parameters of the click-through rate (CTR) estimator in a training-free manner. LLM-HYPER uses few-shot Chain-of-Thought prompting over multimodal ad content (text and images) to infer feature-wise model weights for a linear CTR predictor. By retrieving semantically similar past campaigns via CLIP embeddings and formatting them into prompt-based demonstrations, the LLM learns to reason about customer intent, feature influence, and content relevance. To ensure numerical stability and serviceability, we introduce normalization and calibration techniques that align the generated weights with production-ready CTR distributions. Extensive offline experiments show that LLM-HYPER significantly outperforms cold-start baselines in NDCG@10 by 55.9%. Our real-world online A/B test on one of the top e-commerce platforms in the U.S. demonstrates the strong performance of LLM-HYPER, which drastically reduces the cold-start period and achieves competitive performance. LLM-HYPER has been successfully deployed in production.
Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences. We introduce LLMAdBench, a human-preference benchmark for studying advertising in LLM-generated content. The benchmark isolates a simple but practically important decision: given a user conversation, an LLM response, and a matched advertisement, where should the ad be placed? Our dataset compares pairs of responses that differ only in ad position while holding all other conditions fixed including the user query, base answer, advertisement, and disclosure condition. Human annotators evaluate each pair based on six criteria from both advertiser's and user's perspectives. The resulting benchmark contains more than 18000 human judgments across two disclosure conditions: explicitly labeling the ad as sponsored and merging it into the response without disclosure. We use LLMAdBench to evaluate eight frontier LLMs as preference judges and find that they are not reliable substitutes for human evaluation. Even the most stable models reverse roughly one quarter of their decisions when the presentation order is swapped, agreement across models is low, and their placement preferences differ systematically from those of human annotators. Moreover, LLMAdBench contains substantial learnable signal. In particular, a Qwen3-8B model fine-tuned on the human preferences improves substantially over its base model and outperforms all zero-shot frontier judges on the held-out prediction task. Beyond model evaluation, LLMAdBench provides quantitative evidence on the advertiser-user trade-off and shows that the sponsorship disclosure systematically changes users' preference over ad placement.
Rui Ai, Yuqing Liu, Sitao Qiu +14
Massachusetts Institute of Technology · University of Michigan, Ann Arbor · Shanghai Jiao Tong University +7
Large language models (LLMs) are increasingly used to simulate human populations via persona prompting, often under the assumptions that richer persona descriptions improve behavioral fidelity, similarly sized attribute combinations are equally simulatable, and persona definitions generalize across tasks. In this work, we formalize these assumptions and systematically evaluate them across multiple architectures, scales, and simulation settings. We identify a fundamental limitation we term persona manifold collapse, where increasingly expressive persona specifications lead to systematic contraction of representational and behavioral diversity. Across models, increasing persona complexity consistently reduces inter-persona separation in latent space and weakens behavioral differentiation in downstream simulation tasks. These effects persist across multiple analyses as richer personas fail to preserve human subgroup disagreement, performance varies across attribute combinations of similar size, and adding descriptive detail often degrades rather than improves simulation fidelity. Surprisingly, simple Age-Gender personas consistently outperform richly specified Ideal Customer Profiles (ICPs) across industries, achieving substantially higher downstream prediction accuracy. We find that collapse is not uniform across attributes. Certain combinations remain behaviorally stable and preserve stronger alignment with human responses, forming localized regions we term alignment bridges. Together, our results provide empirical and conceptual foundations for understanding the limits of persona-conditioned simulation, highlighting the need for representation-aware persona construction rather than increasing persona expressivity alone.
Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo