On online advertising platforms, newly introduced promotional ads face the cold-start problem, as they lack sufficient user feedback for model training. In this work, we propose LLM-HYPER, a novel framework that treats large language models (LLMs) as hypernetworks to directly generate the parameters of the click-through rate (CTR) estimator in a training-free manner. LLM-HYPER uses few-shot Chain-of-Thought prompting over multimodal ad content (text and images) to infer feature-wise model weights for a linear CTR predictor. By retrieving semantically similar past campaigns via CLIP embeddings and formatting them into prompt-based demonstrations, the LLM learns to reason about customer intent, feature influence, and content relevance. To ensure numerical stability and serviceability, we introduce normalization and calibration techniques that align the generated weights with production-ready CTR distributions. Extensive offline experiments show that LLM-HYPER significantly outperforms cold-start baselines in NDCG@10 by 55.9%. Our real-world online A/B test on one of the top e-commerce platforms in the U.S. demonstrates the strong performance of LLM-HYPER, which drastically reduces the cold-start period and achieves competitive performance. LLM-HYPER has been successfully deployed in production.
Figures & tables
Figure 1: LLM-HYPER: (a) framework and (b)-(d) prompt details. Weights are normalized for CTR prediction.
Figure 2: LLM-HYPER deployment
Model
AUC
NDCG5
NDCG10
L(ms)
Emb T5
0.4105
0.0174
0.0262
0.357
LLM-R
0.0571
0.00931
0.0221
2.13\mboxE3
LLM-TR
0.0576
0.00886
0.0508
3.78\mboxE3
LH 4o
0.6539
0.0490
0.0764
0.163
LH 5.1
0.6497
0.0427
0.0668
0.158
LH 2.5F
0.6654
0.0435
0.0779
0.160
Table 1: Offline Cold-start Ranking Comparison.
Models
NDCG5
NDCG10
LH 2.5P , zero-shot
0.0376
0.0677
LH 2.5P , zero-shot, −img
0.0165
0.0352
LH 2.5P , 3-shot
0.0421
0.0798
LH 2.5P , 3-shot, −img
0.0267
0.0511
LH 2.5P , 5-shot
0.0456
0.0792
LH 2.5P , 5-shot, −img
0.0292
0.0506
Table 2: CoT and Visual Feature Impact
Figure 3: Explainability Results of Cold Start Model
Figure 4: Counterfactual Robustness of Cold Start Model with accuracy of weight change
Models
Relative CTR score
LLM-HYPER
92%
LR warm
100%
Table 3: Online 30-Day A/B Test Results.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Models
Generation Time(s)
GPT-4o
47.7
GPT-5.1
58.7
Gemini-2.5-Flash
83.3
Gemini-2.5-Pro
102.9
Appendix
Table 4: Offline LLM weight generation time (averaged by each ad)
Ad
Ad Content
Weight
Key Signals
LLM Rationale
Ad 1
Age-appropriate building toys and construction sets
0.95
• Strong: Toy and game category engagement • Strong: Building and construction toy affinity
The dominant feature activation centers on toy and building set interactions. The Ad’s product category and thematic content closely align with this feature’s predictive pattern, indicating strong relevance for engagement prediction.
Ad 2
High-value outdoor furniture with minor recreational overlap
The dominant feature signal centers on indoor play items. The secondary outdoor activity signal is insufficient to predict interest in premium outdoor living categories.
Ad 3
Children’s footwear (Crocs) with shared toy purchaser demographic
The feature captures toy purchasing behavior, which correlates with parents who also purchase children’s apparel and footwear. The cross-category purchasing pattern creates a neutral signal for children’s fashion items despite product category mismatch.
Appendix
Table 5: Feature weight predictions for feature_toys_games feature across different ad contexts.
Original Ad
Target Features
Mod Type
Modified Ad
Sorig
Scf
Δ Score
Patio & Garden
outdoor activity & sports
Enhanced
Game Day Upgrades: Patio & Grill
0.192
0.735
+0.543
Vitamins & Supps
Healthy Living
Neutralized
Daily Essentials
1.055
0.975
−0.080
Fall Home Trends
Home Decor
Diminished
Fall Gutter & Roof Maintenance
3.750
1.150
−2.600
Appendix
Table 6: Examples of counterfactual semantic modifications and the resulting weight changes generated by LLM-HYPER ( LH 2.5P ).
Recommendation systems power engagement and monetization across feeds, ads, and short-video platforms, but translating the latest advances in Large Language Models into Recommendation Systems (RecSys) gains remains rare, particularly in advertising and production-scale real-world industry setups. Prior real-world LLM successes typically fall into three buckets: (a) generative retrieval that directly predicts the next items for candidate generation, (b) late-stage re-ranking that uses LLMs, and (c) auxiliary signal enrichment with LLMs. We introduce a complementary paradigm for ads: a fine-tuned open-source LLM used not as a ranker, but as an ads-specific ancillary predictor, forecasting likely advertisers from user profiles and histories. This LLM-driven advertiser prediction augments conventional candidate generation and provides informative priors to downstream ranking. Developed in a large-scale production advertising system, our approach produces substantial offline improvements and measurable online business impact, demonstrating that LLM world knowledge and predictive capacity can be efficiently harnessed. Beyond validating LLMs for ads applications, our results show that targeted ancillary predictions can unlock end-to-end gains across both retrieval and late-stage ranking, offering a practical path to LLM-enhanced recommendation at scale.
Reconciling platform revenue with user experience in LLM advertising motivates a data-centric foundation. We introduce NaiAD, the first comprehensive dataset for LLM-native advertising comprising 58,999 carefully constructed ad-embedded responses paired with user queries. NaiAD is organized around theoretically grounded evaluation metrics that separately and comprehensively capture user and commercial utility. To mitigate the dimensional collinearity of aligned LLMs, we propose a decoupled generation pipeline that produces structurally diverse samples, ranging from responses that explicitly disentangle stakeholder utilities to responses that are uniformly strong or weak across dimensions. We further provide score labels calibrated by a Variance-Calibrated Prediction-Powered Inference (VC-PPI) framework, aligning automated scoring with human annotations. Mechanistic analyses reveal that successful ad integration relies on reasoning paths that cluster into four distinct semantic strategies. Models leveraging NaiAD internalize these strategies to simultaneously improve user and commercial utility, while enabling independent control over these distinct objectives via in-context learning. Together, these results position NaiAD as a foundational infrastructure for developing future LLM-native ad systems.
Yihang Zhang, Zimeng Huang, Ren Zhai +2
Tsinghua University Beijing, China · College of AI Tsinghua University Beijing, China · Department of Literature, Arts and Communication Anhui International Studies University Anhui, China +1
Traditional ads recommendation systems have primarily focused on optimizing for prediction accuracy of click or conversion events using canonical metrics such as recall or normalized discounted cumulative gain (NDCG). With the hyper-growth of ads inventory and liquidity with generative AI technologies, the prediction stability and predictability is becoming increasingly critical. Intuitively, prediction stability and predictability can be defined to quantify system robustness with respect to minor/noisy input (ads, creatives) perturbations, the lack of which could lead to advertiser perceivable problems such as repeatability, cold start and under-exploration. In this paper, we introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system, and present an online validated semantic candidate generation framework powered by fine-tuned Large Language Models (LLMs) that showed significant improvement along these metrics by fundamentally improving the semantic-awareness of the system. The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the foundation for graph-based expansion, ensuring the retrieved candidates encapsulate semantic variants of an ad, guaranteeing that small creative variants from the advertiser yield consistent and explainable delivery results to the user. We tested this LLM ads retrieval framework in a large-scale industrial ads recommendation system, demonstrating significant improvements across offline and online A/B experiments, showcasing gains in both predictability and traditional performance metrics. Although evaluated in the ads stack, this is a general framework that can be applied broadly to any large-scale recommendation and retrieval systems facing similar scaling and predictability challenges.