PALATE: Pretrained Autoregressive LATent Embeddings

Sizhang Zhao; Praveen Gondhi; Bin Li; Utsaw Kumar

cs.IROct 29, 2026

PALATE: Pretrained Autoregressive LATent Embeddings

Authors: Sizhang Zhao, Praveen Gondhi, Bin Li, Utsaw Kumar

Organizations: DoorDash Seattle, WA, USA · DoorDash San Francisco, CA, USA

Abstract

Local-commerce and food-delivery marketplaces run large ads businesses: across hundreds of thousands of stores, merchants pay to be surfaced on browse and search surfaces, and a ranker decides which ads each consumer sees. Ranking well demands strong per-consumer personalization, yet a consumer's raw interaction history is sparse and high-cardinality. Our production ranker had personalized only through hand-engineered signals: aggregate engagement counts, categorical embeddings, and order-agnostic pooled history, with no learned per-consumer sequence representation. We learn one by training a GPT-style generative model of the consumer's engagement sequence to predict their next few engagements, and read its last-token hidden state as the consumer embedding. That single frozen vector becomes one shared personalization input for every ad surface: a daily job refreshes it, and each ranker consumes it as an ordinary dense feature. Offline it is the largest single-feature gain in our ranking-AUC ladder, a relative Eval-AUC gain of +0.59% on a browse-driven feed and +0.57% on a query-driven search surface. Online, it shipped as one component of bundled ranker retrains on both surfaces: Discovery (revenue +0.79%, conversion rate +1.17%) and Search (revenue +0.94%, conversion rate +1.50%). Three design choices adapt the generative objective to a repurchase-heavy marketplace, and are the core of our contribution: an exploration-weighted loss that up-weights businesses new to the consumer over ones already engaged; a two-stage curriculum: a click pretrain that keeps the representation broad and exploratory, then a conversion fine-tune that sharpens purchase intent; and a designed negative set that pairs a fixed popular-business pool and businesses shown but not ordered with random and in-batch negatives.

Figures & tables

Panel of Figure 1 — Figure 1. The generative user representation and how it is reused: a decoder-only GPT trained on each consumer’s store-engagement sequence, whose last-token hidden state is exported as a frozen 256-d feature and reused by the DCNv2 ranker across both ad surfaces. A diagram showing a consumer store-engagement sequence feeding into a decoder-only GPT model, whose last-token hidden state is exported as a 256-dimensional embedding and fed frozen into a DCNv2 ranker that serves both the Discovery and Search ad surfaces.

Panel of Figure 2 — Figure 2. Model architecture. Per-position store, business, and context embeddings are concatenated (532-d), projected to a 768-d trunk, run through three pre-norm causal decoder blocks, and projected to 256-d. We export the last-token hidden state as the user embedding and the softmax layer’s rows as business embeddings (§ 3.4 ). A diagram of the model architecture showing store, business, and context input embeddings concatenated and projected to a 768-dimensional trunk, passed through three pre-norm causal decoder blocks, then normalized and projected to 256 dimensions to produce per-position hidden states, from which the last-token user embedding and the softmax-layer business embeddings are both extracted.

Design choice	R@10	R@30	MRR	New R@10
$+$ Hard negatives	+14.3%	+9.0%	+8.2%	–
Exploration weighting 3:1	$-$ 7.0%	$-$ 3.5%	$-$ 5.7%	+18.4%
Exploration weighting 9:1	$-$ 20.4%	$-$ 11.1%	$-$ 19.8%	+29.1%
Distinct sequence	$-$ 67.5%	$-$ 46.5%	$-$ 75.5%	–
Time-decay attn (replace pos.)	$-$ 5.0%	$-$ 3.7%	$-$ 5.4%	–
Time-decay attn ( $+$ pos.)	$-$ 3.7%	$-$ 3.1%	$-$ 3.4%	–

Table 1. Recall ablations (% change). New R@10: recall on previously unseen businesses (“–”: not recorded). Time-decay rows use the full-softmax model; full-softmax rows are vs. sampled softmax.

Component added	% impr.
Store emb. $+$ mean-pooled last- $N$ converted-store emb.	+0.33%
$+$ user embedding, uniform weighting	$\approx$ 0.00%
$+$ user embedding, 9:1 weighting, as $\phi(u,c)$ (Eq. 2 )	+0.59%
$+$ DIN attention over the raw history	+0.03%

Table 2. Discovery ranking-AUC ladder: each row’s incremental relative Eval-AUC change over the rows above; row 1 is vs. the retrained production DCNv2. The uniform and 9:1 rows are alternatives, each measured on its own rather than stacked. Search: +0.57% for the user embedding. Single runs.

Metric	Discovery	Search
Revenue	+0.79% $\ast\ast\ast$	+0.94% $\ast\ast\ast$
CVR	+1.17% $\ast\ast\ast$	+1.50% $\ast\ast\ast$
CTR	$-$ 0.10% (n.s.)	+1.38% $\ast\ast\ast$
eCPM	+0.66% $\ast\ast\ast$	+1.56% $\ast\ast\ast$
ROAS	+0.13% (n.s.)	$-$ 0.01% (n.s.)

Table 3. Online A/B lift by surface (% change).

\ast\ast\ast

p{<}0.001

\ast\ast

p{<}0.01

\ast

p{<}0.05

; n.s. = not significant.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 521 papers

Fine-Tuned LLM as a Complementary Predictor Improving Ads System
May 27, 2026Hui Yang, Daiwei He, Kevin Jiang +20LLM Fine-Tuning Online Advertising
Pinterest, Inc., USA
A Cascaded Generative Approach for e-Commerce Recommendations
May 11, 2026Moein Hasani, Hamidreza Shahidi, Trace Levinson +4Recommender Systems Large Language Model-Based Recommendation
Ambience Healthcare San Francisco, USA
Token-Level Advertising
Aug 27, 2026Hanbing Liu, Bowei Zhang, Changyuan Yu +2Online Advertising Mechanism Design
Renmin University of China Beijing, China · Beijing, China · Stanford University Stanford, CA, USA

May 27, 2026J/K move · Enter open · S save

Fine-Tuned LLM as a Complementary Predictor Improving Ads System

Hui Yang, Daiwei He, Kevin Jiang, Taejin Park+19

Pinterest, Inc., USA

LLM Fine-Tuning Online Advertising Recommender Systems Large Language Model-Based Recommendation

Open paper AskPDF arXiv

Recommendation systems power engagement and monetization across feeds, ads, and short-video platforms, but translating the latest advances in Large Language Models into Recommendation Systems (RecSys) gains remains rare, particularly in advertising and production-scale real-world industry setups. Prior real-world LLM successes typically fall into three buckets: (a) generative retrieval that directly predicts the next items for candidate generation, (b) late-stage re-ranking that uses LLMs, and (c) auxiliary signal enrichment with LLMs. We introduce a complementary paradigm for ads: a fine-tuned open-source LLM used not as a ranker, but as an ads-specific ancillary predictor, forecasting likely advertisers from user profiles and histories. This LLM-driven advertiser prediction augments conventional candidate generation and provides informative priors to downstream ranking. Developed in a large-scale production advertising system, our approach produces substantial offline improvements and measurable online business impact, demonstrating that LLM world knowledge and predictive capacity can be efficiently harnessed. Beyond validating LLMs for ads applications, our results show that targeted ancillary predictions can unlock end-to-end gains across both retrieval and late-stage ranking, offering a practical path to LLM-enhanced recommendation at scale.

Abstract

Recommender Systems

Explore similar work

Fine-Tuned LLM as a Complementary Predictor Improving Ads System

A Cascaded Generative Approach for e-Commerce Recommendations

Token-Level Advertising