Recommendation systems may use predicted click-through conversion rate (pCTCVR), the product of predicted click-through rate (pCTR) and post-click conversion rate (pCVR), to guide ranking decisions. Many pCVR models build on the Entire Space Multi-Task Model (ESMM), which jointly trains pCVR and auxiliary pCTR towers using pCTR and pCTCVR losses, but deploys only the pCVR tower. At serving time, this tower may be paired with a separately trained pCTR model, creating a training-serving mismatch in pCTCVR when its predictions differ from those of the auxiliary pCTR tower. In this paper, we use knowledge distillation (KD) from logged serving pCTR scores to supervise ESMM's auxiliary pCTR tower and improve training-serving pCTCVR alignment. We denote variants by KD-α, where α controls the hard-label loss weight. On the Ali-CCP dataset, KD-0.3, the best variant on serving pCTCVR, increases ROC-AUC not only for serving pCTCVR but also for pCVR by 2.1% and 1.6%, respectively, and PR-AUC by 9.9% and 8.8%, respectively, relative to the ESMM baseline. We also test negative downsampling with 5% nonclick retention. Downsampled KD-0.3 increases mean AUC for serving pCTCVR and pCVR by 1.2% and 0.9%, respectively, -and PR-AUC by 8.5% and 4.5%, respectively, relative to downsampled ESMM.
Figures & tables
Figure 1. Full-population KD. ESMM forms auxiliary pCTCVR qivi and uses the logged serving pCTR score pi to supervise qi ; serving combines pi with vi to form pivi . Two vertically arranged training and serving paths. In the upper training path, an ESMM box contains an auxiliary pCTR tower producing qi and a pCVR tower producing vi; their scores enter a multiplication node that produces auxiliary pCTCVR qivi. In the lower serving path, the logged serving pCTR score pi is multiplied by the trained pCVR score vi to produce serving pCTCVR pivi. A dashed arrow from pi to the auxiliary pCTR tower indicates that pi supervises qi through knowledge distillation.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
DCN-V2
pi
0.6310
0.06461
0.9815
ESMM
qi
0.6259
0.06328
0.9822
ESMM+DCN-V2
qi
0.6299
0.06486
0.9804
KD–0.3
qi
0.6301
0.06432
0.9786
Table 1. Full-population Ali-CCP pCTR test means. qi averaged over ten student seeds, and pi is the single seed.
Figure 2. Full-population score alignment for (a) logged serving versus auxiliary pCTR ( p vs. q ) and (b) serving versus auxiliary pCTCVR ( pv vs. qv ). Lower KL and higher Pearson r indicate closer alignment; bars show means ± standard error of the mean (SEM) over ten seeds. Four bar charts compare ESMM, \esmmdcn{}, and KD–0.3. In panel (a), KL(p∥q) decreases and r(p,q) increases from ESMM to \esmmdcn{} to KD–0.3. In panel (b), KL(pv∥qv) likewise decreases and r(pv,qv) increases in the same order. KD–0.3 therefore has the lowest divergence and highest correlation in both comparisons. Error bars show one SEM, and the vertical axes are truncated.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
ESMM
vi
0.6949
0.01310
0.9616
pivi
0.6599
0.0004588
0.9867
ESMM+DCN-V2
vi
0.7027
0.01387
0.9591
pivi
0.6679
0.0004746
0.9854
KD–0.7
vi
0.6956
0.01311
0.9625
pivi
0.6635
0.0004668
0.9873
Table 2. Full-population Ali-CCP test results averaged over ten student seeds. Here, vi denotes pCVR, pi the single-seed logged serving pCTR score, and pivi serving pCTCVR. pCVR is evaluated on clicked impressions, while serving pCTCVR is evaluated on all impressions.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
ESMM R
vi
0.6888
0.01248
0.9635
pivi
0.6540
0.0004514
0.9886
KD R –0.3
vi
0.6949
0.01305
0.9607
pivi
0.6617
0.0004900
0.9855
Table 3. Ali-CCP test means over ten student seeds after training with 5% nonclick retention. Here, vi is the pCVR score, pi is the single seed logged serving pCTR score, and pivi is the serving pCTCVR score. pCVR is evaluated on clicked impressions; serving pCTCVR uses all test impressions.
Offsite conversion rate (OCVR) prediction is an important ranking problem in computational recommendation systems. This task presents a modeling challenge: click signals are abundant and exhibit short temporal horizons, whereas conversion signals are inherently sparse, long-delayed, and frequently unattributed. Despite these statistical disparities, both signal types must inform models that operate within strict serving-latency constraints. Prior pre-training approaches address this heterogeneity with a single, undifferentiated encoder applied uniformly across both data streams. We propose DUET (Dual User Embedding Transformers), a framework that explicitly partitions user behavioral data into two domain-coherent streams -- clicks and conversions -- and pre-trains dedicated transformer encoders with architectures tailored to each stream's statistical characteristics: multi-layer self-attention for the dense click stream and interleaved cross- and self-attention for the sparse conversion stream. The resulting complementary embeddings are jointly consumed by a downstream ranker without exceeding serving-latency budgets. Evaluation demonstrates up to 0.38% normalized entropy (NE) reduction relative to the strongest baseline, and A/B test shows consistent improvements in OCVR prediction accuracy.
Reazul Hasan Russel, Mingwei Tang, Rostam Shirani +18
Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a single network at inference, and the gains of the penalty and of distillation, where measurable, largely add up.
Yifan Wang, Yifei Du
National University of Singapore, Singapore · Department of Mathematics, National University of Singapore, Singapore, Singapore · University of Science and Technology of China, China +1
Click-Through Rate prediction, a core task in recommendation and advertising systems, relies on modeling interactions among sparse categorical features. Explicit cross networks are a central paradigm for CTR prediction, and recent progress has largely come from increasing the interaction capacity of a single predictor through deeper cross networks and more expressive cross operators. We revisit whether continually increasing interaction capacity remains the most effective way to improve predictive performance, and find that its benefits quickly exhibit diminishing returns even as capacity continues to grow. This motivates a complementary scaling direction that we call estimator scaling, where additional resources are used to incorporate multiple related estimators rather than only enlarging a single predictor. Through theoretical analysis, we show that the gains from estimator scaling are governed by the amount of non-shared predictive variation available across estimators. However, exploiting this variation naively can be expensive: independently trained models provide substantial estimator diversity but require deployment cost to grow with ensemble size. This motivates a parameter-efficient realization of estimator scaling that can incorporate diversity from multiple estimator sources without maintaining multiple full models. Building on this view, we introduce RECursive Averaged Predictor (RECAP), a parameter-efficient recursive CTR model that operationalizes estimator scaling at three levels: distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. Experiments across multiple benchmarks establish new state-of-the-art predictive performance on standard benchmarks, while placing the RECAP on a favorable performance-parameter Pareto frontier.