Recommendation systems may use predicted click-through conversion rate (pCTCVR), the product of predicted click-through rate (pCTR) and post-click conversion rate (pCVR), to guide ranking decisions. Many pCVR models build on the Entire Space Multi-Task Model (ESMM), which jointly trains pCVR and auxiliary pCTR towers using pCTR and pCTCVR losses, but deploys only the pCVR tower. At serving time, this tower may be paired with a separately trained pCTR model, creating a training-serving mismatch in pCTCVR when its predictions differ from those of the auxiliary pCTR tower. In this paper, we use knowledge distillation (KD) from logged serving pCTR scores to supervise ESMM's auxiliary pCTR tower and improve training-serving pCTCVR alignment. We denote variants by KD-α, where α controls the hard-label loss weight. On the Ali-CCP dataset, KD-0.3, the best variant on serving pCTCVR, increases ROC-AUC not only for serving pCTCVR but also for pCVR by 2.1% and 1.6%, respectively, and PR-AUC by 9.9% and 8.8%, respectively, relative to the ESMM baseline. We also test negative downsampling with 5% nonclick retention. Downsampled KD-0.3 increases mean AUC for serving pCTCVR and pCVR by 1.2% and 0.9%, respectively, -and PR-AUC by 8.5% and 4.5%, respectively, relative to downsampled ESMM.
Figures & tables
Figure 1. Full-population KD. ESMM forms auxiliary pCTCVR qivi and uses the logged serving pCTR score pi to supervise qi ; serving combines pi with vi to form pivi . Two vertically arranged training and serving paths. In the upper training path, an ESMM box contains an auxiliary pCTR tower producing qi and a pCVR tower producing vi; their scores enter a multiplication node that produces auxiliary pCTCVR qivi. In the lower serving path, the logged serving pCTR score pi is multiplied by the trained pCVR score vi to produce serving pCTCVR pivi. A dashed arrow from pi to the auxiliary pCTR tower indicates that pi supervises qi through knowledge distillation.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
DCN-V2
pi
0.6310
0.06461
0.9815
ESMM
qi
0.6259
0.06328
0.9822
ESMM+DCN-V2
qi
0.6299
0.06486
0.9804
KD–0.3
qi
0.6301
0.06432
0.9786
Table 1. Full-population Ali-CCP pCTR test means. qi averaged over ten student seeds, and pi is the single seed.
Figure 2. Full-population score alignment for (a) logged serving versus auxiliary pCTR ( p vs. q ) and (b) serving versus auxiliary pCTCVR ( pv vs. qv ). Lower KL and higher Pearson r indicate closer alignment; bars show means ± standard error of the mean (SEM) over ten seeds. Four bar charts compare ESMM, \esmmdcn{}, and KD–0.3. In panel (a), KL(p∥q) decreases and r(p,q) increases from ESMM to \esmmdcn{} to KD–0.3. In panel (b), KL(pv∥qv) likewise decreases and r(pv,qv) increases in the same order. KD–0.3 therefore has the lowest divergence and highest correlation in both comparisons. Error bars show one SEM, and the vertical axes are truncated.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
ESMM
vi
0.6949
0.01310
0.9616
pivi
0.6599
0.0004588
0.9867
ESMM+DCN-V2
vi
0.7027
0.01387
0.9591
pivi
0.6679
0.0004746
0.9854
KD–0.7
vi
0.6956
0.01311
0.9625
pivi
0.6635
0.0004668
0.9873
Table 2. Full-population Ali-CCP test results averaged over ten student seeds. Here, vi denotes pCVR, pi the single-seed logged serving pCTR score, and pivi serving pCTCVR. pCVR is evaluated on clicked impressions, while serving pCTCVR is evaluated on all impressions.
Model
Output
AUC ↑
PR-AUC ↑
NE ↓
ESMM R
vi
0.6888
0.01248
0.9635
pivi
0.6540
0.0004514
0.9886
KD R –0.3
vi
0.6949
0.01305
0.9607
pivi
0.6617
0.0004900
0.9855
Table 3. Ali-CCP test means over ten student seeds after training with 5% nonclick retention. Here, vi is the pCVR score, pi is the single seed logged serving pCTR score, and pivi is the serving pCTCVR score. pCVR is evaluated on clicked impressions; serving pCTCVR uses all test impressions.
National University of Singapore, Singapore · Department of Mathematics, National University of Singapore, Singapore, Singapore · University of Science and Technology of China, China +1