FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

Yifan Wang; Yifei Du

cs.IROct 19, 2026

FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

Authors: Yifan Wang, Yifei Du

Organizations: National University of Singapore, Singapore · Department of Mathematics, National University of Singapore, Singapore, Singapore · University of Science and Technology of China, China · Hefei National Research Center for Physical Sciences at the Microscale and School of Physical Sciences; Shanghai Research Center for Quantum Science and CAS Center for Excellence in Quantum Information and Quantum Physics; Hefei National Laboratory, University of Science and Technology of China, Hefei and Shanghai, China

Abstract

Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a single network at inference, and the gains of the penalty and of distillation, where measurable, largely add up.

Figures & tables

Figure 1. Two kinds of ensemble members and their paths into a single deployed tower. Left: one set of weights under different masks; the members are close in prediction space, and their mean and variance enter training in closed form. Right: independently trained solutions; the members are far apart, and their pooled prediction serves as a teacher for a single student tower. In both cases only one tower is served. Conceptual comparison of shared-weight masking members and independently trained members, both feeding a single served tower.

Figure 2. Accuracy of three proxies for the gap

A

of ( 9 ), row by row (Criteo validation split). FLMNet towers trained with

\kappa=0

and

\kappa=2

, three seeds each, on the first

256

validation rows of each seed, pooled. Each proxy is evaluated at the sample mean and moments of

S=2{,}048

sampled masks and divided by the Monte Carlo gap

A_{\rm MC}

from the same masks, so only the treatment of the mask distribution is tested; the closed-form

V

is tested in Table 1 , and the full closed-form penalty in Figure 6 . Second- and fourth-order: ( 11 ) truncated after the

V

term and after the

m_{4}

term, with sampled

m_{3}

and

m_{4}

. Gaussian + GH3: the penalty term of ( 12 ). Thick bars span the 5th to 95th percentiles over rows, thin lines the 1st to 99th, open circles mark the rows beyond, white dots the median, and the dashed line a ratio of

1

; the arrow counts three rows below

0.25

. Training with the second-order truncation in place of GH3 (Criteo,

\kappa=2

, three seeds) changes validation AUC by

0.005\times 10^{-4}

on average, below the seed threshold of

0.94\times 10^{-4}

(Section 5.1 ). Horizontal interval plot with three rows of proxies (second-order truncation, fourth-order truncation, Gaussian with three-point Gauss–Hermite quadrature) and two sub-rows each for kappa equal to 0 and 2, on a logarithmic axis of the per-row ratio of proxy to Monte Carlo gap from 0.25 to 1.15. All medians lie within 1.5 percent of 1. The second-order truncation has a long left tail, with three rows below 0.25 at kappa 0 and a minimum of 0.34 at kappa 2. The fourth-order truncation has a shorter tail, down to 0.31 at kappa 0 and 0.69 at kappa 2. The Gaussian proxy stays between 0.72 and 1.12 at kappa 0 and between 0.89 and 1.04 at kappa 2.

Figure 3. From the published objective to the one we train. Solid arrows are identities. The single dashed arrow is the only approximation in this chain: the mask distribution of

\delta

is replaced by a Gaussian of the same variance, and the resulting integral is evaluated by quadrature. The dotted arrow is not a derivation step: the trained objective needs

\mu

and

V

, which Section 4 computes in one deterministic pass,

\mu

exactly and

V

up to a truncation checked in Table 1 . Seven boxes: a chain of six in two rows, read left to right, and a seventh below the last, with a legend for three arrow types. The published self-ensemble objective at two masks leads to a normalized objective with an explicit distillation weight, then to its infinite-masking limit; a return line labelled with the ambiguity lemma leads to the label loss plus kappa times the ambiguity gap, then to an exact integral form of that gap. A dashed arrow marks the only approximation, which replaces the mask distribution by a Gaussian and integrates by quadrature, giving the objective that is trained. A dotted arrow, marked as a dependency rather than a derivation step, points from the trained objective to a box stating that the mean logit and its variance come from a single deterministic forward pass.

Figure 4. Where the masks act in layer

l

of the tower, drawn for

C=4

fields with

d=3

and

M=1

. Each strip is a vector with one cell per coordinate or gate unit, colored by field; the gate output

1+g_{l}+\eta_{l}

is gray, and all shades are illustrative. Solid arrows carry mask-free quantities: the clean embedding

x_{0}

, which the gate of every layer reads, and the gate units

r_{l}

, which by (C1) are the same for every mask. Each block of

\mathbf{G}_{l}

reads a single field. The two rows of dots are the mask factors,

u_{l}

on the gate units and

v_{l}

on the layer output, and the dropped coordinates are orange; dashed arrows carry quantities that depend on the mask. The mixer

U_{l}

connects every gate unit to all

h

tower coordinates, so dropping one unit (orange curves) perturbs

\eta_{l}

in every coordinate at once. Its connections are drawn as curves rather than arrows; they carry

u_{l}\odot r_{l}

and therefore depend on the mask. The carrier

x^{m}_{l-1}

enters along the right edge, and

\odot

is its coordinatewise product with the gate output. Gray dotted arrows mark the sources of the variance:

u_{l}

through

U_{l}

contributes

\omega K_{l}

and

v_{l}

the factor

1+\omega\,\delta_{jj^{\prime}}

Q_{l}

of ( 20 ). Setting every mask to its mean recovers the clean forward pass, which is the mean logit. A diagram of one tower layer, drawn from bottom to top. The clean embedding x zero, four fields of three coordinates each, feeds four field-local gate blocks G l, one per field, which give the gate units r l. A row of mask dots u l drops some units, and the mixer U l connects every remaining unit to all twelve tower coordinates with curves colored by field; the curves from one dropped unit are highlighted. The result, one plus g l plus eta l, multiplies the carrier x m l minus one, which arrives along the right edge from layers 1 to l minus 1; a second row of mask dots, v l, then drops coordinates, giving x m l, which continues through layers l plus 1 to L to the readout z m. A line on the left edge carries x zero to the gates of the later layers. Dotted arrows from the mixer and from v l lead to a box stating that the variance collects omega K l from u l and one plus omega delta from v l, one factor per layer; another box states that with all masks at one the chain is the clean pass, so the mean equals z.

	Small towers, exhaustive	Trained checkpoints
$\widehat{V}/V$	$0.99982$ – $1.00049$	$0.99965$ $[0.99808,1.00014]$
First order in $u_{l}$ , $V_{\mathrm{lin}}/V$	$0.86625$ – $0.99910$	$0.91955$ $[0.85144,0.95966]$

Table 1. Accuracy of the two truncations against the exact variance ( 21 ). Left: all masks enumerated, with

W^{\mathrm{fg}},\beta^{\mathrm{fg}}

randomly perturbed. Right: six trained checkpoints, one per seed, on the same

4096

validation rows each, in float64; medians over the pooled

24{,}576

rows, with the

1

st and

99

th percentiles in brackets.

Dataset	#Instances	#Fields	#Training	#Validation	#Test
Criteo	46M	39	33M	8.3M	4.6M
Avazu	40M	22	28M	4.0M	8.1M
KKBox	7.4M	13	5.9M	0.74M	0.74M
Frappe	288,609	10	202,027	57,722	28,860
ML-1M	739,012	7	591,209	73,901	73,902

Table 2. Dataset statistics.

Year	Models	Criteo		Avazu
		Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$
2007	LR	0.4609	0.7886	0.3834	0.7463
2010	FM	0.4490	0.8022	0.3788	0.7537
2016	DNN	0.4382	0.8136	0.3774	0.7578
2016	IPNN	0.4384	0.8135	0.3775	0.7569
2017	DeepFM	0.4381	0.8138	0.3775	0.7572

Table 3. Performance comparison on the two advertising benchmarks, Criteo and Avazu. In each column the three best values are shaded, darker for better: best , second , third . Entries are test means over six seeds for EDCN, FinalMLP, FinalNet, QNN-

\alpha

, ECN and our two towers, and over three seeds for the other baselines; the one exception is FLMNet-KD on Avazu, which has three. All models are trained by us at embedding dimension

d{=}10

, the value BARS tunes for Criteo, and baselines whose original papers used a larger dimension are re-run at

d{=}10

. Criteo follows the BARS criteo_x1 protocol. On Avazu, models are trained on the BARS avazu_x1 training split, and its validation and test splits are pooled and re-split at random into two parts of the original sizes (Appendix B.2 ). On Criteo, our reproductions of FinalNet and FinalMLP fall

3.2\times 10^{-4}

and

4.5\times 10^{-4}

in AUC below the values published on BARS, which reports a single tuned run for each of these two models, whereas every entry here is a seed average; Appendix B.5 compares our other reproductions with published values.

Year	Models	KKBox		Frappe		ML-1M
		Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$
2007	LR	0.5745	0.7678	0.3076	0.9356	0.3509	0.8700
2010	FM	0.5065	0.8299	0.1999	0.9691	0.3261	0.8900
2016	DNN	0.4813	0.8502	0.1531	0.9846	0.3149	0.9009
2016	IPNN	0.4796	0.8506	0.1587	0.9835	0.3113	0.9047
2017	DeepFM	0.4783	0.8532	0.1536	0.9843	0.3120	0.9029

Table 4. Performance comparison on the three recommendation benchmarks, KKBox, Frappe, and ML-1M. Shading is as in Table 3 . Entries are test means over six seeds for EDCN, FinalMLP, FinalNet, QNN-

\alpha

, ECN and our two towers, and over three seeds for the other baselines, with these exceptions: FLMNet-CF has eight seeds on ML-1M, and FLMNet-KD has three seeds on every benchmark. All models are trained by us at embedding dimension

d{=}128

on KKBox (

d{=}16

for EulerNet), 10 on Frappe and 16 on ML-1M; KKBox and Frappe follow the BARS kkbox_x1 and frappe_x1 protocols. BARS provides no configuration for ML-1M, so the baselines take the architecture hyper-parameters of their BARS movielenslatest_x1 configurations (ECN and EulerNet those of our KKBox runs). On Frappe, FLMNet-CF and FLMNet-KD agree to four decimals in AUC (0.98594 and 0.98590), and the shading follows the unrounded means. QNN-

\alpha

with four heads, the configuration the QNN authors released for ML-1M, is reported in Appendix B.4 .

Figure 5. Relative improvement over DNN on the five benchmarks, measured by RelaImpr ( 26 ). (a) RelaImpr of each model’s seed-mean test AUC on each benchmark; blue marks a gain over DNN, orange a loss. (b) Mean RelaImpr over the five benchmarks with 95% confidence intervals. The intervals capture seed-to-seed variation only, treating the benchmarks and the DNN reference as fixed, and their degrees of freedom follow the Welch–Satterthwaite approximation; the interval of FLMNet-CF is narrower than its marker. † FLMNet-KD has no interval because its runs are not independent: on every benchmark its students distil from teachers built on overlapping sets of FLMNet-CF towers. LR and FM are omitted here and appear in Tables 3 and 4 , which also give the seed counts. Panel (a) is a heatmap with one row per model and one column per benchmark, each cell giving the model's relative AUC improvement over DNN in percent, shaded blue when positive and orange when negative. Panel (b) shares the rows and plots each model's mean improvement over the benchmarks as a point with a 95% confidence interval (FLMNet-KD without one), with a dashed line at zero for DNN. Our two models occupy the top two rows.

Figure 6. Closed-form penalty against the Monte Carlo gap

A_{\rm MC}

, row by row, for the three towers of Table 6 on Criteo (validation split). On each of the first

256

validation rows, the closed-form penalty is divided by

A_{\rm MC}

, the gap

A

of ( 9 ) estimated from

S=2{,}048

sampled masks, as in Table 5 ; each histogram shows the distribution of this ratio on a logarithmic axis, and the dashed line marks a ratio of

1

. (a) Towers at initialization. (b, c) Towers trained with

\kappa=0

and

\kappa=2

, with the rows of seeds 110000–110002 pooled. Bins are

0.04

decades wide, and each histogram is scaled to its own peak. Median ratio: median over rows, then mean over seeds. Four rows outside the axis range are counted in the edge bins; at initialization, two rows per tower have a non-positive closed-form penalty and are left out of the histogram but not of the median. Three histogram panels, for untrained towers and for towers trained at kappa equal to zero and two, each showing over rows the ratio of the closed-form penalty to the Monte Carlo gap on a logarithmic axis for three towers, with a dashed line at ratio one. For the tower whose gate reads the previous layer, the ratio is already below one at initialization and falls further with training, to a median of 0.039 at kappa equal to two; the other two towers stay near one.

Gate reads	Penalty ratio		$A_{\rm MC}$		AUC $\uparrow$		Gain ( $\times 10^{-4}$ )
Gate reads	$\kappa=0$	$\kappa=2$	$\kappa=0$	$\kappa=2$	$\kappa=0$	$\kappa=2$	AUC $\uparrow$	Logloss $\downarrow$
previous layer	0.609	0.039	22.9	20.7	0.81478	0.81504	$+2.56$	$-4.23$
input embedding	1.000	0.998	21.6	2.27	0.81373	0.81453	$+7.92$	$-9.19$

Table 5. Effect of the gate input on the QNN-

\alpha

backbone on Criteo (validation split, mean over seeds 110000–110002). Both rows share one backbone (

L=4

M=5

, 3.05M tower parameters) and one recipe at

\rho=0.1

, and differ only in what the gate reads: the output of the previous layer, as in QNN-

\alpha

, or the input embedding. Penalty ratio: on each of the first

256

validation rows, the bracketed penalty of ( 12 ), evaluated at the clean logit

z

and the closed-form variance

\widehat{V}

, divided by the Monte Carlo gap

A_{\rm MC}

, the gap

A

of ( 9 ) estimated from

S=2{,}048

sampled masks; median over rows, then mean over seeds. A ratio of

1

means that the closed-form penalty matches

A_{\rm MC}

A_{\rm MC}

columns: averaged over the same rows and then over seeds, in units of

10^{-3}

. Gain:

\kappa=2

minus

\kappa=0

under the same seed, in units of

10^{-4}

; the seed threshold of one such difference (Section 5.1 ) is

0.94

for AUC and

1.0

for Logloss.

Tower	Gate reads	Tower params	AUC $\uparrow$ ( $\kappa=0$ )	$\Delta$ ( $\times 10^{-4}$ )
QNN- $\alpha$	previous layer	3.05M	0.81478	–
QNN- $\alpha$	input embedding	3.05M	0.81373	$-10.5$
FLMNet	input embedding, per field	3.14M	0.81469	$-0.9$

Table 6. Accuracy without the penalty (

\kappa=0

) on Criteo (validation split, mean over seeds 110000–110002). All towers have

L=4

layers and

M=5

branches and are trained with the same recipe at

\rho=0.1

. The first two rows are the rows of Table 5 ; FLMNet differs from the second row in its field-local gates, learned mixer, and field recalibration (Section 4 ).

\Delta

is the AUC difference relative to the first row in units of

10^{-4}

; the seed threshold of one difference is

0.94

Figure 7. Closed-form penalty against sampling-based regularizers on the same tower and backbone (validation split). Each stem shows the AUC of FLMNet-CF at the

\kappa

it uses on that benchmark (given below it) minus that of a sampling-based regularizer, averaged over three seeds, with runs paired by seed; open circles show the individual seed pairs. From left to right in each group, the regularizers are dropout at

\rho=0.1

, the self-ensemble loss (

S=2

), R-Drop with its weight chosen on validation, and dropout at the dose-matched rate

\rho^{\prime}

(dashed). The rate

\rho^{\prime}

gives dropout the dose

\kappa\rho/(1-\rho)

of FLMNet-CF; on Frappe it equals

0.1

, so the dashed stem there reuses the runs of the first stem. The bracketed count above each benchmark is the number of seed pairs, out of the nine under the first three stems, in which the closed form leads. The vertical axis uses a signed square-root scale in units of

10^{-4}

. Single-panel chart with four benchmark groups, Criteo, Avazu, KKBox and Frappe. Each group holds three solid vertical stems for dropout, the self-ensemble loss and R-Drop, followed by a dashed stem for dose-matched dropout. Each stem rises from zero to the mean validation AUC difference between the closed-form penalty and that method, on a signed square-root axis in units of ten to the minus four, with open circles for the three individual seeds. All stems lie above zero; three Avazu seed points lie below zero. A bracket over the three solid stems of each group carries the count of positive seed pairs: 9 of 9, 7 of 9, 9 of 9 and 9 of 9.

Figure 8. Switching the penalty on or off during training (Criteo, constant learning rate,

\kappa=2

, 48 epochs, three seeds; validation split). Each row is one penalty schedule; in (b) and (c), markers give the mean over three seeds and whiskers their range. (a) Schedule: thick bars mark the epochs with the penalty on; shading marks the epochs after the never-penalized run passes its validation peak (epoch 9). (b) AUC gain retained: the gain of each run in peak validation AUC over the never-penalized run, divided by that of the always-penalized run, computed per seed. Lines join the means. The always-penalized run is their shared end point, read as onset at epoch 0 on the switch-on line and as no switch-off on the switch-off line; the switch-on line peaks at onset epoch 4. (c) Growth rate of validation minus training cross-entropy (penalty excluded): the least-squares slope from four epochs after the switch to epoch 48, or from epoch 12 for the never- and always-penalized runs. Grey dotted ticks give the slopes of these two runs (

\kappa=0

and

\kappa=2

) fitted over the same epochs as each row. Three aligned panels on Criteo with one row per training run: never penalized, penalty switched on at epoch 20, 16, 12, 8 or 4, always penalized, and penalty switched off at epoch 16, 12, 8 or 4. Panel a draws each run's penalty schedule as a bar over the epochs, with the epochs after the never-penalized run's validation peak shaded. Panel b plots, for each run, the fraction of the always-penalized AUC gain that the run retains, as a mean marker with a whisker for the seed range and reference lines at zero and one. A blue line joins the switch-on runs to the always-penalized run and bulges to the right, peaking at about 1.28 for onset at epoch 4; an orange line falls from the always-penalized run through the switch-off runs to about 0.10 at epoch 4. Panel c plots each run's growth rate of validation minus training cross-entropy, with grey reference ticks for the never- and always-penalized runs over the same epochs; switch-on runs sit near 0.3 and switch-off runs between 2.5 and 2.9 thousandths per epoch.

Model	AUC $\uparrow$	Logloss $\downarrow$	Gain over direct training	Inference models
Directly trained member	0.81585	0.43630	–	1
Single-teacher student ( $K=1$ )	0.81595	0.43616	$+0.10$ / $-0.14$	1
Multi-teacher student ( $K=5$ )	0.81665	0.43547	$+0.80$ / $-0.83$	1
Teacher ensemble	0.81729	0.43484	$+1.44$ / $-1.46$	5

Table 7. Distilling cross-seed teachers into one tower on Criteo (validation split), with rows ordered by gain. The member row is the mean of six FLMNet-CF seeds with EMA weights. Every student starts from one of these seeds,

\theta_{0}

: the single-teacher row (

K=1

) averages five students, each distilled toward one of the other five seeds, and the multi-teacher student is distilled toward the teacher ensemble, which pools the logits of those five seeds. Students are evaluated on raw weights. The teacher ensemble runs five towers at inference and serves only as a reference. Gains are AUC / Logloss differences from the member mean over all six seeds, in units of

10^{-3}

; the teacher’s gain over the mean of its own five members is the

A_{\rm seed}

of ( 14 ). The starting point

\theta_{0}

has a validation AUC

0.03\times 10^{-3}

below the member mean, so the students’ AUC gains over their own starting point are larger by this amount.

Figure 9. The loss surface around one seed endpoint along distillation and label directions (Criteo; fixed subsets of 200,000 valid and 200,000 train rows). Every run starts from one CF seed endpoint

\theta_{0}

; the five other seeds are the teachers. In (a) and (b) the horizontal axis is the Euclidean distance from

\theta_{0}

over all parameters divided by

\|\theta_{0}\|

(dominated by the tower); each curve follows the straight path from

\theta_{0}

through one trained endpoint (marker) to 1.5 times its length, drawn lighter beyond the endpoint. (a) Change in valid Logloss. Along the distillation directions the loss stays level with or below that of

\theta_{0}

out to about

0.2\|\theta_{0}\|

: the multi-teacher student ends at the lowest point of its path (

-0.72\times 10^{-3}

), and the five single-teacher (

K{=}1

) students end near the rim of this flat stretch (

-0.12

+0.07\times 10^{-3}

). Label-only continuation under five data orders ends only

0.054\|\theta_{0}\|

from

\theta_{0}

, yet

16\times 10^{-3}

higher. (b) Change in train label cross-entropy along the student directions of (a) and along label continuation: it rises toward the students and falls toward label continuation. (c) Cosine between parameter displacements from

\theta_{0}

after their component along

\theta_{0}

is projected out, separately for the tower and the embedding parameters (small filled marker: tower; large open marker: embeddings, drawn in the same column), one column per kind of pair:

K{=}1

students with each other, the multi-teacher student with the mean

K{=}1

displacement, label continuation with

K{=}1

students, and label continuations with each other. Markers give the mean over pairs, 0.43/0.45, 0.95/0.94,

-0.02

-0.03

and 0.98/0.82 (tower/embeddings); the range across pairs is at most 0.04; the dashed line at 0 marks random directions, whose standard deviation is below

10^{-3}

. Parameters are compared only among points that share

\theta_{0}

as origin; the other seeds lie about

1.41\|\theta_{0}\|

from

\theta_{0}

without permutation alignment, an upper bound on their distance after aligning hidden units, and are not drawn. Seed endpoints use EMA weights; students and label continuation use raw weights. In (a) and (c), label continuation is taken at the checkpoint closest to the mean step at which the

K{=}1

students reach their peak validation AUC; in (b), its line runs through its final checkpoint instead. Three-panel figure, in one row, on the loss surface around one trained seed. Panels a and b share a horizontal axis, the parameter distance from the starting seed relative to its norm, and trace straight paths from the starting seed through each trained endpoint. Panel a shows validation log loss: along the five single-teacher student paths and the multi-teacher student path it stays level or dips slightly over a flat stretch out to about 0.2, with the multi-teacher student ending at its lowest point, about 0.7 thousandths below the start; along the label-only continuation paths it climbs off the scale within a distance of about 0.05. Panel b shows training cross-entropy along the same student paths and along label continuation through its final checkpoint: it rises slowly toward the students and drops steeply toward label continuation. Panel c is a narrow dot plot, one column per kind of pair with tower and embedding values in the same column, of cosines between parameter displacements with the starting-point axis removed, for tower and embedding parameters: about 0.43 between single-teacher students, about 0.95 between the multi-teacher student and the mean single-teacher displacement, about zero between label continuation and the students, and about 0.98 among label continuations.

Figure 10. Teacher gain against the number of pooled seeds, and seed disagreement with and without the penalty (validation split). (a) Criteo. Each teacher pools the logits of the first

K

FLMNet-CF seeds in a fixed order (110000, 110001, …), and its gain, the mean member Logloss minus the teacher Logloss, equals

A_{\rm seed}(K)

. The order starts at

\theta_{0}

(seed 110000), so the

K{=}1

teacher is

\theta_{0}

itself, with gain 0; the teacher ensemble of Table 7 instead excludes

\theta_{0}

. The gain rises from

0.91\times 10^{-3}

K=2

1.51\times 10^{-3}

K=6

. The dashed curve is

c(1-1/K)

with

c

fitted at

K=6

alone, and the band marks

\pm 10^{-4}

, the seed threshold of one Logloss difference; the agreement is consistent with the seeds behaving as interchangeable draws. Students continue from

\theta_{0}

toward each teacher; their gain is measured against the EMA weights of

\theta_{0}

rather than the member mean and is

0.59

0.62

times the teacher gain at every

K

from 2 to 6. Students distilled toward a single other seed appear in Table 7 . (b) Seed disagreement

A_{\rm seed}

among three members per benchmark (seeds 110000–110002; Frappe 110008–110010), for FLMNet-CF members divided by that for penalty-free members (

\kappa=0

) with the same seeds.

A_{\rm seed}

equals the Logloss gain of the teacher that pools these three members; the column on the right gives the two values, penalty-free

\to

FLMNet-CF, in

10^{-3}

nats per row. The log axis spans a factor of two either way, and the dashed line at 1 marks no change; each ratio rests on three seeds per side and has no confidence interval. On the same members, the penalty lowers the Gaussian surrogate of the mask gap, which the panel does not show, to

4

–

20\%

of its penalty-free value. Two-panel figure. Panel a plots the Criteo validation Logloss gain of logit-pooled teachers and of distilled students against the number of teacher seeds K from 1 to 6, with a first-order curve c(1-1/K) fitted at K equals 6 and a band of plus or minus the seed threshold; the K equals 1 teacher is the starting point itself and has zero gain, the teacher points lie on the curve, and student gains rise from about 0.57 to 0.90 times ten to the minus three. Panel b is a lollipop chart, on a log axis from 0.5 to 2, of the closed-form to penalty-free ratio of the seed disagreement among three members, one row per benchmark, with a dashed reference line at 1 and the two values listed on the right in units of ten to the minus three: about 0.27 and 0.23 on Avazu, 1.27 and 1.22 on Criteo, 8.5 and 6.8 on KKBox, and 14.8 and 19.2 on Frappe, so the closed-form to penalty-free ratios are 0.84, 0.96, 0.80 and 1.30.

Closed-form penalty	Cross-seed distillation	Criteo		Avazu		KKBox		Frappe
		AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$	AUC $\uparrow$	Logloss $\downarrow$
–	–	0.81469	0.43732	0.76194	0.37471	0.85453	0.47294	0.98670	0.13115
$\surd$	–	0.81586	0.43631	0.76182	0.37479	0.85821	0.46808	0.98703	0.13087
–	$\surd$	0.81569	0.43633	0.76212	0.37443	0.85824	0.46735	0.98671	0.13066
$\surd$	$\surd$	0.81658	0.43555	0.76213	0.37451	0.86046	0.46417	0.98709	0.12709

Table 8. Contribution of each component to validation AUC and Logloss (mean over three seeds; Frappe uses seeds 110008–110010, the other benchmarks 110000–110002). The penalty column marks whether a model is trained with the closed-form penalty at the

\kappa

that FLMNet-CF uses on its benchmark. In the distillation rows a single student continues from a member’s endpoint toward a logit-pooled teacher of three members, and the penalty column then refers to how those members were trained. On Criteo the last row reuses an earlier run in which each student’s teacher pools its own closed-form member with two others drawn at random from seeds 110000–110005 (for seed 110000, seeds 110001 and 110003), whereas in the third row every student’s teacher pools the penalty-free members with seeds 110000–110002. Students are evaluated on raw weights, the other rows on EMA weights. On Criteo and KKBox, the last row retains

87\%

and

80\%

of the sum of the two single-component AUC gains. Best value per column in bold.

Model	Forward passes / step	Step time (ms)	Peak memory (MiB)	Training runs	Inference latency (ms)
QNN- $\alpha$ (one head)	2	45.4	1106	1	5.9
QNN- $\alpha$ (two heads)	2	47.0	1108	1	5.7
FLMNet-CF (ours)	1	46.7	1667	1	5.8
FLMNet-KD (ours)	1	24.1	797	$K+1$	5.8

Table 9. Training and inference cost on Criteo, measured on one Tesla V100 (32 GB) in fp32 at batch size 4,096, with batches preloaded on the device. Forward passes count passes of the network body per optimizer step; QNN-

\alpha

looks up the embeddings once and runs its body twice. Step time is the median over 300 timed steps after 50 warm-up steps, and includes each model’s own training step (the EMA update for FLMNet-CF, the regularizer of FuxiCTR for QNN-

\alpha

). Peak memory is the peak allocated memory in MiB minus the preloaded batches. Inference latency is the median time in ms of one forward pass on a batch of 4,096 rows. The FLMNet-KD step excludes the teacher, whose logits are computed once in advance; its training runs count

K

teachers and one student.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$d$	$C$	$h$	$L$	$M$	$\kappa$	Emb. init std	Tower	Emb.
Criteo	10	39	390	4	5	2	$10^{-4}$	3.14	20.87
Avazu	10	22	220	3	5	0.25	$10^{-4}$	0.77	12.99
KKBox	128	13	1664	2	5	3	0.6	29.84	11.74
Frappe	10	10	100	2	13	1	$10^{-4}$	0.29	0.05
ML-1M	16	7	112	2	13	1	$10^{-4}$	0.38	0.16

Appendix

Table 10. Structure of FLMNet-CF on each benchmark. Parameter counts are for the tower and for the embedding tables, in millions.

Model	Logloss	AUC	AUC s.d.	Seeds
LR	0.3736	0.7514	$1.5\times 10^{-4}$	3
FM	0.3683	0.7602	$5.8\times 10^{-4}$	3
DNN	0.3677	0.7629	$4.3\times 10^{-4}$	3
IPNN	0.3680	0.7618	$1.9\times 10^{-4}$	3
DeepFM	0.3679	0.7625	$1.8\times 10^{-3}$	3
xDeepFM	0.3674	0.7639	$2.0\times 10^{-4}$	3

Appendix

Table 11. Test results on the released Avazu split, before the re-split. Entries are means over the seeds in the last column; standard deviations of test AUC are over the same seeds.

Benchmark	Batch size	Peak LR	Emb. weight decay	EMA $\beta$	LR schedule and stopping rule
Criteo	4096	$10^{-3}$	$10^{-5}$	0.9999	Cosine over 40 epochs, early stopping (patience 2)
Avazu	4096	$5\times 10^{-4}$	$2.5\times 10^{-3}$	0.9999	Cosine over 20 epochs, early stopping (patience 2)
KKBox	10000	$10^{-3}$	$3\times 10^{-4}$	0.9993178	Constant for 22 epochs, cosine over the last 8 of 30, no early stopping
Frappe	1024	$8\times 10^{-3}$	$10^{-3}$	0.99949088	FuxiCTR rule (legacy)
ML-1M	1024	$2\times 10^{-3}$	$5\times 10^{-4}$	0.99942330	FuxiCTR rule (legacy)

Appendix

Table 12. Training settings of FLMNet-CF. The EMA decay

\beta

is applied per optimizer step. On KKBox and ML-1M it corresponds to a horizon of 2.5 and 3 epochs, respectively. The FuxiCTR rule is described in the text; the two boards that use it are marked as a legacy protocol.

Benchmark	Main competitors	FLMNet-CF	FLMNet-KD	Remaining baselines
Criteo	6	6	6	3
Avazu	6	6	3	3
KKBox	6	6	3	3
Frappe	6	6	3	3
ML-1M	6	8	3	3

Appendix

Table 13. Number of random seeds behind each entry of Tables 3 and 4 . FLMNet-KD students on a benchmark are built on overlapping sets of FLMNet-CF towers and are not independent.

Model	Declared	Ours	Gap	Source	Reading
LR	0.751625	0.7513945	$-2.305$	BARS, 1 run	within tolerance
FM	0.761253	0.7602211	$-10.32$	BARS, 1 run	below
DNN	0.763827	0.7628807	$-9.463$	BARS, 1 run	within tolerance
IPNN	0.763014	0.7617708	$-12.43$	BARS, 1 run	below
DeepFM	0.764586	0.7624552	$-21.31$	BARS, mean of 5 runs	below
xDeepFM	0.764205	0.7639083	$-2.967$	BARS, mean of 5 runs	within tolerance

Appendix

Table 14. Test AUC of our reproductions on the released Avazu split against the declared value. Ours is the mean over seeds; gap is ours minus declared, in units of

10^{-4}

. The tolerance is defined in the text. The last column gives the reading of the gap.

Benchmark	Model	Opponent	$n$	$F$	$\bar{\Delta}$	$\theta$	$\bar{\Delta}/\theta$	Sign
Criteo	CF	QNN- $\alpha$ (two heads)	6	0.029	0.605	0.142	4.2	6/6
	KD	QNN- $\alpha$ (two heads)	6	0.029	1.311	0.140	9.4	6/6
Avazu	CF	FinalNet	6	0.547	1.029	0.547	1.9	6/6
	KD	FinalNet	3	0.758	1.325	0.758	1.7	3/3
KKBox	CF	QNN- $\alpha$ (two heads)	5	0.029	0.605	0.070	8.7	5/5
	KD	QNN- $\alpha$ (two heads)	3	0.029	2.919	0.165	17.7	3/3

Appendix

Table 15. Seed-wise test judgements.

\bar{\Delta}

F

and

\theta

are in units of

10^{-3}

AUC; the rule and the floor

F

are defined in the text. Sign: number of seeds with

\Delta_{s}>0

. On Frappe the rule is unpaired and FLMNet-KD is not judged.

CTR Prediction

Click-through rate (CTR) prediction estimates which ads or items users will click to improve ranking, recommendation, and advertising decisions.

See all 19 papers

Sep 29, 2026

Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction

Click-Through Rate prediction, a core task in recommendation and advertising systems, relies on modeling interactions among sparse categorical features. Explicit cross networks are a central paradigm for CTR prediction, and recent progress has largely come from increasing the interaction capacity of a single predictor through deeper cross networks and more expressive cross operators. We revisit whether continually increasing interaction capacity remains the most effective way to improve predictive performance, and find that its benefits quickly exhibit diminishing returns even as capacity continues to grow. This motivates a complementary scaling direction that we call estimator scaling, where additional resources are used to incorporate multiple related estimators rather than only enlarging a single predictor. Through theoretical analysis, we show that the gains from estimator scaling are governed by the amount of non-shared predictive variation available across estimators. However, exploiting this variation naively can be expensive: independently trained models provide substantial estimator diversity but require deployment cost to grow with ensemble size. This motivates a parameter-efficient realization of estimator scaling that can incorporate diversity from multiple estimator sources without maintaining multiple full models. Building on this view, we introduce RECursive Averaged Predictor (RECAP), a parameter-efficient recursive CTR model that operationalizes estimator scaling at three levels: distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. Experiments across multiple benchmarks establish new state-of-the-art predictive performance on standard benchmarks, while placing the RECAP on a favorable performance-parameter Pareto frontier.

Shivang Chopra, Fotis Iliopoulos, Zsolt Kira +1

Georgia Institute of Technology · Google Research

Feature Interaction Modeling Ensemble Learning CTR Prediction

Ask PDF

Jun 20, 2026

Mitigating Early Training Collapse in CTR Models

Deep neural models for click-through rate prediction often exhibit a sharp decline in validation performance immediately after the first training epoch despite continued improvement in training loss. This instability restricts effective learning and limits model performance. In this study, we analyze this behavior using large-scale industrial datasets and evaluate practical mitigation strategies. While reducing the learning rate provides only incremental gains, controlling feature sparsity yields substantial improvements. Removing highly sparse features and aggregating infrequent feature values stabilizes training, extends useful learning beyond a single epoch, and improves both offline evaluation metrics and online system performance.

Ergun Biçici, Erkan Çetinyamaç

Intelligent Application Development, Huawei Türkiye R&D Center, Istanbul, Turkey

CTR Prediction Feature Selection Neural Network Training Dynamics User Preference Modeling

Ask PDF

May 24, 2026

Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration

Scaling test-time compute has proven highly effective for language models, yet this opportunity remains largely unexplored for industrial Click-Through Rate (CTR) prediction. CTR models suffer from a fundamental asymmetry: feature combinations well-represented in training yield confident predictions, while sparsely observed ones produce unreliable outputs. Existing training-phase solutions such as adaptive gating learn a fixed selection function subject to the same sparsity, offering no per-instance recourse at deployment.We propose UTTSI (Uncertainty-Triggered Test-Time Selective Inference), a training-free model-agnostic framework that scales inference depth proportionally to per-instance uncertainty. A dual-signal estimator combining model logit confidence with a data-level frequency prior distinguishes epistemic uncertainty from aleatoric ambiguity. Every instance undergoes adaptive feature filtering to remove unreliable embeddings; uncertain instances additionally receive stochastic feature-path explorations whose predictions are aggregated via consistency-weighted ensembling. Confident instances bypass exploration entirely, keeping average overhead at approximately

2.8\times

base model cost with worst-case latency unchanged.Experiments on four datasets with three backbone architectures demonstrate consistent, statistically significant gains over all training-phase baselines. A seven-day online A/B test further confirms a 5.3% relative CTR gain (

p < 0.01

), establishing selective test-time compute allocation as a practical complement to training-phase advances for CTR prediction.

Moyu Zhang, Yun Chen, Yujun Jin +3

Alibaba Group Beijing, China

Cost-Aware Inference Test-Time Scaling CTR Prediction Recommender Systems User Preference Modeling

Ask PDF

Abstract

CTR Prediction

Explore similar work

Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction

Mitigating Early Training Collapse in CTR Models

Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration