How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration

Rui Liu

cs.LGOct 23, 2026

How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration

Authors: Rui Liu

Organizations: Independent Researcher

Abstract

Offline evaluation of click-through rate (CTR) models still defaults to random splits, although deployed models score future traffic and downstream systems consume calibrated probabilities. On Avazu (40.4M impressions) we train logistic regression (LR), DeepFM, and DCN-v2 under two rolling temporal origins and a random split, three seeds per arm, and recalibrate each test day with Platt scaling and isotonic regression fitted on the preceding hours. At origin 1's nearest test day, random splits report LR's expected calibration error (ECE) at 55-62% of the temporal value in every seed (the deep models' ratio is seed-unstable), and no model's ECE is monotone in temporal distance. With a 6-hour fitting slice, recalibration removes up to 61% of ECE in badly miscalibrated cells but increases it in about a third of applications (14-17 of 48 across seeds). On fixed evaluation sets, a 12-hour slice cuts harm from 13-17 to 3-5 of 36 applications, and 24 hours raises it again. The 12-hour slice subsampled to the 1-hour slice's size (5% of the impressions; 20-25x faster to fit) recovers 88% of the full slice's gain, so impression count alone does not explain it. A slice-ECE gate shows no benefit out-of-origin, and in a simulated auction recalibration moves profit by only about -0.21% to +0.26%. Offline protocols should measure calibration temporally, over multiple origins and seeds; in this data, recalibration fitted on a wide time span beat the freshest few hours, which other logs must confirm.

Figures & tables

Origin	Bucket	Date (weekday)	Eval. rows	Slice rows	Obs. CTR	LR	DeepFM	DCN-v2
						Signed bias, target / slice
1	gap 0	Oct 26 (Sun)	3,148,694	687,198	0.1807	$-0.0044$ / $-0.0033$	$-0.0033$ / $-0.0050$	$-0.0029$ / $-0.0033$
1	gap $+1$	Oct 27 (Mon)	3,225,010	620,216	0.1816	$-0.0080$ / $-0.0074$	$-0.0049$ / $-0.0043$	$-0.0040$ / $-0.0038$
1	gap $+2$	Oct 28 (Tue)	5,287,222	743,264	0.1522	$-0.0031$ / $-0.0036$	$+0.0027$ / $+0.0001$	$+0.0016$ / $+0.0011$
1	gap $+3$	Oct 29 (Wed)	3,832,608	1,007,461	0.1566	$+0.0031$ / $-0.0071$	$+0.0048$ / $-0.0018$	$+0.0054$ / $-0.0028$
2	gap 0	Oct 27 (Mon)	2,575,774	649,236	0.1804	$-0.0091$ / $-0.0071$	$+0.0027$ / $+0.0044$	$-0.0017$ / $-0.0008$

TABLE I: Temporal-arm test buckets at both origins. Gap-0 buckets are evaluated on hours 6–23 of their day (their first six hours are the recalibration slice). Signed bias is the raw model’s mean prediction minus the observed click rate (negative = under-prediction), for the target bucket and its 6-hour fitting slice. Seed 42.

Model	Arm	AUC	Log loss	ECE
LR	Random	0.7455	0.3988	0.0051
LR	Temporal (gap 0)	0.7320	0.4193	0.0092
DeepFM	Random	0.7570	0.3927	0.0038
DeepFM	Temporal (gap 0)	0.7431	0.4135	0.0069
DCN-v2	Random	0.7549	0.3938	0.0051
DCN-v2	Temporal (gap 0)	0.7408	0.4148	0.0070

TABLE II: Random split vs. temporal split at gap 0, origin 1 (raw model). Origin 2’s gap-0 temporal values are LR 0.7457/0.4122/0.0097, DeepFM 0.7562/0.4059/0.0040, DCN-v2 0.7558/0.4064/0.0044 (AUC/log loss/ECE). Seed 42.

Fig. 1: Raw ECE by temporal gap at two training origins: mean of seeds 42/43/44 (lines) and min–max range (bands). LR is seed-stable and non-monotone at both origins; the deep models’ bands overlap across gaps and their seed-42 peaks do not reproduce.

Item	Detail	Time (s)
Training, origin 1	LR / DeepFM / DCN-v2	10.5 / 205.7 / 599.8
Training, origin 2	LR / DeepFM / DCN-v2	13.1 / 257.6 / 703.0
Refit, 1 h (87K impr.)	Platt / iso.	0.085 / 0.032
Refit, 3 h (275K)	Platt / iso.	0.26 / 0.11
Refit, 6 h (0.69–1.01M) †	Platt / iso.	0.93–1.12 / 0.27–0.63
Refit, 12 h (1.86M)	Platt / iso.	1.41 / 0.87

TABLE III: Measured compute on the 2-vCPU VM. Training: wall-clock per model (seed-42 run logs; the full arm including evaluation totals 1,291 s at origin 1 and 1,457 s at origin 2). Refit: median fit time per bucket with median slice volume; 1/3/12/24 h from the seeds-43/44 sweep instrumentation (48 fits per cell, 36 at 24 h). † From an earlier timing probe on two origin-1 slices (median of seven), not directly comparable. ‡ Size-matched control (Section IV-C ), seed 42, gaps

+1

+3

Model	Bucket	AUC	Log loss	$\Delta$ LL (Platt)	$\Delta$ LL (Iso.)	ECE (raw)	ECE (Platt)	ECE (isotonic)
LR	0	0.7320	0.4193	$-0.0001$	$-0.0002$	0.0092	0.0082	0.0037
LR	$+1$	0.7384	0.4175	$-0.0003$	$+0.0001$	0.0118	0.0078	0.0103
LR	$+2$	0.7497	0.3734	$-0.0001$	$+0.0002$	0.0084	0.0071	0.0112
LR	$+3$	0.7352	0.3866	$+0.0005$	$+0.0005$	0.0054	0.0112	0.0100
DeepFM	0	0.7431	0.4135	$-0.0000$	$-0.0000$	0.0069	0.0057	0.0044
DeepFM	$+1$	0.7521	0.4103	$-0.0001$	$-0.0002$	0.0092	0.0072	0.0056

TABLE IV: Temporal arms, full grid (top: origin 1; bottom: origin 2). AUC and log loss are for the raw model;

\Delta

LL is each variant’s signed log-loss change from raw; ECE for raw, Platt, and isotonic variants, fitted per bucket on the preceding 6 hours. Seed 42.

Origin	Calibrator	1 h	3 h	6 h	12 h	24 h
1	Isotonic	$-5$ (10)	$-13$ (11)	$-11$ (9)	$-38$ (1)	$-31$ (6)
1	Platt	$-4$ (8)	$-2$ (9)	$-7$ (6)	$-17$ (4)	$-11$ (11)
2	Isotonic	$+18$ (18)	$+7$ (18)	$+2$ (16)	$-30$ (1)	$-30$ (3)
2	Platt	$0$ (12)	$+6$ (14)	$+3$ (13)	$-14$ (6)	$-9$ (11)
Harms/seed (of 36)		15/15/18	16/16/20	13/14/17	4/5/3	9/11/11
LR harms/seed (of 12)		7/6/7	8/8/8	8/8/8	2/1/1	5/6/6

TABLE V: Slice-length sweep on fixed evaluation sets (gaps

+1

+3

), seeds 42/43/44: mean

\Delta

ECE (

\times 10^{-4}

; negative = recalibration helps) and, in parentheses, harmful applications out of 27 (9 cells

\times

3 seeds). Lower rows: harms per seed (of 36), LR harms (of 12), seed-42 gap-0 harms (of 12), median slice size.

Fig. 2: Recalibration outcome vs. initial miscalibration, both origins (filled markers: origin 1; open markers: origin 2). Harm occurs at low, moderate, and high raw ECE (up to 0.0108) and follows no single threshold across origins. Seed 42, 6-hour slice.

Fig. 3: Calibration gap (observed minus predicted click rate, percentage points) per 15-bin before and after recalibration, in the grid’s best isotonic cell (left) and LR’s worst cell (right); origin 1, seed 42, bins with

\geq

1,000 impressions. Recalibration pulls gaps toward zero in the first case, away in the second.

Model	Bucket	Iso. $\Delta$ %	Platt $\Delta$ %	Oracle
LR	0	$+0.02$	$+0.03$	1.81 $\times$
LR	$+1$	$-0.06$ ∗	$+0.11$ ∗	1.79 $\times$
LR	$+2$	$-0.10$ ∗	$+0.03$	1.95 $\times$
LR	$+3$	$-0.08$ ∗	$-0.10$ ∗	1.97 $\times$
DeepFM	0	$+0.01$	$+0.06$ ∗	1.78 $\times$
DeepFM	$+1$	$+0.06$ ∗	$+0.01$	1.75 $\times$

TABLE VI: Simulated bidding profit: recalibrated strategies as % difference from raw; label oracle as a multiple of raw profit. ∗ = paired-bootstrap 95% CI excludes 0 (24 of 48 deltas). Deltas under 0.005% in magnitude show three decimals. Left: origin 1; right: origin 2.

Post-Hoc Calibration

Post-hoc calibration adjusts a trained model’s probabilities or confidence scores to better match observed accuracy, often without retraining.

See all 82 papers

Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration
May 24, 2026Moyu Zhang, Yun Chen, Yujun Jin +3Cost-Aware Inference Test-Time Scaling
Alibaba Group Beijing, China
CalArena: A Large-Scale Post-Hoc Calibration Benchmark
May 28, 2026Eugène Berta, David Holzmüller, Francis Bach +1Post-Hoc Calibration Probability Calibration
Inria - Ecole Normale Supérieure PSL Research University · Inria
Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction
Sep 29, 2026Shivang Chopra, Fotis Iliopoulos, Zsolt Kira +1Feature Interaction Modeling Ensemble Learning
Georgia Institute of Technology · Google Research

May 24, 2026J/K move · Enter open · S save

Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration

Moyu Zhang, Yun Chen, Yujun Jin, Jinxin Hu+2

Alibaba Group Beijing, China

Cost-Aware Inference Test-Time Scaling CTR Prediction Recommender Systems User Preference Modeling

Open paper AskPDF arXiv

Scaling test-time compute has proven highly effective for language models, yet this opportunity remains largely unexplored for industrial Click-Through Rate (CTR) prediction. CTR models suffer from a fundamental asymmetry: feature combinations well-represented in training yield confident predictions, while sparsely observed ones produce unreliable outputs. Existing training-phase solutions such as adaptive gating learn a fixed selection function subject to the same sparsity, offering no per-instance recourse at deployment.We propose UTTSI (Uncertainty-Triggered Test-Time Selective Inference), a training-free model-agnostic framework that scales inference depth proportionally to per-instance uncertainty. A dual-signal estimator combining model logit confidence with a data-level frequency prior distinguishes epistemic uncertainty from aleatoric ambiguity. Every instance undergoes adaptive feature filtering to remove unreliable embeddings; uncertain instances additionally receive stochastic feature-path explorations whose predictions are aggregated via consistency-weighted ensembling. Confident instances bypass exploration entirely, keeping average overhead at approximately

2.8\times

base model cost with worst-case latency unchanged.Experiments on four datasets with three backbone architectures demonstrate consistent, statistically significant gains over all training-phase baselines. A seven-day online A/B test further confirms a 5.3% relative CTR gain (

p < 0.01

), establishing selective test-time compute allocation as a practical complement to training-phase advances for CTR prediction.

Abstract

Post-Hoc Calibration

Explore similar work

Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction