Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning

Zilin Zhao; Han Yang; Tianpei Yang; Fangsheng Huang; Yanfei Cui; Kan Peng; Yi Li; Yiming Zong

cs.AIAug 28, 2026

Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning

Authors: Zilin Zhao, Han Yang, Tianpei Yang, Fangsheng Huang, Yanfei Cui, Kan Peng, Yi Li, Yiming Zong,

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · ByteDance, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China

Abstract

Complete your ad view and grab a 5-cent bonus! In incentivized advertising, a platform promises users a bonus before observing downstream ad revenue, encouraging them to click and complete ads. It must balance the incentive promised in advance against the revenue realized afterward: insufficient incentives forfeit monetization opportunities, whereas excessive incentives reduce net profit. Because current incentives may also shape user expectations and future engagement, incentive allocation is a sequential decision problem with delayed revenue, cost sensitivity, and carryover effects. Existing work has not studied decision-making algorithms for this setting. Auto-bidding assumes available ad opportunities, while targeted promotion optimizes incentives outside the ad monetization pipeline. We formulate the problem as an MDP and develop an offline model-based RL framework for cost-controllable sequential incentive allocation. It learns a world model of user feedback and ad revenue, then performs conservative policy optimization. An independent counterfactual scorer evaluates each learned policy on held-out logs, enabling pre-launch selection without costly online exposure. Experiments on large-scale industrial data and online A/B tests show that the scorer provides a stable offline signal. The deployment path from causal inference to offline RL and then Offline-MBRL further validates the framework: MB-IQL improves per-user net profit by 7.96% over TD3+BC, whereas reverting to plain IQL reduces it by 6.56% (both $p<0.0001$ ).

Figures & tables

Figure 2. User feedback in incentivized advertising exhibits short-window carryover patterns. Users are often indifferent before the first incentive, so the initial acceptance rate is relatively low. Once accepted, users may enter a short excited stage with higher click propensity. As incentive fatigue accumulates, the willingness to click gradually declines. The data are aggregated from the real interaction trajectories of 100 randomly selected users. Illustration of the indifferent, excited, and fatigue stages in incentivized advertising. The figure shows that the exposure rate starts low, rises after the first exposure, and then decreases as incentive fatigue accumulates.

Figure 3. Offline-MBRL pipeline for incentivized advertising. (i) App logs form offline RL transitions $\mathcal{D}_{\mathrm{off}}$ , with useful request context $\epsilon$ joined offline after timestamp masking for WM and CES. (ii) $\mathcal{D}_{\mathrm{off}}$ trains both the RL policy and the structured user-response WM. (iii) The Actor rolls out with the WM to generate virtual transitions, and policy optimization then uses mixed logged and virtual data with conservative regularization. In the lower panel, $d_{f}$ is the mixed transition distribution and $\rho_{\mathrm{wm}}$ is the WM-induced state–action proposal distribution used by the conservative term. Illustration of the Offline-MBRL workflow. The upper part shows serving logs, table materialization, auxiliary pipeline joins, and initial training of a base RL policy and world model. The lower part shows policy-WM rollouts and fine-tuning with mixed real and synthetic data.

Figure 4. Increasing $\lambda$ makes the learned policy more cost-sensitive and shifts budget toward high-ROI users. (a) relative cost, revenue, and net profit during training; (b) relative budget share across ROI buckets. The left panel shows that increasing the cost coefficient reduces cost and revenue while improving net profit in DY-2. The right panel shows that higher cost coefficients allocate a larger share of budget to high-ROI users.

Env.	MF backbone	+WM method (Ours)	MF net p./u.	+WM net p./u. (Ours)	$\Delta$	Rel. gain
DY-1	TD3+BC	MB TD3+BC	$+13.8\%$	$+19.9\%$	$+6.1$ pp	$+44.2\%$
	IQL	MB IQL	$+16.1\%$	$+21.9\%$	$+5.8$ pp	$+36.0\%$
	CQL	MB CQL	$+16.8\%$	$+20.7\%$	$+3.9$ pp	$+23.2\%$
DY-2	TD3+BC	MB TD3+BC	$+14.1\%$	$+20.9\%$	$+6.8$ pp	$+48.2\%$
	IQL	MB IQL	$+15.9\%$	$+20.8\%$	$+4.9$ pp	$+30.8\%$
	CQL	MB CQL	$+17.0\%$	$+19.2\%$	$+2.2$ pp	$+12.9\%$

Table 1. Adding the structured WM consistently improves matched model-free backbones in CES evaluation. CES metrics aggregate request-level predictions per user; MF denotes model-free; WM denotes world model; net p./u. denotes per-user net profit.

\Delta

is the absolute improvement in percentage points, and Rel. gain is

\Delta

divided by the MF net p./u.

Metric	Exp. A	Exp. B	Exp. C	Exp. D
Period	Mar.	Mar.–Apr.	May–Jun.	Jun.
Online $\pi$	DESCN	TD3+BC	TD3+BC	MB IQL
Candidate $\pi$	TD3+BC	IQL	MB IQL	IQL
Conclusion	+Seq. $\uparrow$	no diff.	+WM $\uparrow$	-WM $\downarrow$
cost/u.	+1.5% (0.5)	+1.1% (1.0)	+1.5% (1.0)	+0.3% (1.0)
net p./u.	+7.7% ( $\sim$ 0)	-0.9% (1.0)	+8.0% ( $\sim$ 0)	-6.6% ( $\sim$ 0)

Table 2. Sequential online A/B tests support the offline ranking at matched cost: MB IQL gains where plain IQL does not, and removing WM reverses the gain. MB denotes model-based; cost/u. denotes per-user incentive cost; parentheses show

p

-values, and

\sim

0 means

p<0.0001

Train	Eval	$\epsilon$	$z$		$y\mid z$		$I\mid z$
			AUC	ECE	AUC	ECE	wMAPE
DY-2	DY-2	N	0.85	0.04	0.86	0.04	0.83
DY-2	DY-2	Y	0.91	0.02	0.93	0.03	0.32
DY-2 in	DY-2 in	Y	0.92	0.02	0.93	0.02	0.30
DY-2 in	DY-2 out	Y	0.78	0.07	0.79	0.07	0.99
DY-2 in	DY-2	Y	0.83	0.06	0.84	0.05	0.61

Table 3. CES factual prediction with and without request context, and under reduced-support shift. AUC and ECE denote area under the ROC curve and expected calibration error; lower ECE and weighted mean absolute percentage error (wMAPE) are better.

Input	Eval	$z$ AUC	$y$ AUC	$s^{\prime}_{\mathrm{c}}\overline{\mathrm{wM}}$	$s^{\prime}_{\mathrm{b}}\overline{\mathrm{wM}}$	$I$ wM
$(s,\epsilon)$	1-step	0.87	0.89	0.33	0.40	0.39
$s$	1-step	0.74	0.74	0.56	0.56	0.84
$s$	a.r.	0.70	0.68	0.69	0.61	1.41

Table 4. WM prediction with observed request context, context removal, and autoregressive (a.r.) rollout.

\mathrm{wM}

denotes wMAPE;

\mathrm{c}

and

\mathrm{b}

denote CPM- and behavior-type state-field groups. Completion metrics

y

are

y\mid z

, and

I

denotes

I\mid z

	TD3+BC	MB TD3+BC	IQL	MB IQL
w/ Q reg.	$+12.6\%$ $\pm 0.7\%$	$+20.9\%$ $\pm 1.1\%$	$+14.0\%$ $\pm 1.1\%$	$+20.8\%$ $\pm 1.2\%$
w/o Q reg.	$+14.1\%$ $\pm 0.7\%$	$-2.3\%$ $\pm 0.9\%$	$+15.9\%$ $\pm 0.7\%$	$+3.5\%$ $\pm 1.0\%$

Table 5. Conservative Q-regularization ablation on DY-2. The regularization benefits the model-based methods but slightly degrades their model-free backbones. Entries report the mean and standard deviation of normalized CES net p./u. across three random-seed runs.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Exp. A	Exp. B	Exp. C	Exp. D
Online $\pi$	DESCN ${}^{\prime}_{A}$	TD3+BC A	TD3+BC A	MB IQL C
Candidate $\pi$	TD3+BC A	IQL B	MB IQL C	IQL D
Env.	DY A	DY B	DY C	DY D
A/B test: cost/u.	$+1.5\%$	$+1.1\%$	$+1.5\%$	$+0.3\%$
A/B test: net p./u.	$+7.7\%$	$-0.9\%$	$+8.0\%$	$-6.6\%$
CES: cost/u.	+2.3%	+2.2%	+0.9%	-0.5%

Appendix

Table 6. Offline backtests broadly follow the online deployment signs on the same policy pairs. Each env. is the corresponding pre-launch DY- slice; subscripts denote original weights of the experiment where the model first appeared; DESCN ′ denotes distillation due to the missing original weight file. CES results aggregate request-level predictions per test user; CES and FQE results are normalized by the online policy.

Metric	DESCN ${}^{\prime}_{A}$	TD3+BC A	IQL B	MB IQL C	IQL D
CES: net p./u.	-2.0%	+1.8%	+2.5%	+7.9%	+2.1%
CES: cost/u.	-3.7%	+2.4%	-0.2%	+1.5%	-0.7%
FQE: net p./u.	-1.5%	+3.0%	+3.1%	+8.4%	+8.0%
FQE: cost/u.	-0.6%	+2.1%	+1.0%	-0.3%	+0.5%

Appendix

Table 7. Evaluating deployed checkpoints on the same DY-2 users provides a common-reference offline comparison. CES aggregates request-level predictions per test user. Results are normalized by ground truth on DY-2.

Metric	TD3+BC $\gamma=0$	TD3+BC half-life $\gamma$	IQL $\gamma=0$	IQL half-life $\gamma$
CES: net p./u.	+9.4%	+11.2%	+12.6%	+15.0%
CES: cost/u.	+1.5%	+2.9%	+1.0%	+0.3%
FQE: net p./u.	+6.3%	+9.8%	+8.5%	+12.6%
FQE: cost/u.	-1.1%	-0.1%	+0.9%	+2.7%

Appendix

Table 8. Sequential policies score higher than their one-step counterparts on DY-2 in this reference evaluation. CES metrics aggregate request-level predictions per test user.

Metric	Exp. A	Exp. F
Period	Mar.	May
Platform	Douyin	Douyin-HS
Online $\pi$	DESCN	MAB
Candidate $\pi$	TD3+BC	TD3+BC
net p./u. ( $p$ )	+7.72% ( $\sim$ 0)	+10.23% (0.01)
cost/u. ( $p$ )	+1.54% (0.45)	+1.82% (0.78)

Appendix

Table 9. TD3+BC improves net profit over one-step causal or bandit online policies in these reference A/B tests. Revenue/u. denotes per-user revenue; Douyin-HS abbreviates Douyin’s Huoshan Version.

Setting	Wall time	Overhead
Training only	20 h	0%
Training + CES	21 h	5%
Training + FQE	30 h	50%

Appendix

Table 10. Wall-time overhead of checkpoint evaluation under the same fixed allocation of 8 GPUs, each with 32 GB of GPU memory and approximately 16 TFLOPS of FP32 compute. The CES time excludes its one-time 1-hour dataset-level training cold start.

Env.	Metric	CES-Fact.	BC	10%BC	DT	TD3+BC	IQL	CQL	MB TD3+BC	MB IQL	MB CQL
DY-1	net p./u.	$+0.2\%$	$+2.0\%$	$+8.5\%$	$+12.3\%$	$+13.8\%$	$+16.1\%$	$+16.8\%$	$+19.9\%$	$+21.9\%$	$+20.7\%$
		$\pm 0.2\%$	$\pm 0.4\%$	$\pm 0.5\%$	$\pm 0.8\%$	$\pm 0.6\%$	$\pm 0.6\%$	$\pm 0.7\%$	$\pm 1.0\%$	$\pm 1.1\%$	$\pm 1.0\%$
	cost/u.	$-0.7\%$	$+2.2\%$	$-1.0\%$	$+0.9\%$	$+2.0\%$	$+0.9\%$	$+2.4\%$	$-2.5\%$	$+1.8\%$	$-1.3\%$
		$\pm 0.2\%$	$\pm 0.2\%$	$\pm 0.5\%$	$\pm 0.4\%$	$\pm 0.7\%$	$\pm 0.6\%$	$\pm 0.3\%$	$\pm 0.7\%$	$\pm 1.0\%$	$\pm 0.7\%$
DY-2	net p./u.	$+1.2\%$	$+3.1\%$	$+8.4\%$	$+12.5\%$	$+14.1\%$	$+15.9\%$	$+17.0\%$	$+20.9\%$	$+20.8\%$	$+19.2\%$
		$\pm 0.2\%$	$\pm 0.5\%$	$\pm 0.5\%$	$\pm 0.8\%$	$\pm 0.7\%$	$\pm 0.7\%$	$\pm 0.5\%$	$\pm 1.1\%$	$\pm 1.2\%$	$\pm 1.3\%$

Appendix

Table 11. Full CES offline evaluation keeps the complete baseline, cost diagnostics, and seed-level variation in one table. All metrics first sum request-level predictions within each test user and then average across users. CES-Fact. uses the logged test actions; each entry reports the mean relative change and

\pm

one sample standard deviation across three random-seed runs.

Metric	Experiment A	Experiment B	Experiment C	Experiment D
Period	Mar	Mar. –Apr.	May – Jun.	Jun.
Unique DIDs per group	$>10$ M	$>10$ M	$>10$ M	$>10$ M
Online policy	DESCN	TD3+BC in Exp. A	TD3+BC in Exp. A	MB IQL in Exp. C
Candidate policy	TD3+BC	IQL	MB IQL	IQL
Per-user send ( $p$ )	+1.3102% (<0.0001)	-1.1025% (0.0007)	-0.1442% (0.9906)	+21.8060% (<0.0001)
Per-user exposure ( $p$ )	+1.4441% (<0.0001)	-1.2222% (0.0003)	-0.0144% (0.9999)	+23.6790% (<0.0001)

Appendix

Table 12. Expanded online A/B metrics.

Figure 5. Timestamp masking prevents future-information leakage when constructing offline request context $\epsilon_{t}$ . The timestamp of each incentivized-ad record masks auxiliary context records that would not yet have been available at that request. The masked context is retained only in the offline transition dataset $\mathcal{D}_{\mathrm{off}}$ to assist the WM and CES; the online Incentive Actor cannot access $\epsilon_{t}$ because of the serving architecture. A pipeline schematic with an incentivized-ad database on the left and several auxiliary context databases on the right. The timestamp from the incentivized-ad record is used to mask the context records before they are combined into an offline transition dataset. The Incentive Actor receives the serving-state path from the incentivized-ad pipeline, while a crossed-out arrow indicates that it cannot receive the offline-only enriched context epsilon.

Figure 6. CES action-invariance diagnostics. Raw $a$ – $\epsilon$ correlations are state-driven; after residualizing on $s$ , $|\mathrm{corr}|<0.1$ (max $0.066$ , mean $0.045$ ). Predicting $a$ improves only modestly from $s$ to $(s,\epsilon)$ (RMSE $828.26\to 804.71$ , $R^{2}$ $0.7133\to 0.7294$ ). A four-panel diagnostic figure. The first panel shows raw correlations between logged incentives and ranking-side request-context features around 0.47 to 0.51. The second panel shows residual correlations after conditioning on state, all within plus or minus 0.1. The last two panels compare logged-incentive prediction from state only and from state plus request context, showing a modest RMSE reduction and R-squared increase.

Figure 7. Cross-support CES diagnostic. Top: mean actions in six eCPM buckets, with

\pm 20\%

DY-2 in (dark green) and

\pm 50\%

DY-2 (light green) supports; cost/u. carries the corresponding per-user payout ranges. Bottom: user-aggregated

\mathrm{CES}_{\mathrm{in}}

(dark) and

\mathrm{CES}_{\mathrm{all}}

(light) agree for BC and IQL (

\lambda=1.5

), but diverge for IQL (

\lambda=8.0

) outside the inner support. A three-by-three diagnostic figure. The first six panels show BC, IQL with lambda 1.5, and IQL with lambda 8.0 across six eCPM buckets and two nested action-support bands. The bottom panels compare cost/u., revenue/u., and net p./u. estimated by CES-in and CES-all. Estimates overlap for BC and IQL with lambda 1.5; for IQL with lambda 8.0, CES-in estimates higher cost/u. and revenue/u. but lower net p./u. than CES-all.

Figure 8. Autoregressive WM rollouts accumulate error, and the degradation is already clear after one to two rollout steps. Panel (a) aggregates prediction performance by rollout step; panel (b) aggregates prediction performance by logged trajectory length. TF denotes teacher forcing. World-model prediction performance aggregated by rollout step and logged trajectory length.

Figure 9. One-step WM predictions stay closer to logged mutable states than selected two-step autoregressive predictions in this qualitative t-SNE view. Points show the mutable subset $s_{\mathrm{rt}}$ of logged states, one-step WM predictions, and two-step autoregressive predictions. t-SNE denotes t-distributed stochastic neighbor embedding. A t-SNE plot of four trajectories and a zoomed trajectory. Black points are logged mutable states, green points are one-step WM predictions, and red points are two-step autoregressive predictions.

Env.	Platform	Explr. base	Explr. range	Valid rec.	$\epsilon$	Test ratio
DY-1	Douyin	TD3+BC ( Fujimoto and Gu, 2021 )	$\pm 50\%$	$\sim$ 200M	Y	$\sim$ 10%
DY-2	Douyin	DESCN ( Zhong et al., 2022 )	$\pm 50\%$	$\sim$ 400M	Y	$\sim$ 10%
DH	Douyin’s Huoshan Version	MAB ( Slivkins, 2019 )	$\pm 70\%$	$\sim$ 50M	N	$\sim$ 10%

Appendix

Table 13. The three industrial environments cover different platforms, behavior policies, and logged exploration settings. Env. denotes environment; Platform denotes the serving app; Explr. base denotes the baseline policy around which action perturbations are applied; Explr. range denotes the logged action-perturbation range; Valid rec. denotes valid records;

\epsilon

denotes the availability of request context; and Test ratio denotes the held-out test proportion.

Method	Medium	Medium-Replay
BC ( Pomerleau, 1989 )	$43.2$	$37.6$
IQL ( Kostrikov et al., 2022 )	$47.4$	$44.2$
TD3+BC ( Fujimoto and Gu, 2021 )	$48.3$	$44.6$
CQL ( Kumar et al., 2020 )	$46.9$	$45.3$
MB IQL	$49.1\pm 1.2$	$46.2\pm 1.8$
MB TD3+BC	$49.9\pm 0.8$	$43.1\pm 1.5$

Appendix

Table 14. D4RL HalfCheetah normalized returns on the Medium and Medium-Replay dataset types. Higher is better.

Online Advertising

Online advertising studies how digital ads are targeted, ranked, priced, measured, and delivered while balancing budgets, business goals, and user interests.

See all 87 papers

Aug 5, 2026

Generative Optimization for Incentivized Advertising with Global Level Constraints

Incentivized advertising allocates monetary or virtual rewards to drive user engagement, where a key challenge is optimizing continuous incentive magnitudes under strict global constraints. This problem is complicated by high-frequency interactions, delayed feedback, and non-Markovian user dynamics such as fatigue, which limit the effectiveness of existing uplift modeling and constrained reinforcement learning approaches. To address these challenges, we propose GOAL, a constraint-aware generative framework that formulates incentive allocation as a conditional sequence generation problem. GOAL directly generates incentive magnitudes conditioned on user histories and system-level global pressure, and integrates a hierarchical causal state encoder to capture both local behavioral dynamics and long-range dependencies. To enable flexible constraint control, we introduce Safe Constrained Policy Optimization (SCPO), which learns a single generative policy that generalizes across a spectrum of ROI constraints without retraining. Experiments on large-scale real-world data and a synthetic fatigue-aware environment show that GOAL improves long-term revenue and user retention while substantially reducing ROI violation rates compared to strong baselines.

Gege Chen, Ning Luo, Hao Jiang +7

University of Electronic Science and Technology of China · Kuaishou Technology

Constrained Generative Modeling Online Advertising Constrained RL

Ask PDF

Apr 7, 2026

JD-BP: A Joint-Decision Generative Framework for Auto-Bidding and Pricing

Auto-bidding services optimize real-time bidding strategies for advertisers under key performance indicator (KPI) constraints such as target return on investment and budget. However, uncertainties such as model prediction errors and feedback latency can cause bidding strategies to deviate from ex-post optimality, leading to inefficient allocation. To address this issue, we propose JD-BP, a Joint generative Decision framework for Bidding and Pricing. Unlike prior methods, JD-BP jointly outputs a bid value and a pricing correction term that acts additively with the payment rule such as GSP. To mitigate adverse effects of historical constraint violations, we design a memory-less Return-to-Go that encourages future value maximizing of bidding actions while the cumulated bias is handled by the pricing correction. Moreover, a trajectory augmentation algorithm is proposed to generate joint bidding-pricing trajectories from a (possibly arbitrary) base bidding policy, enabling efficient plug-and-play deployment of our algorithm from existing RL/generative bidding models. Finally, we employ an Energy-Based Direct Preference Optimization method in conjunction with a cross-attention module to enhance the joint learning performance of bidding and pricing correction. Offline experiments on the AuctionNet dataset demonstrate that JD-BP achieves state-of-the-art performance. Online A/B tests at JD.com confirm its practical effectiveness, showing a 4.70% increase in ad revenue and a 6.48% improvement in target cost.

Linghui Meng, Chun Gan, Shengsheng Niu +8

JD.com China · The Chinese University of Hong Kong China

Automated Bidding Real-Time Bidding Online Advertising Dynamic Pricing

Ask PDF

May 20, 2026

Support-aware offline policy selection for advertising marketplaces

Logged advertising auctions make offline reserve-price evaluation attractive but risky. Replay tables can identify policies with large apparent yield gains, yet they can also hide weak threshold support, multiple-comparison effects, subgroup harm, and bidder-response uncertainty. Existing replay and off-policy evaluation methods estimate or rank policy values, but they do not directly answer the operational question of whether the available evidence is strong enough to justify validation. This paper develops a support-aware offline decision framework for reserve-policy selection. Rather than outputting a single point-estimate winner, the framework converts logged evidence into a conservative decision object consisting of certified policies, statistically dominated alternatives, and unresolved candidates requiring further validation. The main theoretical result gives a unified finite-catalog guarantee showing that, under simultaneous uncertainty control and conservative support gates, the framework preserves the best gate-passing policy while eliminating only policies with certified regret. Supporting results characterize support-localized replay generalization, establish information-theoretic threshold-resolution limits, and quantify when heterogeneous bidder response can overturn localized replay rankings. Experiments on iPinYou real-time-bidding logs show that the leading reserve rule achieves a 47.66% replay lift in season two, a 40.71% simultaneous lower-bound lift, and a 43.87% frozen out-of-time replay lift in season three. The framework reduces a 19-policy catalog to a two-policy validation shortlist while certifying non-harm across 44 advertiser, exchange, and region segments. The results support the central claim that offline reserve-policy evaluation should produce certified validation decisions rather than point-estimate rankings alone.

Prashant Shekhar, Caroline Howard

Department of Mathematics Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

Off-Policy Evaluation Online Advertising Dynamic Pricing Real-Time Bidding

Ask PDF

Abstract

Online Advertising

Explore similar work

Generative Optimization for Incentivized Advertising with Global Level Constraints

JD-BP: A Joint-Decision Generative Framework for Auto-Bidding and Pricing

Support-aware offline policy selection for advertising marketplaces