Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest
Authors: Yichu Zhou, Qinyi Chen, Fei Feng, Jacob Loh, Mehdi Ben Ayed, Lin Yang, Andrei Curelea, Jiajing Xu
Organizations: Pinterest Inc. Seattle, USA · Pinterest Inc. San Francisco, USA · Pinterest Inc. New York, USA · Pinterest Inc. University of California, Los Angeles Los Angeles, USA
Auto-bidding systems in online advertising must balance competing objectives-maximizing budget delivery while minimizing cost per action-across heterogeneous campaigns under non-stationary market conditions. This trade-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns. We present RLBP (Reinforcement Learning for Bidding Parameters), a hybrid architecture deployed at Pinterest that uses RL as a meta-controller to select campaign-specific PID parameters daily while retaining PID for high-frequency bid adjustments. RLBP trains a multi-head Q-network using a relative-lift reward formulation normalized against the production baseline, and performs inference-time fusion that adapts the cost-delivery trade-off to campaign states without retraining. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost-per-click (eCPC) with no revenue degradation, and two months of post-rollout measurements against a holdout also show consistent cost reductions. These results demonstrate that meta-controller RL can successfully augment production control systems at scale.
Figures & tables
Figure 1. Multi-head Q -network architecture.
Figure 2. Reward relative gap distribution comparison between absolute metric values vs. metric lifts. The left panel shows the relative difference between the best and second-best rewards constructed with raw CPC, while the right panel shows the corresponding relative difference by using CPC lift. Two histograms show the relative gap between the best and second-best rewards. Absolute CPC gaps concentrate near zero; CPC lift gaps span more of the interval.
Figure 3. End-to-end training-serving pipeline of RLBP .
Figure 4. Two-layer RL+PID architecture. The RL agent observes campaign state once daily and selects controller gains from a small, bounded set. The PID controller uses those fixed gains for all bid adjustments over the next 24 hours, retaining the existing feedback-control structure.
Reward variant
Revenue lift ( d=0 )
CPC lift ( d>0 )
Absolute reward
0.08%
-2.17%
Relative-lift reward
2.77%
-3.02%
Table 1. Reward formulation ablation. Each row reports IPS lift (%) over the production baseline. Revenue lift is evaluated on the final day of the campaign lifetime ( d=0 ), where the policy prioritizes budget utilization; CPC lift is evaluated on d>0 , where cost efficiency is the primary objective. Bold marks the production choice.
Adaptive function β(d)
Revenue lift
CPC lift
Step
−1.11%
−1.23%
Linear
+0.05%
−0.34 %
Sigmoid
+1.41%
+0.49%
Table 2. Adaptive function ablation. Each row reports IPS lift (%) on a cohort of campaigns tracked across their full last-seven-day lifecycle. Bold marks the production choice.
Figure 5. (a) Three adaptive functions β(d) controlling cost emphasis as a function of lifetime remaining days. (b) Resulting learned policy behavior: fraction of campaigns assigned a cost-reduction action under each adaptive function.
Figure 6. Offline IPS CPC and revenue lift (%) by lifetime remaining days for different adaptive functions, evaluated on campaigns in the last seven days of their lifetime.
Metric
All campaigns
Lifetime-ended campaigns
Revenue
flat
+0.43%
Clicks
+0.19%
+0.25%
eCPC
-0.25%
+0.18%
Table 3. Online A/B results. Bold numbers are statistically significant ( p -value <0.05 ).
Budget level
Historical CPC tier
Metric
Low
High
Low
High
eCPC lift
-0.66%
-0.90%
-0.29%
-0.64%
Clicks lift
+0.02%
+0.81%
+0.12%
+0.10%
Table 4. eCPC and click lifts in online A/B experiments, segmented by budget level and historical CPC tier. Bold numbers are statistically significant ( p -value <0.05 ).
Lifecycle stage
BU
eCPC
CTR
≥7 days remaining
-0.59%
-0.98%
+0.66%
<7 days remaining
+0.60%
-0.72%
+0.00%
Table 5. Post-rollout performance of RLBP relative to the control holdout by two lifecycle stages. BU denotes budget utilization (delivered spend over budget).