Recommendation impressions are a finite resource, hence delivering a recommendation to a user who would discover the content organically yields no incremental value and displaces other recommendations that could. We address this by extending an existing production recommendation model to a causal architecture using holdback data that is already collected as part of routine experimentation infrastructure, requiring no new data collection. A central challenge is that attribution windows differ between treated and holdback observations: treated users are attributed a stream within a short direct-response window, while holdback users are attributed organic streams over a multi-day window. This mismatch makes naive treatment-effect subtraction invalid. We resolve this with a dual-threshold targeting policy that delivers a recommendation only when the probability of a treated stream is high and the probability of organic stream is low. In a production-scale A/B test on millions of Spotify users, this policy reduces recommendation impressions by 7% with no statistically significant reduction in overall recommended content consumption. We further show that joint training with holdback data improves calibration of the treated head relative to the production baseline, and argue this can be taken as evidence that causal models learn more generalisable representations than models trained on observational data alone.
Figures & tables
Figure 1. Recommendation surfaces in Spotify app where the model operates. Recommendation occupies slot indicated by the dashed red box. Screenshot of the Spotify mobile app home screen showing recommendation banners and content cards in the feed.
Figure 2. The baseline production recommendation model: a shared-trunk neural network with task-specific heads, trained exclusively on examples where a recommendation was shown. The primary head predicts p^1(x) . Diagram showing the baseline multi-task recommendation model architecture: a shared trunk neural network feeding into multiple task-specific output heads for stream prediction, click prediction, and other engagement tasks.
Figure 3. The causal recommendation model: the baseline extended with a holdback head. Treated examples update only the treated head and the shared trunk; holdback examples update only the holdback head and the shared trunk. The shared trunk benefits from both treatment conditions, learning representations that capture the interaction between the recommendation and user behaviour. Diagram of the Deep Twin Network causal recommendation model showing the shared trunk with two output heads: a treated head for recommendation-shown predictions and a holdback head for counterfactual organic-stream predictions. Blue arrows indicate treated data flow and orange arrows indicate holdback data flow.
Metric
Δ
95% CI
Recommendation impressions
−7.1%
[−7.2%,−6.9%]
Recommended content consumption
−0.37%
[−0.86%,+0.21%]
Table 1. A/B test: Treatment-Causal vs. Treatment-Model.
Figure 4. Calibration curves for (left) the treated head and (right) the holdback head. Left: the causal treated head (orange) tracks the diagonal more closely than the production baseline (blue), with the largest improvement at higher predicted probabilities. Right: the holdback head is well-calibrated, with minor deviations at the tail attributable to the smaller holdback sample size. Two calibration plots side by side. Left plot shows mean predicted probability vs observed frequency for the causal treated head (orange line) and production baseline (blue line), with the causal head tracking closer to the diagonal. Right plot shows the holdback head calibration curve near the diagonal.