Recommendation impressions are a finite resource, hence delivering a recommendation to a user who would discover the content organically yields no incremental value and displaces other recommendations that could. We address this by extending an existing production recommendation model to a causal architecture using holdback data that is already collected as part of routine experimentation infrastructure, requiring no new data collection. A central challenge is that attribution windows differ between treated and holdback observations: treated users are attributed a stream within a short direct-response window, while holdback users are attributed organic streams over a multi-day window. This mismatch makes naive treatment-effect subtraction invalid. We resolve this with a dual-threshold targeting policy that delivers a recommendation only when the probability of a treated stream is high and the probability of organic stream is low. In a production-scale A/B test on millions of Spotify users, this policy reduces recommendation impressions by 7% with no statistically significant reduction in overall recommended content consumption. We further show that joint training with holdback data improves calibration of the treated head relative to the production baseline, and argue this can be taken as evidence that causal models learn more generalisable representations than models trained on observational data alone.
Figures & tables
Figure 1. Recommendation surfaces in Spotify app where the model operates. Recommendation occupies slot indicated by the dashed red box. Screenshot of the Spotify mobile app home screen showing recommendation banners and content cards in the feed.
Figure 2. The baseline production recommendation model: a shared-trunk neural network with task-specific heads, trained exclusively on examples where a recommendation was shown. The primary head predicts p^1(x) . Diagram showing the baseline multi-task recommendation model architecture: a shared trunk neural network feeding into multiple task-specific output heads for stream prediction, click prediction, and other engagement tasks.
Figure 3. The causal recommendation model: the baseline extended with a holdback head. Treated examples update only the treated head and the shared trunk; holdback examples update only the holdback head and the shared trunk. The shared trunk benefits from both treatment conditions, learning representations that capture the interaction between the recommendation and user behaviour. Diagram of the Deep Twin Network causal recommendation model showing the shared trunk with two output heads: a treated head for recommendation-shown predictions and a holdback head for counterfactual organic-stream predictions. Blue arrows indicate treated data flow and orange arrows indicate holdback data flow.
Metric
Δ
95% CI
Recommendation impressions
−7.1%
[−7.2%,−6.9%]
Recommended content consumption
−0.37%
[−0.86%,+0.21%]
Table 1. A/B test: Treatment-Causal vs. Treatment-Model.
Figure 4. Calibration curves for (left) the treated head and (right) the holdback head. Left: the causal treated head (orange) tracks the diagonal more closely than the production baseline (blue), with the largest improvement at higher predicted probabilities. Right: the holdback head is well-calibrated, with minor deviations at the tail attributable to the smaller holdback sample size. Two calibration plots side by side. Left plot shows mean predicted probability vs observed frequency for the causal treated head (orange line) and production baseline (blue line), with the causal head tracking closer to the diagonal. Right plot shows the holdback head calibration curve near the diagonal.
Large-scale targeting and recommendation systems are typically built around predictive scores fed into heuristic or local allocation. When the business goal is incremental impact, as in marketing campaigns, incentives, and notifications, this paradigm systematically misallocates resources toward users who would have acted anyway. We present a decision-centric framework that instead optimizes causal effects under global constraints, aligning three components under a single objective: a causal neural network with a Transformer backbone for individual treatment-effect estimation, a Bayesian neural-bandit layer for uncertainty-aware exploration, and a dual-based large-scale linear-programming layer for constrained allocation. The framework also supports sequential context and multi-outcome, attribute-conditioned scoring through a Transformer encoder and outcome embeddings. We evaluate it with offline simulations on a public bandit dataset, targeted architectural ablations, and an online A/B test on LinkedIn Feed marketing traffic. We also distill production lessons on causal training-data construction and cost and delivery control, which were critical to successful deployment. The end-to-end treatment policy delivered a statistically significant +7.20% lift in the primary long-term-value metric, demonstrating the feasibility of production-scale causal optimization under business constraints.
Predictive models trained on observational data often fail to generalise when deployed, especially when the training data is a product of the system being optimised. Recommender systems are a canonical example: they are trained on interaction logs confounded by the deployed policy, past user behaviour, and platform filtering. As a result, the training distribution differs from the candidate distribution scored at serving time, a gap that makes offline metrics unreliable predictors of online performance. We address the distribution shift problem with a method motivated by causal representation learning (CRL). We propose an information-theoretic disentanglement criterion and prove that its optimum excludes confounder-dependent non-causal information from the treatment representation. We derive a tractable variational lower bound making the criterion optimisable from finite observational data. The scope of our method is narrower than that of much of the CRL literature, in that we target better generalisation under distribution shift, not full identification of latent causal factors. This narrower target makes the method practical, requiring only existing confounded logs, applying to any standard supervised model, and adding no inference-time cost. Our headline evaluation is an A/B test with millions of users on Spotify, applied to a production ranker for personalised playlist generation. A CRL variant matched in capacity to the production baseline performed on par offline but delivered substantial online gains in listener engagement. Complementary evidence on a synthetic dataset with known causal structure and a semi-synthetic variant of the public KuaiRand recommendation dataset shows similar in-distribution performance to the baseline but gains under distribution shift. Across all three settings, adding our causal disentanglement objective yields better out-of-distribution generalisation.
Yorgos Felekis, Michael O'Riordan, Oriol Corcoll +1
University of Warwick · Spotify · Spotify & University College London
Modern recommender systems are typically based on deep learning (DL) models, where a dense encoder learns representations of users and items. As a result, these systems often suffer from the black-box nature and computational complexity of the underlying models, making it difficult to systematically enhance their recommendation capabilities. To address this problem, we propose Probabilistic Residual Learning (PRL), a causal Bayesian recommendation model that models the residual between ground-truth and base predictions, enabling targeted refinement of existing systems. Specifically, PRL (1) probabilistically groups users for localized residual modeling, (2) models domain-level confounders that influence user and item representations, and (3) aggregates cluster-specific residual predictions over the confounders using do-calculus. Experiments demonstrate that our plug-and-play PRL is compatible with various base deep learning recommender systems, improving their performance while automatically discovering meaningful user clusters.
Wenyuan Wang, Yusong Zhao, Zihao Xu +11
Rutgers University · Piscataway, New Jersey, United States · Meta +4