Stale recommendations are a pervasive challenge and a leading source of user complaints on large-scale content platforms. Items lose relevance through two primary mechanisms: supersession, where emerging updates render prior coverage stale, and relevance decay, where an item's informational value naturally diminishes over its lifecycle. Traditional countermeasures serve as crude proxies: age cutoffs poorly reflect actual relevance loss, while engagement heuristics rely on lagging signals, broadly exposing users to stale content before the system adapts. We present SDF (Supersession-Decay Filtering), a staleness filtering system fully deployed in Google Discover, a personalized recommendation feed with hundreds of millions of daily and billions of monthly active users. SDF targets both mechanisms with complementary filters, each powered by a learned model: a relational staleness model that detects supersession between item pairs, and a predicted traffic ratio (PTR) model that forecasts relevance decay from the item's content, trained on lifetime visit traffic. Applied via disjunction upstream of the ranking stage, SDF prunes stale candidates, measurably reducing downstream serving costs. Online experiments demonstrate that these filters significantly reduce the prevalence of stale content while improving user engagement. Over a two-year production deployment, user-filed staleness reports (in-product user feedback) declined by 54.9% relative to the pre-deployment baseline, establishing SDF as a robust and scalable paradigm for resolving content staleness at industrial scale.
Figures & tables
Figure 1. Two staleness mechanisms. Top: Supersession. Each new update on the Ohtani signing renders prior coverage stale. Bottom: Relevance decay. Time-sensitive items naturally lose value and become stale as they near the end of their useful lifespan, without any superseding counterpart. Two illustrations of content staleness. The top panel is a supersession timeline in which successive news items about Shohei Ohtani's team signing are placed along a left-to-right time axis, with arrows marking that each later item renders the earlier coverage stale. The bottom panel shows user-visit-over-time curves illustrating relevance decay: a time-sensitive item spikes sharply and then goes stale within a day, while a slower-decaying item rises and falls over a longer period before going stale, each marked stale where its traffic tails off.
Figure 2. SDF end-to-end workflow. A candidate item dt (arriving at time t ) is evaluated in parallel. Top: Relational staleness filter for supersession. A student model predicts pairwise staleness, yielding Ssupersession(dt) . Bottom: Intrinsic staleness filter for relevance decay. A multimodal model predicts per-horizon traffic ratios from the item’s content, yielding Sdecay(dt;t′) for decision time t′ . The two outputs are OR-fused into S(dt;t′) ; items flagged ( S(dt;t′)=1 ) are filtered, while the rest remain eligible for recommendation and pass to downstream serving. Block diagram of the SDF end-to-end workflow with two parallel branches evaluating a candidate item. The top branch is the relational staleness filter: a student model classifies whether newer topic-related items render the candidate stale, yielding a supersession indicator. The bottom branch is the intrinsic staleness filter: the item's text, image, and video are combined in a multimodal encoder and passed to a multi-task PTR network that predicts a series of per-horizon traffic ratios; these are then combined into a decay indicator at the current decision time. The two indicators are OR-fused into a joint staleness indicator that determines whether the item stays eligible for recommendation or is excluded from the candidate set.
Acc.
Prec.
Rec.
F1
Disc
83.71
83.68
85.15
84.41
Disc + NLI
83.86
84.25
84.65
84.45
Gen
83.37
81.31
88.14
84.59
Gen + NLI
84.16
84.31
85.28
84.79
Table 1. Relational staleness offline evaluation. Disc (discriminative): class-only supervision; Gen (generative): CoT-style rationale supervision; NLI: auxiliary pair-relational training. Per column, bold marks the best value and underline the second best.
Article R: The Oakland Athletics are looking to continue their winning streak against the struggling Texas Rangers…
Article A: Rangers hope to have stabilized following series victory over Oakland. The Texas Rangers came away with a series victory …
Output: Article R is about the Oakland Athletics’ goal of winning a series against the Texas Rangers. Article A reports on the outcome of that series , making Article R outdated. Therefore, Stale .
Article R: Fixed rate cliff looms over Australian housing market. New CoreLogic report sheds light on emerging risk…
Article A: CoreLogic has released a new report unpacking how the housing market has evolved in the three years since COVID was declared a global pandemic.…
Output: Both articles discuss the Australian housing market. Article R focuses on the fixed rate cliff, while Article A provides an overview of the market’s performance over the past three years. Since Article A does not mention the fixed rate cliff , it does not make Article R stale. Therefore, Fresh .
Table 2. CoT-style decoded outputs from the relational staleness filter. “Article R” is the relevant existing item; “Article A” is the arriving new item.
Figure 3. PTR stale-item classification quality at varying rating-majority cuts; the deployed classifier ( θ=0.9 ) is held fixed. All metrics rise because tighter agreement isolates higher-confidence cases. Grouped bar chart of PTR stale-item classification quality, reporting accuracy, precision, recall, and F1 at three rating-agreement cuts: Majority (at least 3 of 5 raters), Strong (at least 4 of 5), and Unanimous (5 of 5). Within each metric the bars increase as agreement tightens; F1 rises from about 78 percent at the Majority cut to about 88 percent at the Unanimous cut, and precision rises from about 88 percent to about 97 percent.
Filter
Promoted
Demoted
Δstale
Baseline
4.00±0.86%
4.10±0.87%
−0.10±1.22%
Relational
1.68±0.81%
5.87±2.11%
−4.19±2.25%
Intrinsic
7.90±1.67%
11.00±1.94%
−3.10±2.56%
SDF
4.33±0.89%
11.24±2.04%
−6.91±2.23%
Table 3. Prevalence delta of SDF and its component filters individually, alongside an age-cutoff + engagement-cliff baseline. Negative Δstale means the promoted set is less stale than the demoted set. For Δstale , bold marks the best result.
Figure 4. User-filed staleness reports over the two-year SDF deployment in Discover, by mechanism: total (blue), supersession (red), decay (yellow). Top: 7-day rolling average. Bottom: 28-day rolling average. Two line charts of user-filed staleness report volumes over the two-year SDF deployment, each plotting three curves: total reports (blue), supersession reports (red), and decay reports (yellow), against time on the horizontal axis. The top chart shows the 7-day rolling average and the bottom chart the 28-day rolling average. All three curves trend downward over the window, with the total and supersession curves declining most, indicating fewer staleness reports after deployment.
Increasingly, recommender systems are tasked with improving users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a bandit problem with delayed rewards. There is an apparent trade-off in choosing the learning signal: waiting for the full reward to become available might take several weeks, slowing the rate of learning, whereas using short-term proxy rewards reflects the actual long-term goal only imperfectly. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Rewards as well as shorter-term surrogate outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that quickly learns to identify content aligned with long-term success using this new predictive model. We prove a regret bound for our algorithm that depends on the Value of Progressive Feedback, an information-theoretic metric that captures the quality of short-term leading indicators that are observed prior to the long-term reward. We apply our approach to a podcast recommendation problem, where we seek to recommend shows that users engage with repeatedly over two months. We empirically validate that our approach significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards, as demonstrated by an A/B test in a recommendation system that serves hundreds of millions of users.
Kelly W. Zhang, Thomas Baldwin-McDonald, Kamil Ciosek +2
Imperial College London · University of Manchester · Spotify +2
In this paper, we propose a new solution for addressing the content cold-start problem in industry-scale search and recommender systems. Compared to prior approaches, we have made the following new contributions: 1) our solution spans the entire multi-stage funnel and generalizes well for both search and recommendation surfaces, 2) our solution reduces bias favoring existing content, allowing more accurate model prediction across content types and reducing short-term tradeoffs associated with high volumes of explicit content exploration, 3) our solution is evaluated with a scalable measurement framework that enables fast short-term experimentation while validating long-term impact. We have iteratively built and successfully deployed this new system at Pinterest in the past two years and observed significant improvements in fresh content exploration, overall user engagement, and content ecosystem health.
In search and recommendation systems, predictive models often suffer from temporal instability when certain input features introduce volatility in output scores. This instability can degrade model reliability and user experience especially in multi-stage systems where consistent predictions are critical for downstream decision making. We introduce Fortress, a general framework for enhancing model stability and accuracy by identifying and pruning features that contribute to inconsistent prediction scores over time. Fortress leverages historical snapshots temporally partitioned datasets capturing score fluctuations for the same entity across periods and follows a four-step process: (1) collect historical snapshots, (2) identify samples with unstable predictions, (3) isolate and remove instability-inducing features, and (4) retrain models using only stable features. While semantic features from LLMs and BERT-based models improve generalization, they often lack full query or entity coverage. Engagement-based features offer strong predictive power but tend to introduce temporal instability. Fortress mitigates this trade-off by suppressing the volatility of engagement signals while retaining their predictive value leading to more stable and accurate models. We validate Fortress on a query-to-app relevance model in a large-scale app marketplace. Offline experiments demonstrate notable improvements in prediction stability (measured by Coefficient of Variation) and classification performance (measured by PR-AUC).
Milind Pandurang Jagre, Jia Huang, Dayvid V. R. Oliveira +6