Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective

Yuening Li; Zeyu Wang; Zhen Zhang; Jiayi Liu; Zhimeng Jiang; Yuji Roh; Weikai Xu; Mingyan Gao

cs.IROct 9, 2026

Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective

Authors: Yuening Li, Zeyu Wang, Zhen Zhang, Jiayi Liu, Zhimeng Jiang, Yuji Roh, Weikai Xu, Mingyan Gao,

Organizations: Google

Abstract

Training large-scale recommender systems on dynamic industrial platforms, such as Short-Form Video (SFV) services with billions of daily interactions, faces critical stability challenges. Rapid distribution shifts in user data and increasing model complexity frequently trigger a divergence cascade, typically initiating with gradient explosion, propagating to weight explosion, and culminating in loss explosion. This G-W-L cascade severely impedes model convergence and performance. This paper provides a systematic analysis of this G-W-L cascade within an industrial SFV recommender context. We then propose and evaluate a structured mitigation framework where interventions are explicitly mapped to counteract specific stages of the cascade: gradient (G) control, weight/activation (W) stabilization, and loss (L) shaping. Our contribution provides a structured framework for understanding and tackling training divergence, offering practical insights for deploying robust, continuously updated recommenders in large-scale, non-stationary settings. We empirically validate the effectiveness of these strategies on a major industrial SFV recommendation system.

Figures & tables

Panel of Figure 1 — Figure 1. The G-W-L Cascade: A sequential failure chain reaction leading to model divergence. External perturbations (data distribution shifts $\Delta\mathcal{D}_{t}$ or aggressive scaling) trigger gradient explosion ( $g_{t}\uparrow$ ), inducing large parameter updates ( $\|\Delta\theta\|\uparrow$ ) that cause weight explosion ( $w_{t+1}\uparrow$ ). Inflated model weights produce extreme network outputs, driving loss explosion ( $l_{t+1}\uparrow$ ) and culminating in complete model divergence.

Design Name	BC T1	BC T2	Reg T1
	(AUC % $\Delta$ )	(AUC % $\Delta$ )	(RMSE % $\Delta$ )
Baseline	0.00 %	0.00 %	0.00 %
Treatment Gradient	0.05 %	0.19 %	-0.13 %
Treatment Weight	0.03 %	0.06 %	-0.09 %
Treatment Loss	0.21 %	0.38 %	0.16 %
Treatment All	0.25 %	0.38 %	-0.03 %

Table 1. End-Task Performance: Relative Percentage Change from Baseline. AUC: positive %

\Delta

is better. RMSE: negative %

\Delta

is better.

Design Name	$g_{t}$ (% $\Delta$ )	$l_{t}$ (% $\Delta$ )
Baseline	0.0 %	0.0 %
Treatment Gradient	-63.7 %	-3.6 %
Treatment Weight	-19.6 %	-7.2 %
Treatment Loss	-66.7 %	-62.5 %
Treatment All	-83.6 %	-64.8 %

Table 2. Internal Training Dynamics: Relative Percentage Change from Baseline.

g_{t}

is the L2 norm of non-embedding parameter gradients (before clipping);

l_{t}

is the total training loss.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 521 papers

Soft Curriculum Learning for Optimizing Fresh and Generalized Recommendations
Aug 11, 2026Arnab Bhadury, Siyan Zheng, Anlan Yu +6Popularity Bias Long-Tail Recommendation
Google Vancouver, BC, Canada · Google Mountain View, CA, USA
FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
Apr 27, 2026Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade +17Sequential Recommendation Communication-Efficient Distributed Training
1Meta Inc.
MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
Feb 2, 2026Lianhai Ren, Yucheng Ding, Xiao Liu +2Language Model Pretraining LLM Training
Microsoft SIGMA Team, Microsoft Research.

Aug 11, 2026J/K move · Enter open · S save

Soft Curriculum Learning for Optimizing Fresh and Generalized Recommendations

Arnab Bhadury, Siyan Zheng, Anlan Yu, Palaksh Rungta+5

Google Vancouver, BC, Canada · Google Mountain View, CA, USA

Open paper AskPDF arXiv

Large-scale recommender systems, particularly short-form video platforms, are often bottlenecked by massive popularity feedback loops. In such environments, as models recommend popular items, they generate an overwhelming amount of skewed training data for "head" items. This creates a self-reinforcing cycle where retrieval and ranking models memorize "head" item patterns at the expense of generalizing across the vast "tail" of the catalogue. While Curriculum Learning (CL) offers a powerful mechanism to break this feedback loop by systematically exposing models to progressively more difficult and less frequent examples, its adoption in industrial recommendation has been hampered by hardware utilization inefficiencies or the needs for complicated pre-processing techniques because dynamic data rejection algorithms tend to starve hardware accelearators (TPUs/GPUs) by becoming largely CPU-bound. In this work, we introduce a scalable Soft Curriculum Learning framework designed specifically for continuous training setups within industry-scale retrieval and ranking models. By utilizing loss annealing and in-graph weight adjustments rather than rigid data filtering, we break the popularity feedback and enable dynamic curriculum pacing without sacrificing system throughput. We demonstrate empirical evidence through applications across sequence-based retrieval models (such as SASRec), two-tower retrieval models, and large-scale continuous ranking models. Online A/B tests on our short-video platform demonstrate substantial lifts in both overall user satisfaction and fresh content consumption, all without degrading model throughput.

Abstract

Recommender Systems

Explore similar work

Soft Curriculum Learning for Optimizing Fresh and Generalized Recommendations

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration