Training large-scale recommender systems on dynamic industrial platforms, such as Short-Form Video (SFV) services with billions of daily interactions, faces critical stability challenges. Rapid distribution shifts in user data and increasing model complexity frequently trigger a divergence cascade, typically initiating with gradient explosion, propagating to weight explosion, and culminating in loss explosion. This G-W-L cascade severely impedes model convergence and performance. This paper provides a systematic analysis of this G-W-L cascade within an industrial SFV recommender context. We then propose and evaluate a structured mitigation framework where interventions are explicitly mapped to counteract specific stages of the cascade: gradient (G) control, weight/activation (W) stabilization, and loss (L) shaping. Our contribution provides a structured framework for understanding and tackling training divergence, offering practical insights for deploying robust, continuously updated recommenders in large-scale, non-stationary settings. We empirically validate the effectiveness of these strategies on a major industrial SFV recommendation system.
Figures & tables
Figure 1. The G-W-L Cascade: A sequential failure chain reaction leading to model divergence. External perturbations (data distribution shifts ΔDt or aggressive scaling) trigger gradient explosion ( gt↑ ), inducing large parameter updates ( ∥Δθ∥↑ ) that cause weight explosion ( wt+1↑ ). Inflated model weights produce extreme network outputs, driving loss explosion ( lt+1↑ ) and culminating in complete model divergence.
Design Name
BC T1
BC T2
Reg T1
(AUC % Δ )
(AUC % Δ )
(RMSE % Δ )
Baseline
0.00 %
0.00 %
0.00 %
Treatment Gradient
0.05 %
0.19 %
-0.13 %
Treatment Weight
0.03 %
0.06 %
-0.09 %
Treatment Loss
0.21 %
0.38 %
0.16 %
Treatment All
0.25 %
0.38 %
-0.03 %
Table 1. End-Task Performance: Relative Percentage Change from Baseline. AUC: positive % Δ is better. RMSE: negative % Δ is better.
Design Name
gt (% Δ )
lt (% Δ )
Baseline
0.0 %
0.0 %
Treatment Gradient
-63.7 %
-3.6 %
Treatment Weight
-19.6 %
-7.2 %
Treatment Loss
-66.7 %
-62.5 %
Treatment All
-83.6 %
-64.8 %
Table 2. Internal Training Dynamics: Relative Percentage Change from Baseline. gt is the L2 norm of non-embedding parameter gradients (before clipping); lt is the total training loss.
Large-scale recommender systems, particularly short-form video platforms, are often bottlenecked by massive popularity feedback loops. In such environments, as models recommend popular items, they generate an overwhelming amount of skewed training data for "head" items. This creates a self-reinforcing cycle where retrieval and ranking models memorize "head" item patterns at the expense of generalizing across the vast "tail" of the catalogue. While Curriculum Learning (CL) offers a powerful mechanism to break this feedback loop by systematically exposing models to progressively more difficult and less frequent examples, its adoption in industrial recommendation has been hampered by hardware utilization inefficiencies or the needs for complicated pre-processing techniques because dynamic data rejection algorithms tend to starve hardware accelearators (TPUs/GPUs) by becoming largely CPU-bound. In this work, we introduce a scalable Soft Curriculum Learning framework designed specifically for continuous training setups within industry-scale retrieval and ranking models. By utilizing loss annealing and in-graph weight adjustments rather than rigid data filtering, we break the popularity feedback and enable dynamic curriculum pacing without sacrificing system throughput. We demonstrate empirical evidence through applications across sequence-based retrieval models (such as SASRec), two-tower retrieval models, and large-scale continuous ranking models. Online A/B tests on our short-video platform demonstrate substantial lifts in both overall user satisfaction and fresh content consumption, all without degrading model throughput.
Arnab Bhadury, Siyan Zheng, Anlan Yu +6
Google Vancouver, BC, Canada · Google Mountain View, CA, USA
Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.
Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational resources. We study training failures in a 5M-parameter NanoGPT model scaled via μP, identifying two key phenomena preceding collapse: (1) rapid decline in weight matrix stable rank (ratio of squared Frobenius norm to squared spectral norm), and (2) increasing alignment between adjacent layer Jacobians. We prove theoretically that these two conditions jointly cause exponential gradient norm growth with network depth. To break this instability mechanism, we propose MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank. Experiments on models from 5M to 3B parameters demonstrate that MSign effectively prevents training failures with a computational overhead of less than 7.0%.