Exploiting Feature-invariant Scaling Laws for Efficient A/B Experimentation in Recommendation Systems
Organizations: Meta, National University of Singapore · Meta
Abstract
Researchers and engineers improve recommendation systems by performing A/B experiments with new model architectures, optimization methods, and data features. However, running full-scale experiments (e.g., full-model training with online data) for every feature change is prohibitively computationally intensive, while the result from a single small-scale experiment is often misleading. To address this challenge, we show that modern recommendation models exhibit feature-invariant scaling: the change of a model's performance follows a predictable scaling-law even when different feature changes are applied to the baseline model. We introduce an efficient method that exploits scaling-laws and uses statistical inference to predict (with confidence intervals) the full-scale effect of an A/B experiment in recommendation systems. This enables practitioners to efficiently filter out neutral or regressive feature changes with only small-scale experimentation. We validate our method on three distinct recommendation model families---Wukong, HSTU, and LLM-based recommendation models---across more than 40 architectural, optimization, and data changes. Our method accurately distinguishes beneficial feature changes from neutral and regressive ones while using only a fraction of the compute required for full-scale evaluation. It also identifies several changes that improve HSTU's state-of-the-art performance on MovieLens-20M. Our results demonstrate that scaling laws can be practically exploited to make compute-efficient decisions for improving industrial-scale recommendation systems.
Figures & tables
| Method | |
|---|---|
| Baseline | 0.9949 |
| Dropout | 0.9953 |
| SGD Optimizer | 0.9923 |
| Attention | 0.9876 |
| Wukong | |
|---|---|
| Baseline | 0.9949 |
| SGD Momentum | 0.9923 |
| Adjust Dense LR | 0.9803 |
| Drop One Feature | 0.9927 |
| Drop Multiple Features | 0.9920 |
| Dropout | 0.9957 |
| Feature | HR@10 | HR@50 | HR@200 |
|---|---|---|---|
| Baseline ( Zhai et al., 2024 ) | 0.3567 | 0.6139 | 0.8073 |
| Time Gap Bias | 0.3603 | 0.6186 | 0.8112 |
| Dropout 0.1 | 0.3590 | 0.6183 | 0.8107 |
| Pop Debias | 0.3585 | 0.6182 | 0.8102 |
| Wider Head 48 | 0.3584 | 0.6176 | 0.8097 |
| RMS-Norm | 0.3584 | 0.6175 | 0.8094 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Proposal | Description |
|---|---|
| rating_additive | Adds a learned embedding of the observed rating to each historical item. |
| calendar_time | Adds learned hour-of-day, day-of-week, and calendar-year embeddings. |
| item_age | Adds an embedding of the movie’s age at the time of the interaction. |
| genome_content | Projects the MovieLens Tag Genome vector into the item representation. |
| embed_dim_15x | Increases the embedding and residual-stream width from 256 to 384. |
| extra_block | Adds one HSTU block, increasing the full-scale depth from 16 to 17. |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 564 papers