Modern computational advertising platforms typically rely on recommendation systems to predict user responses, such as click-through rates, conversion rates, and other optimization events. To support a wide variety of product surfaces and advertiser goals, these platforms frequently maintain an extensive ecosystem of machine learning (ML) models. However, operating at this scale creates significant development and efficiency challenges. Substantial engineering effort is required to regularly refresh ML models and propagate new techniques, which results in long latencies when deploying ML innovations across the ecosystem. We present a large-scale empirical study comparing model performance, efficiency, and ML technique propagation between a standardized model-building approach and independent per-model optimization in recommendation systems. To facilitate this standardization, we propose the Standard Model Template (SMT) -- a framework that generates high-performance models adaptable to diverse data distributions and optimization events. By utilizing standardized, composable ML model components, SMT reduces technique propagation complexity from O(n⋅2k) to O(n+k) where n is the number of models and k the number of techniques. Evaluating an extensive suite of models over four global development cycles within Meta's production ads ranking ecosystem, our results demonstrate: (1) a 0.63% average improvement in cross-entropy at neutral serving capacity, (2) a 92% reduction in per-model iteration engineering time, and (3) a 6.3× increase in technique-model pair adoption throughput. These findings challenge the conventional wisdom that diverse optimization goals inherently require diversified ML model design.
Figures & tables
Figure 1: The conventional model iteration process at Meta. For n models and k candidate techniques, this per-model development cycle yields an O(n⋅2k) search complexity. For each model, engineers conduct trial-and-error hyperparameter optimization across all k techniques, isolate the subset that improves performance, and evaluate all possible combinations of these techniques (up to 2k ) before selecting the final candidate for deployment.
Figure 2: The template defines a modular, layered architecture, standardizing components from feature selection and representation across different feature types (numerical, categorical, embedding, sequence, etc.), through feature interaction, to prediction. Components at each layer have standardized interfaces allowing composition and substitution, enabling template evolution as state-of-the-art changes.
Paradigm
Scale Variants
Use Case
Single Tower
Small, Medium, Large
Final stage ranking
Two Tower
Small, Medium, Large
Early stage ranking
Table 1: SMT supports both single-tower and two-tower architectures of varying sizes to support a wide array of use cases and model sizes across ranking and retrieval stages.
Figure 3: The Small, Medium, and Large templates provide comprehensive coverage across a wide variety of ranking and retrieval models across a wide range of model FLOPs.
Figure 4: Sample per-technique group-average NE delta correlation between the Representative Model Set (RMS, x-axis) and the held-out Test Set (y-axis) for the best splitting strategy. Each point represents one technique. Alignment with the dashed y=x line indicates that the technique NE gains observed on the RMS are predictive of those on the test set.
Symbol
Description
Tj∈T
The j -th candidate ML technique
Mi∈MR
The i -th model in the representative set
θh
A hyperparameter configuration for technique Tj
Pibase∈R
Baseline performance of model Mi (higher = better)
Pi,j,h∈R
Performance of Mi with tech. Tj and hyperparameter config θh
Δi,j,h
Performance Delta: Pi,j,h−Pibase
Table 2: Notation for technique generalization.
Figure 5: Multi-Model Optimization (MMO): ML techniques are jointly optimized across multiple representative models using Bayesian Optimization for hyperparameter tuning and aggregation of metrics across models, producing an optimal set of hyperparameters θh .
Figure 6: During template iteration, eligible techniques undergo Multi-Model Optimization. Techniques meeting selection criteria are codified into the Standard Model Template with version control for backward compatibility.
Figure 7: The model iteration process in a standardized environment. By eliminating the need for per-model customization, each model simply instantiates the latest template version to seamlessly incorporate a suite of previously generalized ML techniques.
Period
Model Share (%)
Mean NE (%)
Max NE (%)
Cycle 1
14.2
0.36
0.95
Cycle 2
20.1
0.56
1.12
Cycle 3
19.8
0.68
1.38
Cycle 4
29.4
0.78
1.94
Total
51.8
0.63
1.94
Table 3: Offline performance improvement measured across four development cycles indicates cumulative average NE improvement of 0.63% with consistent improvement across 51.8% of models in the production ecosystem.
Period
Model Share (%)
Online Lift (%)
Cycle 1
14.2
0.11
Cycle 2
20.1
0.19
Cycle 3
19.8
0.27
Cycle 4
29.4
0.29
Total
51.8
0.86
Table 4: Online metric lift is consistent and increasing across cycles, totaling 0.86% across 4 such development cycles.
Metric
SMT
Custom
Techniques shipped per model (average)
1.26X
X
Models reached per cycle (average)
5Y
Y
Total (technique, model) pairs
6.3XY
XY
Table 5: Technique propagation throughput. SMT demonstrates a 6.3× increase in total ⟨technique, model⟩ pairs shipped compared to independently optimized baselines. This surge is largely attributable to a 5× increase in the number of models iterated.