Modern computational advertising platforms typically rely on recommendation systems to predict user responses, such as click-through rates, conversion rates, and other optimization events. To support a wide variety of product surfaces and advertiser goals, these platforms frequently maintain an extensive ecosystem of machine learning (ML) models. However, operating at this scale creates significant development and efficiency challenges. Substantial engineering effort is required to regularly refresh ML models and propagate new techniques, which results in long latencies when deploying ML innovations across the ecosystem. We present a large-scale empirical study comparing model performance, efficiency, and ML technique propagation between a standardized model-building approach and independent per-model optimization in recommendation systems. To facilitate this standardization, we propose the Standard Model Template (SMT) -- a framework that generates high-performance models adaptable to diverse data distributions and optimization events. By utilizing standardized, composable ML model components, SMT reduces technique propagation complexity from O(n⋅2k) to O(n+k) where n is the number of models and k the number of techniques. Evaluating an extensive suite of models over four global development cycles within Meta's production ads ranking ecosystem, our results demonstrate: (1) a 0.63% average improvement in cross-entropy at neutral serving capacity, (2) a 92% reduction in per-model iteration engineering time, and (3) a 6.3× increase in technique-model pair adoption throughput. These findings challenge the conventional wisdom that diverse optimization goals inherently require diversified ML model design.
Figures & tables
Figure 1: The conventional model iteration process at Meta. For n models and k candidate techniques, this per-model development cycle yields an O(n⋅2k) search complexity. For each model, engineers conduct trial-and-error hyperparameter optimization across all k techniques, isolate the subset that improves performance, and evaluate all possible combinations of these techniques (up to 2k ) before selecting the final candidate for deployment.
Figure 2: The template defines a modular, layered architecture, standardizing components from feature selection and representation across different feature types (numerical, categorical, embedding, sequence, etc.), through feature interaction, to prediction. Components at each layer have standardized interfaces allowing composition and substitution, enabling template evolution as state-of-the-art changes.
Paradigm
Scale Variants
Use Case
Single Tower
Small, Medium, Large
Final stage ranking
Two Tower
Small, Medium, Large
Early stage ranking
Table 1: SMT supports both single-tower and two-tower architectures of varying sizes to support a wide array of use cases and model sizes across ranking and retrieval stages.
Figure 3: The Small, Medium, and Large templates provide comprehensive coverage across a wide variety of ranking and retrieval models across a wide range of model FLOPs.
Figure 4: Sample per-technique group-average NE delta correlation between the Representative Model Set (RMS, x-axis) and the held-out Test Set (y-axis) for the best splitting strategy. Each point represents one technique. Alignment with the dashed y=x line indicates that the technique NE gains observed on the RMS are predictive of those on the test set.
Symbol
Description
Tj∈T
The j -th candidate ML technique
Mi∈MR
The i -th model in the representative set
θh
A hyperparameter configuration for technique Tj
Pibase∈R
Baseline performance of model Mi (higher = better)
Pi,j,h∈R
Performance of Mi with tech. Tj and hyperparameter config θh
Δi,j,h
Performance Delta: Pi,j,h−Pibase
Table 2: Notation for technique generalization.
Figure 5: Multi-Model Optimization (MMO): ML techniques are jointly optimized across multiple representative models using Bayesian Optimization for hyperparameter tuning and aggregation of metrics across models, producing an optimal set of hyperparameters θh .
Figure 6: During template iteration, eligible techniques undergo Multi-Model Optimization. Techniques meeting selection criteria are codified into the Standard Model Template with version control for backward compatibility.
Figure 7: The model iteration process in a standardized environment. By eliminating the need for per-model customization, each model simply instantiates the latest template version to seamlessly incorporate a suite of previously generalized ML techniques.
Period
Model Share (%)
Mean NE (%)
Max NE (%)
Cycle 1
14.2
0.36
0.95
Cycle 2
20.1
0.56
1.12
Cycle 3
19.8
0.68
1.38
Cycle 4
29.4
0.78
1.94
Total
51.8
0.63
1.94
Table 3: Offline performance improvement measured across four development cycles indicates cumulative average NE improvement of 0.63% with consistent improvement across 51.8% of models in the production ecosystem.
Period
Model Share (%)
Online Lift (%)
Cycle 1
14.2
0.11
Cycle 2
20.1
0.19
Cycle 3
19.8
0.27
Cycle 4
29.4
0.29
Total
51.8
0.86
Table 4: Online metric lift is consistent and increasing across cycles, totaling 0.86% across 4 such development cycles.
Metric
SMT
Custom
Techniques shipped per model (average)
1.26X
X
Models reached per cycle (average)
5Y
Y
Total (technique, model) pairs
6.3XY
XY
Table 5: Technique propagation throughput. SMT demonstrates a 6.3× increase in total ⟨technique, model⟩ pairs shipped compared to independently optimized baselines. This surge is largely attributable to a 5× increase in the number of models iterated.
The rapid deployment of machine learning systems across cloud, edge, and enterprise environments has brought model optimization to the forefront of systems-engineering. Despite a rich literature spanning quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference-time optimization, practitioners are often left navigating these techniques through heuristics rather than principled methodology. We argue that optimization should be formulated as a constraint-driven, multi-objective engineering decision and introduce a unified framework that characterizes any production deployment along five interacting constraint dimensions: data availability, latency budget, memory budget, accuracy tolerance, and retraining budget. Building on this taxonomy, we synthesize empirical gains reported across the research literature and map them to operational constraints rather than algorithmic categories. To ensure practical relevance, we selected these techniques by reviewing recent literature for methods that report measurable improvements against critical deployment bottlenecks. We propose a prescriptive decision framework and provide optimization pipelines for four representative industrial scenarios to illustrate it in practice. To the best of our knowledge, this work provides one of the first structured attempts to formalize model optimization as a constraint-aware, multi-objective engineering process, synthesizing quantitative evidence from the research literature.
Modern industrial ads ranking stacks are increasingly bottlenecked not by model capacity or training compute, but by the throughput of human ML iteration - the cycles of research, implementation, training, debugging, evaluation, and launch required to surface a single statistically significant improvement. A typical ranking stack contains numerous differentiated models with heterogeneous data, architectures, and infrastructure constraints, and each cycle takes days to weeks of senior engineer attention per model. As a result, techniques that have proven effective on one model diffuse into others slowly and unevenly, leaving substantial recoverable signal unexplored. We present Agentic ML Exploration (A-MLE), an autonomous LLM-agent system that systematically explores ML techniques across a portfolio of ads ranking models. A-MLE decomposes ML iteration into five stages involving hypothesis generation, exploration strategy, experiment execution, result analysis and shared knowledge substrate which are orchestrated by a single agent that invokes domain-specific skills and agentic workflows against a sandboxed execution layer, with human-in-the-loop checkpoints at each stage boundary. We deploy A-MLE across a representative set of large-scale ads ranking models and evaluate it along a tiered capability framework (tool availability, autonomous workflow execution, and open-ended exploration). We further report a controlled cross-LLM study using a fixed agent loop, which surfaces qualitative differences in execution reliability and exploration aggressiveness across the Claude Sonnet, Gemini, and GPT families. We discuss failure modes and the design choices that govern reliability. Our findings suggest that agentic exploration is a practical force multiplier for ML engineers in industrial recommenders, especially for the long tail of models that rarely receive expert attention.
Evaluating candidate changes (hypotheses) to recommendation models requires full-scale training runs consuming thousands of GPU-hours, severely limiting the hypothesis space engineers can feasibly explore. We introduce Predictable Scaling (PS), a systematic framework that exploits empirically verified regularity conditions to predict full-scale model performance from small-scale proxies. PS constructs scaled-down variants via composable scaling functions along distinct resource axes, and projects metrics to the target scale via curve fitting with calibrated uncertainty estimates. We compare three prediction strategies of increasing expressiveness: even the simplest (raw proxy) achieves prediction error within the empirical noise floor of repeated-run variance; the most expressive (Gaussian Process) provides an indicative 15% relative MAE improvement and calibrated uncertainty. Evaluation on two ranking models of increasing complexity---a task-specific model and a foundation model---demonstrates that PS correctly ranks actionable hypotheses across diverse hypothesis types at 79-88% compute reductions.