Recent advances in Large Language Models (LLMs) have inspired a surge of scaling research in industrial search, advertising, and recommendation systems. However, existing approaches focus mainly on architectural improvements, overlooking the critical synergy between data and architecture design. We observe that scaling model parameters alone exhibits diminishing returns, and that the performance degradation caused by complex heterogeneous data distributions is often irrecoverable through model design alone. In this paper, we propose UniScale, a novel co-design framework that jointly optimizes data and architecture to unlock the full potential of model scaling. UniScale includes two core parts: (1) ES3 (Entire-Space Sample System), a high-quality data scaling system that expands the training signal beyond conventional sampling strategies through intra-domain expansion with hierarchical label attribution and cross-domain searchification; and (2) HHSFT (Heterogeneous Hierarchical Sample Fusion Transformer), a novel architecture that effectively models the complex heterogeneous distribution of scaled data via Heterogeneous Hierarchical Feature Interaction and Entire Space User Interest Fusion, thereby surpassing the performance ceiling of structure-only model tuning. Extensive experiments on large-scale industrial datasets demonstrate that UniScale achieves significant improvements and exhibits clear scaling trends. Online A/B tests on a real-world E-commerce search platform confirm that UniScale consistently outperforms strong production baselines, achieving 1.70% and 2.04% increases in user purchase and Gross Merchandise Value (GMV).
Figures & tables
Figure 1. Overall Architecture of Proposed UniScale, which combines Entire Space Sample System (ES³) for high quality data scaling and Heterogeneous Hierarchical Sample Fusion Transformer (HHSFT) for fully incorporating user interests expressed by entire space user behavior.
Figure 2. Overview of the Entire-Space Sample System (ES 3 ). ES 3 constructs a unified, bias-mitigated training set by synergistically integrating samples across the entire user behavior space. The framework comprises two core modules: (i) Intra-domain Sample and Label Expansion; (ii) Cross-domain Sample Searchification.
Figure 3. Overview of Heterogeneous Hierarchical Sample Fusion Transformer (HHSFT) architecture. The HHSFT is designed to exploit model and data scaling in ranking systems through two primary stages. First, the Heterogeneous Hierarchical Feature Interaction (HHFI) stage employs heterogeneous feature attention layers with token-specific projection matrices and FFNs to preserve domain-unique distributions, followed by a global feature attention layer to capture high-order cross-domain interactions. Second, the Entire Space User Interest Fusion (ESUIF) stage comprises: Domain-Routed Expert Fusion (DREF) utilizes sample routing constraints to disentangle shared knowledge from domain-specific patterns; and Domain-Aware Personalized Gated Attention (DAPGA) adaptively modulates cross-domain information transfer via personalized gating.
Stage
Requests
Samples
Click Pos.
Baseline (Search-exposed-only)
1.0 ×
1.0 ×
1.0 ×
+ Unexposed Expansion
1.0 ×
3.0 ×↑
1.0 ×
+ Hierarchical Label Attribution
1.0 ×
3.0 ×
2.0 ×↑
+ Cross-domain Searchification
2.0 ×↑
5.0 ×↑
4.0 ×↑
Table 1. Impact of ES 3 on Training Data Scale
Model
AUC
GAUC
Params(M)
TFLOPs
DLRM-MLP (base)
-
-
15
0.42
DCNv2
+0.08%
+0.01%
24
0.65
AutoInt
+0.26%
+0.14%
150
1.19
HiFormer
+0.54%
+0.49%
170
1.98
Wukong
+0.21%
+0.13%
32
0.94
Rankmixer
+0.38%
+0.32%
140
1.93
Table 2. Performance comparison of ranking models (best values in bold)
Setting
MLP
Transformer
Heterogeneous Attention
HHFI
AUC gain
-
+0.17%
+0.41%
+0.80%
Table 3. Ablation on components of HHFI.
Sample
Setting
AUC
GAUC
HR@5
search
HHFI
-
-
-
HHFI+DREF+DAPGA
+0.02%
+0.01%
+0.02%
ES³
HHFI
-0.43%
-0.24%
-4.76%
HHFI+DREF
+0.19%
+0.25%
+0.39%
HHFI+DAPGA
+0.22%
+0.18%
+0.17%
HHFI+DREF+DAPGA
+0.32%
+0.26%
+0.44%
Table 4. Ablation on components of ESUIF.
Figure 4. AUC gain vs Dense Parameters Scale Ratio
In real-world deployments of large language models (LLMs), balancing inference quality and computational cost has become a central challenge. Existing approaches tackle this trade-off along two largely independent dimensions: model routing, which switches among models of different scales to match request complexity, and test-time scaling (TTS), which adjusts inference-time compute within a fixed model for fine-grained control. However, this decoupled design introduces inherent limitations. Model routing yields coarse-grained, discrete performance changes due to the sparse set of model scales, while single-model TTS often encounters capacity ceilings and exhibits diminishing returns as compute increases. Moreover, treating the two mechanisms separately restricts adaptability in dynamic inference environments. To overcome these limitations, we introduce Unified Inference Scaling (UIS), which unifies model routing and TTS in a single optimization space. Building on this formulation, we propose UniScale, an online framework that models adaptive UIS as a contextual multi-armed bandit problem and learns inference policies via LinUCB. The framework incorporates efficiency-aware learning and cost modeling to ensure stable and scalable optimization over high-dimensional action spaces. Evaluation shows that UniScale effectively exploits the synergy in the UIS space to deliver a fine-grained and consistently better quality-cost trade-off across diverse, dynamic inference scenarios.
Kaiyu Huang, Xingyu Wang, Mingze Kong +6
School of Computer Science and Technology, Tongji University · Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen · School of Data Science, The Chinese University of Hong Kong, Shenzhen +2
Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.
Zhiliang Chen, Sebastian Ament, David Eriksson +4
Meta · National University of Singapore · Atomic Machines
The rapid scaling of large language model training requires distributing GPU resources across multiple data center buildings and regions. We refer to such paradigm as "scale-across" training. As infrastructure expands, the system design space becomes increasingly intricate, encompassing new model architectures, hardware heterogeneity, and evolving communication patterns. Drawing from Meta's production experience, we highlight the complexities of deploying training jobs across a few data centers housing hundreds of thousands of GPUs. To accelerate exploration of the large design space and to enable efficient training for frontier model development, we conduct in-depth characterization of three key design dimensions: parallelism placement, parallelism scheduling, and network layer technologies. We then propose ScaleAcross Explorer, an optimizer that considers the interplay of design dimensions and holistically optimizes scale-across training. Testbed experiments and simulations demonstrate up to 64.62% training speedups over production configuration and up to 37.59% training speedups over the state-of-the-art baseline across a wide range of design points.