The sparse path, comprising sparse operations and collective communication, is a critical component of large-scale recommendation models. However, these operations are often represented as opaque nodes in the training computation graph, hindering whole-graph optimization. We present SparseLift, a low-overhead compiler for the sparse path of large-scale recommendation training. It exposes sparse computation and communication to whole-graph analysis to expand the optimization space and improve training efficiency. By analyzing model structure and runtime information, SparseLift reconstructs opaque nodes as analyzable subgraphs, recovering the resource semantics of sparse computation and communication within the graph. Building on the recovered semantics, we design incremental cost models and a constraint-aware planner to automatically select and organize subgraph optimizations into an execution plan that is reused during subsequent steady-state training. We evaluate SparseLift on an industrial video-platform recommendation task with 100B sparse parameters and the public DLRM benchmark. SparseLift improves steady-state throughput for ordinary industrial training steps by 16.5% over the native baseline. On DLRM, it achieves 1.60× and 1.26× model-execution speedups over DLRM-Baseline and TorchRec, respectively.
Figures & tables
Figure 1: Schematic of the sparse path and optimization opportunities in the industrial video-platform task. A2A: all-to-all; T: embedding table.
Figure 2: SparseLift architecture: sparse semantic lifting, incremental cost-based decisions, and constraint-aware planning. The resulting execution plan is reused during steady-state training. SRSV: Sparse Resource Semantic View; Ti : Embedding table i .
Optimization
Recovered relation
Feasibility checks
Cost inputs and decision
Communication aggregation
Collectives with matching semantic roles
Communication group; data type; layout mapping; collective ordering
Inputs: per-member traffic; packing/unpacking cost. Choice: roles to aggregate.
Subgraph parallelism
Independent table-local operations
Dependencies; workspace requirements; stream concurrency support
Inputs: per-member workloads; orchestration overhead. Choice: group count G .
Figure 5: Industrial stage breakdown and ablation. (a) Base/Lift: Baseline/SparseLift; error bars: total-step standard deviations over four measurements. (b) Percentages show latency increases upon individual strategy removal from Full.
Figure 6: Industrial throughput on 8 GPUs. Points aggregate nine ordinary steps; bands/dashes show IQRs/medians after step 60.
System
ROC-AUC
AP
LogLoss
MAE
RMSE
DLRM-Baseline
0.7884
0.5832
0.46541
—
—
SparseLift
0.7883
0.5829
0.46575
1.975
3.645
Table 4: DLRM quality after one epoch. Loss MAE/RMSE compare 24 matched steps with DLRM-Baseline, scaled by 105 .
Figure 7: DLRM model-execution performance, reported as five-run medians. (a) SparseLift speedups over DLRM-Baseline and TorchRec. (b) Each system’s single-GPU-relative speedup per GPU. (c) Throughput with 25th–75th percentile error bars.
Figure 8: Industrial training loss. Faint/solid curves show per-step values/25-step centered means.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
Warmup and measured executions
Communication aggregation
One warmup pair, then two measured pairs per compatible role or complete forward chain.
Subgraph parallelism
One warmup per variant, then a bidirectional sweep with two region observations per group count.
Code generation
One warmup per implementation, then complete-region measurements in the order ref→gen→gen→ref .
Latency hiding
One warmup step, then two valid complete-step observations per schedule.
Appendix
Table 5: Reference calibration budgets per measurement point and attempt. Warmup is excluded from observations.
System
25th pct.
Median
75th pct.
Baseline
4260.4
4334.6
4370.3
SparseLift
4982.8
5050.7
5095.8
Appendix
Table 6: Industrial ordinary-step throughput on 8 GPUs, in samples/s, over 94 steady-state periods per system.
GPUs
System
Sparse forward
Other forward
Sparse backward
Other backward
Other
Total step
SD
1
Baseline
26.75
112.50
25.50
213.25
50.75
428.75
0.50
1
SparseLift
2.50
110.75
9.75
214.25
32.50
369.75
1.50
8
Baseline
44.25
113.75
28.00
222.75
42.75
451.50
7.14
8
SparseLift
3.13
108.88
9.13
227.88
29.50
378.50
2.38
16
Baseline
48.00
110.50
39.50
229.00
47.25
474.25
3.40
16
SparseLift
6.00
105.75
23.50
233.25
33.50
402.00
3.56
Appendix
Table 7: Industrial stage timings in ms over four retained observations per row. SD is the sample standard deviation of total step time. Component and total means are rounded independently.
Rank
Incumbent
Lookahead
Increment
0
418.464
383.889
−34.575
1
418.752
384.108
−34.644
2
418.752
383.996
−34.756
3
418.684
384.095
−34.589
4
418.953
383.948
−35.005
5
418.676
384.198
−34.479
Appendix
Table 8: Industrial lookahead calibration in ms. Each rank reports its minimum of two measurements; the aggregate increment is F(T1)−F(T0) .
Configuration
n
Step (ms)
Increase
Baseline
4
451
19.3%
Full
4
378
—
Without aggregation
4
383
1.3%
Without parallelism
4
382
1.1%
Without latency hiding
3
426
12.7%
Without code generation
4
389
2.9%
Appendix
Table 9: Industrial strategy ablation on 8 GPUs. n counts retained step observations; increases are relative to Full and are not additive.
Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.
Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.
Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph's backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to 9.51% while keeping normalized entropy close to the baseline.