SparseLift: Automatically Optimizing Opaque Sparse Paths in Industrial-Scale Recommendation Model Training

Changqing Shi; Anan Liu; Genbao Chen; Junchao Yang

cs.LGNov 1, 2026

SparseLift: Automatically Optimizing Opaque Sparse Paths in Industrial-Scale Recommendation Model Training

Authors: Changqing Shi, Anan Liu, Genbao Chen, Junchao Yang

Organizations: Tencent, Beijing, China.

Abstract

The sparse path, comprising sparse operations and collective communication, is a critical component of large-scale recommendation models. However, these operations are often represented as opaque nodes in the training computation graph, hindering whole-graph optimization. We present SparseLift, a low-overhead compiler for the sparse path of large-scale recommendation training. It exposes sparse computation and communication to whole-graph analysis to expand the optimization space and improve training efficiency. By analyzing model structure and runtime information, SparseLift reconstructs opaque nodes as analyzable subgraphs, recovering the resource semantics of sparse computation and communication within the graph. Building on the recovered semantics, we design incremental cost models and a constraint-aware planner to automatically select and organize subgraph optimizations into an execution plan that is reused during subsequent steady-state training. We evaluate SparseLift on an industrial video-platform recommendation task with 100B sparse parameters and the public DLRM benchmark. SparseLift improves steady-state throughput for ordinary industrial training steps by 16.5% over the native baseline. On DLRM, it achieves $1.60\times$ and $1.26\times$ model-execution speedups over DLRM-Baseline and TorchRec, respectively.

Figures & tables

Figure 1: Schematic of the sparse path and optimization opportunities in the industrial video-platform task. A2A: all-to-all; T: embedding table.

Figure 2: SparseLift architecture: sparse semantic lifting, incremental cost-based decisions, and constraint-aware planning. The resulting execution plan is reused during steady-state training. SRSV: Sparse Resource Semantic View;

T_{i}

: Embedding table

i

Optimization	Recovered relation	Feasibility checks	Cost inputs and decision
Communication aggregation	Collectives with matching semantic roles	Communication group; data type; layout mapping; collective ordering	Inputs: per-member traffic; packing/unpacking cost. Choice: roles to aggregate.
Subgraph parallelism	Independent table-local operations	Dependencies; workspace requirements; stream concurrency support	Inputs: per-member workloads; orchestration overhead. Choice: group count $G$ .
Code generation	Structurally compatible repeated computation	Value/gradient correspondence; address isolation; interface compatibility	Inputs: workload distribution; region execution time. Choice: implementation $v$ .
Latency hiding	Release-to-use relations	Input readiness; parameter version; batch identity; buffer lifetime	Input: complete schedule execution time. Choice: enable/disable.

Table 1: Mapping recovered semantics to optimization feasibility and cost-based decisions.

Figure 3: Lifted graph slice and read-only SRSV. Gray: dataflow; purple: execution constraints; blue: compiler projection or handoff. A2A: all-to-all.

Figure 4: Latency hiding and its execution constraints. Dashed edges denote workspace availability, parameter-version, and result-consumption constraints.

Component	Configuration
GPUs	$8\times$ H20 (96 GiB each)
CPUs	$2\times$ EPYC 9K84 (96 cores each)
Host memory	2.2 TiB
Intra-node interconnect	NVLink
Network port rate	200 Gb/s Ethernet
PyTorch / CUDA	2.6.0 / 12.6

Table 2: Experimental platform specifications per node.

Strategy	Industry	DLRM
Aggregation	size/key/embed/grad	Embeddings
Parallelism	$2/2/1/4$ groups a	Keep ref.
Code generation	Gradient preparation	Keep ref.
Latency hiding	One-step lookahead	Keep ref.
a Input preparation / output assembly / gradient preparation / gradient merging. Generated gradient preparation requires one group.

Table 3: Workload-specific execution choices. Keep ref. denotes retaining the reference execution.

Figure 5: Industrial stage breakdown and ablation. (a) Base/Lift: Baseline/SparseLift; error bars: total-step standard deviations over four measurements. (b) Percentages show latency increases upon individual strategy removal from Full.

Figure 6: Industrial throughput on 8 GPUs. Points aggregate nine ordinary steps; bands/dashes show IQRs/medians after step 60.

System	ROC-AUC	AP	LogLoss	MAE	RMSE
DLRM-Baseline	0.7884	0.5832	0.46541	—	—
SparseLift	0.7883	0.5829	0.46575	1.975	3.645

Table 4: DLRM quality after one epoch. Loss MAE/RMSE compare 24 matched steps with DLRM-Baseline, scaled by

10^{5}

Figure 7: DLRM model-execution performance, reported as five-run medians. (a) SparseLift speedups over DLRM-Baseline and TorchRec. (b) Each system’s single-GPU-relative speedup per GPU. (c) Throughput with 25th–75th percentile error bars.

Figure 8: Industrial training loss. Faint/solid curves show per-step values/25-step centered means.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Warmup and measured executions
Communication aggregation	One warmup pair, then two measured pairs per compatible role or complete forward chain.
Subgraph parallelism	One warmup per variant, then a bidirectional sweep with two region observations per group count.
Code generation	One warmup per implementation, then complete-region measurements in the order $\mathrm{ref}\to\mathrm{gen}\to\mathrm{gen}\to\mathrm{ref}$ .
Latency hiding	One warmup step, then two valid complete-step observations per schedule.

Appendix

Table 5: Reference calibration budgets per measurement point and attempt. Warmup is excluded from observations.

System	25th pct.	Median	75th pct.
Baseline	4260.4	4334.6	4370.3
SparseLift	4982.8	5050.7	5095.8

Appendix

Table 6: Industrial ordinary-step throughput on 8 GPUs, in samples/s, over 94 steady-state periods per system.

GPUs	System	Sparse forward	Other forward	Sparse backward	Other backward	Other	Total step	SD
1	Baseline	26.75	112.50	25.50	213.25	50.75	428.75	0.50
1	SparseLift	2.50	110.75	9.75	214.25	32.50	369.75	1.50
8	Baseline	44.25	113.75	28.00	222.75	42.75	451.50	7.14
8	SparseLift	3.13	108.88	9.13	227.88	29.50	378.50	2.38
16	Baseline	48.00	110.50	39.50	229.00	47.25	474.25	3.40
16	SparseLift	6.00	105.75	23.50	233.25	33.50	402.00	3.56

Appendix

Table 7: Industrial stage timings in ms over four retained observations per row. SD is the sample standard deviation of total step time. Component and total means are rounded independently.

Rank	Incumbent	Lookahead	Increment
0	418.464	383.889	$-34.575$
1	418.752	384.108	$-34.644$
2	418.752	383.996	$-34.756$
3	418.684	384.095	$-34.589$
4	418.953	383.948	$-35.005$
5	418.676	384.198	$-34.479$

Appendix

Table 8: Industrial lookahead calibration in ms. Each rank reports its minimum of two measurements; the aggregate increment is

F(T_{1})-F(T_{0})

Configuration	$n$	Step (ms)	Increase
Baseline	4	451	19.3%
Full	4	378	—
Without aggregation	4	383	1.3%
Without parallelism	4	382	1.1%
Without latency hiding	3	426	12.7%
Without code generation	4	389	2.9%

Appendix

Table 9: Industrial strategy ablation on 8 GPUs.

n

counts retained step observations; increases are relative to Full and are not additive.

Setting	Value
Dense input features	13
Bottom MLP widths	512–256–64–256
Top MLP widths	512–256–1
Feature interaction	Pairwise dot products
Random seed	123

Appendix

Table 10: Additional DLRM settings. MLP widths specify layer outputs, excluding input dimensions.

Compiler Optimization

See all 32 papers

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
Apr 27, 2026Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade +17Sequential Recommendation Communication-Efficient Distributed Training
1Meta Inc.
Sparser, Faster, Lighter Transformer Language Models
Mar 24, 2026Edoardo Cetin, Stefano Peluchetti, Emilio Castillo +3LLM Inference Acceleration Sparse Neural Networks
Sakana AI · NVIDIA
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
Aug 19, 2026Ergan Shang, Flavio Sales TruzziDeep Learning Optimization Backpropagation
Carnegie Mellon University · Meta Inc.

Apr 27, 2026J/K move · Enter open · S save

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade, Yanli Zhao+16

1Meta Inc.

Sequential Recommendation Communication-Efficient Distributed Training Distributed Optimization Distributed Training

Open paper AskPDF arXiv

Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.

Abstract

Compiler Optimization

Explore similar work

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

Sparser, Faster, Lighter Transformer Language Models

ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems