Modern recommendation systems face substantial memory pressure, largely from the sparse embedding layer that maps high-cardinality categorical features into dense vectors. In-training embedding pruning (ITEP) removes less important embedding rows during training, yielding end-to-end memory savings but sometimes degrading accuracy because per-table pruning ratios are set by size-based heuristics. We use feature importance (FI) to guide these ratios (FI-ITEP), assigning higher pruning to tables tied to less important features. Across three large-scale recommendation models, FI-ITEP achieves 20--40% total memory reduction while matching or improving accuracy relative to percentile-based ITEP. Because the feature-importance signal requires minimal resources to compute, these memory savings translate into concrete systems benefits: the same models train on fewer GPU trainers (e.g., 112 to 80) with higher per-GPU throughput, saving 1,000+ GPU-hours per model launch. We reproduce these trends on the open-source DLRM (Criteo 1TB) and quantify the end-to-end training-time and GPU-cost tradeoff---including the burden of computing FI---offering practical guidance for embedding pruning in real-world systems.
Figures & tables
Model
# of Sparse Features
Raw Emb Size
# of GPUs
FI-ITEP (Shuffle)
(Approximate Value)
(Approximate Value)
Memory Saving
Avg. Eval NE Gain (%)
Rec. Model 1
555
390 GB
32
≈ 13.6%
36.4%
Rec. Model 2
1012
1.1 TB
128
≈ 14.2%
51.3%
Rec. Model 3
423
80 GB
32
≈ 8.3%
32.0%
Table 1: A summary of the impact of FI-ITEP on DLRMs of varying sizes. The "Memory Saving" and "Avg. Eval NE Gain (%)" columns indicate relative percentage improvements over the percentile-based ITEP for these models.
Figure 1: Example pruning schedules for Recommendation Model 1 under the percentile, shuffle-FI, and SVS-FI strategies; color encodes each table’s assigned pruning ratio. Embedding tables with low feature importance above a predefined minimum table size are assigned higher pruning ratios.
Memory Reduction
Batches
FI-ITEP AUROC
Percentile-ITEP AUROC
Δ AUROC
50%
200
0.6395
0.6176
+0.0218
50%
1000
0.6920
0.6851
+0.0068
80%
200
0.6268
0.6223
+0.0045
80%
1000
0.6932
0.6932
+0.0001
Table 2: FI-ITEP vs. Percentile-ITEP on the open-source DLRM (Criteo 1TB) at matched memory reduction. “Memory Reduction” is the fraction of sparse-embedding memory removed.
Configuration
Memory
Training Time
Speedup vs. Baseline
Baseline
1,476 MB
130.7s
—
FI-ITEP 50%
738 MB
107.1s
18% faster
Percentile-ITEP 50%
738 MB
107.7s
18% faster
FI-ITEP 80%
295 MB
93.4s
29% faster
Percentile-ITEP 80%
295 MB
112.2s
14% faster
Table 3: Training-time speedup on the open-source DLRM (1000 batches) at matched memory footprint.
Metric
Baseline (Unpruned)
FI-ITEP (96 Trainers)
FI-ITEP (80 Trainers)
Number of trainers (GPUs)
112
96
80
P90 Global QPS
985K
879K
760K
P90 QPS/GPU
8,800
9,161
9,501
P90 Per-GPU QPS %
0%
4%
8%
Equivalent GPU-Hour Saving
0 hr
54 hr
108 hr
GPU Max Utilization
75%
80%
93%
Table 4: End-to-end GPU savings with FI-ITEP via trainer reduction on an industry-scale model.
Metric
Model 1
Model 2
Model 3
Total Sparse Features
555
1012
423
Total training data
55 GB
80 GB
50 GB
FI data sampled
1–2 GB
1–2 GB
1–2 GB
FI computation time
1h 37m
1h 19m
45m
Full run duration
38h
11h 58m
15h 20m
Total FI overhead
4.9%
11%
4.3%
Table 5: Feature Importance (FI) computation overhead across industry-scale models.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Example feature importance values computed using shuffle and Shapley Value Sampling (SVS) for Recommendation Model 1. A floor is applied to raise all zero feature importance values to the minimum nonzero feature importance value.
Model 1
Model 2
Model 3
Metric
1 mo
2 mo
1 mo
2 mo
1 mo
2 mo
Top-100 FI score correlation (Spearman)
0.90
0.78
0.85
0.67
0.92
0.81
% change in top-100 features
9.5%
21.2%
11.8%
34.1%
8.9%
20.5%
Appendix
Table 6: FI score drift (Shapley sampling) across one- and two-month retraining intervals.
Early ranking stages in recommendation systems precompute item embeddings and cache them in-model for scoring within strict latency constraints. Because this cache exists only at serving time, outside the training loop, training and serving use different item representations, a structural discrepancy that limits quality and adds operational fragility. We show that co-designing the training and serving paths removes this representation discrepancy at its source. We introduce the memory layer, an in-model key-value embedding cache co-trained with the model: the item tower writes embeddings during training and the model reads them at serving, one source of truth for item representations by construction. Always-on embeddings cover items not yet cached, so every item receives a prediction, and the design consolidates three separate trainer-to-predictor update paths into a single self-contained pipeline. Deployed in production on Instagram Reels, the memory layer raises prediction coverage from 96% to 100%, improves embedding freshness from O(5 min) to O(20 s), and narrows the training-serving Normalized Entropy (NE) gap by up to 86%, yielding over 2× recall for the freshest content and a 5-6% cold start engagement lift. Because embeddings are produced during training, the system needs no separate bulk-evaluation or publish-time recomputation, cutting training-and-publish computational cost by 30% at neutral serving computational cost.
Large-scale recommendation systems face "Memory Wall" bottlenecks due to massive, dense embedding tables. While generative retrieval uses discrete tokens for IDs, high-dimensional context still relies on inefficient dense formats. Inspired by computer vision data compression, we propose Dual-purpose Semantic IDs to achieve LLM-level I/O efficiency. Our methodology uses hierarchical quantization to condense continuous embeddings into discrete Semantic IDs performing two concurrent roles: (1) Collaborative Identity: modeling user-item interactions via learnable embedding table; and (2) Content Reconstruction: using a lightweight Semantic Decoder for on-the-fly embedding approximation. This approach replaces massive vector storage with on-demand reconstruction, reducing system overhead and data footprints. We demonstrate the efficacy of our framework through offline evaluations and successful online deployment in production-scale ranking and retrieval systems at a major video sharing platform, showing that discrete tokens are indeed all you need for highly efficient, content-rich recommendation.
Recommender systems have advanced markedly over the past decade by transforming each user/item into a dense embedding vector with deep learning models. At industrial scale, embedding tables constituted by such vectors of all users/items demand a vast amount of parameters and impose heavy compute and memory overhead during training and inference, hindering model deployment under resource constraints. Existing solutions towards embedding compression either suffer from severely compromised recommendation accuracy or incur considerable computational costs. To mitigate these issues, this paper presents BACO, a fast and effective framework for compressing embedding tables. Unlike traditional ID hashing, BACO is built on the idea of exploiting collaborative signals in user-item interactions for user and item groupings, such that similar users/items share the same embeddings in the codebook. Specifically, we formulate a balanced co-clustering objective that maximizes intra-cluster connectivity while enforcing cluster-volume balance, and unify canonical graph clustering techniques into the framework through rigorous theoretical analyses. To produce effective groupings while averting codebook collapse, BACO instantiates this framework with a principled weighting scheme for users and items, an efficient label propagation solver, as well as secondary user clusters. Our extensive experiments comparing BACO against full models and 18 baselines over benchmark datasets demonstrate that BACO cuts embedding parameters by over 75% with a drop of at most 1.85% in recall, while surpassing the strongest baselines by being up to 346X faster.
Runhao Jiang, Renchi Yang, Donghao Wu
Hong Kong Baptist University · The Chinese University of Hong Kong