Not All Candidates are Created Equal: A Heterogeneity-Aware Approach to Pre-ranking in Recommender Systems

Pengfei Tong; Siyuan Chen; Chenwei Zhang; Bo Wang; Qi Pi; Pixun Li; Zuotao Liu

cs.IRMar 4, 2026

Not All Candidates are Created Equal: A Heterogeneity-Aware Approach to Pre-ranking in Recommender Systems

Authors: Pengfei Tong, Siyuan Chen, Chenwei Zhang, Bo Wang, Qi Pi, Pixun Li, Zuotao Liu

Organizations: ByteDance

Abstract

Most large-scale recommender systems follow a multi-stage cascade of retrieval, pre-ranking, ranking, and re-ranking. A key challenge at the pre-ranking stage arises from the heterogeneity of training instances sampled from coarse-grained retrieval results, fine-grained ranking signals, and exposure feedback. Our analysis reveals that prevailing pre-ranking methods, which indiscriminately mix heterogeneous samples, suffer from gradient conflicts: hard samples dominate training while easy ones remain underutilized, leading to suboptimal performance. We further show that the common practice of uniformly scaling model complexity across all samples is inefficient, as it overspends computation on easy cases and slows training without proportional gains. To address these limitations, this paper presents Heterogeneity-Aware Adaptive Pre-ranking (HAP), a unified framework that mitigates gradient conflicts through conflict-sensitive sampling coupled with tailored loss design, while adaptively allocating computational budgets across candidates. Specifically, HAP disentangles easy and hard samples, directing each subset along dedicated optimization paths. Building on this separation, it first applies lightweight models to all candidates for efficient coverage, and further engages stronger models on the hard ones, maintaining accuracy while reducing cost. This approach not only improves pre-ranking effectiveness but also provides a practical perspective on scaling strategies in industrial recommender systems. HAP has been deployed in the Toutiao production system for 9 months, yielding up to 0.4% improvement in user app usage duration and 0.05% in active days, without additional computational cost. We also release a large-scale industrial hybrid-sample dataset to enable the systematic study of source-driven candidate heterogeneity in pre-ranking.

Figures & tables

Figure 1. Gradient conflicts across negatives of varying difficulty, illustrating the inherent heterogeneity of candidates.

Figure 2. Architecture of HAP’s training sample collection and model training. Samples are drawn from complete requests, including exposed positives and negatives, and negatives from retrieval, pre-ranking , and ranking stages. HAP adopts the DAMR architecture: the lightweight model learns from all sample types via GHCL, scores retrieval candidates, and filters easy ones, while the expressive model trains exclusively on hard samples using a deeper, attention-based network.

Loss	Sample	Difficulty	Grad Norm	Logits	Cosine
BCE	EN	Hardest	1.101	0.322	–
	RN	Harder	0.254	0.232	0.816
	PRN	Easier	0.140	0.191	0.496
	GN	Easiest	0.060	0.173	0.237
InfoNCE	EN	Hardest	1.750	0.326	–
	RN	Harder	0.401	0.235	0.820

Table 1. Comparison of gradient characteristics across negatives of varying difficulty under BCE and InfoNCE losses. Sample types: exposed (EN), ranking (RN), pre-ranking (PRN), and global random (GN).

Figure 3. Model AUC vs. sample difficulty, with "Exposed" for exposed negatives and "Rank-

N

"/ "Prerank-

N

" for negatives ranked after position

N

; smaller

N

implies higher difficulty.

Figure 4. HAP’s online deployment in Toutiao: a two-stage pre-ranking pipeline with a lightweight model for easy candidates and an expressive model for hard ones; samples are grouped and common features are shared for efficiency.

Dataset	Multi-Stage	Complete	Multi-Type	Industrial
	Coverage	Rank Info	Feedback	-Scale
MovieLens ( Harper and Konstan, 2015 )	✗	✗	✗	✗
Amazon ( Ni et al., 2019 )	✗	✗	✗	✗
Yelp ( Asghar, 2016 )	✗	✗	✗	✗
Taobao ( Zhu et al., 2018 )	✗	✗	✓	✓
Tenrec ( Yuan et al., 2022 )	✗	✗	✓	✓

Table 2. Comparisons of ToutiaoRec with existing datasets.

Model	TEN	TRN	TPRN	TGN	THard	TEasy
DSSM	0.6901	0.8247	0.9008	0.9785	0.7527	0.9257
COLD	0.6936	0.8303	0.9038	0.9803	0.7601	0.9261
COPR	0.7221	0.8533	0.9172	0.9871	0.7824	0.9402
HCCP	0.7244	0.8653	0.9218	0.9874	0.7917	0.9441
HAP	0.7377	0.8693	0.9236	0.9878	0.8023	0.9468
$\text{HAP}_{\text{lite}}$	0.7331	0.8666	0.9225	0.9872	0.7998	0.9452

Table 3. Offline AUC comparison of SoTA pre-ranking models on various test sets. The best result is in bold and the second-best result is underline , respectively.

NegType	Model	TEN	TRN	TPRN	TGN	THard	TEasy
SingleNeg	$\text{BCE}_{\text{GN}}$	0.5930	0.6859	0.7717	0.9992	0.6391	0.8680
	$\text{BCE}_{\text{PRN}}$	0.6186	0.7732	0.9616	0.9406	0.6955	0.9502
	$\text{BCE}_{\text{RN}}$	0.6238	0.9017	0.8595	0.8861	0.7623	0.8728
	$\text{BCE}_{\text{EN}}$	0.7591	0.6815	0.7121	0.7815	0.7207	0.7484
	$\text{InfoNCE}_{\text{GN}}$	0.5941	0.6834	0.7677	0.9991	0.6378	0.8634
	$\text{InfoNCE}_{\text{PRN}}$	0.6148	0.7564	0.9531	0.9348	0.6845	0.9487

Table 4. Comparison of negative sampling strategies. For each negative type (GN, PRN, RN, EN), SingleNeg evaluates BCE loss versus the combined InfoNCE+BCE objective. MultiNeg combines all four. Best results in bold and underline .

Figure 5. Ablations of GHCL in HAP: removing GHCL leads to a notable AUC decrease across all test sets.

Model	THard a	TEasy a	CTR c	Compute Cost b	FLOPs ( $10^{6}$ )	Latency ( $ms$ )	Cost Ratio c
Unified Large Model	0.8158	0.9586	+0.55%	38.1	70	225	+15%
HAP	0.8197	0.9579	+3.22%	39.2	89	203	-6%

Table 5. Ablations of DAMR in HAP: Online A/B test results on Toutiao, trained on the large-scale logs.

Panel of Figure 6 — Figure 6. Gradient statistics of heterogeneous negatives. GHCL stabilizes directions and magnitudes, mitigating conflicts compared to existing methods.

Figure 7. Online performance–efficiency trade-off of HAP under varying gating thresholds.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Table 6. Performance and Efficiency Trade-offs of Models with Different Complexities

	Model	TEN	TRN	TPRN	TGN	THard	TEasy
AUC $\uparrow$	DSSM	0.6901	0.8247	0.9008	0.9785	0.7527	0.9257
	COLD	0.6936	0.8303	0.9038	0.9803	0.7601	0.9261
	COPR	0.7221	0.8533	0.9172	0.9871	0.7824	0.9402
	HCCP	0.7244	0.8653	0.9218	0.9874	0.7917	0.9441
	HAP	0.7377	0.8693	0.9236	0.9878	0.8023	0.9468
	$\text{HAP}_{\text{lite}}$	0.7331	0.8666	0.9225	0.9872	0.7998	0.9452

Appendix

Table 7. Comparative Evaluation of HAP Against Industrial Recommender Baselines. The best results are highlighted in bold and underline . Metrics with

\uparrow

indicate higher is better, while

\downarrow

indicates lower is better.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 508 papers

Feb 24, 2026

Generative Pseudo-Labeling for Pre-Ranking with LLMs

Pre-ranking is a critical stage in industrial recommendation systems, tasked with efficiently scoring thousands of recalled items for downstream ranking. A key challenge is the train-serving discrepancy: pre-ranking models are trained only on exposed interactions, yet must score all recalled candidates -- including unexposed items -- during online serving. This mismatch not only induces severe sample selection bias but also degrades generalization, especially for long-tail content. Existing debiasing approaches typically rely on heuristics (e.g., negative sampling) or distillation from biased rankers, which either mislabel plausible unexposed items as negatives or propagate exposure bias into pseudo-labels. In this work, we propose Generative Pseudo-Labeling (GPL), a framework that leverages large language models (LLMs) to generate unbiased, content-aware pseudo-labels for unexposed items, explicitly aligning the training distribution with the online serving space. By offline generating user-specific interest anchors and matching them with candidates in a frozen semantic space, GPL provides high-quality supervision without adding online latency. Deployed in a large-scale production system, GPL improves click-through rate by 3.07%, while significantly enhancing recommendation diversity and long-tail item discovery.

Junyu Bi, Xinting Niu, Daixuan Cheng +4

Alibaba Group Beijing, China · Renmin University Beijing, China

Pseudo-Labeling Recommender Systems

Ask PDF

Jul 30, 2026

Heterogeneous Ranking in Industrial-Scale Recommender Systems: A Case Study

Heterogeneous recommendation feeds present complex challenges that extend beyond those found in highly homogeneous environments (e.g., music-only or video-only closed-ecosystem platforms). In Google Discover, a unified feed integrates diverse content sourced from the decentralized open web, including web articles, long-form and short-form videos, user-generated content (UGC), and beyond. Different content types exhibit distinct feature densities and user interaction patterns. Building a unified ranking model that sustains high performance across such heterogeneity, while avoiding negative transfer or majority bias, remains a significant industrial challenge. This paper presents an end-to-end case study on the industrial-scale multi-task ranking of heterogeneous feeds, grounded in real-world deployment. We introduce HA-MoE, a heterogeneity-adaptive multi-gated mixture-of-experts architecture that incorporates explicit heterogeneity context into both gating networks and expert representations. This approach enables effective specialization without significantly increasing operational overhead. To support reliable deployment, we introduce LENS, a lightweight observability framework that provides interpretable diagnostics of expert specialization and tracks this functional heterogeneity across continuous retraining. We evaluate our method using Dual-Level AUC (DL-AUC), a heterogeneity-aware evaluation metric that combines global ranking performance with cross-segment ranking correctness. Offline evaluations on a large-scale industrial dataset demonstrate consistent improvements over baseline models. Furthermore, online A/B testing confirms gains in feed activity and exploration metrics. Together, offline and online results validate the effectiveness of our approach for managing heterogeneity in industrial-scale recommender systems.

Di Bai, Jintao Liu, Zhenwei Tang +3

Google LLC Mountain View, CA, USA · Google LLC Toronto, ON, Canada · Google LLC New York, NY, USA +1

Learning to Rank Multi-Task Learning Recommender Systems Mixture-of-Experts Models

Ask PDF

Sep 10, 2026

Recommendation Retrievers Need Verifiers: Universal Generative Reranking for Sequential Recommendations

First-stage recommenders in multi-stage systems produce a ranked candidate list from which a limited prefix is forwarded to downstream rankers. Because each forwarded item must be processed by more expensive ranking stages, this shortlist cannot be arbitrarily large. The first-stage objective is therefore high coverage of relevant items within the forwarded prefix, commonly measured by Recall@

k

. A relevant item may be available deeper in the retrieved list but absent from the shorter prefix that is actually consumed. This paper studies post-hoc verification for promoting such candidates into the consumed shortlist without retraining or replacing the retriever. We introduce a lightweight generative verifier for retrieval models. Given a retriever state and a candidate item, the verifier scores the item through the likelihood of its identifier tokens. It is trained post hoc with next-token cross entropy, requires no sampled negatives or candidate pool during training, and scores only the retriever's top-

K

candidates at inference. The interface is minimal: the retriever supplies a query state and candidate items, and the item representation can use any fixed tokenization. Across Amazon product recommendation and YaMBDa music recommendation, the same verifier training recipe improves Recall@10 for SASRec, GRU4Rec, NextItNet, and MiniOneRec. Ablations show that the improvements are not explained solely by injecting item-content features into the retriever, supporting verification as a post-hoc output-side adaptation mechanism.

Benyu Zhang, Qiang Zhang, Rui Li +3

Meta MRS

Sequential Recommendation Recommender Systems

Ask PDF

Abstract

Recommender Systems

Explore similar work

Generative Pseudo-Labeling for Pre-Ranking with LLMs

Heterogeneous Ranking in Industrial-Scale Recommender Systems: A Case Study

Recommendation Retrievers Need Verifiers: Universal Generative Reranking for Sequential Recommendations