Search-Aware On-Policy Self-Distillation with Sibling Alignment for Generative Recommendation
Authors: Rongfeng Guo, Zhuang Zhuang, Ji Dai, Peng Zhao, Shijie Li, Jie Chen, Teng Long, Vincent Tao Hu, Fei Pan
Organizations: Kuaishou Technology, Beijing, China · University of Amsterdam, Amsterdam, Netherlands · Huazhong University of Science and Technology, Wuhan, China
Generative recommendation retrieves items by autoregressively generating semantic IDs, typically via beam search. On-policy self-distillation (OPSD) enhances training supervision by transferring privileged self-teacher knowledge at student-generated states. However, beam-search retention depends on competition among prefixes: visited states differ in importance, and initially reliable decisions may approach the pruning boundary as training evolves. Moreover, conventional distribution matching couples target-token confidence with preferences among competing continuations, potentially weakening relational supervision when the self-teacher strongly favors the target. To address these challenges, we propose Search-Aware On-Policy Self-Distillation (SA-OPSD), a framework that transfers privileged self-teacher knowledge to the states visited during the student's own beam search. Specifically, we introduce Evolving-Boundary Sampling, which prioritizes search-critical states by considering both current pruning-boundary proximity and initial-reference retention-margin decline under a controlled distillation budget. Furthermore, we develop Confidence-Decoupled Branch Distillation, which separates target-probability calibration from informative sibling relations to better preserve the teacher's preferences among competing branches. Experiments on Industrial, Office, and Instruments demonstrate strong performance for SA-OPSD.
Figures & tables
Figure 1. Path-wise and student beam-search distillation. (a) Two illustrative path-wise settings supervise the ground-truth SID path or a single student-sampled path. (b) SA-OPSD selects GT-prefix and non-GT-prefix parents from competing beam-search states. Scores and highlighted parents are illustrative. EBS sampling is defined in Sec. 3.3 . Two-part comparison of path-wise distillation and student beam-search states. The left part displays target-path and single-trajectory supervision; the right part shows beam width two, cumulative prefix scores, retained and pruned candidates, and selected parent states.
Figure 2. Overview of SA-OPSD. Only the student performs beam search. EBS selects parent states using current pruning-boundary proximity and reference-margin decline. A frozen privileged self-teacher receives the same student prefix and the ground-truth item title, and CDBD transfers its conditional branch preferences. The student generates beam-search states. EBS selects parent states, then the privileged self-teacher evaluates legal next-token continuations. CDBD applies relational and target-mass supervision at GT-prefix states and full KL at non-GT-prefix states before the student update.
Statistic
Industrial
Office
Instruments
#Users
7,694
8,328
7,537
#Items
3,685
3,459
3,399
#Interactions
50,354
54,226
57,482
Train samples
36,259
38,924
39,956
Valid samples
4,532
4,866
4,994
Test samples
4,533
4,866
4,995
Table 1. Statistics of the three datasets.
Method
Industrial
Office
Instruments
HR@5
NDCG@5
HR@10
NDCG@10
HR@5
NDCG@5
HR@10
NDCG@10
HR@5
NDCG@5
HR@10
NDCG@10
Sequential recommenders
GRU4Rec
0.0774
0.0598
0.0999
0.0669
0.0789
0.0595
0.1019
0.0669
0.0661
0.0538
0.0871
0.0605
Caser
0.0717
0.0555
0.0942
0.0628
0.0865
0.0664
0.1093
0.0737
0.0628
0.0510
0.0845
0.0580
SASRec
0.0909
0.0748
0.1088
0.0806
0.0949
0.0805
0.1120
0.0858
0.0793
0.0675
0.0971
0.0733
BERT4Rec
0.0869
0.0709
0.1110
0.0784
0.0912
0.0753
0.1173
0.0836
0.0759
0.0652
0.0980
0.0725
Table 2. Retrieval performance on Industrial, Office, and Instruments (HR/NDCG, higher is better). Best and second-best values in each metric are bold and underlined.
Strategy
Industrial
Office
Instruments
Single Rollout
0.1168
0.1233
0.1093
Multi Rollout
0.1198
0.1266
0.1122
Dense Beam (Full KL)
0.1251
0.1277
0.1149
SA-OPSD (Ours)
0.1263
0.1334
0.1192
Table 3. NDCG@10 under trajectory-based and beam-search-based distillation.
Figure 3. Industrial and Office NDCG@10 ablations. OPD and OPSD + Full KL are reference controls. Base, w/o Floor, w/o Degradation, and SA-OPSD isolate reference-margin decline priority ( β>0 ) and the non-target relation floor ( ωmin>0 ). All four retain the target-mass gate. Grouped bars compare Industrial and Office on a shared NDCG at ten axis. Industrial NDCG at ten: OPD 0.1168, OPSD plus Full KL 0.1185, Base 0.1220, without Floor 0.1235, without Degradation 0.1240, SA-OPSD 0.1263. Office NDCG at ten: OPD 0.1224, OPSD plus Full KL 0.1250, Base 0.1282, without Floor 0.1300, without Degradation 0.1309, SA-OPSD 0.1334.
Sampling Strategy
NDCG@10
Regression (%)
Uniform sampling
0.1216
16.1%
Current boundary only
0.1240
14.5%
Boundary + shuffled reference declines
0.1248
14.4%
EBS (Ours)
0.1263
10.7%
Table 4. State selection on Industrial. Regression (%) is the fraction of initially correct Top-10 cases lost after training.
Figure 4. Industrial search diagnostics. (a) Sibling-order agreement: bars show percentages, and the +3.3 pp and +8.4 pp annotations denote absolute percentage-point gains over Full KL. (b) Depth-four GT-prefix survival: bars show percentages, while the line shows relative gains (%) over Full KL. The left panel shows sibling-order agreement of 61.9, 65.2, and 70.3 percent for Full KL, Gated KL, and CDBD. The right panel shows depth-four target-prefix survival of 26.3, 28.4, and 30.2 percent for Full KL, Base, and SA-OPSD, plus a relative-improvement line of 0.0, 8.0, and 14.8 percent.
Figure 5. Training efficiency. (a) Training Cost Compared with Dense Beam (Full KL). (b) Time per 1k training interactions for OPSD + Full KL and SA-OPSD on Industrial. Panel a shows the Dense Beam (Full KL) configuration at 17.77 GPU-hours with 470,800 teacher-state evaluations, and SA-OPSD at 5.79 GPU-hours with 66,742 evaluations, a 67.4 percent training cost reduction. Panel b shows the per-one-thousand-interaction timing curves near 3.3 minutes for OPSD plus Full KL and SA-OPSD.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Industrial: subsequent first-pruning risk across current retention-margin bins, stratified by low, medium, and high reference-margin decline. Legend values are simple, unweighted averages over the four bins. Three lines display subsequent first-pruning risk across four current-margin bins, from zero to 0.05. Risks decrease with increasing current margin in every reference-margin decline group. High degradation has the greatest risk at each margin bin, followed by medium and low degradation.
Figure 7. Industrial NDCG@10 under one-at-a-time sweeps of (a) the EBS coefficient β and (b) the CDBD relation floor ωmin , with the other parameter fixed. For beta equal to 0, 0.2, 0.4, and 0.8, NDCG@10 is 0.1240, 0.1250, 0.1263, and 0.1253. For omega-minimum equal to 0, 0.05, 0.15, and 0.30, NDCG@10 is 0.1235, 0.1247, 0.1263, and 0.1255.
Generative recommendation models user histories and item catalogs to generate personalized items, lists, or queries rather than only scoring candidates.
Sequential recommendation predicts a user's next interaction from historical behavior. Recent LLM-based methods recast this task as generative recommendation and increasingly adopt reinforcement learning such as GRPO for post-training. However, these methods suffer from two limitations, namely credit-blind optimization, where a sequence-level reward is shared across all generated tokens, and collaborative-blind supervision, where each training signal is derived from a single user and ignores behavioral patterns shared across similar users. To address both issues, we propose Retrieval-Privileged on-policy Self-Distillation (RPSD). For credit-blind optimization, we introduce on-policy self-distillation (OPSD), in which a teacher with privileged knowledge re-evaluates the student's on-policy response, and the token-level confidence difference between the student and teacher reshapes the shared sequence-level advantage into token-specific credit. However, the privileged knowledge in vanilla OPSD is still derived from a single user, leaving collaborative-blind supervision unresolved. We therefore ground the teacher in collaborative behavioral evidence from similar users. Specifically, we use temporally aware optimal transport (OT) to compare interaction trajectories across users by jointly considering item similarity and temporal structure, and further apply diversity-aware selection to retain complementary trajectories as privileged evidence. In this way, on-policy self-distillation provides fine-grained token-level credit, while OT enriches the teacher with structured cross-user evidence. To the best of our knowledge, this is the first work to bring on-policy self-distillation to sequential recommendation and to use cross-user behavior as its privileged signal. Extensive experiments across multiple real-world datasets demonstrate that RPSD achieves state-of-the-art performance.
Dongyue Li, Zebang Cheng, Zhongyi Deng +3
Nagoya University · Shenzhen University · Sichuan Agricultural University +2
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target--student probability mismatches translate into updates. We propose Self-Referenced On-Policy Self-Distillation (SR-OPSD), which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward Rényi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the Rényi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.
Zhuo Sun, Entong Li, Yanlong Zhao +9
Shanghai University of Finance and Economics · Imperial College London · Independent Researcher +6
On-policy distillation is an efficient alternative to reinforcement learning, offering dense token-level training signals. However, its reliance on a stronger external teacher has driven recent work on on-policy self-distillation, where the same model serves as both teacher and student under different prompt contexts. Yet, existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model. This approach often suffers from training instability and can degrade reasoning performance over time. Moreover, self-distillation from the same model with prompt augmentation lacks the exploratory diversity provided by a genuine external teacher. To address these limitations, we move beyond fixed-teacher KL matching and propose Preference-Based Self-Distillation (PBSD), which revisits on-policy self-distillation through a reward-regularized perspective. Instead of directly matching the teacher distribution, we derive a reward-regularized objective whose analytic optimum is a reward-reweighted teacher distribution, yielding a target policy provably superior to the original teacher under this objective. Practically, PBSD optimizes preference gaps between teacher and student samples while maintaining on-policy student sampling. We support this framework with a statistical analysis of the induced preference-learning problem, formally establishing when on policy self-distillation is preferable to learning from an external teacher in our setting. Experiments on mathematical reasoning and tool-use benchmarks across multiple model scales demonstrate that PBSD consistently achieves the strongest average performance among comparable baselines, showing improved training stability over prior self-distillation baselines while preserving token efficiency.