RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport
Authors: Dongyue Li, Zebang Cheng, Zhongyi Deng, Hongboxu, Yi Bin, Fei Ma
Organizations: Nagoya University · Shenzhen University · Sichuan Agricultural University · Guangdong Laboratory of Artificial Intelligence and Digital Economy · Tongji University
Sequential recommendation predicts a user's next interaction from historical behavior. Recent LLM-based methods recast this task as generative recommendation and increasingly adopt reinforcement learning such as GRPO for post-training. However, these methods suffer from two limitations, namely credit-blind optimization, where a sequence-level reward is shared across all generated tokens, and collaborative-blind supervision, where each training signal is derived from a single user and ignores behavioral patterns shared across similar users. To address both issues, we propose Retrieval-Privileged on-policy Self-Distillation (RPSD). For credit-blind optimization, we introduce on-policy self-distillation (OPSD), in which a teacher with privileged knowledge re-evaluates the student's on-policy response, and the token-level confidence difference between the student and teacher reshapes the shared sequence-level advantage into token-specific credit. However, the privileged knowledge in vanilla OPSD is still derived from a single user, leaving collaborative-blind supervision unresolved. We therefore ground the teacher in collaborative behavioral evidence from similar users. Specifically, we use temporally aware optimal transport (OT) to compare interaction trajectories across users by jointly considering item similarity and temporal structure, and further apply diversity-aware selection to retain complementary trajectories as privileged evidence. In this way, on-policy self-distillation provides fine-grained token-level credit, while OT enriches the teacher with structured cross-user evidence. To the best of our knowledge, this is the first work to bring on-policy self-distillation to sequential recommendation and to use cross-user behavior as its privileged signal. Extensive experiments across multiple real-world datasets demonstrate that RPSD achieves state-of-the-art performance.
Figures & tables
Figure 1: RPSD combines token-level credit assignment with retrieved evidence. (a) Conceptual differences between RPSD and GRPO. (b) Training reward and token entropy of GRPO and RPSD. (c) Comparison between RPSD and RPSD w/o OT, which only takes the ground-truth item as privileged knowledge of the teacher.
Figure 2: Overview of RPSD. The student samples on-policy responses from the original recommendation prompt, while a retrieval-privileged teacher re-scores the same trajectory using behavioral evidence only during training to derive token-level credit.
Model
MovieLens-1M
MicroLens
Netflix
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
Non-LLM
GRU4Rec ( Hidasi et al., 2016 )
0.5507
0.7131
0.4762
0.5236
0.5080
0.6440
0.4369
0.4756
0.5152
0.7152
0.3659
0.4494
SASRec ( Kang and McAuley, 2018 )
0.6073
0.7498
0.5090
0.5638
0.5243
0.6597
0.4439
0.4958
0.5560
0.7570
0.4042
0.4786
LightGCN ( He et al., 2020 )
0.5808
0.7167
0.4894
0.5330
0.5138
0.6604
0.4344
0.4913
0.5430
0.7420
0.3928
0.4702
CL4SRec ( Xie et al., 2022 )
0.6221
0.7557
0.5306
0.5861
0.5482
0.6826
0.4694
0.5186
0.5788
0.7873
0.4374
0.5091
LATTICE ( Zhang et al., 2021 )
0.6131
0.7558
0.5223
0.5762
0.5306
0.6612
0.4531
0.5036
0.5635
0.7955
0.4110
0.5038
Table 1: Recommendation performance comparisons on three different datasets ( nc=10 ). Bold indicates the best results among all methods.
MovieLens-1M
Netflix
Model
HR@5
HR@10
NG@5
NG@10
HR@5
HR@10
NG@5
NG@10
CL4SRec ( Xie et al., 2022 )
0.1427
0.1991
0.1084
0.1198
0.1080
0.1872
0.0671
0.0937
TallRec ( Bao et al., 2023 )
0.1317
0.1930
0.0930
0.1108
0.0980
0.1790
0.0591
0.0836
MLLMRec-R1 ( Wang et al., 2026 )
0.2444
0.2983
0.1806
0.1969
0.1715
0.2560
0.1199
0.1512
Qwen3-4B + SFT
0.0795
0.1134
0.0504
0.0596
0.1155
0.1945
0.0663
0.0910
Qwen3-4B + GRPO
0.2440
0.2974
0.1802
0.1958
0.1706
0.2541
0.1149
0.1503
Table 2: Recommendation performance with a larger candidate set ( nc=100 ).
Variants
MovieLens-1M
MicroLens
Netflix
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
GRPO
0.7432
0.8261
0.6333
0.6495
0.6621
0.7764
0.5841
0.6330
0.6690
0.8510
0.5490
0.6198
RPSD w/o OT
0.7639
0.8716
0.6593
0.7007
0.6672
0.7801
0.5881
0.6360
0.7090
0.8660
0.5864
0.6509
RPSD (Full)
0.7871
0.8716
0.6678
0.7007
0.6836
0.7809
0.6036
0.6417
0.8020
0.9245
0.6952
0.7448
Table 3: Ablation studies on the contribution of each proposed module in RPSD .
Variants
MovieLens-1M
MicroLens
Netflix
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
HR@3
HR@5
NG@3
NG@5
w/o Re-ranking
0.7498
0.8451
0.6398
0.6789
0.6380
0.7629
0.5556
0.6062
0.6950
0.8640
0.5727
0.6404
w/o Temporal Cost
0.7481
0.8600
0.6421
0.6885
0.6305
0.7577
0.5517
0.6039
0.6950
0.8545
0.5739
0.6393
w/o Recency Weighting
0.7655
0.8600
0.6556
0.6875
0.6358
0.7681
0.5538
0.6094
0.6785
0.8500
0.5563
0.6258
w/o Diversity Selection
0.7457
0.8451
0.6414
0.6815
0.6171
0.7487
0.5460
0.6010
0.6810
0.8575
0.5592
0.6249
w/o Centering & Clipping
0.7639
0.8563
0.6499
0.6833
0.6432
0.7554
0.5634
0.6094
0.6920
0.8480
0.5662
0.6272
Table 4: Ablation studies on different designs of retrieval-based privileged evidence.
Figure 3: Cross-domain recommendation performance. All methods are trained on MovieLens and evaluated on MicroLens and Netflix.
Figure 4: Training diagnostics of RPSD. (a) The rule-based reward and the within-group standard deviation of that reward. (b) The centered teacher–student gap δi,t and the student’s token entropy. (c) The KL divergence to the reference model.
Figure 5: Performance at different training steps of SFT, GRPO, and RPSD. RPSD consistently achieves stronger ranking performance and converges to the best final results.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
#Users
#Items
#Interactions
Density
MicroLens
25,411
41,081
223,263
2.1×10−4
Netflix
13,187
17,366
68,933
3.0×10−4
MovieLens
6,040
3,952
1,000,209
4.2×10−2
Appendix
Table 5: Statistics of the experimental datasets.
Figure 6: RPSD training diagnostics on MovieLens. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap δi,t averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight wi,t , which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 7: Training stability of RPSD on MovieLens. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose G responses all receive the same reward. Such groups have zero reward variance and provide no advantage signal. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
Figure 8: RPSD training diagnostics on Netflix. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap δi,t averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight wi,t , which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 9: Training stability of RPSD on Netflix. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose G responses all receive the same reward. Such groups have zero reward variance and provide no advantage signal. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
Figure 10: RPSD training diagnostics on MicroLens. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap δi,t averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight wi,t , which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 11: Training stability of RPSD on MicroLens. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose G responses all receive the same reward. Such groups have zero reward variance and provide no advantage signal. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
Figure 12: Training diagnostics of RPSD w/o OT on MovieLens-1M. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 13: Training stability of RPSD w/o OT on MovieLens-1M. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
Figure 14: Training diagnostics of RPSD w/o OT on MicroLens. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 15: Training stability of RPSD w/o OT on MicroLens. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
Figure 16: Training diagnostics of RPSD w/o OT on Netflix. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.
Figure 17: Training stability of RPSD w/o OT on Netflix. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.
MicroLens
Model
HR@3
HR@5
NG@3
NG@5
Qwen3-4B + SFT
0.5187
0.6526
0.4339
0.4982
Qwen3-4B + GRPO
0.6621
0.7764
0.5841
0.6330
Qwen3-4B + OPSD
0.5669
0.6942
0.4809
0.5404
Qwen3-4B + RPSD (Ours)
0.6836
0.7809
0.6036
0.6417
Appendix
Table 6: Performance comparison with vanilla OPSD.
Netflix
Model
HR@3
HR@5
NG@3
NG@5
GRPO
0.6690
0.8510
0.5490
0.6198
Response-level RPSD
0.6975
0.8655
0.5775
0.6439
Token-level RPSD (Ours)
0.8020
0.9245
0.6952
0.7448
Appendix
Table 7: Ablation on the Response-level RPSD and Token-level RPSD (Ours).
Sequential recommender systems have achieved significant success in modeling temporal user behavior but remain limited in capturing rich user semantics beyond interaction patterns. Large Language Models (LLMs) present opportunities to enhance user understanding with their reasoning capabilities, yet existing integration approaches create prohibitive inference costs in real time. To address these limitations, we present a novel knowledge distillation method that utilizes textual user profile generated by pre-trained LLMs into sequential recommenders without requiring LLM inference at serving time. The resulting approach maintains the inference efficiency of traditional sequential models while requiring neither architectural modifications nor LLM fine-tuning.
Conventional sequential recommendation models have achieved remarkable success in mining implicit behavioral patterns. However, these architectures remain structurally blind to explicit user intent: they struggle to adapt when a user's immediate goal (e.g., expressed via a natural language prompt) deviates from their historical habits. While Large Language Models (LLMs) offer the semantic reasoning to interpret such intent, existing integration paradigms force a dilemma: LLM-as-a-recommender paradigm sacrifices the efficiency and collaborative precision of ID-based retrieval, while Reranking methods are inherently bottlenecked by the recall capabilities of the underlying model. In this paper, we propose Decoupled Promptable Sequential Recommendation (DPR), a model-agnostic framework that empowers conventional sequential backbones to natively support Promptable Recommendation, the ability to dynamically steer the retrieval process using natural language without abandoning collaborative signals. DPR modulates the latent user representation directly within the retrieval space. To achieve this, we introduce a Fusion module to align the collaborative and semantic signals, a Mixture-of-Experts (MoE) architecture that disentangles the conflicting gradients from positive and negative steering, and a three-stage training strategy that progressively aligns the semantic space of prompts with the collaborative space. Extensive experiments on real-world datasets demonstrate that DPR significantly outperforms state-of-the-art baselines in prompt-guided tasks while maintaining competitive performance in standard sequential recommendation scenarios.
Fuyuan Lyu, Chenglin Luo, Qiyuan Zhang +6
McGill & Mila - Quebec AI Institute Montreal, Canada · Shenzhen Technological University Shenzhen, China · City Univerisity of Hong Kong Hong Kong, China +3
Large language models (LLMs) have been applied to sequential recommendation by formulating it as a natural language task. Previous work has improved personalization by incorporating collaborative and sequential signals through input conditioning or LLM fine-tuning. However, existing approaches often rely on one or more of the following: LLM fine-tuning, additional architectural modules, representation distillation, or item-level conditioning over long interaction histories, increasing training complexity and deployment cost. We propose REPREC, a lightweight framework that reformulates LLM-based sequential recommendation through lightweight user representation alignment. REPREC maps a fixed-size user embedding from a frozen sequential encoder into a small set of learned soft tokens through a lightweight MLP injector that conditions a frozen LLM, leaving both pretrained backbones unchanged while training only the injector. We conducted exhaustive experiments on multiple benchmark datasets and demonstrate that REPREC consistently outperforms LoRA while remaining compatible with different pretrained sequential encoders and LLM backbones, enabling a modular and production-friendly recommendation pipeline without modifying either pretrained component. The gains are particularly pronounced for casual and core users across all datasets, highlighting REPREC's effectiveness in low-data regimes. Finally, when trained on short prompt histories and evaluated with longer contexts, REPREC maintains 85-100% of LoRA's performance while reducing per-epoch training time by an average of 1.51X, demonstrating an effective balance between recommendation quality and computational efficiency for production deployment. The code is available at https://github.com/phdbotcode/REPREC
Harshini Kavuru, Dwipam Katariya, Giri Iyengar +3
The Ohio State University, USA · Capital One, AI Foundations, USA