RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

Dongyue Li; Zebang Cheng; Zhongyi Deng; Hongboxu; Yi Bin; Fei Ma

cs.AIOct 27, 2026

RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

Authors: Dongyue Li, Zebang Cheng, Zhongyi Deng, Hongboxu, Yi Bin, Fei Ma

Organizations: Nagoya University · Shenzhen University · Sichuan Agricultural University · Guangdong Laboratory of Artificial Intelligence and Digital Economy ·

Abstract

Sequential recommendation predicts a user's next interaction from historical behavior. Recent LLM-based methods recast this task as generative recommendation and increasingly adopt reinforcement learning such as GRPO for post-training. However, these methods suffer from two limitations, namely credit-blind optimization, where a sequence-level reward is shared across all generated tokens, and collaborative-blind supervision, where each training signal is derived from a single user and ignores behavioral patterns shared across similar users. To address both issues, we propose Retrieval-Privileged on-policy Self-Distillation (RPSD). For credit-blind optimization, we introduce on-policy self-distillation (OPSD), in which a teacher with privileged knowledge re-evaluates the student's on-policy response, and the token-level confidence difference between the student and teacher reshapes the shared sequence-level advantage into token-specific credit. However, the privileged knowledge in vanilla OPSD is still derived from a single user, leaving collaborative-blind supervision unresolved. We therefore ground the teacher in collaborative behavioral evidence from similar users. Specifically, we use temporally aware optimal transport (OT) to compare interaction trajectories across users by jointly considering item similarity and temporal structure, and further apply diversity-aware selection to retain complementary trajectories as privileged evidence. In this way, on-policy self-distillation provides fine-grained token-level credit, while OT enriches the teacher with structured cross-user evidence. To the best of our knowledge, this is the first work to bring on-policy self-distillation to sequential recommendation and to use cross-user behavior as its privileged signal. Extensive experiments across multiple real-world datasets demonstrate that RPSD achieves state-of-the-art performance.

Figures & tables

Figure 1: RPSD combines token-level credit assignment with retrieved evidence. (a) Conceptual differences between RPSD and GRPO. (b) Training reward and token entropy of GRPO and RPSD. (c) Comparison between RPSD and RPSD w/o OT, which only takes the ground-truth item as privileged knowledge of the teacher.

Figure 2: Overview of RPSD. The student samples on-policy responses from the original recommendation prompt, while a retrieval-privileged teacher re-scores the same trajectory using behavioral evidence only during training to derive token-level credit.

	Model	MovieLens-1M				MicroLens				Netflix
	Model	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5
Non-LLM	GRU4Rec ( Hidasi et al., 2016 )	0.5507	0.7131	0.4762	0.5236	0.5080	0.6440	0.4369	0.4756	0.5152	0.7152	0.3659	0.4494
	SASRec ( Kang and McAuley, 2018 )	0.6073	0.7498	0.5090	0.5638	0.5243	0.6597	0.4439	0.4958	0.5560	0.7570	0.4042	0.4786
	LightGCN ( He et al., 2020 )	0.5808	0.7167	0.4894	0.5330	0.5138	0.6604	0.4344	0.4913	0.5430	0.7420	0.3928	0.4702
	CL4SRec ( Xie et al., 2022 )	0.6221	0.7557	0.5306	0.5861	0.5482	0.6826	0.4694	0.5186	0.5788	0.7873	0.4374	0.5091
	LATTICE ( Zhang et al., 2021 )	0.6131	0.7558	0.5223	0.5762	0.5306	0.6612	0.4531	0.5036	0.5635	0.7955	0.4110	0.5038

Table 1: Recommendation performance comparisons on three different datasets (

n_{c}=10

). Bold indicates the best results among all methods.

	MovieLens-1M				Netflix
Model	HR@5	HR@10	NG@5	NG@10	HR@5	HR@10	NG@5	NG@10
CL4SRec ( Xie et al., 2022 )	0.1427	0.1991	0.1084	0.1198	0.1080	0.1872	0.0671	0.0937
TallRec ( Bao et al., 2023 )	0.1317	0.1930	0.0930	0.1108	0.0980	0.1790	0.0591	0.0836
MLLMRec-R1 ( Wang et al., 2026 )	0.2444	0.2983	0.1806	0.1969	0.1715	0.2560	0.1199	0.1512
Qwen3-4B + SFT	0.0795	0.1134	0.0504	0.0596	0.1155	0.1945	0.0663	0.0910
Qwen3-4B + GRPO	0.2440	0.2974	0.1802	0.1958	0.1706	0.2541	0.1149	0.1503

Table 2: Recommendation performance with a larger candidate set (

n_{c}=100

Variants	MovieLens-1M				MicroLens				Netflix
Variants	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5
GRPO	0.7432	0.8261	0.6333	0.6495	0.6621	0.7764	0.5841	0.6330	0.6690	0.8510	0.5490	0.6198
RPSD w/o OT	0.7639	0.8716	0.6593	0.7007	0.6672	0.7801	0.5881	0.6360	0.7090	0.8660	0.5864	0.6509
RPSD (Full)	0.7871	0.8716	0.6678	0.7007	0.6836	0.7809	0.6036	0.6417	0.8020	0.9245	0.6952	0.7448

Table 3: Ablation studies on the contribution of each proposed module in RPSD .

Variants	MovieLens-1M				MicroLens				Netflix
Variants	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5	HR@3	HR@5	NG@3	NG@5
w/o Re-ranking	0.7498	0.8451	0.6398	0.6789	0.6380	0.7629	0.5556	0.6062	0.6950	0.8640	0.5727	0.6404
w/o Temporal Cost	0.7481	0.8600	0.6421	0.6885	0.6305	0.7577	0.5517	0.6039	0.6950	0.8545	0.5739	0.6393
w/o Recency Weighting	0.7655	0.8600	0.6556	0.6875	0.6358	0.7681	0.5538	0.6094	0.6785	0.8500	0.5563	0.6258
w/o Diversity Selection	0.7457	0.8451	0.6414	0.6815	0.6171	0.7487	0.5460	0.6010	0.6810	0.8575	0.5592	0.6249
w/o Centering & Clipping	0.7639	0.8563	0.6499	0.6833	0.6432	0.7554	0.5634	0.6094	0.6920	0.8480	0.5662	0.6272

Table 4: Ablation studies on different designs of retrieval-based privileged evidence.

Figure 3: Cross-domain recommendation performance. All methods are trained on MovieLens and evaluated on MicroLens and Netflix.

Figure 4: Training diagnostics of RPSD. (a) The rule-based reward and the within-group standard deviation of that reward. (b) The centered teacher–student gap

\delta_{i,t}

and the student’s token entropy. (c) The KL divergence to the reference model.

Figure 5: Performance at different training steps of SFT, GRPO, and RPSD. RPSD consistently achieves stronger ranking performance and converges to the best final results.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Users	#Items	#Interactions	Density
MicroLens	25,411	41,081	223,263	$2.1\times 10^{-4}$
Netflix	13,187	17,366	68,933	$3.0\times 10^{-4}$
MovieLens	6,040	3,952	1,000,209	$4.2\times 10^{-2}$

Appendix

Table 5: Statistics of the experimental datasets.

Figure 6: RPSD training diagnostics on MovieLens. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap

\delta_{i,t}

averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight

w_{i,t}

, which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 7: Training stability of RPSD on MovieLens. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose

G

responses all receive the same reward. Such groups have zero reward variance and provide no advantage signal. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.

Figure 8: RPSD training diagnostics on Netflix. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap

\delta_{i,t}

averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight

w_{i,t}

, which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 9: Training stability of RPSD on Netflix. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose

G

Figure 10: RPSD training diagnostics on MicroLens. We track three quantities that are internal to RPSD. (left) The teacher-student confidence gap

\delta_{i,t}

averaged over completion tokens, which measures how much the teacher model differs from the student model. (middle) The advantage-aware token weight

w_{i,t}

, which sets how strongly the shared advantage is reshaped. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 11: Training stability of RPSD on MicroLens. (left) Rule-based reward. (middle) Group agreement, i.e., the fraction of sampled groups whose

G

Figure 12: Training diagnostics of RPSD w/o OT on MovieLens-1M. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 13: Training stability of RPSD w/o OT on MovieLens-1M. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.

Figure 14: Training diagnostics of RPSD w/o OT on MicroLens. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 15: Training stability of RPSD w/o OT on MicroLens. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.

Figure 16: Training diagnostics of RPSD w/o OT on Netflix. The teacher is privileged with the ground-truth item alone. (left) The teacher–student confidence gap averaged over completion tokens. (middle) The advantage-aware token weight. (right) The KL divergence to the reference policy. Blue curves are the raw logged values and orange curves are moving averages.

Figure 17: Training stability of RPSD w/o OT on Netflix. (left) Rule-based reward. (middle) Group agreement. (right) Policy entropy over completion tokens. Blue curves are the raw logged values and orange curves are moving averages.

	MicroLens
Model	HR@3	HR@5	NG@3	NG@5
Qwen3-4B + SFT	0.5187	0.6526	0.4339	0.4982
Qwen3-4B + GRPO	0.6621	0.7764	0.5841	0.6330
Qwen3-4B + OPSD	0.5669	0.6942	0.4809	0.5404
Qwen3-4B + RPSD (Ours)	0.6836	0.7809	0.6036	0.6417

Appendix

Table 6: Performance comparison with vanilla OPSD.

	Netflix
Model	HR@3	HR@5	NG@3	NG@5
GRPO	0.6690	0.8510	0.5490	0.6198
Response-level RPSD	0.6975	0.8655	0.5775	0.6439
Token-level RPSD (Ours)	0.8020	0.9245	0.6952	0.7448

Appendix

Table 7: Ablation on the Response-level RPSD and Token-level RPSD (Ours).

Sequential Recommendation

Sequential recommendation predicts what a user will choose next from ordered interactions, adapting to changing interests and context.

See all 141 papers

Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation
Apr 23, 2026Nikita Severin, Danil Kartushov, Vladislav Urzhumov +8Sequential Recommendation Language Model Distillation
Independent researcher, Belgrade, Serbia · ITMO University, Saint Petersburg, Russia · Innopolis University, Innopolis, Russia +3
Give Users the Wheel: Towards Promptable Recommendation Paradigm
Feb 21, 2026Fuyuan Lyu, Chenglin Luo, Qiyuan Zhang +6Sequential Recommendation User Preference Modeling
McGill & Mila - Quebec AI Institute Montreal, Canada · Shenzhen Technological University Shenzhen, China · City Univerisity of Hong Kong Hong Kong, China +3
REPREC: Representation Driven Parameter-Efficient Recommendation System
Jul 24, 2026Harshini Kavuru, Dwipam Katariya, Giri Iyengar +3Sequential Recommendation Large Language Model-Based Recommendation
The Ohio State University, USA · Capital One, AI Foundations, USA

Apr 23, 2026J/K move · Enter open · S save

Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

Nikita Severin, Danil Kartushov, Vladislav Urzhumov, Vladislav Kulikov+7

Independent researcher, Belgrade, Serbia · ITMO University, Saint Petersburg, Russia · Innopolis University, Innopolis, Russia+3

Sequential Recommendation Language Model Distillation Large Language Model-Based Recommendation User Preference Modeling Knowledge Distillation

Open paper AskPDF arXiv

Sequential recommender systems have achieved significant success in modeling temporal user behavior but remain limited in capturing rich user semantics beyond interaction patterns. Large Language Models (LLMs) present opportunities to enhance user understanding with their reasoning capabilities, yet existing integration approaches create prohibitive inference costs in real time. To address these limitations, we present a novel knowledge distillation method that utilizes textual user profile generated by pre-trained LLMs into sequential recommenders without requiring LLM inference at serving time. The resulting approach maintains the inference efficiency of traditional sequential models while requiring neither architectural modifications nor LLM fine-tuning.

Abstract

Sequential Recommendation

Explore similar work

Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

Give Users the Wheel: Towards Promptable Recommendation Paradigm

REPREC: Representation Driven Parameter-Efficient Recommendation System