cs.LGNov 15, 2026

Exploration of Reinforcement Learning in Generative Recommendation Models

Authors: Pengfei Zheng, Wuchao Li, Shiyao Wang, Honghui Bao, Hongyi Xie, Jiaxin Deng, Xingmei Wang, Jinghao Zhang, 

Organizations: University of Science and Technology of China, Hefei, China · Kuaishou, Beijing, China · Unaffiliated, Beijing, China · State Key Laboratory of Cognitive Intelligence

Abstract

Traditional cascaded recommender systems balance accuracy and efficiency over large item spaces, but misaligned stage-wise objectives hinder end-to-end optimization. Generative recommendation unifies these stages in an autoregressive framework, providing a natural interface for reinforcement learning (RL) to optimize the entire generation process. However, directly applying RL often leads to unstable optimization and degraded recommendation quality. We revisit Group Relative Policy Optimization (GRPO) and other ratio-based objectives through a unified logit-gradient view. We identify three issues: (i) clipped surrogates create dead-gradient regions, under-utilizing samples; (ii) negative-advantage updates can become effectively unbounded as importance ratios grow, allowing a small subset of negatives to dominate optimization; and (iii) ratio-based control lacks sensitivity to absolute policy confidence, so updates on high-confidence cases may fail to vanish. Motivated by this diagnosis, we propose Gradient-Bounded Policy Optimization (GBPO), a simple objective with stop-gradient normalization that is simultaneously (a) fully gradient-active, (b) sign-symmetrically bounded for both positive and negative updates, and (c) explicitly confidence-weighted under current policy probabilities. This ensures that updates gracefully vanish for π ⁣→ ⁣1π\!\to\!1 positives and π ⁣→ ⁣0π\!\to\!0 negatives. Across the evaluated settings, GBPO achieves competitive recommendation performance with more stable training than competing RL methods.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation

    Jun 7, 2026Kewei Xu, Junbo Qi, Yanyan Zou +3RL for RecommendationGroup Relative Policy Optimization

  2. Learning to Reason with Predictive Gains for Generative Recommendation

    Nov 16, 2026Zhuoyang Li, Teng Shi, Jiakai Tang +10Generative RecommendationLarge Language Model-Based Recommendation