Exploration of Reinforcement Learning in Generative Recommendation Models
Authors: Pengfei Zheng, Wuchao Li, Shiyao Wang, Honghui Bao, Hongyi Xie, Jiaxin Deng, Xingmei Wang, Jinghao Zhang, Yufei Ye, Ruiming Tang, Guorui Zhou, Han Li, Kun Gai, Defu Lian
Organizations: University of Science and Technology of China, Hefei, China · Kuaishou, Beijing, China · Unaffiliated, Beijing, China · State Key Laboratory of Cognitive Intelligence
Traditional cascaded recommender systems balance accuracy and efficiency over large item spaces, but misaligned stage-wise objectives hinder end-to-end optimization. Generative recommendation unifies these stages in an autoregressive framework, providing a natural interface for reinforcement learning (RL) to optimize the entire generation process. However, directly applying RL often leads to unstable optimization and degraded recommendation quality. We revisit Group Relative Policy Optimization (GRPO) and other ratio-based objectives through a unified logit-gradient view. We identify three issues: (i) clipped surrogates create dead-gradient regions, under-utilizing samples; (ii) negative-advantage updates can become effectively unbounded as importance ratios grow, allowing a small subset of negatives to dominate optimization; and (iii) ratio-based control lacks sensitivity to absolute policy confidence, so updates on high-confidence cases may fail to vanish. Motivated by this diagnosis, we propose Gradient-Bounded Policy Optimization (GBPO), a simple objective with stop-gradient normalization that is simultaneously (a) fully gradient-active, (b) sign-symmetrically bounded for both positive and negative updates, and (c) explicitly confidence-weighted under current policy probabilities. This ensures that updates gracefully vanish for π→1 positives and π→0 negatives. Across the evaluated settings, GBPO achieves competitive recommendation performance with more stable training than competing RL methods.
Figures & tables
Figure 1. Comparison of Gradient Behavior Between GRPO and GBPO. Left: GRPO enforces bounded, confidence-sensitive updates for positive samples but leaves negative-sample gradients unconstrained. Right: GBPO guarantees bounded gradients for both update signs, with magnitudes that vanish as absolute confidence increases. This design naturally prioritizes hard, informative examples while preventing unbounded logit gradients. Comparison of Gradient Behavior Between GRPO and GBPO.
Figure 2. Comparison of coefficient maps ( κ ) for different policy optimization objectives, showing gradient activity regions across importance ratios and advantages. Heatmaps showing the gradient activity regions (kappa) for different policy optimization objectives across importance ratios and advantages.
Method
W/ All Samples?
Pos-Grad Bounded?
Neg-Grad Bounded?
Pos Conf.-Weighted?
Neg Conf.-Weighted?
GRPO
No
Yes
No
Partial
No
DAPO
No
Yes
No
Partial
No
GSPO
No
Yes
No
Partial
No
GPPO
Yes
Yes
No
Yes
No
Dual-Clip PPO
No
Yes
Yes
Partial
No
GBPO (ours)
Yes
Yes
Yes
Yes
Yes
Table 1. Qualitative comparison of policy optimization objectives. Detailed justifications are provided in Appendix A.11 . “W/ All Samples?” indicates whether the objective maintains active gradients for all sampled tokens (i.e., avoiding dead-gradient regions). “Pos/Neg grad bounded” asks whether the objective prevents arbitrarily large logit gradients. “Conf.-weighted” indicates whether the gradient magnitude vanishes smoothly as a function of absolute policy confidence ( π→1 for positives, π→0 for negatives). Methods that achieve vanishing solely through confidence-driven decay are marked “Yes”; those whose vanishing is partly caused by ratio-triggered dead-gradient cutoffs are marked “Partial”. For GBPO, both token- and sequence-level variants satisfy the listed qualitative properties.
Figure 3. Sequence-level GBPO gradient envelopes, normalized by ∣A^i∣/G . For selected-token probability p and identifier length n , the positive bound is (1−p)/n and the negative bound is n−1∑j=1npj/n . Negative-side attenuation weakens as the length increases. Positive- and negative-advantage selected-logit gradient bounds for identifier lengths 1, 4, and 8.
Beauty
Toys
Sports
Method
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
GRU4Rec ( Hidasi et al., 2016 )
0.0395
0.0584
0.0265
0.0326
0.0330
0.0490
0.0228
0.0279
0.0190
0.0312
0.0122
0.0161
SASRec ( Kang and McAuley, 2018 )
0.0402
0.0607
0.0254
0.0320
0.0448
0.0626
0.0300
0.0358
0.0199
0.0301
0.0106
0.0141
HGN ( Ma et al., 2019 )
0.0319
0.0536
0.0196
0.0266
0.0326
0.0517
0.0192
0.0254
0.0183
0.0313
0.0109
0.0150
BERT4Rec ( Sun et al., 2019 )
0.0232
0.0396
0.0146
0.0199
0.0215
0.0332
0.0131
0.0168
0.0102
0.0175
0.0065
0.0088
HSTU ( Zhai et al., 2024 ) (SFT)
0.0457
0.0627
0.0327
0.0382
0.0427
0.0576
0.0322
0.0371
0.0211
0.0317
0.0141
0.0175
Table 2. Main results on Amazon Beauty, Toys, and Sports. We report HitRate (H) and NDCG (N) at {5,10} following ( Rajput et al., 2023 ) . All numbers are reported at the best validation checkpoint.
Beauty
Toys
Sports
RL Method
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
GRPO † ( Shao et al., 2024 )
0.0405 †
0.0569 †
0.0285 †
0.0338 †
0.0334 †
0.0469 †
0.0234 †
0.0278 †
0.0297 †
0.0455 †
0.0191 †
0.0242 †
DAPO ( Yu et al., 2025 )
0.0411
0.0593
0.0284
0.0343
0.0286 †
0.0400 †
0.0204 †
0.0240 †
0.0304 †
0.0467 †
0.0196 †
0.0248 †
GSPO † ( Zheng et al., 2025 )
0.0405 †
0.0569 †
0.0285 †
0.0338 †
0.0334 †
0.0469 †
0.0234 †
0.0278 †
0.0297 †
0.0455 †
0.0191 †
0.0242 †
GPPO ( Su et al., 2025 )
0.0464
0.0719
0.0301
0.0384
0.0455
0.0704
0.0296
0.0376
0.0284
0.0436
0.0178
0.0227
Dual-Clip PPO † ( Ye et al., 2020 )
0.0521 †
0.0771 †
0.0360 †
0.0439 †
0.0524 †
0.0743 †
0.0356 †
0.0426 †
0.0289 †
0.0463 †
0.0188 †
0.0244 †
Table 3. RL objective comparison on HSTU backbone. All methods share the same HSTU backbone with direct item-ID prediction, candidate generation ( G=32 ), frozen DIN reward, and identical optimization settings. Best results among stable (non-collapsed) methods are in bold .
Dataset
Variant
H@5
H@10
N@5
N@10
Beauty
Sequence
0.0523
0.0753
0.0327
0.0401
Token
0.0517
0.0773
0.0328
0.0411
Toys
Sequence
0.0543
0.0792
0.0350
0.0431
Token
0.0541
0.0800
0.0344
0.0428
Sports
Sequence
0.0300
0.0489
0.0187
0.0248
Token
0.0275
0.0457
0.0170
0.0228
Table 4. Performance comparison between token-level and sequence-level GBPO.
Beauty
K=1
K=3
K=5
K=8
K=10
K=16
H@10
0.0707
0.0753
0.0753
0.0772
0.0764
0.0665
N@10
0.0400
0.0423
0.0401
0.0396
0.0378
0.0319
Toys
K=1
K=3
K=5
K=8
K=10
K=16
H@10
0.0661
0.0755
0.0792
0.0748
0.0751
0.0682
N@10
0.0384
0.0427
0.0431
0.0393
0.0373
0.0344
Sports
K=1
K=3
K=5
K=8
K=10
K=16
Table 5. Impact of the top- K positive reward threshold on recommendation performance (Top-10).
Method
K=1
K=3
K=5
K=8
K=10
K=16
GBPO H@10
0.0661
0.0755
0.0792
0.0748
0.0751
0.0682
GBPO N@10
0.0384
0.0427
0.0431
0.0393
0.0373
0.0344
GRPO H@10
0.0599
0.0609
0.0630
0.0637
0.0585
0.0548
GRPO N@10
0.0316
0.0322
0.0320
0.0322
0.0300
0.0275
Table 6. Comparative performance of GBPO and GRPO under different reward thresholds K (Toys dataset, Top-10).
Method
G=8
G=16
G=32
G=64
G=128
GBPO H@10
0.0752
0.0776
0.0792
0.0782
0.0767
GBPO N@10
0.0409
0.0425
0.0431
0.0425
0.0412
GRPO H@10
0.0523
0.0589
0.0630
0.0630
0.0645
GRPO N@10
0.0268
0.0310
0.0320
0.0330
0.0342
Table 7. Impact of candidate set size G on recommendation performance (Toys dataset, Top-10).
Dataset
Method
H@5
H@10
NDCG@5
NDCG@10
Beauty
GBPO
0.0511
0.0774
0.0325
0.0410
KD
0.0529
0.0766
0.0368
0.0444
S-DPO †
0.0440
0.0694
0.0296
0.0377
Toys
GBPO
0.0541
0.0782
0.0350
0.0428
KD
0.0506
0.0745
0.0349
0.0426
S-DPO †
0.0347
0.0559
0.0222
0.0289
Table 8. Single-run comparison of GBPO and non-RL post-training methods on TIGER (seed 42). Best values within each dataset are bold.
Method
H@5
H@10
NDCG@5
NDCG@10
D/N
SFT
0.04561
0.07589
0.02901
0.03874
GRPO
0.05281±0.00037
0.08825±0.00109
0.03326±0.00040
0.04466±0.00060
3/0
DAPO
0.05264±0.00073
0.08716±0.00125
0.03306±0.00056
0.04415±0.00074
3/0
GSPO
0.05577±0.00036
0.09080±0.00093
0.03475±0.00017
0.04598±0.00026
3/0
GPPO
0.05382±0.00152
0.08920±0.00061
0.03348±0.00125
0.04485±0.00079
3/0
Dual-Clip
0.05419±0.00079
0.08779±0.00101
0.03392±0.00057
0.04475±0.00052
3/2
Table 9. Steam results with the TIGER backbone and DIN rewards. SFT is the fixed initialization; RL results are mean ± sample standard deviation over seeds 42–44. The highest RL means are bold; the second-highest are underlined.
Figure 4. Comparison of training dynamics across RL objectives in Amazon Sports. Validation HitRate@10 over training steps comparing various RL objectives.
Figure 5. Comparison of entropy evolution across RL methods in Amazon Sports. Entropy evolution over training epochs for different reinforcement learning methods.
Figure 6. Proportion of Samples with Discarded Gradients across RL Methods in Amazon Sports. Clipping-based baselines exhibit a consistently non-zero discarded ratio, indicating under-utilization of samples, whereas GBPO stays near zero by design (no clip), enabling all sampled tokens to contribute learning signal. Plot showing the proportion of samples with discarded gradients across training for various RL methods, illustrating GBPO's full sample utilization.
Figure 7. Gradient-Norm Ratio (Negative vs. Positive Advantage) across RL Methods in Amazon Sports. The ratio gneg/gpos quantifies the dominance of negative updates (clipped to 20 for visualization, which is sufficient to show the dominance of negative gradients). Spikes in this ratio often precede training collapse. Plot of the gradient-norm ratio between negative and positive advantage updates, showing stability differences among RL methods.
Beauty
Toys
Sports
RL Method
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
GRPO ( Shao et al., 2024 )
0.0469
0.0719
0.0298
0.0379
0.0385
0.0630
0.0242
0.0320
0.0258
0.0402
0.0160
0.0206
DAPO ( Yu et al., 2025 )
0.0463
0.0723
0.0300
0.0384
0.0390
0.0636
0.0250
0.0329
0.0255
0.0394
0.0158
0.0203
GSPO ( Zheng et al., 2025 )
0.0491
0.0747
0.0315
0.0397
0.0479
0.0715
0.0308
0.0384
0.0285
0.0444
0.0180
0.0232
GPPO ( Su et al., 2025 )
0.0454
0.0729
0.0287
0.0375
0.0429
0.0667
0.0269
0.0346
0.0275
0.0447
0.0172
0.0228
Dual-Clip PPO ( Ye et al., 2020 )
0.0486
0.0732
0.0317
0.0396
0.0439
0.0668
0.0280
0.0354
0.0265
0.0432
0.0169
0.0223
Table 10. RL objective comparison on TIGER backbone. All methods report HitRate (H) and NDCG (N). They use constrained SID decoding, beam-search candidate generation ( G=32 beams), frozen DIN reward, and identical optimization settings unless stated otherwise.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Method
H@5
H@10
NDCG@5
NDCG@10
Beauty
GRPO †
0.0413
0.0647
0.0259
0.0334
DAPO †
0.0415
0.0640
0.0267
0.0339
GSPO †
0.0427
0.0662
0.0278
0.0353
GPPO †
0.0372
0.0624
0.0235
0.0316
Dual-Clip †
0.0422
0.0667
0.0280
0.0359
GBPO
0.0405
0.0687
0.0243
0.0333
Appendix
Table 11. Single-run results with continuous DIN sigmoid rewards on TIGER. Best values within each dataset are bold.
Dataset
Method
H@5
H@10
NDCG@5
NDCG@10
Beauty
GRPO †
0.0470
0.0753
0.0310
0.0401
DAPO †
0.0474
0.0744
0.0313
0.0400
GSPO †
0.0523
0.0788
0.0340
0.0426
GPPO †
0.0478
0.0745
0.0297
0.0383
Dual-Clip †
0.0516
0.0793
0.0339
0.0428
GBPO
0.0522
0.0796
0.0327
0.0415
Appendix
Table 12. Single-run results with a SASRec-style teacher and binary rewards on TIGER. Best values within each dataset are bold.
Dataset
Method
H@5
H@10
NDCG@5
NDCG@10
D/N
Beauty
SFT
0.0438
0.0682
0.0295
0.0374
GRPO
0.0478±0.0014
0.0734±0.0006
0.0315±0.0010
0.0397±0.0007
3/0
DAPO
0.0475±0.0003
0.0732±0.0015
0.0316±0.0003
0.0398±0.0006
3/0
GSPO
0.0492±0.0007
0.0761±0.0016
0.0317±0.0004
0.0404±0.0003
3/0
GPPO
0.0462±0.0003
0.0731±0.0008
0.0292±0.0003
0.0379±0.0004
3/0
Dual-Clip
0.0487±0.0010
0.0736±0.0016
0.0324±0.0003
0.0404±0.0005
3/0
Appendix
Table 13. TIGER results under default binary-DIN rewards. RL scores are mean ± sample standard deviation over seeds 42–44. SFT is the fixed initialization reference. The highest RL means within each dataset are bold.
Dataset
Method
H@5
H@10
NDCG@5
NDCG@10
D/N
Beauty
SFT
0.0443
0.0617
0.0316
0.0372
GRPO
0.0483±0.0012
0.0686±0.0006
0.0331±0.0008
0.0396±0.0006
3/0
DAPO
0.0496±0.0009
0.0704±0.0010
0.0339±0.0004
0.0406±0.0005
3/0
GSPO
0.0483±0.0012
0.0686±0.0006
0.0331±0.0008
0.0396±0.0006
3/0
GPPO
0.0467±0.0002
0.0724±0.0006
0.0307±0.0003
0.0390±0.0003
0/0
Dual-Clip
0.0543±0.0018
0.0784±0.0022
0.0369±0.0008
0.0447±0.0008
1/2
Appendix
Table 14. HSTU results under default binary-DIN rewards. RL scores are mean ± sample standard deviation over seeds 42–44. SFT is the fixed initialization reference. The highest RL means within each dataset are bold.
Backbone
Dataset
Baseline
H@5
H@10
NDCG@5
NDCG@10
TIGER
Beauty
GRPO
0.66654↑
0.16281↑
0.77618↑
0.12631↑
DAPO
0.47382↑
0.06816↑
0.77618↑
0.12631↑
GSPO
0.98035↑
1.00000↑
1.00000↑
1.00000↑
GPPO
0.28041↑
0.28040↑
0.28041↑
0.26830↑
Dual-Clip
0.77618↑
0.56326↑
1.00000↑
0.98035↑
Toys
GRPO
0.01528↑
0.00692↑
0.01488↑
0.00492↑
Appendix
Table 15. Holm-adjusted p -values for GBPO versus five RL baselines. Bold indicates p<0.05 ; ↑/↓ denotes a higher/lower GBPO mean in the corresponding result table.
Reinforcement learning (RL) presents a promising avenue for enhancing generative recommendation beyond supervised imitation, leveraging reward signals to guide policy improvement. However, its efficacy is critically contingent on the trustworthiness of the reward model for the samples it evaluates. In practice, production rankers, the widely adopted reward models, are trained on exposure-biased logs, leading to sample-dependent inaccuracies that violate this assumption. Our stratified analysis uncovers a consistent pattern: reward guidance is most beneficial when the policy exhibits uncertainty and the ranker can effectively discriminate the ground-truth item from rollout negatives. On other samples, the reward signal is either negligible or detrimental, highlighting the risk of uniform RL application. To address such an issue, we introduce AdaGRPO, a novel framework that treats reward-guided optimization as selective admission rather than uniform pressure. Training is anchored in supervised negative log-likelihood, while the GRPO objective is gated by a binary, per-sample clip determined by two rollout diagnostics: policy-side difficulty and reward discriminability. Instances failing either diagnostic default to pure supervision, ensuring stability and mitigating the amplification of noisy gradients. We validate AdaGRPO on a large-scale e-commerce dataset. At the best intermediate checkpoint, it elevates HR@10 from 11.01% to 12.18% while constraining hallucination below 0.22%, and maintains robustness at the final checkpoint (HR@10 11.63%, hallucination 0.27%), outperforming fixed NLL--GRPO mixtures across the retrieval--validity frontier. In production A/B tests, AdaGRPO achieves statistically significant gains in click-through rate and dwell time, confirming its practical utility.
Kewei Xu, Junbo Qi, Yanyan Zou +3
Waseda University Tokyo, Japan · University of Electronic Science and Technology of China Chengdu, China
Generic group-based RL assumes that sampled rollout groups are already usable learning signals. We show that this assumption breaks down in sparse-hit generative recommendation, where many sampled groups never become learnable at all. We propose ReCast, a repair-then-contrast learning-signal framework that first restores minimal learnability for all-zero groups and then replaces full-group reward normalization with a boundary-focused contrastive update on the strongest positive and the hardest negative. ReCast leaves the outer RL framework unchanged, modifies only within-group signal construction, and partially decouples rollout search width from actor-side update width. Across multiple generative recommendation tasks, ReCast consistently outperforms OpenOneRec-RL, achieving up to 36.6% relative improvement in Pass@1. Its matched-budget advantage is substantially larger: ReCast reaches the baseline's target performance with only 4.1% of the rollout budget, and this advantage widens with model scale. The same design also yields direct system-level gains, reducing actor-side update time by 16.60x, lowering peak allocated memory by 16.5%, and improving actor MFU by 14.2%. Mechanism analysis shows that ReCast mitigates the persistent all-zero / single-hit regime, restores learnability when natural positives are scarce, and converts otherwise wasted rollout budget into more stable policy updates. These results suggest that, for generative recommendation, the decisive RL problem is not only how to assign rewards, but how to construct learnable optimization events from sparse, structured supervision.
Reasoning-enhanced generative recommendation (GR) has attracted increasing attention for leveraging the reasoning capabilities of large language models (LLMs) to improve next-item prediction. However, existing methods primarily optimize reasoning through Group Relative Policy Optimization (GRPO) with rule-based rewards determined by final recommendation performance, which faces two key limitations: (i) unmeasured predictive contributions within reasoning, where the predictive contributions of individual reasoning steps are not directly reflected in optimization signals, and (ii) weakly discriminative learning signals within groups, where different reasoning trajectories within the group often receive indistinguishable rewards, limiting effective policy optimization. In this paper, we propose GainReason, a simple yet effective training framework that optimizes reasoning through Predictive Gain (PG). Specifically, PG measures changes in target Semantic ID (SID) log-likelihood induced by individual reasoning sentences, providing contribution-aware and discriminative rewards. We further theoretically establish the alignment between expected PG and the recommendation ranking objective. Building on this signal, we develop PG-based Reasoning Optimization (PGRO) to optimize reasoning generation with sentence-level rewards, supported by a custom causal attention mechanism for efficient PG computation. Additionally, an auxiliary PG-based Reasoning Utilization (PGRU) task adaptively strengthens reasoning-conditioned prediction to complement PGRO. Extensive experiments on three public benchmarks demonstrate that GainReason consistently outperforms competitive baselines and achieves substantial performance gains from reasoning.
Zhuoyang Li, Teng Shi, Jiakai Tang +10
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China · Alibaba International Digital Commercial Group, Hangzhou, China