Generative recommenders decode the semantic ID of a user's next item with a language model, and post-training them with reinforcement learning (RL) after supervised fine-tuning (SFT) has become standard practice. The gains are mostly reported on the first ten results, yet in deployment such models hand hundreds of candidates to a downstream ranker or reward model. We show that RL post-training has a hidden cost: it improves accuracy at the top of the list (hit@K for small K) but, on all five datasets we study, lowers the recall of the candidate list (hit@K for K in the hundreds). The model becomes a better ranker and a worse retriever. Controlled experiments trace the loss to the negative advantages that RL assigns to the candidates it samples, the rollout. The root cause is that only one next item is logged per user, so RL treats every other candidate as wrong. Yet these candidates are the model's most plausible alternatives, and often the next items of other users with similar histories. Penalising them pushes down exactly what similar users choose next, and these items drop out of those users' candidate lists. This diagnosis points directly to a remedy: out-of-rollout anchoring, a single additional loss term that preserves the SFT model's ordering of the items outside the rollout. It keeps the gain at the top and restores recall to the SFT level on all five datasets, and it is effective under other RL configurations as well. To our knowledge, this is the first systematic study of this recall loss: we measure it, identify its cause, and show that, once the cause is known, a simple remedy suffices.
Figures & tables
Figure 1. Change in hit@ K after GRPO post-training on Office and Industrial, relative to the SFT model that RL started from (exact ranking of the full catalogue; mean of three seeds, bands show their range). Line chart of the change in hit at K against the cut-off K on a logarithmic axis for Office and Industrial. Both curves are above zero for small K and fall below zero beyond K of about 20 and 50, reaching minus 5.1 and minus 1.5 points at K equal to 500.
Figure 2. (a) A generative recommender is deployed as the retrieval stage of a cascade (highlighted: the common case), where recall at large K matters, or end to end, where accuracy at the top matters (example systems by the role their papers describe). (b) The effect studied here: after RL post-training, targets near the top move up and targets further down drop out of the top K . Two panels. Panel a shows the classical retrieval and ranking cascade, a generative model replacing the retrieval stage, which is highlighted as the common case, and a generative model replacing the whole cascade. Panel b shows two rankings side by side, under SFT and after RL: a target near the top moves up and a target further down leaves the top K.
System
Platform
Objective
Candidates
Retrieval stage: candidates are passed to a ranker
TSGR ( Zhan et al., 2026 )
Taobao
GRPO
1,500
GrowthGR ( Wang et al., 2026a )
Taobao
GRPO
1,000
EG-GRPO ( Zhu et al., 2026 )
Tmall
GRPO
100
GenR-PO ( Li et al., 2024 )
JD
DPO
100
PinGER ( Li et al., 2026 )
Pinterest
DPO
100
Table 1. RL and preference post-training of generative recommenders in prior work. Candidates: number generated per request (passed to the next stage or, failing that, the largest beam width); n/s: not stated. † Online results are reported with and without a reward model that selects among the generated candidates.
Dataset
Items
Test
Split
In rollout
Office
3,459
4,866
chronological
37%
Industrial
3,686
4,533
chronological
42%
Beauty
12,101
5,000
leave-one-out
24%
Toys
11,924
5,000
leave-one-out
16%
Sports
18,357
5,000
leave-one-out
11%
Table 2. Datasets. “In rollout”: share of training prompts whose rollout contains the target at the start of RL.
NDCG
hit@ K
@10
5
10
50
100
200
500
Office
+0.95
+0.71
+0.44
−1.52
−2.77
−4.14
−5.14
Industrial
+0.43
+0.54
+0.51
+0.02
−1.27
−1.04
−1.51
Beauty
+0.38
+0.28
+0.47
+0.33
−0.31
−0.39
−0.49
Toys
+0.40
+0.30
+0.50
+0.24
−0.40
−0.62
−0.04
Sports
+0.01
+0.20
−0.18
−0.47
−0.25
−0.19
−0.67
Table 3. Change after one epoch of GRPO relative to SFT, in points (mean of three seeds). Table 5 gives the absolute values.
Figure 3. Why recall degrades (illustrative example). (1) A target in the rollout gets a positive advantage and the other candidates negative ones, which improves the top of the list. (2) A plausible item such as black ink is sampled for many users and is rarely the target, so penalties accumulate on it and its score also falls for similar users. (3) Where the rollout misses black ink no gradient counteracts the fall, and for a test user whose target it is, rarely sampled items overtake it. Three cards. Card one shows a rollout for a training user in which the target, colour ink, gets a plus sign and three other candidates including black ink get minus signs. Card two shows sixteen squares of which fifteen are negative, standing for the advantages black ink collects over all rollouts that sample it. Card three shows a test user whose target is black ink: under SFT it is above the top K line, after RL two rarely sampled items have moved above it and it falls below the line.
Office
Industrial
Rank under SFT
median rank
demoted
median rank
demoted
1
1 → 1
13%
1 → 1
19%
2–10
4 → 3
42%
5 → 5
47%
11–50
26 → 44
62%
25 → 30
56%
51–100
72 → 136
64%
72 → 90
57%
101–200
146 → 262
65%
147 → 166
55%
Table 4. Where the targets move under GRPO: test prompts grouped by the rank of their target under SFT, with the median rank before and after RL and the share whose rank worsens (mean of three seeds).
Figure 4. Evidence for the cause (Office; change in hit@ K against SFT; a, c: synced reference, one run). (a) Keeping only the positive advantages removes the loss and the gain; keeping only the negative ones produces a much larger loss. (b) DPO loses recall up to K=500 with negatives from the SFT beam and only at intermediate K with random ones (three seeds, bands: range). (c) Test prompts grouped by the rollout record of their target, computed before RL: the loss falls on targets with a negative record. Three line charts of the change in hit at K against K. Panel a: with positive advantages only the curve stays near zero, GRPO falls to minus 6 points and negative advantages only fall to minus 20 points at K of 500. Panel b: DPO with beam negatives falls to minus 4 points on average, with random negatives it returns to zero at K of 500. Panel c: targets with a strongly negative rollout record lose 27 points at K of 500, mildly negative ones 9 points, and targets with a positive record gain a few points.
Figure 5. Out-of-rollout anchoring on one training prompt. GRPO handles the rollout as before. In addition, M items sampled outside the rollout are scored by the frozen SFT model and by the policy, and the policy is penalised if their relative scores change. A rollout of four candidates, the target with a plus sign and three others with minus signs, labelled inside the rollout, GRPO unchanged. Below a dotted line, five items outside the rollout, three of them highlighted as anchor items that must keep the relative order the SFT model gave them, labelled with the anchor term lambda times a KL divergence.
Figure 6. GRPO with and without the anchor. (a, b) Change in hit@ K against SFT (mean of three seeds, bands show their range). (c) Office test prompts grouped by the rollout record of their target (Section 4.3 ): change in hit@500 against SFT (one seed). Three panels. Panels a and b are line charts of the change in hit at K against K for Office and Industrial. The GRPO curve starts above zero and falls to minus 5.1 points on Office and minus 1.5 on Industrial at K of 500; the curve of GRPO with the anchor starts at the same height and stays near zero, ending at minus 0.7 on Office and plus 0.4 on Industrial. Panel c is a bar chart of the change in hit at 500 for four groups of Office test prompts: for targets with a strongly negative rollout record, minus 23.4 points under GRPO and minus 3.1 with the anchor; for mildly negative ones, minus 5.4 and plus 0.6; for the two groups with a positive record, small gains under both.
hit@ K (%)
NDCG
3
5
10
50
100
200
300
500
@10
Office
SFT
11.76
13.42
15.97
25.73
31.94
39.56
44.88
52.84
11.59
GRPO
12.66
14.13
16.41
24.22
29.16
35.42
40.29
47.69
12.55
GRPO + anchor
12.76
14.41
16.89
26.27
32.17
39.29
44.40
52.14
12.64
anchor effect
+0.10
+0.29
+0.48
+2.05
+3.01
+3.87
+4.11
+4.45
+0.09
Industrial
SFT
8.54
10.21
13.19
23.21
30.20
38.05
43.97
53.23
9.05
Table 5. SFT, GRPO and GRPO with the out-of-rollout anchor (mean of three seeds; best value per dataset and column in bold). Anchor effect: paired difference between the two RL models in points.
Office
Industrial
@50
@100
@500
@50
@100
@500
SFT
25.73
31.94
52.84
23.21
30.20
53.23
Synced reference
23.76
27.81
46.40
24.22
29.36
51.71
+ anchor
25.94
31.28
52.28
23.58
29.38
53.98
anchor effect
+2.18 ∗
+3.47 ∗
+5.88 ∗
−0.64
+0.02
+2.27 ∗
REINFORCE++
20.86
24.39
37.92
21.51
26.34
46.94
Table 6. The anchor on other base configurations: hit@ K in % (one seed). Anchor effect: paired difference in points; a star marks a significant difference. Within each pair the better value is bold when the difference is significant.
Aligning generative recommender systems to user preferences via post-training is critical for closing the gap between next-item prediction and actual recommendation quality. Existing post-training methods are ill-suited for production-scale systems: RLHF methods reward hack due to noisy user feedback and unreliable reward models, offline RL alternatives require propensity scores that are unavailable, and online interaction is infeasible. We identify exponential reward-weighted SFT with weights w=exp(r/λ) as uniquely suited to this setting, and provide the theoretical and empirical foundations that explain why. By optimizing directly on observed rewards without querying a learned reward model, the method is immune to reward hacking, requires no propensity scores, and is fully offline. We prove the first policy improvement guarantees for this setting under noisy rewards, showing that the gap scales only logarithmically with catalog size and remains informative even for large item catalogs. Crucially, we show that temperature λ explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding. Experiments on three open-source and one proprietary dataset against four baselines confirm that exponential reward weighting is simple, scalable, and consistently outperforms RLHF-based alternatives.
Recent studies increasingly explore Large Language Models (LLMs) as a new paradigm for recommendation systems due to their scalability and world knowledge. However, existing work has three key limitations: (1) most efforts focus on retrieval and ranking, while the reranking phase, critical for refining final recommendations, is largely overlooked; (2) LLMs are typically used in zero-shot or supervised fine-tuning settings, leaving their reasoning abilities, especially those enhanced through reinforcement learning (RL) and high-quality reasoning data, underexploited; (3) items are commonly represented by non-semantic IDs, creating major scalability challenges in industrial systems with billions of identifiers. To address these gaps, we propose the Generative Reasoning Reranker (GR2), an end-to-end framework with a three-stage training pipeline tailored for reranking. First, a pretrained LLM is mid-trained on semantic IDs encoded from non-semantic IDs via a tokenizer achieving ≥99% uniqueness. Next, a stronger larger-scale LLM generates high-quality reasoning traces through carefully designed prompting and rejection sampling, which are used for supervised fine-tuning to impart foundational reasoning skills. Finally, we apply Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO), enabling scalable RL supervision with verifiable rewards designed specifically for reranking. Experiments on two real-world datasets demonstrate GR2's effectiveness: it surpasses the state-of-the-art OneRec-Think by 2.4% in Recall@5 and 1.3% in NDCG@5. Ablations confirm that advanced reasoning traces yield substantial gains across metrics. We further find that RL reward design is crucial in reranking: LLMs tend to exploit reward hacking by preserving item order, motivating conditional verifiable rewards to mitigate this behavior and optimize reranking performance.
Generative recommenders return a limited candidate set and may omit observed targets before reranking. A training strategy appends these missed targets to reranker training lists, although inference still ranks only original candidates. This operation simultaneously changes retrieved-target weight, adds supervision over appended targets, and makes the two groups compete for probability. An append/no-append comparison therefore cannot explain changes in returned-item rankings. We construct three matched losses that hold retrieved-target weight fixed while introducing appended-target supervision and group competition separately. The intermediate loss trains within both groups but normalizes them separately, preventing training-only targets from competing with inference candidates. Experiments with a released OneRec model and locally trained Amazon generators show that this competition can harm returned-item ranking. In four prespecified Amazon Video Games comparisons, removing it improved full-target normalized discounted cumulative gain (FT-NDCG) by 7.8--22.2%; 95% intervals over users and three of four intervals over training runs excluded zero. A conservative development-set rule selected appended-target training for two of three generators in one held-out category and rejected it for all three in another, avoiding a 1.7% loss. Candidate completion should therefore be evaluated for each generator rather than applied automatically.
Xuesi Wang, Yangbin Shi, Xiaolin Zheng
Independent Researcher Shanghai, China · Zhejiang University Hangzhou, China