Learning to Reason with Predictive Gains for Generative Recommendation
Authors: Zhuoyang Li, Teng Shi, Jiakai Tang, Weijie Yu, Guofu Xie, Xiao Zhang, Lei Kou, Yang Yu, Lei Shen, Xin Chen, Bing Wang, Xiaoyi Zeng, Jun Xu
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China · Alibaba International Digital Commercial Group, Hangzhou, China
Reasoning-enhanced generative recommendation (GR) has attracted increasing attention for leveraging the reasoning capabilities of large language models (LLMs) to improve next-item prediction. However, existing methods primarily optimize reasoning through Group Relative Policy Optimization (GRPO) with rule-based rewards determined by final recommendation performance, which faces two key limitations: (i) unmeasured predictive contributions within reasoning, where the predictive contributions of individual reasoning steps are not directly reflected in optimization signals, and (ii) weakly discriminative learning signals within groups, where different reasoning trajectories within the group often receive indistinguishable rewards, limiting effective policy optimization. In this paper, we propose GainReason, a simple yet effective training framework that optimizes reasoning through Predictive Gain (PG). Specifically, PG measures changes in target Semantic ID (SID) log-likelihood induced by individual reasoning sentences, providing contribution-aware and discriminative rewards. We further theoretically establish the alignment between expected PG and the recommendation ranking objective. Building on this signal, we develop PG-based Reasoning Optimization (PGRO) to optimize reasoning generation with sentence-level rewards, supported by a custom causal attention mechanism for efficient PG computation. Additionally, an auxiliary PG-based Reasoning Utilization (PGRU) task adaptively strengthens reasoning-conditioned prediction to complement PGRO. Extensive experiments on three public benchmarks demonstrate that GainReason consistently outperforms competitive baselines and achieves substantial performance gains from reasoning.
Figures & tables
Figure 1. Comparison between rule-based and predictive gain (PG)-based reasoning optimization. The former relies on the performance of recommendations after reasoning, without measuring the predictive contributions within reasoning or adequately distinguishing rewards across reasoning trajectories. In contrast, PG evaluates the predictive contributions of individual reasoning sentences, providing more predictive contribution-aware and discriminative learning signals.
Figure 2. Analysis of reasoning optimization during training on Industrial. (Left) Ratio of reasoning sentences with positive to negative gains in target SID likelihood. (Right) Ratio of rollout groups with all zero-advantages. Compared with rule-based methods, GainReason achieves a higher ratio of positive to negative predictive gains and provides more discriminative learning signals, with no zero-advantage groups observed during training.
Figure 3. Overview of GainReason. PG-based Reasoning Optimization (PGRO) assigns sentence-level incremental predictive gains as rewards to optimize reasoning generation through group-relative policy optimization. An auxiliary PG-based Reasoning Utilization (PGRU) task adaptively enhances reasoning-conditioned target SID prediction by emphasizing trajectories with near-zero net predictive gains. Itemic alignment and reasoning activation are omitted for clarity.
Figure 4. The custom causal attention for efficient PG computation. Shaded cells denote allowed attention. Prompt tokens and user history are omitted for clarity.
Dataset
#Users
#Items
#Inter.
Avg. Len.
Sparsity
Industrial
7,694
3,686
49,371
3.98
99.8259%
Movies
14,216
13,190
148,099
6.01
99.9210%
Yelp
19,940
15,390
211,844
6.14
99.9310%
Table 1. Statistics of the benchmark datasets.
Models
Industrial
Movies
Yelp
N@5
N@10
HR@5
HR@10
N@5
N@10
HR@5
HR@10
N@5
N@10
HR@5
HR@10
Traditional Sequential Recommenders
GRU4Rec
0.0541
0.0580
0.0673
0.0797
0.0077
0.0088
0.0107
0.0141
0.0138
0.0195
0.0226
0.0393
BERT4Rec
0.0609
0.0642
0.0767
0.0872
0.0071
0.0094
0.0111
0.0180
0.0150
0.0211
0.0244
0.0434
SASRec
0.0727
0.0776
0.0992
0.1145
0.0118
0.0150
0.0187
0.0289
0.0157
0.0219
0.0244
0.0439
Generative Recommenders
Table 2. Overall performance on three recommendation benchmarks with different categories of methods. The best and second-best results are highlighted in bold and underline , respectively.
Figure 5. Changes in recommendation performance from enabling reasoning across methods and datasets.
Model
Industrial
Yelp
N@5
HR@5
N@5
HR@5
GainReason (Ours)
0.0985
0.1227
0.0204
0.0320
After Preparation
0.0881
0.1109
0.0169
0.0273
w/o PGRO
0.0892
0.1125
0.0191
0.0304
w/o PGRU
0.0954
0.1202
0.0200
0.0315
w/o PG-adaptive Weighting
0.0963
0.1211
0.0201
0.0315
Table 3. Ablation results on Industrial and Yelp. “w/o” denotes removing a component.
Figure 6. Target SID log-likelihood before and after reasoning and recommendation performance dynamics during reasoning optimization on Industrial.
Figure 7. Training time per step for OneRec-Think, GainReason, and GainReason T (left), and recommendation performance of the two GainReason variants on Industrial (right). GainReason and GainReason T use sentence-level and trajectory-level PG rewards, respectively.
Figure 8. Case study comparing the reasoning behaviors of OneRec-Think and GainReason. Sentences with positive and negative PG are shown in red and gray , respectively. Shaded text highlights attributes related to the user history or target item, while underlined text indicates incorrect statements.
Generative recommendation models user histories and item catalogs to generate personalized items, lists, or queries rather than only scoring candidates.
Large Language Models (LLMs) are increasingly adopted as backbones for Generative Recommendation (GR), promising access to pretrained world knowledge. Yet reliably invoking this knowledge for GR remains poorly understood. A key obstacle is that LLM-based GR typically represents items with Semantic IDs (SIDs), disrupting LLMs' natural-language reasoning interface because these tokens are unseen by the LLM during pretraining. Existing approaches address this with expensive multi-stage pipelines that ground SIDs and elicit explicit rationales, but offer limited insight into when and why each stage is necessary. In this work, we systematically decompose explicit reasoning training pipelines for LLM-based GR, revealing three key limitations: weakened world-knowledge verbalization, misalignment between SID and natural-language token embedding spaces, and sensitivity to rationale quality, all of which hurt explicit reasoning performance. To circumvent these issues, we propose PauseRec, a lightweight implicit reasoning paradigm tailored for GR. PauseRec is exceptionally practical, avoiding costly reasoning trace acquisition and reasoning alignment training, leading to a multitude of benefits: (1) it outperforms standard explicit CoT methods by up to 6.22%, (2) it reduces training cost by up to 65% GPU hours, and (3) it speeds up inference by up to 71.3%. These results position PauseRec as a lightweight alternative to explicit rationale generation, enabling more effective and efficient LLM-based GR.
Yinhan He, Liam Collins, Bhuvesh Kumar +3
University of Virginia, Charlottesville, VA, USA · Snap Inc., Santa Monica, CA, USA
Recent studies increasingly explore Large Language Models (LLMs) as a new paradigm for recommendation systems due to their scalability and world knowledge. However, existing work has three key limitations: (1) most efforts focus on retrieval and ranking, while the reranking phase, critical for refining final recommendations, is largely overlooked; (2) LLMs are typically used in zero-shot or supervised fine-tuning settings, leaving their reasoning abilities, especially those enhanced through reinforcement learning (RL) and high-quality reasoning data, underexploited; (3) items are commonly represented by non-semantic IDs, creating major scalability challenges in industrial systems with billions of identifiers. To address these gaps, we propose the Generative Reasoning Reranker (GR2), an end-to-end framework with a three-stage training pipeline tailored for reranking. First, a pretrained LLM is mid-trained on semantic IDs encoded from non-semantic IDs via a tokenizer achieving ≥99% uniqueness. Next, a stronger larger-scale LLM generates high-quality reasoning traces through carefully designed prompting and rejection sampling, which are used for supervised fine-tuning to impart foundational reasoning skills. Finally, we apply Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO), enabling scalable RL supervision with verifiable rewards designed specifically for reranking. Experiments on two real-world datasets demonstrate GR2's effectiveness: it surpasses the state-of-the-art OneRec-Think by 2.4% in Recall@5 and 1.3% in NDCG@5. Ablations confirm that advanced reasoning traces yield substantial gains across metrics. We further find that RL reward design is crucial in reranking: LLMs tend to exploit reward hacking by preserving item order, motivating conditional verifiable rewards to mitigate this behavior and optimize reranking performance.
Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically outcome-reward algorithm with exact-match feedback on the generated SID. However, in large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct; when a generated SID mismatches, outcome-reward cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes reinforcement-learning training and consistently improves over existing generative recommendation baselines, with the largest gains where sparse exact-match feedback makes reasoning-step credit assignment important. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output.