FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Yijun Pan; Weikang Qiu; Qiyao Ma; Mingxuan Ju; Tong Zhao; Neil Shah; Rex Ying

cs.LGMar 12, 2026

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Authors: Yijun Pan, Weikang Qiu, Qiyao Ma, Mingxuan Ju, Tong Zhao, Neil Shah, Rex Ying

Organizations: Yale University · University of California, Davis · Snap Inc.

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives for diverse recommendation scenarios, yet most traditional recommenders are optimized for a single static target and struggle to reconfigure behavior on demand. Recent advances in reinforcement-learning-based post-training have unlocked strong instruction-following and reasoning capabilities in LLMs, suggesting a principled route for aligning them to complex recommendation goals. Motivated by this, we study closed-set autoregressive ranking, where an LLM generates a permutation over a fixed candidate set conditioned on user context and an explicit need instruction. However, applying RL to this setting faces two key obstacles: (i) sequence-level rewards yield coarse credit assignment that fails to provide fine-grained training signals, and (ii) interaction feedback is sparse and noisy, which together lead to inefficient and unstable updates. We propose FlexRec, a post-training RL framework that addresses both issues with (1) a causally grounded item-level reward based on counterfactual swaps within the remaining candidate pool, and (2) critic-guided, uncertainty-aware scaling that explicitly models reward uncertainty and down-weights low-confidence rewards to stabilize learning under sparse supervision. Across diverse recommendation scenarios and objectives, FlexRec achieves substantial gains: it improves NDCG@5 by up to 59% and Recall@5 by up to 109.4% in need-specific ranking, and further achieves up to 24.1% Recall@5 improvement under generalization settings, outperforming strong traditional recommenders and LLM-based baselines.

Figures & tables

Figure 1: Overall framework of FlexRec. Given a candidate set and an explicit user need, an LLM recommender generates ranked rollouts. An item-level reward is computed by evaluating the marginal contribution of individual item placements via counterfactual swaps (top right). A critic predicts both the expected interaction reward and its uncertainty for unobserved interactions. These estimates are used to form uncertainty-aware advantages, down-weighting unreliable signals during optimization (bottom).

	KuaiRec					MovieLens-1M
Maximizing Interest	N@5	N@10	N@30	R@5	MRR@5	N@5	N@10	N@30	P@5	MRR@5
Traditional Rerankers
BERT4Rec ( Sun et al., 2019 )	0.415	0.479	0.706	0.182	0.578	0.502	0.560	0.796	0.128	0.231
STAR ( Lee et al., 2025 )	0.384	0.457	0.697	0.170	0.541	0.500	0.562	0.797	0.129	0.234
Zero-Shot LLMs
GPT-4o	0.376	0.451	0.695	0.154	0.505	0.508	0.564	0.800	0.139	0.244

Table 1: Performance on Maximizing Interest across KuaiRec and MovieLens-1M. FlexRec achieves superior single-need ranking performance compared with baselines.

Figure 2: Performance across all three needs on KuaiRec. FlexRec is trained jointly on all needs. Joint training yields consistently stronger performance across needs, supporting FlexRec as an all-purpose recommender conditioned by need instructions.

Figure 3: Validation NDCG@5 and NDCG@10 during training under different reward designs, showing that FlexRec’s swap-based item-level reward outperforms sequence-level GRPO.

Figure 4: Validation NDCG@5 and NDCG@10 during training under different reward signals (UA denotes uncertainty-aware update), showing that a learned critic with uncertainty-aware GRPO achieves the strongest and most stable learning.

Traditional Rerankers
Product Search	N@5	N@10	N@30	P@5	R@5
BERT4Rec ( Sun et al., 2019 )	—	—	—	—	—
STAR ( Lee et al., 2025 )	0.412	0.466	0.634	0.220	0.533
Zero-Shot LLMs
GPT-4o	0.502	0.530	0.675	0.264	0.647
Qwen2.5-3B-Instruct	0.449	0.493	0.650	0.235	0.585

Table 3: Performance on Amazon Product Search (BERT4Rec not applicable to non-sequential setting), showing that FlexRec is effective not only for sequential recommendation but also for non-sequential ranking tasks such as product search.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	KuaiRec				ML-1M
Method	MSE $\downarrow$	MAE $\downarrow$	Pearson (Mean) $\uparrow$	Pearson (Var) $\uparrow$	MSE $\downarrow$	MAE $\downarrow$	Pearson (Mean) $\uparrow$	Pearson (Var) $\uparrow$
Critic	0.328	0.400	$0.374^{*}$	$0.300^{*}$	0.317	0.271	$0.510^{*}$	$0.132^{*}$

Appendix

Table 4: Critic performance on engagement prediction (watch ratio for KuaiRec and rating for ML-1M). We report mean prediction accuracy and uncertainty quality measured by the correlation between predicted variance and squared prediction error. An asterisk indicates statistical significance (

p<0.05

Component	Specification
User history encoder	MLP with ReLU activations
Candidate encoder	Fixed item embeddings
Hidden dimension	256
Interaction features	$\hat{u},\hat{c},\hat{u}\odot\hat{c},\hat{u}^{\top}\hat{c}$
Output heads	Mean head + log-variance head
Normalization	LayerNorm

Appendix

Table 5: Architecture of the critic model used to estimate engagement mean and uncertainty. where

\hat{u}

represents (learnt) user embedding and

\hat{c}

represents (fixed) candidate item embedding.

Traditional Rerankers
	KuaiRec					MovieLens-1M
Generalization Setting	Train: Max-Interest $\rightarrow$ Test: Trend Promotion					Train: Max-Interest $\rightarrow$ Test: Trend Promotion
Models	N@5	N@10	N@30	R@5	MRR@5	N@5	N@10	N@30	R@5	MRR@5
BERT4Rec ( Sun et al., 2019 )	0.479	0.520	0.774	0.123	0.085	0.542	0.601	0.820	0.181	0.212
STAR ( Lee et al., 2025 )	0.579	0.624	0.821	0.164	0.095	0.604	0.655	0.844	0.228	0.280
Zero-Shot LLMs

Appendix

Table 6: Need generalization results (train: Maximizing Interest , test: Trend Promotion ) indicate that FlexRec transfers strong ranking abilities effectively across needs.

Alg.	Reward Formulation	N@5	N@10	N@30	R@5	MRR@5
GRPO	Independent contribution	0.380	0.455	0.695	0.167	0.533
	Swap-based contribution (non-causal)	0.461	0.514	0.731	0.214	0.716
	CausalSwap reward (ours)	0.607	0.636	0.790	0.336	0.851
PPO	Independent contribution	0.390	0.462	0.698	0.175	0.546
	Swap-based contribution (non-causal)	0.383	0.455	0.695	0.169	0.534
	CausalSwap reward (ours)	0.621	0.661	0.803	0.338	0.850

Appendix

Table 7: Ablation of reward formulations for GRPO and PPO on the KuaiRec Maximizing Interest task under the dense-interaction setting.

CF-based
Reward Signal	N@5	N@10	N@30	R@5	MRR@5
User-KNN CF	0.410	0.484	0.707	0.194	0.570
Item-KNN CF	0.417	0.480	0.707	0.208	0.603
Critic-based
Raw critic	0.566	0.591	0.775	0.280	0.831
FlexRec: Uncertainty-aware Critic	0.595	0.623	0.786	0.319	0.825

Appendix

Table 8: Ablation of reward signals. Critic-based rewards outperform CF heuristics, and variance-aware updates further improve performance.

Baselines
Maximizing Interest (KuaiRec)	N@5	N@10	N@30	R@5	MRR@5
BERT4Rec	0.415	0.479	0.706	0.182	0.578
STAR (retriever)	0.384	0.457	0.697	0.170	0.541
GPT-4o	0.376	0.451	0.695	0.154	0.505
Qwen2.5-3B	0.375	0.451	0.693	0.160	0.515
Post-Trained LLMs (Dense)

Appendix

Table 9: Performance on the Maximizing Interest need on dense version of KuaiRec.

Explore similar work

ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning
Apr 9, 2026Jiani Huang, Shijie Wang, Liangbo Ning +2Large Language Model-Based Recommendation RL Fine-Tuning
The Hong Kong Polytechnic University
Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
Oct 13, 2026Xin Li, Hao Jiang, Jinghao Guo +2Large Language Model-Based Recommendation RL for Language Models
Nanyang Technological University, Singapore
GR2: Generative Reasoning Re-ranker
Feb 8, 2026Mingfu Liang, Yufei Li, Jay Xu +20RL for Language Models LLM Reranking
Meta AI

Apr 9, 2026J/K move · Enter open · S save

ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning

Jiani Huang, Shijie Wang, Liangbo Ning, Wenqi Fan+1

The Hong Kong Polytechnic University

Large Language Model-Based Recommendation RL Fine-Tuning RL for Language Model Reasoning Reward Shaping

Open paper AskPDF arXiv

With the rise of LLMs, there is an increasing need for intelligent recommendation assistants that can handle complex queries and provide personalized, reasoning-driven recommendations. LLM-based recommenders show potential but face challenges in multi-step reasoning, underscoring the need for reasoning-augmented systems. To address this gap, we propose ReRec, a novel reinforcement fine-tuning (RFT) framework designed to improve LLM reasoning in complex recommendation tasks. Our framework introduces three key components: (1) Dual-Graph Enhanced Reward Shaping, integrating recommendation metrics like NDCG@K with Query Alignment and Preference Alignment Scores to provide fine-grained reward signals for LLM optimization; (2) Reasoning-aware Advantage Estimation, which decomposes LLM outputs into reasoning segments and penalizes incorrect steps to enhance reasoning of recommendation; and (3) Online Curriculum Scheduler, dynamically assess query difficulty and organize training curriculum to ensure stable learning during RFT. Experiments demonstrate that ReRec outperforms state-of-the-art baselines and preserves core abilities like instruction-following and general knowledge. Our codes are available at https://github.com/jiani-huang/ReRec.

Abstract

Explore similar work

ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning

Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

GR2: Generative Reasoning Re-ranker