Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

Xin Li; Hao Jiang; Jinghao Guo; Weisi Lin; Chau Yuen

cs.CLOct 13, 2026

Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

Authors: Xin Li, Hao Jiang, Jinghao Guo, Weisi Lin, Chau Yuen

Organizations: Nanyang Technological University, Singapore

Abstract

A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.

Figures & tables

Figure 1: One request, five responders and one generated list. Rank-GRPO credits the list against one drawn responder, so its reward depends on which one was drawn; ReMaRk credits it once against the union of all responders’ items.

Figure 2: Rank-GRPO scores each sampled list against one drawn responder; ReMaRk scores it once against all of them. Top: one row per responder, drawn by row sampling, rewarded by that responder alone. Bottom: one row per request, drawn uniformly, with binary item rewards against the union of the responders’ items. Both keep Rank-GRPO’s rank-wise group normalization (per-group scaling) and GRPO update.

	Reddit-v2	Books	Music
training requests	72,233	32,539	19,159
training rows	383,013	149,679	142,396
validation requests	1,752	1,254	1,966
test requests	2,050	1,150	1,951
responders: mean (median)	5.30 (2)	4.60 (3)	7.43 (5)
single-responder (%)	41.0	18.1	18.3

Table 1: Training-split statistics and the cost of per-responder rows (definitions in Appendix H ).

	Reddit-v2							Reddit-Books		Music
	Qwen3.5					Llama		Qwen3.5		Qwen3.5
	2B	4B	9B	27B	35B-A3B	3B	8B	2B	4B	2B
SFT	15.40	16.53	17.23	17.90	17.34	15.38	16.84	20.67	21.73	9.31
Rank-GRPO	20.97	23.56	24.05	25.17	24.74	21.93	24.22	29.34	30.40	13.40
Per-resp., batch	20.43	23.16	23.95	23.96	24.58	22.28	23.89	29.35	30.51	13.71
ReMaRk (ours)	22.53	25.10	25.51	26.62	26.20	23.56	25.22	30.16	31.49	14.48

Table 2: Per-row test R@20 (%), 2,000 steps;

\Delta

: ReMaRk minus the better per-responder arm; seed 3407 (App. H ).

Figure 3: Best validation R@20 so far against training GPU-hours for Reddit-v2 pairs trained until neither improves; each line ends at the selected checkpoint.

Table 6

Figure 4: The gain follows the responder noise removed. (a) Test gain of ReMaRk over Rank-GRPO by the request’s number of responders (seed 3407, 95% CIs). (b, c) Dose-response at 2B on validation, batch scaling: R@20 when the reward averages

m=1,2,4

or all responders, spaced by the share of noise removed (three seeds; line: least squares), for all and for single-responder requests.

Figure 5: Test gain of ReMaRk over the better per-responder baseline in every setting, with 95% CIs. Filled: per row (seed 3407); small dots: other seeds (Llama-3.2-3B seed 42: over Rank-GRPO); open: per request.

pairwise preference for ReMaRk
	9B, 2k steps	4B, plateau
GPT-5.6	0.56 [0.52,0.61]	0.54 [0.49,0.59]
Gemini 3.1 Pro	0.58 [0.54,0.63]	0.52 [0.48,0.57]
precision@20, ReMaRk $-$ Rank-GRPO
GPT-5.6	+1.27 [0.16,2.53]	+0.58 [ $-$ 0.40,1.61]
Gemini 3.1 Pro	+1.03 [ $-$ 0.28,2.36]	+0.82 [ $-$ 0.61,2.11]

Table 6: Blind LLM judges on Reddit-v2 test: share of pairwise wins (ties 0.5) and precision@20 difference (points).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Resampling unit
	Reddit-v2			Reddit-Books
	2B	4B	9B	2B
requests (primary)	+1.57 [1.10,2.06]	+1.54 [1.10,2.00]	+1.46 [1.02,1.93]	+0.81 [0.19,1.46]
Reddit submissions	+1.57 [1.09,2.07]	+1.54 [1.08,2.00]	+1.46 [1.01,1.92]	–
without the 42 train-overlap requests	+1.56 [1.10,2.03]	+1.55 [1.09,2.05]	+1.47 [1.04,1.92]	–
Metric

Appendix

Table 8: Robustness of the gain over the better per-responder baseline.

	Reddit-v2			Reddit-Books
	2B	4B	9B	2B
R@5	+0.73 [0.36,1.10]	+0.65 [0.24,1.06]	+0.75 [0.36,1.14]	+1.13 [0.62,1.67]
R@10	+1.22 [0.80,1.63]	+1.11 [0.69,1.51]	+1.13 [0.71,1.56]	+0.78 [0.25,1.35]
R@20	+1.57 [1.10,2.06]	+1.54 [1.10,2.00]	+1.46 [1.02,1.93]	+0.81 [0.19,1.46]
N@20	+0.76 [0.53,0.99]	+0.82 [0.61,1.04]	+0.84 [0.62,1.07]	+0.65 [0.36,0.96]
Hit@20	+1.96 [1.37,2.56]	+1.58 [1.06,2.07]	+1.47 [0.96,2.03]	+0.85 [0.11,1.58]

Appendix

Table 9: ReMaRk minus the better per-responder baseline at other cutoffs (test, per row).

pair	ReMaRk	Rank-GRPO	ReMaRk	Rank-GRPO	per row	per request	share
	best validation R@20 (step)		test R@20		$\Delta$ test R@20		GPU-h
2B	26.15 (4,000)	25.82 (5,250)	22.72	22.58	+0.14 [ $-$ 0.30,0.59]	$-$ 0.03 [ $-$ 0.84,0.78]	52%
2B, seed 42	26.23 (4,500)	26.10 (7,250)	23.29	22.87	+0.42 [ $-$ 0.05,0.90]	+0.43 [ $-$ 0.45,1.29]	63%
4B	28.33 (3,000)	27.89 (5,250)	25.34	24.51	+0.83 [0.39,1.29]	+1.10 [0.33,1.90]	45%
9B	28.89 (2,750)	29.22 (7,250)	25.85	25.77	+0.08 [ $-$ 0.34,0.50]	+0.50 [ $-$ 0.27,1.25]	–
9B, seed 42	29.00 (2,750)	29.09 (8,000)	25.93	25.88	+0.05 [ $-$ 0.38,0.46]	+0.20 [ $-$ 0.62,0.98]	–

Appendix

Table 10: Plateau pairs: both arms trained until 1,500 steps pass without a new validation best (at most 8,000 steps), each tested once at its best checkpoint; Reddit-v2 unless noted. GPU-h share: ReMaRk’s GPU-hours to reach Rank-GRPO’s best validation score as a share of Rank-GRPO’s (interpolated; –: not reached). Seed 3407 unless noted.

Figure 6: Validation R@20 every 250 RL steps (step 0: SFT). Dashed: 4,000-step runs. Dotted curves (Reddit-v2 2B seeds 3407 and 42, 4B, 9B; Reddit-Books 2B): runs trained until 1,500 steps pass without a new validation best or to 8,000 steps, with stars at the selected steps (2B seed 3407: ReMaRk 4,000, Rank-GRPO 5,250; seed 42: 4,500 and 7,250, after which Rank-GRPO collapses to 3.68; 4B: 3,000 and 5,250, with a transient dip of ReMaRk at 2,500; 9B: 2,750 and 7,250; Reddit-Books: 2,000 and 3,250). Horizontal dotted line: ReMaRk’s best within 2,000 steps. Faint: runs of the same configuration with every checkpoint kept (Appendix C.3 , test at the matching step).

Figure 7: Reddit-Music (2B): validation R@20 every 250 RL steps. Solid: 2,000-step runs; dotted: runs trained until 1,500 steps pass without a new validation best (ReMaRk stopped at 2,750, Rank-GRPO at 6,250), with stars at the selected steps (1,250 and 4,750).

Figure 8: Raw validation curves behind Figure 3 (seed 3407). Solid: validation R@20 of each checkpoint; dashed: best so far.

	tail	torso	head
2B	+0.46	+2.09	+3.36
4B	+0.89	+1.24	+2.66
9B	+0.61	+2.33	+1.40
Reddit-Books 2B	+0.24	+2.02	$-$ 0.09
Reddit-Books 4B	+0.62	+2.07	$-$ 0.11

Appendix

Table 13: Per-row test R@20 gain (points) of ReMaRk by item popularity.

Reddit-v2, Qwen3.5 (10,972 rows, 2,050 requests)
		per row		per request
backbone	method	R@20	N@20	R@20	N@20	exact-20
–	Popularity (train)	2.75	1.29	2.68	1.23	–
–	kNN prior ( $k{=}200$ )	15.94	7.76	15.99	7.90	–
–	GPT-5.6 zero-shot	22.63	11.61	23.24	12.37	–
–	Qwen3.5-27B zero-shot	15.22	7.58	15.07	7.72	–

Appendix

Table 14: Full test results on Reddit-v2, Qwen3.5.

backbone	method	R@20	N@20	R@20	N@20	exact-20
		per row		per request
Reddit-v2, Llama (10,972 rows, 2,050 requests)
Llama-3.2-3B	Rank-GRPO, released (15.8k steps)	23.65	12.78	25.33	14.50	47.7
Llama-3.2-3B	SFT	15.38	8.37	14.66	8.28	99.8
	Rank-GRPO	21.93	11.51	22.97	12.59	99.0
	per-responder, batch	22.28	11.69	23.89	12.89	99.0

Appendix

Table 15: Full test results on Reddit-v2, Llama.

	aggregation	scale	sampling	R@20	req. R@20	R@20, $n{=}1$	R@20, $n{\geq}5$	exact-20
0	SFT (no RL)			17.09	16.16	13.75	17.30	99.9
1	per-responder	group	row (Rank-GRPO)	24.19	24.55	23.75	24.16	100.0
2	per-responder	batch	row	23.96	23.83	22.40	24.00	100.0
3	per-responder	batch	request	23.95	25.83	27.31	23.54	99.9
4	mean	batch	row	25.15	25.89	25.44	24.99	100.0
5	mean	batch	request	24.94	26.90	28.46	24.49	99.8

Appendix

Table 16: Validation version of Table 5 .

Reddit-Books (6,333 rows, 1,150 requests)
		per row		per request
backbone	method	R@20	N@20	R@20	N@20	exact-20
–	Popularity (train)	5.77	2.60	5.32	2.39	–
–	kNN prior ( $k{=}200$ )	21.60	11.00	21.58	10.94	–
–	GPT-5.6 zero-shot	23.66	12.43	25.41	14.02	–
–	Qwen3.5-27B zero-shot	13.31	7.88	13.36	8.33	–

Appendix

Table 17: Full test results on Reddit-Books and Reddit-Music.

	aggregation	scale	sampling	R@20	req. R@20	R@20, $n{=}1$	R@20, $n{\geq}5$
0	SFT (no RL)			15.40	15.32	14.75	15.57
1	per-responder	group	row (Rank-GRPO)	20.97	20.94	20.05	21.12
2	per-responder	batch	row	20.43	19.92	18.35	20.71
3	per-responder	batch	request	20.93	22.19	23.16	20.73
4	mean	batch	row	21.79	22.25	22.63	21.90
5	mean	batch	request	21.95	23.60	25.06	21.63

Appendix

Table 18: Ablations by number of responders

n

(2B, Reddit-v2 test, R@20).

difference (rows of Table 5 )	R@20	req. R@20	R@20, $n{=}1$	R@20, $n{\geq}5$
ReMaRk vs. Rank-GRPO (7 $-$ 1)	+1.57 [1.10,2.06]	+2.29 [1.43,3.17]	+3.70 [1.95,5.53]	+1.37 [0.86,1.89]
scale, bounded sum (7 $-$ 6)	+0.69 [0.32,1.06]	+0.86 [0.20,1.54]	+1.29 [ $-$ 0.04,2.69]	+0.63 [0.21,1.03]
bounded sum, batch scale vs. Rank-GRPO (6 $-$ 1)	+0.87 [0.47,1.28]	+1.43 [0.66,2.22]	+2.41 [0.61,4.09]	+0.74 [0.32,1.18]
reward, row sampling (4 $-$ 2)	+1.35 [0.91,1.78]	+2.32 [1.46,3.18]	+4.28 [2.38,6.25]	+1.19 [0.68,1.70]
reward, request sampling (5 $-$ 3)	+1.02 [0.53,1.50]	+1.42 [0.63,2.24]	+1.90 [0.37,3.42]	+0.90 [0.40,1.42]
reward, request sampling, bounded sum (6 $-$ 3)	+0.91 [0.43,1.34]	+0.19 [ $-$ 0.61,0.98]	$-$ 0.70 [ $-$ 2.21,0.84]	+1.13 [0.64,1.62]

Appendix

Table 19: Paired differences between the 2B ablation arms (test, points).

RL (all arms)
learning rate	$10^{-6}$ , constant; AdamW $(0.9,0.99)$
KL coefficient	$10^{-3}$
clipping $\epsilon_{\mathrm{low}}/\epsilon_{\mathrm{high}}$	0.06 / 0.08
group size $G$	8
requests / lists per step	48 / 384
policy updates per batch	1 (on-policy)

Appendix

Table 20: Hyperparameters.

		R@20	req. R@20	R@20, $n{\geq}5$
0	SFT	21.65	21.11	21.50
1	Rank-GRPO	30.53	29.86	30.45
2	per-resp., batch	30.65	30.06	30.53
4	mean, $\propto n$	31.50	30.99	31.37
5	mean, request	31.33	31.13	31.01
6	bounded sum, batch	31.43	30.62	31.56

Appendix

Table 21: Reddit-Books validation (2B).

RL step	Rank-GRPO	ReMaRk	$\Delta$
250	9.45	10.11	$+0.65$
500	9.89	12.71	$+2.82$
750	10.80	13.76	$+2.96$
1,000	11.60	13.88	$+2.29$
1,250	12.02	14.41	$+2.39$
1,500	12.41	14.15	$+1.74$

Appendix

Table 22: Reddit-Music (2B): validation and test R@20.

arm (2B)	3407	42	1234	seed 3407	3 seeds
	R@10 by seed			$\Delta$
(a) Rank-GRPO	42.73	41.81	42.54
(b) exact RB	44.39	44.33	44.17	+1.65 [0.24,3.08]	+1.94 [0.80,3.07]
(c) mean, batch	42.37	43.86	44.37	$-$ 0.36 [ $-$ 2.02,1.15]	+1.17 [ $-$ 1.01,3.35]
(d) ReMaRk	45.77	46.15	46.17	+3.04 [1.46,4.64]	+3.67 [2.64,4.69]

Appendix

Table 24: ProtoQA test results (558 questions, per-respondent recall@10).

Explore similar work

GR2: Generative Reasoning Re-ranker
Feb 8, 2026Mingfu Liang, Yufei Li, Jay Xu +20RL for Language Models LLM Reranking
Meta AI
FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning
Mar 12, 2026Yijun Pan, Weikang Qiu, Qiyao Ma +4Recommender Systems Large Language Model-Based Recommendation
Yale University · University of California, Davis · Snap Inc.
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM Evaluation Large Language Model-Based Recommendation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Feb 8, 2026J/K move · Enter open · S save

GR2: Generative Reasoning Re-ranker

Mingfu Liang, Yufei Li, Jay Xu, Kavosh Asadi+19

Meta AI

RL for Language Models LLM Reranking Recommender Systems Large Language Model-Based Recommendation LLM Reasoning

Open paper AskPDF arXiv

Recent studies increasingly explore Large Language Models (LLMs) as a new paradigm for recommendation systems due to their scalability and world knowledge. However, existing work has three key limitations: (1) most efforts focus on retrieval and ranking, while the reranking phase, critical for refining final recommendations, is largely overlooked; (2) LLMs are typically used in zero-shot or supervised fine-tuning settings, leaving their reasoning abilities, especially those enhanced through reinforcement learning (RL) and high-quality reasoning data, underexploited; (3) items are commonly represented by non-semantic IDs, creating major scalability challenges in industrial systems with billions of identifiers. To address these gaps, we propose the Generative Reasoning Reranker (GR2), an end-to-end framework with a three-stage training pipeline tailored for reranking. First, a pretrained LLM is mid-trained on semantic IDs encoded from non-semantic IDs via a tokenizer achieving

\ge

99% uniqueness. Next, a stronger larger-scale LLM generates high-quality reasoning traces through carefully designed prompting and rejection sampling, which are used for supervised fine-tuning to impart foundational reasoning skills. Finally, we apply Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO), enabling scalable RL supervision with verifiable rewards designed specifically for reranking. Experiments on two real-world datasets demonstrate GR2's effectiveness: it surpasses the state-of-the-art OneRec-Think by 2.4% in Recall@5 and 1.3% in NDCG@5. Ablations confirm that advanced reasoning traces yield substantial gains across metrics. We further find that RL reward design is crucial in reranking: LLMs tend to exploit reward hacking by preserving item order, motivating conditional verifiable rewards to mitigate this behavior and optimize reranking performance.

Abstract

Explore similar work

GR2: Generative Reasoning Re-ranker

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation