ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training

Rui Ai; Yu Pan; David Simchi-Levi; Chonghuan Wang

cs.AIMar 31, 2026

ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training

Authors: Rui Ai, Yu Pan, David Simchi-Levi, Chonghuan Wang

Organizations: MIT · University of Sydney · University of Texas at Dallas

Abstract

In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collective utility of the entire set rather than individual candidates independently. However, existing reinforcement learning post-training paradigms, such as Group Relative Policy Optimization (GRPO), typically assign the same set-level scalar reward to every candidate in the set. This leads to noisy training signals where poor candidates free-ride on the high reward produced by a single strong peer, resulting in suboptimal exploration. To address this, we propose Shapley-Enhanced GRPO (ShapE-GRPO). By leveraging the permutation-invariant nature of set-level utility, we derive a Shapley-enhanced formulation from cooperative game theory to decompose set-level rewards into granular, candidate-specific signals. We show that our formulation preserves the fundamental axioms of the Shapley value while remaining computationally efficient with polynomial-time complexity. Empirically, ShapE-GRPO consistently outperforms standard GRPO across diverse datasets with accelerated convergence during training.

Figures & tables

Prompt:

Please recommend two or three places near Seattle suitable for a trip.

Response:

Considering +5.0 different +5.0 preferences, +5.0 I +5.0 recommend +5.0 the +5.0 following +5.0 locations +5.0 that +5.0 cover +5.0 natural +5.0 scenery, +5.0 cultural +5.0 landmarks, +5.0 and +5.0 technological +5.0 marvels: +5.0 Hiking +7.5 at +7.5 Mount +7.5 Rainier; +7.5 Touring +4.5 the +4.5 Boeing +4.5 Factory; +4.5 Experiencing +3.0 German +3.0 culture +3.0 in +3.0 Leavenworth. +3.0 <EOS> 0.0

Figure 1: When the user considers the ratings for the three suggestions to be 5.0, 4.0, and 3.0, respectively, our ShapE-GRPO performs candidate-level (with broadcasting to token-level) reward allocation as above. However, GRPO only uses a single +5.0 sequence-level reward shared across tokens.

Dataset	Qwen3-8B	GRPO	WTA	ShapE-GRPO
ACLSum ( $\uparrow$ )	0.654	0.658	0.642	0.672
DS-1000 ( $\uparrow$ )	0.398 4 4 footnotemark: 4	0.465	0.418	0.510
Netflix w/ history ( $\uparrow$ )	2.872 5 5 5 We limit the maximum output length to 4096 tokens during training. Since the pretrained model often exceeds this limit, we allow up to 16384 tokens when evaluating the pretrained model. When restricted to 4096 tokens, the pretrained model achieves a score of 0.273.	3.710	3.550	4.141
Netflix w/o history ( $\uparrow$ )	3.395	3.627	2.497	3.821

Table 1: Performance comparison across four datasets using Qwen3-8B . Winner-Takes-All assigns candidate-level reward only to the highest-scoring candidate, whereas ShapE-GRPO allocates reward by scaled Shapley values. ShapE-GRPO consistently improves over both the standard GRPO baseline and the Winner-Takes-All baseline.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Response reward as a function of the number of candidates across different datasets using Qwen3-8b .

Hyperparameter	Value
Training epochs	1
Train batch size	2
Learning rate	1e-6
Max response length	4096
Sampling temperature	0.7
Top- $p$	0.9

Appendix

Table 2: Main training hyperparameters used in our experiments.

Dataset	Llama 3.1‑8B‑Instruct	GRPO	ShapE-GRPO
ACLSum ( $\uparrow$ )	0.664	0.707	0.780
DS-1000 ( $\uparrow$ )	0.180	0.205	0.278
Netflix w/ history ( $\uparrow$ )	1.885	3.783	4.027
Netflix w/o history ( $\uparrow$ )	2.130	3.783	4.071

Appendix

Table 3: Performance comparison across four datasets using Llama 3.1‑8B‑Instruct . ShapE-GRPO consistently improves over both the pretrained model and the standard GRPO baseline.

Explore similar work

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization
Oct 8, 2026Pengxin Guo, Shuang Zeng, Zonggen Li +3Federated RL RL for Language Model Reasoning
The University of Hong Kong · Sun Yat-sen University
Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning
Jul 21, 2026Wentao Zhang, Haoyu Zhang, Xinke Jiang +7RL for Language Model Reasoning Shapley Value Attribution
ShanghaiTech University · City University of Hong Kong · Peking University +2
Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
Nov 11, 2025Chih-Hsuan, Yang, Tanwi Mallick +5Multi-Agent LLM Systems Cooperative Game Theory
Bella · Argonne National Laboratory · & University of Chicago

Oct 8, 2026J/K move · Enter open · S save

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng+2

The University of Hong Kong · Sun Yat-sen University

Federated RL RL for Language Model Reasoning Communication-Efficient Distributed Training Group Relative Policy Optimization

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) global reward calibration that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) adaptive sparse communication that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by

32\times

and supporting up to

621\times

compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO.

Abstract

Explore similar work

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents