PubSwap: Public-Data Off-Policy Coordination for Federated RLVR

Anupam Nayak; Baris Askin; Muhammed Ustaomeroglu; Carlee Joe-Wong; Gauri Joshi

cs.LGApr 14, 2026

PubSwap: Public-Data Off-Policy Coordination for Federated RLVR

Authors: Anupam Nayak, Baris Askin, Muhammed Ustaomeroglu, Carlee Joe-Wong, Gauri Joshi

Organizations: Carnegie Mellon University

Abstract

Reasoning post-training with reinforcement learning from verifiable rewards (RLVR) is typically studied in centralized settings, yet many realistic applications involve decentralized private data distributed across organizations. Federated training is a natural solution, but scaling RLVR in this regime is challenging: full-model synchronization is expensive, and performing many local steps can cause severe client drift under heterogeneous data. We propose a federated RLVR framework that combines LoRA-based local adaptation with public-data-based off-policy steps to improve both communication efficiency and cross-client coordination. In particular, a small shared public dataset is used to periodically exchange and reuse response-level training signals across organizations, providing a lightweight anchor toward a more globally aligned objective without exposing private data. Our method selectively replaces locally incorrect responses with globally correct ones during public-data steps, thereby keeping training closer to the local policy while still benefiting from cross-client coordination. Across mathematical and medical reasoning benchmarks and models, our method consistently improves over standard baselines. Our results highlight a simple and effective recipe for federated reasoning post-training: combining low-rank communication with limited public-data coordination.

Figures & tables

Figure 1: The figure illustrates our proposed PubSwap method, which alternates training on both public and private data. During a local step, each client performs a GRPO step on minibatches sampled from its own private dataset. During the public step, every client generates $K$ responses for the same shared batch of prompts and sends these generations to the server. Based on the response aggregation method, the server then returns the responses to each client, which are used them to perform GRPO update. Models are aggregated after each communication round.

Model	Method	MATH				DeepMath
Model	Method	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120
Qwen3-1.7B	Base model	55.2	55.2	55.2	55.2	14.9	14.9	14.9	14.9
	FedAvg-GRPO	77.9	75.6	75.5	75.9	48.9	52.7	50.4	50.7
	FedProx-GRPO	77.5	76.0	76.5	75.6	50.6	52.3	48.0	47.7
	FedAvg- PubSwap	77.0	76.7	76.9	76.6	51.1	53.0	53.3	55.8
Qwen2.5- Math-1.5B	Base	58.4	58.4	58.4	58.4	34.9	34.9	34.9	34.9

Table 1: Pass @1 performance comparison on MATH ( Hendrycks et al., 2021 ) and DeepMath ( He et al., 2025 ) across different numbers of local steps (

\tau

). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap . We tune baselines to the best performance, more details are presented in the Appendix.

Method	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120
Base model	51.1	51.1	51.1	51.1
FedAvg-GRPO	66.9	66.5	67.0	67.4
FedAvg- PubSwap	70.0	65.4	70.2	70.5

Table 2: Pass@1 performance of the model Qwen3-4B-Instruct-2507 on DeepMath across different numbers of local steps (

\tau

). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .

Method	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120
Base model	49.2	49.2	49.2	49.2
FedAvg-GRPO	58.7	58.2	57.9	56.0
FedAvg- PubSwap	57.5	59.4	58.5	58.1

Table 3: Pass@1 performance of the model Llama3.2-3B-Instruct on medical reasoning across different numbers of local steps (

\tau

). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .

Method	$\tau$ =40	$\tau$ =90	$\tau$ =120
Base model	14.9	14.9	14.9
FedAvg+GRPO	52.0	51.6	51.3
PubSwap	53.0	56.7	54.9

Table 4: Pass@1 performance of the model Qwen3-1.7B on DeepMath across different numbers of local steps (

\tau

) for a higher heterogeneity split.

Method	$\tau$ =80	$\tau$ =90	$\tau$ =120
FedAvg-GRPO	48.0	50.4	50.7
FedProx-GRPO	46.5	48.0	47.7
$\tau_{\texttt{swap}}$ = 2	52.2	53.3	55.8
$\tau_{\texttt{swap}}$ = 4	54.3	54.3	52.5
$\tau_{\texttt{swap}}$ = 8	53.1	53.4	52.6

Table 5: Pass@1 performance of Qwen3-1.7B across swap periods on DeepMath.

Method	$\tau$ =90	$\tau$ =120
FedAvg-GRPO	50.5	49.3
FedProx-GRPO	48.0	49.4
$\tau_{\texttt{swap}}$ = 2	51.0	53.1
$\tau_{\texttt{swap}}$ = 4	52.3	53.6
$\tau_{\texttt{swap}}$ = 8	51.9	51.7

Table 6: Pass@1 performance of Qwen2.5-Math-1.5B across different swap periods on DeepMath.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Bubble plot (quantized by 100 samples) of per client topic distribution heterogeneity for the MATH dataset experiments

Figure 3: Bubble plot (quantized by 100 samples) of per client topic distribution (topic number) heterogeneity for the DeepMath dataset experiments

Figure 4: Bubble plot (quantized by 100 samples) of per client topic distribution heterogeneity for the high heterogeneity DeepMath dataset experiments (Table 4 )

Model	Method	MATH				DeepMath
Model	Method	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120
Qwen3-1.7B	Base model	55.2	55.2	55.2	55.2	14.9	14.9	14.9	14.9
	FedAvg-GRPO	78.3	76.1	76.3	75.9	49.5	52.7	50.4	50.7
	FedProx-GRPO	77.5	76.7	76.5	75.6	52.0	52.3	48.0	47.7
	FedAvg- PubSwap	77.0	76.9	76.9	76.6	53.8	54.6	53.3	55.8
Qwen2.5- Math-1.5B	Base model	58.4	58.4	58.4	58.4	34.9	34.9	34.9	34.9

Appendix

Table 7: Best checkpoint pass@1 performance comparison on MATH ( Hendrycks et al., 2021 ) and DeepMath ( He et al., 2025 ) across different numbers of local steps (

\tau

). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .

Method	$\tau$ =10	$\tau$ =40	$\tau$ =90	$\tau$ =120
Base model	49.2	49.2	49.2	49.2
FedAvg-GRPO	59.7	58.9	57.9	57.7
FedAvg- PubSwap	58.3	59.5	58.5	58.1

Appendix

Table 8: Best checkpoint pass@1 performance of the model Llama3.2-3B-Instruct on medical reasoning across different numbers of local steps (

\tau

). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .

Parameter	Value	Parameter	Value
Pretrained Model	Qwen3-1.7B	Num Clients	4
Grad epochs per GRPO step	2	Train Batch Size	8
Max prompt length	1024	Max response length	2048
LoRA rank	32	LoRA alpha	64
LoRA modules	all linear	Gen. per prompt	8
Learning rate	$1\times 10^{-5}$	Clip ratio low	0.2

Appendix

Table 9: Configuration for Qwen3-1.7B

Parameter	Value	Parameter	Value
Pretrained Model	Qwen2.5-Math-1.5B	Num Clients	4
Grad epochs per GRPO step	2	Train Batch Size	8
Max prompt length	1024	Max response length	2048
LoRA rank	32	LoRA alpha	64
LoRA modules	all linear	Gen. per prompt	8
Learning rate	$1\times 10^{-5}$	Clip ratio low	0.2

Appendix

Table 10: Configuration for Qwen2.5-Math-1.5B

Parameter	Value	Parameter	Value
Pretrained Model	Qwen3-4B-Instruct-2507	Num Clients	4
Grad epochs per GRPO step	2	Train Batch Size	4
Max prompt length	1024	Max response length	2048
LoRA rank	32	LoRA alpha	64
LoRA modules	all linear	Gen. per prompt	8
Learning rate	$1\times 10^{-5}$	Clip ratio low	0.2

Appendix

Table 11: Configuration for Qwen3-4B-Instruct

Parameter	Value	Parameter	Value
Pretrained Model	Llama-3.2-3B-Instruct	Num Clients	4
Grad epochs per GRPO step	2	Train Batch Size	8
Max prompt length	1024	Max response length	2048
LoRA rank	32	LoRA alpha	64
LoRA modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj	Gen. per prompt	8
Learning rate	$1\times 10^{-5}$	Clip ratio low	0.2

Appendix

Table 12: Configuration for Llama-3.2-3B-Instruct

Federated RL

Federated reinforcement learning lets distributed agents learn policies collaboratively while keeping their local data and experience private.

See all 34 papers

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization
Oct 8, 2026Pengxin Guo, Shuang Zeng, Zonggen Li +3Federated RL RL for Language Model Reasoning
The University of Hong Kong · Sun Yat-sen University
FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models
Aug 7, 2025Chuan Li, Qianyi Zhao, Fengran Mo +1Communication-Efficient Distributed Training Medical LLMs
East China Normal University, China · University of Montreal, Canada
Efficient Federated RLHF via Zeroth-Order Policy Optimization
Apr 20, 2026Deyi Wang, Qining Zhang, Lei YingFederated RL Communication-Efficient Distributed Training
University of Michigan, Ann Arbor

Oct 8, 2026J/K move · Enter open · S save

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng+2

The University of Hong Kong · Sun Yat-sen University

Federated RL RL for Language Model Reasoning Communication-Efficient Distributed Training Group Relative Policy Optimization

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) global reward calibration that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) adaptive sparse communication that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by

32\times

and supporting up to

621\times

compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO.

Abstract

Federated RL

Explore similar work

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models

Efficient Federated RLHF via Zeroth-Order Policy Optimization