cs.LGApr 14, 2026

PubSwap: Public-Data Off-Policy Coordination for Federated RLVR

Authors: Anupam Nayak, Baris Askin, Muhammed Ustaomeroglu, Carlee Joe-Wong, Gauri Joshi

Organizations: Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reasoning post-training with reinforcement learning from verifiable rewards (RLVR) is typically studied in centralized settings, yet many realistic applications involve decentralized private data distributed across organizations. Federated training is a natural solution, but scaling RLVR in this regime is challenging: full-model synchronization is expensive, and performing many local steps can cause severe client drift under heterogeneous data. We propose a federated RLVR framework that combines LoRA-based local adaptation with public-data-based off-policy steps to improve both communication efficiency and cross-client coordination. In particular, a small shared public dataset is used to periodically exchange and reuse response-level training signals across organizations, providing a lightweight anchor toward a more globally aligned objective without exposing private data. Our method selectively replaces locally incorrect responses with globally correct ones during public-data steps, thereby keeping training closer to the local policy while still benefiting from cross-client coordination. Across mathematical and medical reasoning benchmarks and models, our method consistently improves over standard baselines. Our results highlight a simple and effective recipe for federated reasoning post-training: combining low-rank communication with limited public-data coordination.

Figures & tables

Figure 1: The figure illustrates our proposed PubSwap method, which alternates training on both public and private data. During a local step, each client performs a GRPO step on minibatches sampled from its own private dataset. During the public step, every client generates KK responses for the same shared batch of prompts and sends these generations to the server. Based on the response aggregation method, the server then returns the responses to each client, which are used them to perform GRPO update. Models are aggregated after each communication round.
ModelMethodMATHDeepMath
τ\tau =10τ\tau =40τ\tau =90τ\tau =120τ\tau =10τ\tau =40τ\tau =90τ\tau =120
Qwen3-1.7BBase model55.255.255.255.214.914.914.914.9
FedAvg-GRPO77.975.675.575.948.952.750.450.7
FedProx-GRPO77.576.076.575.650.652.348.047.7
FedAvg- PubSwap77.076.776.976.651.153.053.355.8
Qwen2.5- Math-1.5BBase58.458.458.458.434.934.934.934.9
Table 1: Pass @1 performance comparison on MATH ( Hendrycks et al., 2021 ) and DeepMath ( He et al., 2025 ) across different numbers of local steps ( τ\tau ). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap . We tune baselines to the best performance, more details are presented in the Appendix.
Methodτ\tau =10τ\tau =40τ\tau =90τ\tau =120
Base model51.151.151.151.1
FedAvg-GRPO66.966.567.067.4
FedAvg- PubSwap70.065.470.270.5
Table 2: Pass@1 performance of the model Qwen3-4B-Instruct-2507 on DeepMath across different numbers of local steps ( τ\tau ). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .
Methodτ\tau =10τ\tau =40τ\tau =90τ\tau =120
Base model49.249.249.249.2
FedAvg-GRPO58.758.257.956.0
FedAvg- PubSwap57.559.458.558.1
Table 3: Pass@1 performance of the model Llama3.2-3B-Instruct on medical reasoning across different numbers of local steps ( τ\tau ). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .
Methodτ\tau =40τ\tau =90τ\tau =120
Base model14.914.914.9
FedAvg+GRPO52.051.651.3
PubSwap53.056.754.9
Table 4: Pass@1 performance of the model Qwen3-1.7B on DeepMath across different numbers of local steps ( τ\tau ) for a higher heterogeneity split.
Methodτ\tau =80τ\tau =90τ\tau =120
FedAvg-GRPO48.050.450.7
FedProx-GRPO46.548.047.7
τswap\tau_{\texttt{swap}} = 252.253.355.8
τswap\tau_{\texttt{swap}} = 454.354.352.5
τswap\tau_{\texttt{swap}} = 853.153.452.6
Table 5: Pass@1 performance of Qwen3-1.7B across swap periods on DeepMath.
Methodτ\tau =90τ\tau =120
FedAvg-GRPO50.549.3
FedProx-GRPO48.049.4
τswap\tau_{\texttt{swap}} = 251.053.1
τswap\tau_{\texttt{swap}} = 452.353.6
τswap\tau_{\texttt{swap}} = 851.951.7
Table 6: Pass@1 performance of Qwen2.5-Math-1.5B across different swap periods on DeepMath.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Bubble plot (quantized by 100 samples) of per client topic distribution heterogeneity for the MATH dataset experiments
Figure 3: Bubble plot (quantized by 100 samples) of per client topic distribution (topic number) heterogeneity for the DeepMath dataset experiments
Figure 4: Bubble plot (quantized by 100 samples) of per client topic distribution heterogeneity for the high heterogeneity DeepMath dataset experiments (Table 4 )
ModelMethodMATHDeepMath
τ\tau =10τ\tau =40τ\tau =90τ\tau =120τ\tau =10τ\tau =40τ\tau =90τ\tau =120
Qwen3-1.7BBase model55.255.255.255.214.914.914.914.9
FedAvg-GRPO78.376.176.375.949.552.750.450.7
FedProx-GRPO77.576.776.575.652.052.348.047.7
FedAvg- PubSwap77.076.976.976.653.854.653.355.8
Qwen2.5- Math-1.5BBase model58.458.458.458.434.934.934.934.9
Appendix
Table 7: Best checkpoint pass@1 performance comparison on MATH ( Hendrycks et al., 2021 ) and DeepMath ( He et al., 2025 ) across different numbers of local steps ( τ\tau ). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .
Methodτ\tau =10τ\tau =40τ\tau =90τ\tau =120
Base model49.249.249.249.2
FedAvg-GRPO59.758.957.957.7
FedAvg- PubSwap58.359.558.558.1
Appendix
Table 8: Best checkpoint pass@1 performance of the model Llama3.2-3B-Instruct on medical reasoning across different numbers of local steps ( τ\tau ). The Balanced method and a swap period of 2 is used here for response aggregation with PubSwap .
ParameterValueParameterValue
Pretrained ModelQwen3-1.7BNum Clients4
Grad epochs per GRPO step2Train Batch Size8
Max prompt length1024Max response length2048
LoRA rank32LoRA alpha64
LoRA modulesall linearGen. per prompt8
Learning rate1×10−51\times 10^{-5}Clip ratio low0.2
Appendix
Table 9: Configuration for Qwen3-1.7B
ParameterValueParameterValue
Pretrained ModelQwen2.5-Math-1.5BNum Clients4
Grad epochs per GRPO step2Train Batch Size8
Max prompt length1024Max response length2048
LoRA rank32LoRA alpha64
LoRA modulesall linearGen. per prompt8
Learning rate1×10−51\times 10^{-5}Clip ratio low0.2
Appendix
Table 10: Configuration for Qwen2.5-Math-1.5B
ParameterValueParameterValue
Pretrained ModelQwen3-4B-Instruct-2507Num Clients4
Grad epochs per GRPO step2Train Batch Size4
Max prompt length1024Max response length2048
LoRA rank32LoRA alpha64
LoRA modulesall linearGen. per prompt8
Learning rate1×10−51\times 10^{-5}Clip ratio low0.2
Appendix
Table 11: Configuration for Qwen3-4B-Instruct
ParameterValueParameterValue
Pretrained ModelLlama-3.2-3B-InstructNum Clients4
Grad epochs per GRPO step2Train Batch Size8
Max prompt length1024Max response length2048
LoRA rank32LoRA alpha64
LoRA modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_projGen. per prompt8
Learning rate1×10−51\times 10^{-5}Clip ratio low0.2
Appendix
Table 12: Configuration for Llama-3.2-3B-Instruct
More in this topic

Federated RL

Federated reinforcement learning lets distributed agents learn policies collaboratively while keeping their local data and experience private.

See all 34 papers

Explore similar work

CardsList
Oct 8, 2026

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) global reward calibration that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) adaptive sparse communication that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by 32×32\times and supporting up to 621×621\times compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO.

Pengxin Guo, Shuang Zeng, Zonggen Li +3

The University of Hong Kong · Sun Yat-sen University

Federated RLRL for Language Model ReasoningCommunication-Efficient Distributed TrainingGroup Relative Policy Optimization
AskPDF
Aug 7, 2025

FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models

Enhancing LLM reasoning in federated settings is nontrivial due to stringent computational, communication, and privacy constraints, especially in healthcare, where clinically consequential decisions require not only accuracy but also interpretable, auditable rationales to meet safety, accountability, and regulatory requirements. Conventional federated fine-tuning largely imitates final answers rather than cultivating step-by-step reasoning, often relying on privacy-sensitive centralized distillation and still incurring substantial communication overhead. We address this gap with \ours{}, a federated reasoning framework that combines lightweight chain-of-thought resampling with a compact discriminator for selection, and client-aware LoRA stacking with weighted classifier aggregation to accommodate heterogeneity while reducing aggregation noise and communication; clients generate candidate chains and supervision locally, and only lightweight modules are aggregated on the server. Experiments on medical reasoning benchmarks show consistent gains under tight resource budgets while keeping data local and respecting privacy, offering an interpretable and resource-efficient solution. Our code is made publicly available at https://github.com/DIaacKr/FedCoT

Chuan Li, Qianyi Zhao, Fengran Mo +1

East China Normal University, China · University of Montreal, Canada

Communication-Efficient Distributed TrainingMedical LLMsLLM ReasoningFederated Learning
AskPDF
Apr 20, 2026

Efficient Federated RLHF via Zeroth-Order Policy Optimization

This paper considers reinforcement learning from human feedback in a federated learning setting with resource-constrained agents, such as edge devices. We propose an efficient federated RLHF algorithm, named Partitioned, Sign-based Stochastic Zeroth-order Policy Optimization (Par-S2^2ZPO). The algorithm is built on zeroth-order optimization with binary perturbation, resulting in low communication, computation, and memory complexity by design. Our theoretical analysis establishes an upper bound on the convergence rate of Par-S2^2ZPO, revealing that it is as efficient as its centralized counterpart in terms of sample complexity but converges faster in terms of policy update iterations. Our experimental results show that it outperforms a FedAvg-based RLHF on four MuJoCo RL tasks.

Deyi Wang, Qining Zhang, Lei Ying

University of Michigan, Ann Arbor

Federated RLCommunication-Efficient Distributed TrainingZeroth-Order OptimizationPolicy LearningRL from Human Feedback
AskPDF