Group Relative Policy Optimization

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

14 new papers

A weekly snapshot of new work published in Group Relative Policy Optimization.

Period ending 2026-09-14

6 new papers

A weekly snapshot of new work published in Group Relative Policy Optimization.

Period ending 2026-09-07

21 new papers

A weekly snapshot of new work published in Group Relative Policy Optimization.

Inside this field

Focused directions

Focused topic

Group Relative Policy Optimization

Within Group Relative Policy Optimization

287 papersExplore

Focused topic

Multi-Objective Policy Optimization

Within Group Relative Policy Optimization

122 papersExplore

Focused topic

Policy Gradient

Within Group Relative Policy Optimization

64 papersExplore

Focused topic

Proximal Policy Optimization

Within Group Relative Policy Optimization

60 papersExplore

Focused topic

Flow-Grpo

Within Group Relative Policy Optimization

42 papersExplore

532 papers

Latest in Group Relative Policy Optimization

Browse all topics →

On Advantage Estimates for Max@K Policy Gradients
Jun 4, 2026Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6Policy Gradient Inference-Time
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
Jun 4, 2026Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili +1Group Relative Policy Optimization Flow-Grpo
SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter
Jun 4, 2026Powei Chang, Jinpeng Zhang, Chaoqun Sun +6Group Relative Policy Optimization Policy Gradient
TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents
Jun 4, 2026Chengqi Dong, Chuhuai Yue, Hang He +6Credit Assignment Group Relative Policy Optimization
Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces
Jun 3, 2026Nicolás Astorga, Nabeel Seedat, Mihaela van der SchaarPrompt Optimization Llm-Based Optimization Modeling
Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
Jun 3, 2026Chirag Chawla, Rohan Charudatt Salvi, Madhav S. BaidyaGroup Relative Policy Optimization Policy Gradient
Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents
Jun 3, 2026Dae Yon Hwang, Raunaq Suri, Valentin Villecroze +4Agentic Reinforcement-Learning Framework Large Language Model Agents
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
Jun 3, 2026Saket Reddy, Ke Yang, ChengXiang ZhaiGroup Relative Policy Optimization Reinforcement Learning From Human Feedback
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Jun 3, 2026Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2Group Relative Policy Optimization Verifiable Rewards
Policy Gradient for Continuous-Time Robust Markov Decision Processes
Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaMarkov Decision Processes Policy Gradient
POLARIS: Guiding Small Models to Write Long Stories
Jun 2, 2026Rishanth Rajendhran, Jenna Russell, Mohit Iyyer +1Narratives Small Models
Physics-Guided Policy Optimization with Self-Distillation
Jun 2, 2026Ke Wang, Yuning Wu, Haoran Liu +3Unsupervised On-Policy Self-Distillation Multi-Objective Policy Optimization
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Jun 2, 2026Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5Proximal Policy Optimization Multi-Objective Policy Optimization
GPU-Parallel Multi-Task Reinforcement Learning with Demonstration Guided Policy Optimization
Jun 2, 2026Rui Zhang, Qiwei Wu, Zhengyu Zhang +5Multi-Task Reinforcement Learning Multi-Objective Policy Optimization
SD-GRPO: Verifiable Segment Decomposition for Long-Form Vision-Language Generation
Jun 2, 2026Hyunwoong Kim, Seongeun Lee, Hannah Yun +2Large Vision Language Models Group Relative Policy Optimization
FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data
Jun 2, 2026Pengyu Chen, Shaowei Li, Kai Wang +4Federated Reinforcement Learning Group Relative Policy Optimization
Self-Distilled Policy Gradient
Jun 2, 2026Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1Unsupervised On-Policy Self-Distillation On-Policy Distillation
Hint-Guided Diversified Policy Optimization for LLM Reasoning
Jun 2, 2026Zhiyu Cao, Kaixin Wu, Mingjie Zhong +4Large Language Model Reasoning Reinforcement Learning With Verifiable Reward
Towards Precise Intent-Aligned VLA Aerial Navigation via Expert-Guided GRPO
Jun 1, 2026Tianyang Chen, Wenjun Li, Xin zhou +2Aerial Vision-Language Navigation Vision-Language-Action Models
Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing
Jun 1, 2026Azal Ahmad Khan, Ammar Ahmed, Zeshan Fayyaz +3Group Relative Policy Optimization Deep Reinforcement Learning
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Jun 1, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Group Relative Policy Optimization Preference Alignment
RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
May 31, 2026Yixiu Mao, Yun Qu, Qi Wang +2Reinforcement Learning With Verifiable Reward Group Relative Policy Optimization
Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specifications
May 30, 2026Vignesh Subramanian, Subhajit Roy, Suguman BansalBehavior Cloning Deep Reinforcement Learning
Beyond Pure Sampling: Hybrid Optimization Mechanisms for Non-Convex Model Predictive Control
May 30, 2026Yuichiro Aoyama, Minchan Jung, Akash Ratheesh +1Model Predictive Control Proximal Policy Optimization
Improving Visual Representation Alignment Generation with GRPO
May 30, 2026Shentong Mo, Sukmin YunVision Encoders Group Relative Policy Optimization
Interpretable Policy Distillation for Power Grid Topology Control
May 30, 2026Aleksandra Dmitruka, Karlis FreivaldsProximal Policy Optimization Power Systems
LithoGRPO: Fast Inverse Lithography via GRPO Reinforced Flow Matching
May 29, 2026Yao Lai, Xuyuan Xiong, Zeyue Xue +7Lithography Industrial Automation
CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
May 29, 2026Yang Li, Gongle Xue, Yijia Guo +3Reinforcement Learning With Verifiable Reward Unsupervised On-Policy Self-Distillation
EchoRL: Reinforcement Learning via Rollout Echoing
May 29, 2026Jinhe Bi, Aniri, Minglai Yang +9Reinforcement Learning Post-Training Reinforcement Learning With Verifiable Reward
Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments
May 29, 2026Nishant Kumar, Enrique Areyan Viqueira, Amy GreenwaldPolicy Gradient
Distilling LLM Feedback for Lean Theorem Proving
May 29, 2026Gaetan Narozniak, Gérard Biau, Rémi Munos +2Token-Level Supervision Group Relative Policy Optimization
Safe Equilibrium Policy Optimization for Strategic Agent Policies
May 29, 2026Karthika Arumugam, Kiran Kumar Manku, Amit DhandaGroup Relative Policy Optimization Optimal Policies
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
May 29, 2026Yiming Ren, Yiran Xu, Zicheng Lin +8Group Relative Policy Optimization Multi-Objective Policy Optimization
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillMulti-Objective Policy Optimization Deep Reinforcement Learning
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadGradient Descent Adam
VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangGroup Relative Policy Optimization Reinforcement Learning With Verifiable Reward
HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime
May 28, 2026Mohamed Sana, Nicola Piovesan, Antonio De Domenico +2Sparse Rewards Multi-Objective Policy Optimization
ESPO: Early-Stopping Proximal Policy Optimization
May 28, 2026Zihang Li, Rui Zhou, Yingcheng Shi +8Proximal Policy Optimization
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Group Relative Policy Optimization Multi-Objective Policy Optimization
OISD: On-Policy Internal Self-Distillation of Language Models
May 27, 2026Xinyu Liu, Darryl Cherian Jacob, Yang Zhou +2Unsupervised On-Policy Self-Distillation On-Policy Distillation
TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoning
May 27, 2026Chusen Li, Zhou Liu, Shuigeng Zhou +1Deep Reinforcement Learning Group Relative Policy Optimization
ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation
May 27, 2026Hongru Hou, Tiehua Mei, Denghui Geng +5Policy Gradient Deep Reinforcement Learning
Long Live The Balance: Information Bottleneck Driven Tree-based Policy Optimization
May 27, 2026Hao Jiang, Shurui Li, Tianpeng Bu +7Multi-Objective Policy Optimization Exploration-Exploitation Dilemma
BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
May 27, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2Group Relative Policy Optimization Code Completion
DecomposeRL: Learning to Ask Useful, Informative, and Diverse Questions for Semi-Supervised, Traceable Claim Verification
May 27, 2026Shubhashis Roy Dipta, Ankur Padia, Francis FerraroNoisy Claim Verification Group Relative Policy Optimization
Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization
May 26, 2026Cihan Xiao, Yiwen Shao, Chenxing Li +5Audio Representation Cross-Modal Learning
Training-Inference Kernel Contracts: Bounding Divergence in Post-Training and Deployment
May 26, 2026Bruce Changlong Xu, Lan WuTraining-Free Inference Framework LLM Post-Training Methods
FSA-GRPO: Teaching Auditory LLMs to Use Few-shot Demonstrations
May 26, 2026Haolong Zheng, Siyin Wang, Xulin Fan +2Large Audio Language Models Few-Shot Learning
Touch-R1: Reinforcing Touch Reasoning in MLLMs
May 26, 2026Yingxin Lai, Yafei Zhou, Fucai Zhu +2Tactile Perception Group Relative Policy Optimization
Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning
May 26, 2026Yilong Li, Suman Banerjee, Tong CheCode Generation Multi-Objective Policy Optimization
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
May 26, 2026Xin Cheng, Shuo He, Lang Feng +4Group Relative Policy Optimization Credit Assignment
Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problems
May 26, 2026Wolfgang Maass, Sabine JanzenExponential Horizon Dependence Policy Gradient
Bilevel Optimization over Saddle Points of Zero-Sum Markov Games
May 26, 2026Zihao Zheng, Irwin King, Songtao LuBilevel Optimization Policy Gradient
Sample Complexity of Policy Gradient for Log-Growth Control
May 26, 2026Qiuhua Pan, Yukai Shen, Liwei Zhang +2Policy Gradient Lyapunov Function
Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
May 26, 2026Woojeong Kim, Ziyi Yang, Jing Nathan Yan +1Group Relative Policy Optimization Rollout
Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient
May 26, 2026Haoxiang You, Yilang Liu, Davis Zong +5Visuomotor Policy Policy Gradient
Credit-assigned Policy Gradient for Early Stage Retrieval in Two-stage Ranking
May 25, 2026Haruka Kiyohara, Mihaela Curmei, Ariel Evnine +7Reranking Policy Gradient
Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
May 25, 2026Xiaoyuan Cheng, Wenxuan Yuan, Zhancun Mu +5Diffusion Policies Multi-Objective Policy Optimization
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
May 25, 2026Zhaoyu Zhu, Rui Gao, Shuang LiPolicy Gradient Entropy-Regularized Equilibrium Selection
Merge-Bench: Resolve Merge Conflicts with Large Language Models
May 25, 2026Benedikt Schesch, Michael D. ErnstModel Merging Machine Learning

Jun 4, 2026 · cs.LGJ/K move · Enter open · S save

On Advantage Estimates for Max@K Policy Gradients

Shota Takashiro, Soichiro Nishimori, Paavo Parmas, Yongmin Kim+5

The University of Tokyo

Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is to optimize inference-time objectives such as pass@K and max@K directly, yet existing policy-gradient estimators for these objectives use different signals, baselines, and normalizations, making their relationships unclear. We study this issue through baseline design and advantage centering. Starting from the advantage estimator of a leading method in the field, we show that it is policy-gradient unbiased but yields a non-centered advantage. We then introduce a Leave-Two-Out baseline that preserves policy-gradient unbiasedness while making realized batch advantages exactly centered. The resulting method, MaxPO, has an efficient quadratic-time implementation and integrates naturally into group-based RL for LLM post-training. We further derive the canonical finite-batch advantage for max@K, providing a unified view of existing advantage estimators. Empirically, we verify that the L2O baseline reduces gradient variance and outperforms non-centered alternatives.

Policy Gradient Inference-Time

Open paper Ask about it PDF