Group Relative Policy Optimization

Also known as GRPO

Group Relative Policy Optimization studies how to improve a model or agent by sampling several responses to the same prompt and using their relative rewards to guide policy updates. Its critic-free, group-normalized approach is widely used for reinforcement learning with verifiable rewards in reasoning tasks such as GSM8K and MATH, and has been adapted to tool-use agents, vision-language models, and diffusion-based image and video generation. Current methods refine reward design and credit assignment, adjust sampling and curricula, and modify normalization, clipping, or entropy controls to improve learning efficiency and stability. A central challenge is that sparse or uniform group rewards can provide little or misleading signal, while choices such as response-length normalization and diversity control can trade off update reliability, exploration, and performance.

Latest papers

341 papers53 papers in the last four weeks, up 33% on the four weeks before. 0.3% of all new papers.

JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks
Nov 3, 2026Shanka Subhra Mondal, Jiayun WangRL for GUI Agents Long-Horizon Agent Tasks
Center for Advanced AI, Accenture
When Rewards Depend on the Group, GRPO Plays a Tournament
Nov 2, 2026Junyi Gao, Yu Shi, Pingzhao Hu +1Group Relative Policy Optimization Reward Design for RL
Centre for Medical Informatics, University of Edinburgh · Health Data Research UK · Biostatistics Division, Dalla Lana School of Public Health, University of Toronto +1
One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets
Oct 31, 2026Jiaying Zhou, Jiachen Liu, Qingchao ChenGroup Relative Policy Optimization Policy Gradient
Peking University
Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards
Oct 31, 2026Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao +2Reinforcement Learning RL for Language Model Reasoning
RIKEN-AIP · Northeastern University · UNC-Chapel Hill +1
Distributional Reward Calibration and Budget-Neutral Credit Assignment for Multimodal RLVR
Oct 29, 2026Lang Chen, Zongxin Xu, Borui Wang +2Reinforcement Learning Token-Level Credit Assignment
Shenzhen University
Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents
Oct 29, 2026Yuxiang Chai, Quyu Kong, Panrong Tong +2RL for GUI Agents Reinforcement Learning
Alibaba Token Hub, Alibaba Group · MMLab, The Chinese University of Hong Kong
When Does Closed-Loop Curriculum Adaptation Help GRPO? An Analysis on Countdown
Oct 28, 2026Yuv Raj Pant, Moshiur Rahman Tonmoy, Kaiqun FuRL for Language Model Reasoning Curriculum RL
Department of Computer Science, Texas Christian University · Department of Electrical Engineering and Computer Science, Texas A&M University - Kingsville
Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training
Oct 27, 2026Faeze Moradi Kalarde, Armin Toroghi, Nam Ho Koh +4RL Post-Training Group Relative Policy Optimization
University of Toronto · Vector Institute of AI · Korea Advanced Institute of Science & Technology
SetGRPO: Calibrating Grader-Induced Policy Optimization
Oct 27, 2026Jiale Huang, Zixu Li, Zhiwei Chen +3Reinforcement Learning Group Relative Policy Optimization
Shandong University · National University of Singapore
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
Oct 26, 2026Shurun Li, Michael Hua Wang, Haibo Zeng +1Reward Hacking Reinforcement Learning with Verifiable Rewards
REASONS Lab, Institute for Network Sciences and Cyberspace, Tsinghua University · Virginia Tech
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Oct 26, 2026Sohyun Son, Yerim Hong, Jaeyun Park +1Reinforcement Learning RL for Language Model Reasoning
Kyung Hee University, Yongin, Republic of Korea
Correlation-Aware Exploration for Long-Trace Reasoning
Oct 26, 2026Junxing Hu, Zicheng Zhang, Zhiqian Zhang +4RL for Language Model Reasoning RL for Language Models
Microsoft
Direction-Aware Token Credit Assignment with Reward-Model Gradients
Oct 23, 2026Yikuan Hu, Jifeng Zhu, Minle Su +6Token-Level Credit Assignment Policy Gradient Methods
Nanjing University · National University of Singapore · Tsinghua University +1
Pseudo-Labels as Rewards, Not Targets: Group-Relative Policy Optimization for Semi-Supervised ASR
Oct 23, 2026Shashi Kumar, Yanis Labrak, Sergio Burdisso +4Automatic Speech Recognition Domain Adaptation for ASR
Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic +1
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Oct 23, 2026Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang +3Policy Optimization Group Relative Policy Optimization
The University of Hong Kong · Tsinghua University
Group Relative Policy Optimisation Improves Multilingual Speech Recognition in Low-Resource Languages
Oct 22, 2026Tim Wenjie Hu, Evelyn Nafula Ouma, Benjamin Akera +1Automatic Speech Recognition Multilingual ASR
Sunbird AI, Uganda
Uni-Mem: Learning a Long-Horizon Agentic Memory Manager with Unified Online Policy Optimization
Oct 21, 2026Zhiqian Xia, Jielong Tang, Ling-An Zeng +8Memory Management for LLM Agents RL for LLM Agents
Southeast University · Huawei Technologies, Noah’s Ark Lab · Sun Yat-sen University +1
Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Oct 21, 2026Xin Zhang, Haochen Wang, Beibei Lin +4Multimodal Grounding Multimodal Understanding
Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs
Oct 21, 2026Yeonbum Kim, Hyun Kyu Lee, Sung Whan YoonRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea · Department of Electrical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea.
FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit
Oct 19, 2026Jiaxuan Sun, Biao Wang, Shuai LiRL for Video Reasoning Video Reasoning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tencent Hy
Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning
Oct 19, 2026Lakshya ChaudhryFine-Tuning RL Fine-Tuning
Purdue University
Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts
Oct 18, 2026Tomas Duncun, Jonah Ellison, Owen Pell +4Adaptive Sampling Sample-Efficient RL
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom
Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training
Oct 18, 2026Jonah Ellison, Mateo Ruiz, Owen Pell +4RL for Language Model Reasoning RL Post-Training
University of Michigan · University of Edinburgh
EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents
Oct 18, 2026Sijia Luo, Mengjie Gao, Yuxiao He +4RL for LLM Agents Step-Level Credit Assignment in RL
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China · Ant Group, Beijing, China +2
ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
Oct 17, 2026Diyi Hu, Ruizhong Qiu, Lin Qiu +2RL for Language Models RL for Language Model Reasoning
Independent Researcher · University of Illinois Urbana-Champaign · Meta
MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
Oct 16, 2026Jihao Huang, Guangjia Chai, Zhirui Chen +9RL for Language Model Reasoning LLM Reasoning
Rethinking GRPO for Large Reasoning Model Safety Alignment
Oct 16, 2026Jiawei Chen, Yi Zhang, Jiawei Du +9Safety-Preserving Fine-Tuning of Language Models LLM Safety Alignment
East China Normal University · Baidu Inc. · Zhongguancun Academy +5
SCOPE: Semantic Bounded Credit Optimization for Prompt Engineering in Text-to-image Generation
Oct 15, 2026Yufei Jiao, Jinfa Huang, Zeyue Xue +5Prompt Optimization for T2I Generation RL for Image Generation
Xiamen University · University of Rochester · The University of Hong Kong +1
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Oct 14, 2026Sheldon Yu, Tong Yu, Nikki Lijing Kuang +6RL for Language Model Reasoning RL Post-Training
University of California, San Diego · Adobe Research · University of Virginia +2
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Oct 14, 2026Xinrui Lin, Fan Li, Shumin Wang +2Pairwise Preference Learning RL for Video Generation
University of Science and Technology of China, Hefei, China · Huawei Noah’s Ark Lab

Nov 3, 2026J/K move · Enter open · S save

JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks

Shanka Subhra Mondal, Jiayun Wang

Center for Advanced AI, Accenture

RL for GUI Agents Long-Horizon Agent Tasks Hierarchical MARL Group Relative Policy Optimization

Open paper AskPDF arXiv

Multimodal web agents have shown promising capabilities for interacting with websites through graphical user interfaces (GUIs), yet realistic web tasks often require long-horizon, multi-step planning that remains challenging for monolithic policies. Such agents must jointly infer high-level plans, navigate through intermediate states, track progress over extended interaction histories, and execute visually grounded actions. Although prior work has explored modular multi-agent systems with a separate planner, the planner is typically not updated jointly with the actor. We investigate whether explicitly separating and jointly training planning and execution provides a stronger inductive bias for long-horizon web navigation. We introduce JHiMA-WebRL, a joint hierarchical multi-agent reinforcement learning framework for web agents that couples a task decomposer with an actor. To train the decomposer, we use a strong teacher model to construct structured subtask supervision from multimodal GUI trajectories, and use these annotations to fine-tune both the decomposer and the subtask-conditioned actor. We then extend Group Relative Policy Optimization (GRPO) to jointly optimize the decomposer and actor in the interactive WebArena environment through hierarchical rollout grouping and role-specific advantages. On WebArena-Lite-v2, JHiMA-WebRL-14B, consisting of a 7B actor and a 7B decomposer, achieves an average success rate of 37.2%, compared with 24% for a ScaleCUA-32B agent that we fine-tune with RL on the same training tasks and evaluate under the same protocol. Ablations show that jointly updating the decomposer during RL outperforms updating the actor alone. These results suggest that jointly learned role factorization with smaller models can be a competitive alternative to scaling monolithic agents for long-horizon web tasks.