Group Relative Policy Optimization

Follow this topic Ask about this topic

Recent momentum

-15%

22 papers in the last 28 days · 0.4% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Group Relative Policy Optimization.

Period ending 2026-09-07

12 new papers

A weekly snapshot of new work published in Group Relative Policy Optimization.

287 papers

Latest in Group Relative Policy Optimization

Browse all topics →

Reinforcement Learning with Decomposed Subtasks
Sep 22, 2026Mattie Terzolo, Mikolaj Sacha, Ayan Sinha +1Group Relative Policy Optimization
Marginally Correct Tool Caches Can Reverse Group-Normalized Policy Updates
Sep 22, 2026Shivam GuptaGroup Relative Policy Optimization Long Rollouts
Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?
Sep 21, 2026Nazmus SakibLong Rollouts Group Relative Policy Optimization
EPIG-Tree: Compute-Optimal Branching for Gradient-Efficient Reinforcement Learning
Sep 17, 2026Nikita Khomich, Leopold Hermansson, Ido HakimiGroup Relative Policy Optimization Autoregressive Rollout
GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control
Sep 17, 2026Pengqin Wang, Qiming Zhang, Shaojie Shen +1Group Relative Policy Optimization Robot Policies
EmbodiedMind: Adaptive Data Curation and Prefix-Tree Reinforcement Learning for Efficient Embodied Intelligence
Sep 17, 2026Feifan Wang, Zongbing Zhang, Yu Zhang +8Deep Reinforcement Learning Long-Horizon Planning
Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR
Sep 16, 2026Xiuwen ZhengAutoregressive Decoding Low-Latency Systems
HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments
Sep 14, 2026Quoc-Vinh Lai-Dang, Hyo-Sang ShinReinforcement Learning With Verifiable Reward Multi-Objective Policy Optimization
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs
Sep 14, 2026Naihao Deng, Samee Arif, Shuaichen Chang +2Fairness Constraints Large Language Model Alignment
Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
Sep 9, 2026Haoyue Liu, Xiaoyu Ma, Ye Chen +2Group Relative Policy Optimization External Reward Models
ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
Sep 8, 2026Tommy Sha, Skylar Zhai, Siqi ZhaoReinforcement Learning With Verifiable Reward Group Relative Policy Optimization
SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
Sep 8, 2026Shengtian Yang, Ziyu Xiong, Yu Li +3Group Relative Policy Optimization Multi-Agent Large Language Model
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Sep 7, 2026Chenqian Le, Jiayi Cheng, Qijia He +3Agentic Reinforcement-Learning Framework Group Relative Policy Optimization
Spurious Advantage Hidden in GRPO
Sep 3, 2026Jiamian Wang, Samyadeep Basu, Koustava Goswami +2Group Relative Policy Optimization Advantage
Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
Sep 3, 2026Hyun Bin Park, Du-Seong ChangGroup Relative Policy Optimization Agentic Search
TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
Sep 3, 2026Jinwei GanGroup Relative Policy Optimization Credit Assignment
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation Teacher
On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers
Sep 1, 2026Vignesh Prabhakar, Jialing Pan, Anil Babu AnkisettipalliOn-Policy Distillation Reranking
Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
Sep 1, 2026Prakhar Gupta, Vaibhav GuptaLLM Post-Training Methods Supervised Fine-Tuning
A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss
Sep 1, 2026Suryaansh Jain, Rahasya Barkur, Vishal G +8Remote Sensing Image Change Captioning Fine-Grained Perception
CRAFT: Fine-Tuning Pre-hoc Explainability in AI-native 6G RAN
Sep 1, 2026Pranshav Gajjar, Vijay K ShahOpen Radio Access Networks Mobile Networks
Group Adaptive Clipping Policy Optimization
Aug 31, 2026Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan +1Group Relative Policy Optimization Gradient Clipping
When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models
Aug 31, 2026Joonyong Park, Jerry LiTraining-Free Reward Alignment Group Relative Policy Optimization
DiffPDE: Masked Diffusion Language Models as PDE Solver
Aug 31, 2026Wenxuan Guo, Yuyang Hong, Lubin Fan +4Partial Differential Equation Benchmarks Partial Differential Equations
Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving
Aug 31, 2026Tian Zhang, Zhuo Huang, Hongrui Ye +3Flow-Matching Vision-Language-Action Group Relative Policy Optimization
Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Trajectory-Level Credit
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Aug 13, 2026Yubo Zhang, Xinhong Ma, Zezhong Tan +1Unsupervised On-Policy Self-Distillation Group Relative Policy Optimization
GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation
Aug 12, 2026Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam +3Finance Domain Large Language Model Decision-Making
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Aug 12, 2026Kai Yang, Jingwei Xu, Wanyu Wang +4Group Relative Policy Optimization Autoregressive Rollout
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Utkarsh Tyagi +6Rubric-Based Reinforcement Learning Rubrics
SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation
Aug 10, 2026Jefferson Hernandez, Jaywon Koo, Zilin Xiao +2Group Relative Policy Optimization Advantage
MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Experts
Aug 10, 2026Peiwen Li, Shiyang Zhang, Yangtian Zhang +3Multi-Agent Systems Task-Level
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
Aug 7, 2026Yiyao Zhang, Diksha Goel, Hussain Ahmad +2Judges Llm-As-A-Judge
AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward
Aug 7, 2026Mingyang Wu, Kaituo Feng, Bohao Li +3Video Captioning Group Relative Policy Optimization
GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization
Aug 6, 2026Sajjad Ghiasvand, Nader SehatbakhshLanguage Model Evasion Attacks Anonymization
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
Aug 6, 2026Yingqing Guo, Hui Yuan, Zijian He +2Flow-Based Policy Reward Gradients
Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO
Aug 5, 2026Xuzheng Yang, Jun Ling, Tao Huang +2Refusals Mllm-Based
ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation
Aug 5, 2026Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun +5Image Generation Multimodal Agents
When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO
Aug 4, 2026Zhe Cao, Miaowen Wen, Fangjiong ChenReinforcement Learning With Verifiable Reward Credit Assignment
Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
Aug 4, 2026Yongshi Ye, Liang Zhang, Yidong Chen +2Reinforcement Learning With Verifiable Reward Group Relative Policy Optimization
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Aug 4, 2026Wen Wang, Jiahua Bao, Tu Yongsiqi +8Sparse Rewards Group Relative Policy Optimization
Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
Aug 3, 2026Lecheng Yan, Jianze Lin, Yichong Zhang +5Video Editing Agentic Video Generation
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Aug 1, 2026Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9Group Relative Policy Optimization On-Policy Distillation
MDGAM-Based Cooperative Task Scheduling for Communication-Constrained Distributed Multi-Agent Systems
Aug 1, 2026Licheng Wang, Mingtao Huang, Yuan ShenMulti-Agent Reinforcement Learning Multi-Robot Systems
Element-Aware Group Learning for E-Commerce Image Generation
Aug 1, 2026Jingtong Chen, Jiahui Wang, Xue Zhao +2Image Generation Group Relative Policy Optimization
Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
Jul 30, 2026Songshuo Lu, Zhi Chen, Yaohua TangTeacher Recursive Reasoner
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Jul 30, 2026Xilin Jiang, Riki Shimizu, Sukru Samet Dindar +3Conversational Artificial Intelligence Speech Large Language Models
ReCo: Reweighting GRPO Against Distributional Concentration
Jul 29, 2026Junoh Park, Junseo Hwang, Wonguk Cho +1Group Relative Policy Optimization Reweighting
FAS-R1: A Unified Multi-Task MLLM for Reasoning Face Anti-Spoofing
Jul 29, 2026Hongyang Wang, Yichen Shi, Hongrui Li +3Spoofed Speech Authenticity
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Jul 28, 2026Bo-Wen Zhang, Junwei He, Wen Wang +5Rubric-Based Reinforcement Learning Credit Assignment
ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning
Jul 28, 2026Jiaqi Zhang, Tong Chen, Junliang Yu +2Agentic Reasoning Multi-Objective Policy Optimization
Training Language Models to Cooperate with Inference-Time Controllers
Jul 26, 2026Moumita Choudhury, Vanshaj Khattar, Jing Liu +4Large Language Model Post-Training Inference-Time Control
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Jul 25, 2026Fei Ding, Yongkang Zhang, Yuhao Liao +2Group Relative Policy Optimization Multi-Objective Policy Optimization
Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning
Jul 25, 2026Xiaokun Wang, Siyu Song, Wentao Liu +1Socratic Dialogue Large Language Model Alignment
Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
Jul 24, 2026Heyang Jiang, Henry Liu, Baharan MirzasoleimanReinforcement Learning With Verifiable Reward Long Rollouts
Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization Agentic Reinforcement-Learning Framework
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Jul 21, 2026Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou +4Reinforcement Learning With Verifiable Reward Group Relative Policy Optimization
DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization
Jul 21, 2026Thanni Adewuyi, Angelica Obayi, Andem Aniekan +10Medical Vision-Language Models Chest Radiography
Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM
Jul 21, 2026Xiaomeng Hu, Jiaqi Hu, Hao Chen +4Large Language Model Hallucination Large Language Model Reasoning
RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts
Jul 20, 2026Yuxin Xiong, Xunyi Jiang, Rohan Surana +8Group Relative Policy Optimization Reinforcement Learning With Verifiable Reward

Sep 22, 2026 · cs.AIJ/K move · Enter open · S save

Reinforcement Learning with Decomposed Subtasks

Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich

Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.