cs.LGSep 12, 2026

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards

Authors: Juntao Yu

Organizations: Fudan University, Shanghai, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in {0.1, 0.3, 0.5}, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

Figures & tables

Figure 1: Same model, data and hyperparameters; only the group-gating rule differs. Right: the mechanism—both unsafe arms are driven to very short final responses (roughly 3–18 tokens).
Figure 2: Endpoint EM versus phantom strength λ\lambda . Error bars on the λ=0.30\lambda{=}0.30 anchor are sample standard deviations across four runs per arm.
Figure 3: The same three arms across model scale and task difficulty. On MATH, unsafe arms at both scales reach EM 00 within 20–30 steps, while outcome filtering keeps learning.
Figure 4: Confirmation on verl’s official recipe/dapo trainer with physical group deletion and batch refilling. Right: one generation batch always suffices for the shaped-score metric, whereas the accuracy metric exercises the refill path in 29 of 40 steps.
Figure 5: De-noised held-out accuracy under a faulty verifier.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Explore similar work

CardsList
Sep 29, 2026

STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

Reward hacking occurs when policy optimization exploits a brittle reward interface or an overly permissive proxy objective, improving the training score without improving the underlying response quality. This phenomenon is amplified in group-relative policy optimization: an unsupported reward can shift the group baseline and alter the updates of other rollouts, while post-hoc or purely relative weighting cannot represent group-wide uncertainty. We propose Self-Tuned Anchored Reliability Group-Relative Policy Optimization (STAR-GRPO), a reliability-first advantage estimator based on paired assessments of the same rollout. STAR separates the quality signal from its learning influence: score disagreement determines rollout reliability, relative reliability enters a self-tuned robust location--scale fit before group normalization, and absolute group reliability attenuates the resulting bounded advantage. The analysis establishes coordinate and second-moment bounds, characterizes exact centering through the weighted location equation, and gives reliability-dependent attenuation guarantees for outlying rewards. We evaluate STAR-GRPO in two complementary reward-hacking regimes. In token-interface exploitation, STAR prevents runaway optimization of the deployed-interface score while improving the canonical quality signal. In rubric-proxy overoptimization for medical reasoning, STAR improves independent semantic evaluation, narrows the proxy--judge discrepancy, and reduces overclaim while optimizing the same task proxy. Together, these results show that reliability-first normalization offers a principled way to limit unsupported reward influence on both group baselines and policy updates, while retaining the task reward as the optimization target.

Wan Tian, Zhongyi Li, Xiang Xu +3

Peking University · Beihang University · Nanjing University

Reward HackingGroup Relative Policy Optimization
AskPDF
May 8, 2026

Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage, A=2r−1A=2r-1, performs pass@GG failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with p<0.0001p<0.0001. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@kk analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.

Wenhua Nie, Jianan Wu, Junlin Liu +6

National Taiwan University

Reinforcement LearningGroup Relative Policy OptimizationReinforcement Learning with Verifiable Rewards
AskPDF
Feb 6, 2026

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-NN transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at N=8N{=}8, our method improves average math pass@256 from 64.1 →\rightarrow 70.3 (GRPO), 69.3 →\rightarrow 72.5 (DAPO), and 73.2 →\rightarrow 76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4

T-Tech · Saint Petersburg Electrotechnical University “LETI”

Reinforcement LearningRL for Language Model ReasoningGroup Relative Policy OptimizationReinforcement Learning with Verifiable Rewards
AskPDF