cs.LGSep 12, 2026

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards

Authors: Juntao Yu

Organizations: Fudan University, Shanghai, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in {0.1, 0.3, 0.5}, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

Figures & tables

Figure 1: Same model, data and hyperparameters; only the group-gating rule differs. Right: the mechanism—both unsafe arms are driven to very short final responses (roughly 3–18 tokens).
Figure 2: Endpoint EM versus phantom strength λ\lambda . Error bars on the λ=0.30\lambda{=}0.30 anchor are sample standard deviations across four runs per arm.
Figure 3: The same three arms across model scale and task difficulty. On MATH, unsafe arms at both scales reach EM 00 within 20–30 steps, while outcome filtering keeps learning.
Figure 4: Confirmation on verl’s official recipe/dapo trainer with physical group deletion and batch refilling. Right: one generation batch always suffices for the shaped-score metric, whereas the accuracy metric exercises the refill path in 29 of 40 steps.
Figure 5: De-noised held-out accuracy under a faulty verifier.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Explore similar work

CardsList

  1. STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

    Sep 29, 2026Wan Tian, Zhongyi Li, Xiang Xu +3Reward HackingGroup Relative Policy Optimization

    Peking University · Beihang University · Nanjing University

  2. Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

    May 8, 2026Wenhua Nie, Jianan Wu, Junlin Liu +6Reinforcement LearningGroup Relative Policy Optimization

    National Taiwan University

  3. F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

    Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement LearningRL for Language Model Reasoning

    T-Tech · Saint Petersburg Electrotechnical University “LETI”