cs.LGNov 10, 2026

Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

Authors: Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan

Abstract

In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin M=p1−p2M=p_1-p_2 of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95% CI [4.1,9.0][4.1, 9.0]), and a Qwen3-4B-Base replication improves for all three seeds (+4.75+4.75). The gain is largest on problems that GRPO solves only some of the time and grows with kk in pass@kk, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

    Oct 17, 2026Diyi Hu, Ruizhong Qiu, Lin Qiu +2RL for Language ModelsRL for Language Model Reasoning

  2. Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

    Jun 17, 2026Yingyu Shan, Yuhang Guo, Zihao Cheng +7Token-Level Credit AssignmentRL for Language Model Reasoning