cs.LGNov 5, 2026

Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning

Authors: Amit Sagtani, Saloni Garg

Organizations: Santa Clara, California, USA

Abstract

Reinforcement learning from verifiable rewards (RLVR) is usually evaluated as an alternative to supervised fine-tuning (SFT), or as a stage that follows it, with the implicit assumption that the RL stage contributes gains of its own. We measure that contribution directly, in a controlled setting where both stages can be run to completion and compared exactly. We built gpu-train-debug, a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks across 30 archetypes, graded by a programmatic verifier with partial credit, and calibrated so that a mid-tier model scores in a 0.60 to 0.75 band. A pilot produced a surprise: 1,200 gold demonstrations nearly solved the benchmark (SFT 0.785 on the hard split), while 15 steps of GRPO from the base model did not move it (0.314 vs 0.311 base). We therefore amended the study, in writing and before the full run, to ask the sharper question: what does verifier-based RL add beyond demonstrations? At the pre-registered scale (60 GRPO steps, learning rate 1e-6), the answer is: no detectable improvement in-distribution (+0.22 points, 95% CI [-0.06, +0.51]), and a modest but measurable improvement under distribution shift (+0.33 points on a held-out OOD split, CI [+0.02, +0.68]). SFT accounted for most of the available improvement, while verifier-based RL produced only modest additional gains under distribution shift. The residual errors SFT leaves are mostly systematic, identical across samples, which likely reduces the within-group learning signal available to GRPO. A full 60-step cold-start GRPO arm confirms the pilot at the full horizon. The whole study cost $8.69 of compute and is fully reproducible from archived generations, frozen configs, and tagged code.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Quantifying Empirical Compute-Supervision Tradeoffs in RLVR

    May 24, 2026Ryo Mitsuhashi, Patrick Chen, Isabelle Tseng +2Reinforcement LearningRL for Language Model Reasoning

  2. An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

    Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code GenerationReinforcement Learning

  3. Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

    Apr 6, 2026Kazuki Egashira, Mark Vero, Jasper Dekoninck +3Reinforcement Learning with Verifiable RewardsRL for Language Model Reasoning