cs.LGNov 11, 2026

Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff

Authors: Nathaniel Bottman, Alexei Oblomkov

Organizations: Incubilate · University of Massachusetts, Amherst

Abstract

Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay bb for a verdict that a checker cannot verify ("this program is correct") and rr for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when p~ r/b>1+κ\tilde{p}\,r/b>1+κ. Here p~\tilde{p} is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and κκ is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises p~\tilde{p}, so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term κκ starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with pp the model's success rate before RL, no run above pr/b=1pr/b=1 ends collapsed, and runs under the checker can survive slightly below it. At pr/b=3pr/b=3, above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

    Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code GenerationReinforcement Learning

  2. Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

    Oct 1, 2025Xin-Qiang Cai, Wei Wang, Feng Liu +3Reinforcement LearningRL for Language Model Reasoning