Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay
b for a verdict that a checker cannot verify ("this program is correct") and
r for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when
p~r/b>1+κ. Here
p~ is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and
κ is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises
p~, so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term
κ starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with
p the model's success rate before RL, no run above
pr/b=1 ends collapsed, and runs under the checker can survive slightly below it. At
pr/b=3, above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.