Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
Organizations: Incubilate · University of Massachusetts, Amherst
Abstract
Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay for a verdict that a checker cannot verify ("this program is correct") and for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when . Here is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises , so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with the model's success rate before RL, no run above ends collapsed, and runs under the checker can survive slightly below it. At , above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.
Figures & tables
| Datalog | code | abstention | |
|---|---|---|---|
| class at stake | underivable instances | buggy programs | math problems |
| unchecked verdict ( ) | DERIVABLE, unproved | CORRECT | “I don’t know” |
| attempt ( ) | UNDERIVABLE with a closure | BUGGY with a counterexample | an answer |
| checker | closure check | differential oracle | reference answer |
| other class | derivable instances | correct programs | none |
| Facts: p1, p2 Goal: p4 | |
|---|---|
| derivable instance | underivable twin |
| Rules: p3 :- p1, p2 | Rules: p3 :- p1, p2 |
| Rules: p4 :- p3 | Rules: p4 :- p6 |
| Rules: p5 :- p6 | Rules: p5 :- p6 |
| ANSWER: DERIVABLE | ANSWER: UNDERIVABLE |
| PROOF: | CLOSURE: |
| bracket | |||||
|---|---|---|---|---|---|
| task | base | collapses | survives | seeds | |
| Datalog | depth 2 | 0.77 | 0.80 | 0.95 | 2, 4 |
| depth 2, sharpened | 0.86 | 0.60 | 1.10 | 1, 1 | |
| depth 3 | 0.66 | 0.80 | 1.07 | 1, 2 | |
| depth 3, sharpened | 0.77 | – | 1.07 | –, 3 | |
| code | 3B coder | 0.735 | 0.95 | 2.00 | 1, 1 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| base | reward | seed | honest verdicts (%) at each eval (step: value) | outcome | ||
|---|---|---|---|---|---|---|
| depth 2, =0.77 | 0.6, 0.311 | checked | 0.40 | 1 | 20: 65, 40: 0, 60: 0, 80: 0, 100: 0 | collapses |
| 0.6, 0.466 | checked | 0.60 | 1 | 20: 59, 40: 0, 60: 0, 80: 0, 100: 0 | collapses | |
| 0.6, 0.621 | checked | 0.80 | 1 | 20: 85, 40: 1, 60: 0, 80: 0, 100: 0 | collapses | |
| 0.6, 0.621 | checked | 0.80 | 1 ∗ | 20: 89, 40: 2, 60: 0, 80: 0, 100: 0 | collapses | |
| 0.6, 0.621 | checked | 0.80 | 2 | 10: 95, 20: 81, 30: 12, 40: 1, 50: 0, 60: 0, 70: 0, 80: 0, 90: 0, 100: 0 | collapses | |
| 0.6, 0.621 | lottery | 0.80 | 1 | 20: 68, 40: 0, 60: 0, 80: 0, 100: 0 | collapses |
| model | BUGGY recall (%) at each evaluation (step: value) | correct called correct (%) | validity of attempts (%) | outcome | ||
|---|---|---|---|---|---|---|
| 3B coder | 0.204 | 0.50 | 10: 19, 20: 0, 30: 0 | 100 | 60 | collapses |
| 0.327 | 0.80 | 10: 20, 20: 0, 30: 0 | 100 | 33 | collapses | |
| 0.388 | 0.95 | 10: 22, 20: 0, 30: 0, 40: 0, 60: 0, 80: 0, 100: 0 | 100 | 75 | collapses | |
| 0.51 | 1.25 | 10: 26, 20: 1, 30: 0, 40: 0, 60: 1, 80: 32, 100: 39 | 97 | 81 | undecided | |
| 0.817 | 2.00 | 10: 51, 20: 50, 30: 70, 40: 82, 60: 80, 80: 81, 100: 74 | 79 | 80 | survives | |
| 1.225 | 3.00 | 10: 87, 20: 97, 30: 100, 40: 99, 60: 90, 80: 90, 100: 93 | 45 | 76 | survives |
| DERIVABLE | UNDERIVABLE | |||
| instance | valid proof | no valid proof | valid closure | no valid closure |
| derivable | – | |||
| underivable | – | |||
| model | regime | BUGGY recall | correct called correct | opening | |
|---|---|---|---|---|---|
| 3B coder | 0.50 | below 1 | 66, 19 (step 10), 0 (step 30) | 80, 100 | falls, collapses |
| 0.80 | below 1 | 66, 20 (step 10), 0 (step 30) | 80, 100 | falls, collapses | |
| 0.95 | below 1 | 66, 22 (step 10), 0 (step 100) | 80, 100 | falls, collapses | |
| 1.25 | 1 to | 66, 26 (step 10), 39 (step 100) | 80, 97 | falls, then rises | |
| 2.00 | just below | 66, 51 (step 10), 74 (step 100) | 80, 79 | falls, then rises | |
| 3.00 | above 2.2 | 66, 87 (step 10), 93 (step 100) | 80, 45 | rises |
| run | seed, step | honest | refit probe | frozen probe | frozen AUROC | frozen AUROC, best layer | base layer |
|---|---|---|---|---|---|---|---|
| depth 2, checked, | 1, 100 | 0 | 97.5 | 50.8 | 0.916 | 0.990 (layer 25) | 27 |
| depth 2, checked, | 1, 100 | 0 | 97.5 | 52.1 | 0.935 | 0.997 (layer 25) | 27 |
| depth 2, checked, | 1, 100 | 0 | 97.5 | 54.2 | 0.966 | 0.997 (layer 25) | 27 |
| depth 2, checked, | 2, 100 | 0 | 97.5 | 64.2 | 0.976 | 0.995 (layer 25) | 27 |
| depth 2, lottery, | 1, 100 | 0 | 97.9 | 52.1 | 0.959 | 0.995 (layer 25) | 27 |
| depth 2, checked, | 1, 100 | 76 | 98.3 | 91.7 | 0.998 | 0.999 (layer 25) | 27 |
| seed | steps | checked | lottery | difference | |
| 0.80 | 1 | 31–40 | 0.88 (108) | 0.70 (20) | |
| 0.93 | 1 | 91–100 | 0.85 (1814) | 0.90 (20) | |
| 0.95 | 1 | 51–60 | 0.85 (845) | 0.96 (24) | |
| 0.95 | 2 | 151–160 | 0.89 (1695) | 0.64 (22) | |
| 0.95 | 3 | 61–70 | 0.86 (1454) | 0.81 (32) | |
| 0.95 | 4 | 41–50 | 0.83 (1073) | 0.81 (201) |
Reinforcement Learning with Verifiable Rewards
Reinforcement learning with verifiable rewards trains models using outcomes checked by code, proofs, simulators, or structured criteria.
See all 734 papers