Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.
Figures & tables
Figure 1: The policy improves because of the reward. Qwen3-1.7B trained with GRPO on MATH (training split), changing only the reward. Every evaluation uses reference answers, never the reward model. Left: accuracy after 100 steps, with 95% paired-bootstrap intervals for the gain over the base model. Right: our reward improves the policy at every checkpoint and ends level with the answer key. Random reward does not help. Each evaluated checkpoint carries about ±1.8 points of 95% uncertainty.
Hypothesis
Test (falsifying control)
Verdict
H1
RL with the calibrated decision reward makes the policy more accurate on held-out problems, and the reward, not RL itself, is the cause.
E6: GRPO, ground-truth eval (random-reward floor)
Supported
H2
The reference-free reward trains the policy as well as the answer key.
E6 (answer-key ceiling); E6b
Supported on MATH; Partly on hard problems
H3
The reward stays calibrated and unhacked on the policy’s own samples, better than a softmax judge on the same backbone.
E6 on-policy logs (same-backbone softmax)
Supported
H4
The decision model locates reasoning errors at least as well as a dedicated process reward model, with no step training.
With equal calibration labels, decision probabilities are better reward probabilities for correctness than softmax probabilities and scalar-RM scores.
E1, E2 (same labels, same backbone, head refit)
Supported vs. softmax; Partly vs. scalar RM
H6
On hard problems, step-level decision credit gives a better policy than the answer key alone, and a policy better than the softmax judge on the full set.
E6b; E6 (softmax judge)
Partly
Table 1: Hypotheses, how each is tested, and the outcome. Supported : significant in the predicted direction. Partly : the predicted direction, not yet significant, or significant only on some measures.
Work
Calibrated
Debiased
Typed / multi-option
Step-level
Online LLM RL
d-RLAIF [ 27 ]
✗
labels only
score scale
✗
✓
GenRM [ 77 ]
✗
✗
binary
✗
✗
Reward Bridge [ 58 ]
✗
✗
binary
✗
✓
CCR [ 53 ]
✗
✗
binary
✗
✓
Probe judges [ 69 ]
not measured
n/a
binary
✗
✓
OPRM [ 6 ]
✓
✗
score scale
✗
✗
Table 2: Probability readouts used as RL rewards. ✓ = present, ✗ = absent. Compiled from the cited papers.
Figure 2: RLCDR in one picture. Each policy answer is wrapped in a typed question. The decision model returns a distribution over the options in one forward pass. Averaging over option orders removes position bias, and a two-parameter fit on a small labeled set makes the probability calibrated. The calibrated probability is the reward.
Figure 3: The experimental program. Stage 1 asks whether the decision model is a good reward on fixed data (E1, E2, E4). Stage 2 asks whether training with it improves a real model, with controls that isolate the reward (E6). Stage 3 tests the regime where the answer key gives no signal (E6b). The boxes at the bottom name the hypotheses ( Table 1 ) that each experiment tests.
Exp.
Data (size)
Compared
Labels
Main metric
E1
PPE best-of-K (5,120), JudgeBench (620), RM-Bench (3,981), RewardBench 2 (1,723)
FxAI-DM, softmax, Skywork-V2-8B
0–1,600
Brier, NLL, AUROC
E2
Same as E1, hidden states saved
6 head variants
25–1,600
Brier
E4
ProcessBench (3,400 solutions, 16,548 steps)
FxAI-DM, softmax, Qwen2.5-Math-PRM-7B
0–400
First-error F1
E6
MATH train (5,584) → MATH500, AIME24/25
5 rewards ( Section 4 )
1,024 (frozen)
pass@1, pass@ k
E6b
1,758 hard MATH problems; 4,451 + 800 policy answers
FxAI-DM only; answer key ± step credit
1,024 (frozen)
pass@1, re-solve drop
Table 3: Every experiment at a glance. “Labels” is the number of labelled examples any method may use for calibration. All splits are by question; nothing is fitted on evaluation items.
Reward used for RL
MATH500
M500-clean
AIME24
AIME25
M500 pass@4
None (base model)
72.2
72.3
11.3
9.4
85.2
Random (floor)
72.8
72.8
12.3
8.5
85.8
Softmax judge (same backbone)
75.5
75.5
12.9
10.0
88.4
RLCDR (ours)
76.8
76.9
16.5
10.8
88.4
Answer key (uses reference answers)
76.9
77.1
14.4
13.1
89.2
Table 4: Ground-truth evaluation after 100 GRPO steps (pass@1, %; MATH500 pass@4 in the last column). Only the reward differs between rows. Bold : best reward that does not use reference answers.
Comparison
Δ
95% CI
p
Gain over base model
RLCDR − base
+4.65
[+2.80,+6.50]
<10−4
Answer key − base
+4.75
[+3.00,+6.55]
<10−4
Softmax − base
+3.35
[+1.55,+5.15]
3×10−4
Random − base
+0.65
[−1.30,+2.50]
0.50
What causes the gain
RLCDR − random
+4.00
[+2.05,+6.05]
10−4
RLCDR − answer key
−0.10
[−1.75,+1.50]
0.90
Table 5: Paired differences on MATH500 after 100 steps (points; 500 problems × 4 samples). The second block isolates the causes.
Base
Random
Softmax
RLCDR
Answer key
MATH500 mean tokens
837
863
856
1,056
1,049
MATH500 truncated
4.3%
5.0%
4.0%
9.7%
9.2%
AIME24 truncated
37%
32%
37%
62%
64%
Table 6: Response length after 100 steps. Mean MATH500 tokens; share of responses truncated at 3,072 tokens.
Figure 4: Every gain over the base model, with 95% paired-bootstrap intervals (after 100 steps; number of problems in brackets). An interval that does not cross the black zero line is a significant gain. Our reward and the answer key are indistinguishable on every slice; random reward is indistinguishable from no training on every slice.
Figure 5: MATH500 gain over base by difficulty level (after 100 steps). Our reward’s gain rises with difficulty, from about +3 points on levels 2–3 to +6.9 on level 5 (44.8% → 51.7%), tracking the answer key. Level 1 has only 43 problems that the base model already solves 96% of the time, so its differences ( ±2 points) are noise.
Figure 6: Accuracy of the policy’s own training samples during RL (256 samples per step, averaged over 10 steps, labelled by the answer key but never shown to the non-answer-key rewards). Every informative reward pulls away from the random-reward run after about 20 steps. Note: training accuracy is measured on training problems; Table 4 gives held-out results.
Figure 7: The reward measured on the policy’s own samples during RL (25,600 samples per run, labelled with the answer key, which neither reward saw). (a) Our reward’s probabilities track the true accuracy of the policy’s answers about twice as closely. (b) The gap between reward and truth stays flat over training. Our reward is slightly cautious (never overpaying), while the softmax judge undervalues correct answers about twice as much. (c) Within each GRPO group, our reward ranks a correct answer above a wrong one more often at every difficulty level. This within-group ordering is what drives GRPO’s update.
Figure 8: The decision model finds the step where a failed solution goes wrong. Including the step it flags roughly halves the chance of recovering, whereas a random step has no effect. The effect is 45% larger than for the softmax judge, and the prefix our model keeps before the flagged step is more useful (13.3% vs. 10.6% recovery).
Answer key
FxAI-DM only (reference-free)
Answer key + FxAI-DM step credit
Score of an answer
1 if correct, else 0
rout+(1−rout)0.5prog
1 if correct, else 0.5prog
Group with all 8 answers wrong
no gradient
graded
graded
Uses the reference answer
yes
no
yes (0/1 part only)
Table 7: What each hard-problem reward sees. rout is the calibrated FxAI-DM probability that the final answer is correct; prog is the critic’s progress score ( Section 3 ).
Policy (50 steps, hard problems only)
MATH500
Level 5
AIME24
AIME25
Base (Qwen3-1.7B)
72.2
44.8
11.3
9.4
Answer key
75.3
50.6
14.8
9.2
FxAI-DM only (reference-free)
74.1
48.9
13.1
10.0
Answer key + FxAI-DM step credit
75.7
52.1
12.9
13.1
FxAI-DM only − base
+1.9[+0.2,+3.8]
+4.1[−0.2,+8.4]
+1.9
+0.6
FxAI-DM only − answer key
−1.2[−2.9,+0.5]
−1.7[−5.8,+2.4]
−1.7
+0.8
Table 8: RL on hard problems (pass@1, %, after 50 steps; paired 95% bootstrap CIs; level 5 has 134 MATH500 problems). Pooled over both AIME sets, only the hybrid policy improves significantly on base ( +2.7[+1.0,+4.7] , p=0.006 ; answer key +1.7 , p=0.26 ; FxAI-DM only +1.2 , p=0.24 ).
Figure 9: Hard-problem RL (50 steps on the 1,758 problems the base model mostly fails). (a) On held-out MATH500, the reference-free reward (dark blue) is above base at every level but below the answer key on levels 2, 4 and 5. Answer key + step credit (light blue) gives the best level-5 accuracy (52.1% vs. 50.6%; base 44.8%). (b) On the training problems, the reference-free run learns more slowly and rises only in the last 20 steps. The hybrid is slightly behind the answer key here, so its held-out advantage is not from fitting the training set better.
Method
Labels
GSM8K
MATH
Olympiad
Omni-MATH
Average
Softmax judge (same 27B backbone)
100
0.725
0.721
0.644
0.637
0.682
Qwen2.5-Math-PRM-7B (step-trained)
100
0.811
0.773
0.659
0.645
0.722
FxAI-DM decision (ours)
100
0.816
0.799
0.736
0.737
0.772
FxAI-DM decision, zero-shot
0
0.858
0.799
0.705
0.716
0.770
FxAI-DM one-question critic, zero-shot
0
0.861
0.789
0.719
0.687
0.764
Softmax one-question critic, zero-shot
0
0.790
0.731
0.655
0.610
0.696
Table 9: Finding the first wrong step (ProcessBench F1, 3,400 human-labelled solutions; 20 random half splits; “labels” = solutions used to fit the calibration and the cutoff). FxAI-DM was never trained on step labels.
Figure 10: Step-level rewards on ProcessBench (100 calibration labels, mean of 20 splits). The decision model, never trained on step labels, beats the dedicated step-trained PRM on every subset, and its lead is largest on the hardest ones (Olympiad +7.7, Omni-MATH +9.2 F1). Its step probabilities also rank better (AUROC) and, after calibration, are the most accurate (Brier).
Figure 11: Equal-label comparison of reward probabilities. Left: on PPE (one sampled policy answer per item, the RL reward case), the decision readout with zero labels is already better than either baseline with 1,600 labels. Right: the decision readout wins where the question is whether an answer is correct. A preference-trained scalar RM wins on preference-pair benchmarks that match its training format.
FxAI-DM
Softmax judge
FxAI-DM
Domain
Acc.
ECE raw
ECE MATH-fit
> 0.9
ECE raw
ECE MATH-fit
100-label refit
MATH
0.36
0.084
0.048
0.87 (128)
0.133
0.062
–
MMLU-Pro
0.48
0.096
0.074
0.94 (224)
0.094
0.115
–
IFEval
0.50
0.086
0.075
0.91 (197)
0.199
0.300
–
GPQA
0.39
0.118
0.088
0.76 (49)
0.100
0.110
0.075
MBPP+ (code)
0.60
0.162
0.142
0.74 (487)
0.211
0.233
0.090
Table 10: One calibration, five domains. PPE, 1,024 policy answers per domain. ECE before calibration (raw) and after the two Platt parameters fitted on MATH only, the fit used for RL (MATH is in-sample). “ > 0.9”: accuracy among answers the raw readout scores above 0.9 ( n in parentheses). Last column: Platt refit on 100 in-domain labels, mean over 20 splits.
Benchmark
Method
Acc ↑
AUROC ↑
Brier ↓
NLL ↓
ECE
Wins
PPE (policy answers)
FxAI-DM decision
0.735
0.809
0.184
0.562
0.058
–
Softmax (same backbone)
0.658
0.709
0.222
0.642
0.059
0/20
Scalar RM (Skywork-V2-8B)
0.668
0.708
0.217
0.636
0.073
0/20
JudgeBench
FxAI-DM decision
0.780
–
0.297
0.446
0.066
–
Softmax (same backbone)
0.708
–
0.373
0.556
0.067
0/20
Scalar RM
0.744
–
0.352
0.523
0.063
1/20
Table 11: Full equal-label comparison ( N=100 calibration labels, mean over 20 held-out splits). “Wins” is the number of splits in which the method’s Brier beats FxAI-DM. AUROC applies to pointwise PPE only.
Figure 12: Why the decision readout is the better reward for correctness. (a) It separates correct from wrong policy answers best on MATH, MMLU-Pro and IFEval; the scalar RM is better on code and GPQA, our weakest domains. (b) Before debiasing, the softmax readout’s bias against the first option slot is 4–6 × larger than the decision readout’s. Without debiasing the softmax judge loses 5.4 accuracy points on PPE; the decision readout loses at most 0.8.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Features + head
PPE
JudgeBench
RM-Bench
RewardBench 2
FxAI-DM + Platt (shipped head, 2 parameters)
0.179
0.328
0.157
0.212
FxAI-DM head refit
0.178
0.318
0.155
0.185
FxAI-DM new head on features
0.177
0.322
0.154
0.193
Base Qwen3.8-27B, new head (probe judge)
0.198
0.366
0.186
0.191
Skywork-Reward-V2-8B + Platt
0.210
0.372
0.078
0.153
Appendix
Table 13: Head refit with equal labels (held-out Brier ↓ ; 1,600 labels for PPE and RM-Bench, 400 for JudgeBench and RewardBench 2).