Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is
Organizations: FUTURIXAI RESEARCH, FUTURIXAI.COM
Abstract
Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.
Figures & tables
| Hypothesis | Test (falsifying control) | Verdict | |
|---|---|---|---|
| H1 | RL with the calibrated decision reward makes the policy more accurate on held-out problems, and the reward, not RL itself, is the cause. | E6: GRPO, ground-truth eval (random-reward floor) | Supported |
| H2 | The reference-free reward trains the policy as well as the answer key. | E6 (answer-key ceiling); E6b | Supported on MATH; Partly on hard problems |
| H3 | The reward stays calibrated and unhacked on the policy’s own samples, better than a softmax judge on the same backbone. | E6 on-policy logs (same-backbone softmax) | Supported |
| H4 | The decision model locates reasoning errors at least as well as a dedicated process reward model, with no step training. | E4 ProcessBench; E6b prefix restarts (random step) | Supported |
| H5 | With equal calibration labels, decision probabilities are better reward probabilities for correctness than softmax probabilities and scalar-RM scores. | E1, E2 (same labels, same backbone, head refit) | Supported vs. softmax; Partly vs. scalar RM |
| H6 | On hard problems, step-level decision credit gives a better policy than the answer key alone, and a policy better than the softmax judge on the full set. | E6b; E6 (softmax judge) | Partly |
| Work | Calibrated | Debiased | Typed / multi-option | Step-level | Online LLM RL |
|---|---|---|---|---|---|
| d-RLAIF [ 27 ] | ✗ | labels only | score scale | ✗ | ✓ |
| GenRM [ 77 ] | ✗ | ✗ | binary | ✗ | ✗ |
| Reward Bridge [ 58 ] | ✗ | ✗ | binary | ✗ | ✓ |
| CCR [ 53 ] | ✗ | ✗ | binary | ✗ | ✓ |
| Probe judges [ 69 ] | not measured | n/a | binary | ✗ | ✓ |
| OPRM [ 6 ] | ✓ | ✗ | score scale | ✗ | ✗ |
| Exp. | Data (size) | Compared | Labels | Main metric |
|---|---|---|---|---|
| E1 | PPE best-of-K (5,120), JudgeBench (620), RM-Bench (3,981), RewardBench 2 (1,723) | FxAI-DM, softmax, Skywork-V2-8B | 0–1,600 | Brier, NLL, AUROC |
| E2 | Same as E1, hidden states saved | 6 head variants | 25–1,600 | Brier |
| E4 | ProcessBench (3,400 solutions, 16,548 steps) | FxAI-DM, softmax, Qwen2.5-Math-PRM-7B | 0–400 | First-error F1 |
| E6 | MATH train (5,584) MATH500, AIME24/25 | 5 rewards ( Section 4 ) | 1,024 (frozen) | pass@1, pass@ |
| E6b | 1,758 hard MATH problems; 4,451 + 800 policy answers | FxAI-DM only; answer key step credit | 1,024 (frozen) | pass@1, re-solve drop |
| Reward used for RL | MATH500 | M500-clean | AIME24 | AIME25 | M500 pass@4 |
|---|---|---|---|---|---|
| None (base model) | 72.2 | 72.3 | 11.3 | 9.4 | 85.2 |
| Random (floor) | 72.8 | 72.8 | 12.3 | 8.5 | 85.8 |
| Softmax judge (same backbone) | 75.5 | 75.5 | 12.9 | 10.0 | 88.4 |
| RLCDR (ours) | 76.8 | 76.9 | 16.5 | 10.8 | 88.4 |
| Answer key (uses reference answers) | 76.9 | 77.1 | 14.4 | 13.1 | 89.2 |
| Comparison | 95% CI | |||
| Gain over base model | RLCDR base | +4.65 | ||
| Answer key base | +4.75 | |||
| Softmax base | +3.35 | |||
| Random base | +0.65 | 0.50 | ||
| What causes the gain | RLCDR random | +4.00 | ||
| RLCDR answer key | 0.90 |
| Base | Random | Softmax | RLCDR | Answer key | |
|---|---|---|---|---|---|
| MATH500 mean tokens | 837 | 863 | 856 | 1,056 | 1,049 |
| MATH500 truncated | 4.3% | 5.0% | 4.0% | 9.7% | 9.2% |
| AIME24 truncated | 37% | 32% | 37% | 62% | 64% |
| Answer key | FxAI-DM only (reference-free) | Answer key + FxAI-DM step credit | |
|---|---|---|---|
| Score of an answer | 1 if correct, else 0 | 1 if correct, else | |
| Group with all 8 answers wrong | no gradient | graded | graded |
| Uses the reference answer | yes | no | yes (0/1 part only) |
| Policy (50 steps, hard problems only) | MATH500 | Level 5 | AIME24 | AIME25 |
|---|---|---|---|---|
| Base (Qwen3-1.7B) | 72.2 | 44.8 | 11.3 | 9.4 |
| Answer key | 75.3 | 50.6 | 14.8 | 9.2 |
| FxAI-DM only (reference-free) | 74.1 | 48.9 | 13.1 | 10.0 |
| Answer key + FxAI-DM step credit | 75.7 | 52.1 | 12.9 | 13.1 |
| FxAI-DM only base | ||||
| FxAI-DM only answer key |
| Method | Labels | GSM8K | MATH | Olympiad | Omni-MATH | Average |
|---|---|---|---|---|---|---|
| Softmax judge (same 27B backbone) | 100 | 0.725 | 0.721 | 0.644 | 0.637 | 0.682 |
| Qwen2.5-Math-PRM-7B (step-trained) | 100 | 0.811 | 0.773 | 0.659 | 0.645 | 0.722 |
| FxAI-DM decision (ours) | 100 | 0.816 | 0.799 | 0.736 | 0.737 | 0.772 |
| FxAI-DM decision, zero-shot | 0 | 0.858 | 0.799 | 0.705 | 0.716 | 0.770 |
| FxAI-DM one-question critic, zero-shot | 0 | 0.861 | 0.789 | 0.719 | 0.687 | 0.764 |
| Softmax one-question critic, zero-shot | 0 | 0.790 | 0.731 | 0.655 | 0.610 | 0.696 |
| FxAI-DM | Softmax judge | FxAI-DM | |||||
|---|---|---|---|---|---|---|---|
| Domain | Acc. | ECE raw | ECE MATH-fit | 0.9 | ECE raw | ECE MATH-fit | 100-label refit |
| MATH | 0.36 | 0.084 | 0.048 | 0.87 (128) | 0.133 | 0.062 | – |
| MMLU-Pro | 0.48 | 0.096 | 0.074 | 0.94 (224) | 0.094 | 0.115 | – |
| IFEval | 0.50 | 0.086 | 0.075 | 0.91 (197) | 0.199 | 0.300 | – |
| GPQA | 0.39 | 0.118 | 0.088 | 0.76 (49) | 0.100 | 0.110 | 0.075 |
| MBPP+ (code) | 0.60 | 0.162 | 0.142 | 0.74 (487) | 0.211 | 0.233 | 0.090 |
| Benchmark | Method | Acc | AUROC | Brier | NLL | ECE | Wins |
|---|---|---|---|---|---|---|---|
| PPE (policy answers) | FxAI-DM decision | 0.735 | 0.809 | 0.184 | 0.562 | 0.058 | – |
| Softmax (same backbone) | 0.658 | 0.709 | 0.222 | 0.642 | 0.059 | 0/20 | |
| Scalar RM (Skywork-V2-8B) | 0.668 | 0.708 | 0.217 | 0.636 | 0.073 | 0/20 | |
| JudgeBench | FxAI-DM decision | 0.780 | – | 0.297 | 0.446 | 0.066 | – |
| Softmax (same backbone) | 0.708 | – | 0.373 | 0.556 | 0.067 | 0/20 | |
| Scalar RM | 0.744 | – | 0.352 | 0.523 | 0.063 | 1/20 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Features + head | PPE | JudgeBench | RM-Bench | RewardBench 2 |
|---|---|---|---|---|
| FxAI-DM + Platt (shipped head, 2 parameters) | 0.179 | 0.328 | 0.157 | 0.212 |
| FxAI-DM head refit | 0.178 | 0.318 | 0.155 | 0.185 |
| FxAI-DM new head on features | 0.177 | 0.322 | 0.154 | 0.193 |
| Base Qwen3.8-27B, new head (probe judge) | 0.198 | 0.366 | 0.186 | 0.191 |
| Skywork-Reward-V2-8B + Platt | 0.210 | 0.372 | 0.078 | 0.153 |
Reward Modeling
Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.
See all 344 papers