TRUER: Test-Time Reward Correction Using an Error-Aware Rectifier for Reinforcement Learning
Organizations: Shanghai Jiao Tong University
Abstract
Test-time reinforcement learning enables language models to adapt using rewards derived from their own answers. However, majority voting can reinforce incorrect solutions even when correct alternatives are available. Our training audit shows that false positives (incorrect responses receiving positive rewards) dominate reward errors. We focus on a practical regime where ground-truth rewards are difficult to obtain at scale, while a limited labeled source set is available. We introduce TRUER, a framework that uses limited supervision to train a reward rectifier that distinguishes correctness from consensus. To achieve this goal, we train the rectifier with an emphasis on incorrect majorities and correct minorities, targeting cases where consensus and correctness disagree. After that, we freeze the rectifier and use it to correct rewards during subsequent test-time adaptation. By improving reward accuracy and thereby model performance, TRUER consistently outperforms naive TTRL in both metrics across all evaluated configurations.
Figures & tables
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Target | Seed | Updates | Rectifier | Step | Deploy |
|---|---|---|---|---|---|---|
| Math-1.5B | MATH500 | 42 | 90 | Qwen3-4B | 150 | Weighted |
| Qwen3-1.7B | AMC23 | 42 | 120 | Qwen3-1.7B | 198 | Weighted |
| Qwen3-1.7B | AMC23 | 42 | 120 | Qwen3-4B | 25 | Weighted |
| Qwen3-1.7B | AIME24 | 42 | 120 | Qwen3-1.7B | 260 | Weighted |
| Qwen3-1.7B | AIME24 | 42 | 120 | Qwen3-4B | 312 | — |
| Qwen3-4B | AMC23 | 43 | 120 | Qwen3-4B | 175 | Weighted |
| Seed | Naive TTRL | TRUER | Oracle | |
|---|---|---|---|---|
| 42 | 56.09 | 59.38 | 64.61 | +3.28 |
| 43 | 55.23 | 61.02 | 64.30 | +5.78 |
| 44 | 54.84 | 59.69 | 65.31 | +4.84 |
| Mean | 55.39 | 60.03 | 64.74 | +4.64 |
| Setting | Value |
|---|---|
| Model | Qwen2.5-Math-1.5B, original pretrained weights |
| Policy trainable parameters | All parameters; four-GPU FSDP2 |
| Policy optimizer | AdamW; weight decay 0.01; gradient clipping 1.0 |
| Policy learning rate | Peak ; three warmup steps; cosine to step 90 |
| Reference policy / KL | Frozen initial policy; separate KL loss coefficient 0.001 |
| Training batch / responses | 32 questions / 32 responses per question |
| Category | Eligible candidates | Priority | Loss weight |
|---|---|---|---|
| Majority-correct | 1,621 | 1 | 0.2813 |
| Majority-wrong | 440 | 3 | 3.1091 |
| Minority-correct | 153 | 3 | 8.9412 |
| Minority-wrong | 1,434 | 1 | 0.3180 |
| Step | Method | Greedy | Pass@1 | Pass@2 | Majority@32 | Pass@32 |
|---|---|---|---|---|---|---|
| 15 | TTRL | 73.51 | 58.46 | 72.43 | 81.89 | 92.70 |
| 15 | Rank1-only | 72.70 | 57.90 | 72.33 | 82.16 | 92.43 |
| 15 | TRUER | 72.43 | 59.33 | 73.07 | 80.27 | 91.89 |
| 15 | Oracle | 72.16 | 58.56 | 72.49 | 81.62 | 92.16 |
| 30 | TTRL | 72.43 | 65.49 | 76.99 | 84.05 | 92.97 |
| 30 | Rank1-only | 72.97 | 65.46 | 76.77 | 82.70 | 93.24 |
| Method | Greedy | Pass@1 | Pass@2 | Majority@32 | Pass@32 |
|---|---|---|---|---|---|
| Base | 45.00 | 46.09 | 59.97 | 67.50 | 90.00 |
| TTRL | 57.50 | 56.09 | 61.67 | 65.00 | 77.50 |
| TRUER | 60.00 | 59.38 | 66.67 | 67.50 | 92.50 |
| Oracle | 55.00 | 64.61 | 72.91 | 77.50 | 92.50 |
| Model | Unparsed | Length-limit hits | Mean output tokens |
|---|---|---|---|
| Base | 32.97% | 11.25% | 1,102.7 |
| TTRL90 | 1.78% | 2.58% | 682.6 |
| TRUER90 | 1.98% | 2.52% | 684.1 |
| Quantity | TTRL | TRUER |
|---|---|---|
| TP | 56,183 | 55,400 |
| FP | 4,363 | 1,081 |
| TN | 30,498 | 33,395 |
| FN | 1,116 | 2,284 |
| Overall reward accuracy | 94.05% | 96.35% |
| FPR | 12.52% | 3.14% |
| Policy | TP | FP | Active groups | Precision (%) | TP kept (%) | FP removed (%) |
|---|---|---|---|---|---|---|
| TTRL (no filter) | 56,596 | 4,012 | 2,557 | 93.38 | 100.0 | 0.0 |
| Vote margin | 54,133 | 1,299 | 1,815 | 97.66 | 95.6 | 67.6 |
| Vote margin | 53,143 | 1,014 | 1,714 | 98.13 | 93.9 | 74.7 |
| Vote margin | 52,110 | 928 | 1,636 | 98.25 | 92.1 | 76.9 |
| Absolute support | 43,692 | 496 | 1,179 | 98.88 | 77.2 | 87.6 |
| TRUER (ours) | 55,400 | 1,081 | 2,105 | 98.09 | 97.9 | 73.1 |
| Contrast | Metric | Difference | 95% interval |
|---|---|---|---|
| TRUER rank1 | Greedy | +0.27 | [-2.43, +2.97] |
| TRUER rank1 | Pass@1 | +0.42 | [-0.37, +1.28] |
| TRUER rank1 | Pass@2 | +0.25 | [-0.54, +1.11] |
| TRUER rank1 | Majority@32 | +1.08 | [-1.35, +3.51] |
| TRUER rank1 | Pass@32 | -2.43 | [-4.32, -0.81] |
| Rank1 TTRL | Greedy | +0.00 | [-2.70, +2.70] |
| Response source | Accuracy | Wrong accepted | Correct withheld | Invalid |
|---|---|---|---|---|
| Base model | ||||
| TTRL model | ||||
| Oracle model |
| Target | Rectifier | Naive TTRL | Untrained | Trained |
|---|---|---|---|---|
| AMC23 | Qwen3-4B | 56.09 | 59.22 | 59.38 |
| AIME24 | Qwen3-1.7B | 20.21 | 18.96 | 21.46 |
| AIME24 | Qwen3-4B | 20.21 | 23.02 | 24.79 |
| Naive TTRL | TRUER | ||||
|---|---|---|---|---|---|
| Model / target (rectifier) | Majority | +Rectifier | Majority | +Rectifier | |
| Math-1.5B / MATH500 (4B) | 370 | 83.24 | 88.11 | 84.86 | 86.76 |
| Qwen3-1.7B / AMC23 (1.7B) | 40 | 65.00 | 65.00 | 67.50 | 70.00 |
| Qwen3-1.7B / AMC23 (4B) | 40 | 65.00 | 70.00 | 67.50 | 70.00 |
| Qwen3-1.7B / AIME24 (1.7B) | 30 | 26.67 | 30.00 | 26.67 | 33.33 |
| Qwen3-4B / AMC23 (4B) | 40 | 82.50 | 85.00 | 90.00 | 95.00 |
| Selection rule | Base | TTRL | TRUER | Oracle |
|---|---|---|---|---|
| Original majority | 67.50 | 65.00 | 67.50 | 77.50 |
| Sequential rank1-to-rank2 | 72.50 | 67.50 | 67.50 | 80.00 |
| Top2: one representative | 72.50 | 67.50 | 65.00 | 80.00 |
| Top2: all candidates | 70.00 | 65.00 | 70.00 | 80.00 |
| All32: all candidates | 70.00 | 65.00 | 75.00 | 80.00 |
| Selection rule | TRUER90 | TRUER110 | Oracle90 |
|---|---|---|---|
| Original majority | 84.86 | 85.14 | 85.41 |
| Rank1-to-rank2 cascade | 85.68 | 86.22 | 86.22 |
| Top2, score first | 85.68 | 85.95 | 86.76 |
| Top3, score first | 85.68 | 85.68 | 87.84 |
| Naive TTRL | TRUER | |||
| Model / target (rectifier) | Judgments | Output tokens | Judgments | Output tokens |
| Math-1.5B / MATH500 (4B) | 11,463 | 7,409,697 | 11,457 | 7,382,694 |
| Qwen3-1.7B / AMC23 (1.7B) | 1,261 | 1,501,618 | 1,207 | 1,366,736 |
| Qwen3-1.7B / AMC23 (4B) | 1,261 | 1,110,284 | 1,173 | 939,509 |
| Qwen3-1.7B / AIME24 (1.7B) | 864 | 208,525 | 768 | 169,730 |
| Qwen3-4B / AMC23 (4B) | 1,194 | 936,227 | 1,170 | 925,414 |
| Pass@1 (%) | GPU-hours | ||||||
|---|---|---|---|---|---|---|---|
| Model / task | Seed | ||||||
| Qwen3-1.7B / AMC23 | 42 | 56.09 | 59.14 | 59.38 | 9.78 | 27.65 | 27.31 |
| Qwen3-1.7B / AIME24 | 42 | 20.21 | 23.75 | 24.79 | 19.70 | 39.39 | 38.96 |
| Qwen3-4B / AMC23 | 43 | 79.69 | 78.52 | 82.34 | 27.10 | 30.64 | 30.21 |
| Qwen3-4B / MATH-L5 | 43 | 79.82 | 80.36 | 80.68 | 56.45 | 68.75 | 67.84 |
| Llama-3.1-8B / AMC23 | 43 | 49.45 | 47.50 | 60.23 | 29.26 | 63.69 | 62.97 |
| Active groups | ||||||
|---|---|---|---|---|---|---|
| Model / task | Target | Added steps | Pass@1 (%) | GPU-hours | ||
| Qwen3-1.7B / AMC23 | 636 | 930 | 935 | 65 | 61.02 | 43.46 |
| Qwen3-1.7B / AIME24 | 530 | 1,179 | 1,180 | 144 | 27.92 | 121.80 |
| Qwen3-4B / AMC23 | 569 | 651 | 653 | 21 | 81.72 | 38.00 |
| Qwen3-4B / MATH-L5 | 1,094 | 1,197 | 1,197 | 24 | 80.74 | 76.15 |
| Llama-3.1-8B / AMC23 | 415 | 630 | 630 | 139 | 64.53 | 140.13 † |
RL for Language Models
Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.
See all 460 papers