Test-time reinforcement learning enables language models to adapt using rewards derived from their own answers. However, majority voting can reinforce incorrect solutions even when correct alternatives are available. Our training audit shows that false positives (incorrect responses receiving positive rewards) dominate reward errors. We focus on a practical regime where ground-truth rewards are difficult to obtain at scale, while a limited labeled source set is available. We introduce TRUER, a framework that uses limited supervision to train a reward rectifier that distinguishes correctness from consensus. To achieve this goal, we train the rectifier with an emphasis on incorrect majorities and correct minorities, targeting cases where consensus and correctness disagree. After that, we freeze the rectifier and use it to correct rewards during subsequent test-time adaptation. By improving reward accuracy and thereby model performance, TRUER consistently outperforms naive TTRL in both metrics across all evaluated configurations.
Figures & tables
Figure 1: TRUER overview and learning dynamics. Top: limited source labels train an error-aware rectifier, which is then frozen to correct rewards during test-time adaptation. Bottom: starting from the same sampled answer distribution, naive TTRL can reinforce a wrong majority, whereas TRUER can recover a correct minority and support policy learning.
Figure 2: Available correct answers do not guarantee reliable rewards. (a) Pass@32 and Majority@32 use the same candidate pool at each naive-TTRL endpoint. The gap represents correct candidates missed by voting. (b) In the Qwen2.5-Math-1.5B/MATH500 training audit, false positives dominate reward errors across disjoint training windows, including late training.
Table 2: TRUER main results (%). Columns report the model/task, rectifier, Pass@1, Pass@32, and reward accuracy. In each Pass@1 bar, the gray line spans the TTRL–Oracle gap and the blue line shows the TTRL–TRUER gain. Labels give the gain in percentage points (above), and the full gap in percentage points and the fraction closed (below).
Figure 3: Reward correction on MATH500. Qwen2.5-Math-1.5B with a Qwen3-4B rectifier. (a) FP/FN counts on identical responses; retention is relative to majority voting. (b) Reward accuracy over 15-update windows and endpoint Pass@1 on each method’s own trajectory.
Figure 4: From correct minority to majority. (a) Each training visit pairs naive TTRL (left, hatched) with TRUER (right, solid); stars mark correct rank2 selection. (b) Separately sampled endpoint responses. Each bar contains 32 responses; “Other responses” includes incorrect and unparsed answers.
Figure 5: Rectifier reuse at deployment. For TTRL, TRUER, and Oracle, darker segments end at Pass@1 and lighter extensions reach Majority@32. Purple bars show TRUER with score-weighted answer aggregation on identical 32-candidate pools. Panels use different truncated y-axis ranges. Four configurations are shown; Appendix E reports all six.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Target
Seed
Updates
Rectifier
Step
Deploy
Math-1.5B
MATH500
42
90
Qwen3-4B
150
Weighted
Qwen3-1.7B
AMC23
42
120
Qwen3-1.7B
198
Weighted
Qwen3-1.7B
AMC23
42
120
Qwen3-4B
25
Weighted
Qwen3-1.7B
AIME24
42
120
Qwen3-1.7B
260
Weighted
Qwen3-1.7B
AIME24
42
120
Qwen3-4B
312
—
Qwen3-4B
AMC23
43
120
Qwen3-4B
175
Weighted
Appendix
Table 3: Configurations in the nine-row main table. Seeds describe policy training. Rectifier steps identify the frozen checkpoint, not additional policy updates. A dash denotes a configuration outside the matched deployment comparison.
Seed
Naive TTRL
TRUER
Oracle
Δ
42
56.09
59.38
64.61
+3.28
43
55.23
61.02
64.30
+5.78
44
54.84
59.69
65.31
+4.84
Mean
55.39
60.03
64.74
+4.64
Appendix
Table 4: AMC23 training repetitions: Qwen3-1.7B with the fixed 4B rectifier. Pass@1 (%) at update 120. Each row contains independently trained policies using the same training seed. Δ is TRUER minus naive TTRL in percentage points, computed before rounding.
Setting
Value
Model
Qwen2.5-Math-1.5B, original pretrained weights
Policy trainable parameters
All parameters; four-GPU FSDP2
Policy optimizer
AdamW; weight decay 0.01; gradient clipping 1.0
Policy learning rate
Peak 5×10−7 ; three warmup steps; cosine to step 90
Reference policy / KL
Frozen initial policy; separate KL loss coefficient 0.001
Training batch / responses
32 questions / 32 responses per question
Appendix
Table 5: Qwen2.5-Math-1.5B/MATH500 training settings.
Category
Eligible candidates
Priority
Loss weight
Majority-correct
1,621
1
0.2813
Majority-wrong
440
3
3.1091
Minority-correct
153
3
8.9412
Minority-wrong
1,434
1
0.3180
Appendix
Table 6: Rectifier supervision for Qwen2.5-Math-1.5B/MATH500. Correctness labels come from source ground truth; category membership is not provided as input.
Figure 6: Reward intervention and policy learning. TRUER moves Pass@1 and Majority@32 toward the ground-truth-reward reference, while Pass@32 shows a different ordering. Each line is one training seed. The figure shows all recorded checkpoints and does not select the best intermediate result.
Figure 7: Paired uncertainty in the MATH500 learning curves. Left: model Pass@1. Right: TRUER minus TTRL with pointwise 95% question-bootstrap intervals, conditional on the saved samples and one training seed.
Step
Method
Greedy
Pass@1
Pass@2
Majority@32
Pass@32
15
TTRL
73.51
58.46
72.43
81.89
92.70
15
Rank1-only
72.70
57.90
72.33
82.16
92.43
15
TRUER
72.43
59.33
73.07
80.27
91.89
15
Oracle
72.16
58.56
72.49
81.62
92.16
30
TTRL
72.43
65.49
76.99
84.05
92.97
30
Rank1-only
72.97
65.46
76.77
82.70
93.24
Appendix
Table 7: All recorded model checkpoints. Percentages, frozen scorer. No best-checkpoint selection is used for the main endpoint.
Figure 8: All 30 AMC23 TRUER checkpoints. Solid curves are separately completed, matched evaluations of each checkpoint without rectifier-based selection. Dashed lines mark the final epoch 30 TTRL and Oracle scores; their intermediate sampled checkpoints were not retained. Greedy uses the same complete evaluation pipeline rather than the separate inline training validation.
Figure 9: Where the same questions move after adaptation. Rows are base correct-response counts; columns are counts at update 90. Both panels contain all 370 questions and use the same bins and color scale. Movement to the right indicates more frequent correct answers.
Model
Unparsed
Length-limit hits
Mean output tokens
Base
32.97%
11.25%
1,102.7
TTRL90
1.78%
2.58%
682.6
TRUER90
1.98%
2.52%
684.1
Appendix
Table 9: Parsing and generation lengths under the base comparison. All scores retain the full denominator of 11,840 responses; unparsed responses are not discarded.
Figure 10: Cleaner rewards do not automatically supply more corrective signal. Both panels use TRUER’s fixed training response pool. Left: retained advantage mass and active groups relative to raw majority labeling. Right: the nested conditions needed to rescue a correct rank2. Counts are question visits, not unique questions.
Quantity
TTRL
TRUER
TP
56,183
55,400
FP
4,363
1,081
TN
30,498
33,395
FN
1,116
2,284
Overall reward accuracy
94.05%
96.35%
FPR
12.52%
3.14%
Appendix
Table 10: Training reward audit over all 90 updates, evaluated on each arm’s own responses. For an isolated labeling comparison, use the shared-pool numbers in the main text.
Policy
TP
FP
Active groups
Precision (%)
TP kept (%)
FP removed (%)
TTRL (no filter)
56,596
4,012
2,557
93.38
100.0
0.0
Vote margin δ=8
54,133
1,299
1,815
97.66
95.6
67.6
Vote margin δ=10
53,143
1,014
1,714
98.13
93.9
74.7
Vote margin δ=12
52,110
928
1,636
98.25
92.1
76.9
Absolute support ≥24
43,692
496
1,179
98.88
77.2
87.6
TRUER (ours)
55,400
1,081
2,105
98.09
97.9
73.1
Appendix
Table 11: Label-free confidence filters on the frozen pool of Section 5.3 . Only vote counts enter the selection. Precision is TP/(TP+FP) ; the last two columns are relative to unfiltered TTRL majority. The learned cascade removes 73.1% of false positives while keeping 97.9% of correct positive rewards and 82.3% of active groups.
Figure 11: Label-free confidence filters against the learned rectifier. Each curve sweeps one filter family over its threshold. At comparable FP removal, TRUER retains more correct rewards and active groups. Panel (b) restricts the pool to updates 61–90, where false positives dominate reward errors.
Contrast
Metric
Difference
95% interval
TRUER − rank1
Greedy
+0.27
[-2.43, +2.97]
TRUER − rank1
Pass@1
+0.42
[-0.37, +1.28]
TRUER − rank1
Pass@2
+0.25
[-0.54, +1.11]
TRUER − rank1
Majority@32
+1.08
[-1.35, +3.51]
TRUER − rank1
Pass@32
-2.43
[-4.32, -0.81]
Rank1 − TTRL
Greedy
+0.00
[-2.70, +2.70]
Appendix
Table 12: Paired 90-step contrasts involving rank1-only. Differences and intervals are percentage points. These were recomputed from the saved question-level scores after the rank1 run completed.
Response source
Accuracy ↑
Wrong accepted ↓
Correct withheld ↓
Invalid ↓
Base model
74.39→88.05
38.47→14.80
7.65→8.16
3.09→0.33
TTRL model
71.85→86.52
52.11→18.35
4.89→9.64
3.97→0.16
Oracle model
82.89→90.79
48.31→16.92
3.19→5.89
1.84→0.26
Appendix
Table 13: Original versus trained rectifier on identical AMC23 responses. Each entry is original → trained, in percent. Accuracy includes invalid outputs as errors. Wrong-answer acceptance is conditional on a wrong candidate; correct-answer withholding includes explicit rejection and invalid outputs on correct candidates. The rectifier was trained on disjoint AMC2024 questions.
Figure 12: Task-specific training improves the 4B rectifier’s judgment tradeoff. Original and trained models judge the same candidates. Training reduces wrong-answer acceptance but also withholds more correct answers.
Target
Rectifier
Naive TTRL
Untrained
Trained
AMC23
Qwen3-4B
56.09
59.22
59.38
AIME24
Qwen3-1.7B
20.21
18.96
21.46
AIME24
Qwen3-4B
20.21
23.02
24.79
Appendix
Table 14: Adaptation with and without rectifier training. Final model Pass@1 (%) without rectifier-based answer selection. All policies are Qwen3-1.7B; untrained denotes the original pretrained rectifier without task-specific training.
Figure 13: Filtering without recovery. (a) Answer distributions at all eight recorded TRUER training visits; crosses mark abstention. (b) Separately sampled endpoint responses for Base, naive TTRL, and TRUER. The named wrong answer is zero. Rejecting this answer supplies no replacement target, and the model remains incorrect.
Naive TTRL
TRUER
Model / target (rectifier)
N
Majority
+Rectifier
Majority
+Rectifier
Math-1.5B / MATH500 (4B)
370
83.24
88.11
84.86
86.76
Qwen3-1.7B / AMC23 (1.7B)
40
65.00
65.00
67.50
70.00
Qwen3-1.7B / AMC23 (4B)
40
65.00
70.00
67.50
70.00
Qwen3-1.7B / AIME24 (1.7B)
30
26.67
30.00
26.67
33.33
Qwen3-4B / AMC23 (4B)
40
82.50
85.00
90.00
95.00
Appendix
Table 15: Deployment under one shared selection rule (%). Parentheses identify the rectifier size. Both adaptation methods use the same frozen rectifier within each row. “+Rectifier” applies Equation 7 to the corresponding model’s fixed 32-response pool, with majority fallback.
Selection rule
Base
TTRL
TRUER
Oracle
Original majority
67.50
65.00
67.50
77.50
Sequential rank1-to-rank2
72.50
67.50
67.50
80.00
Top2: one representative
72.50
67.50
65.00
80.00
Top2: all candidates
70.00
65.00
70.00
80.00
All32: all candidates
70.00
65.00
75.00
80.00
Appendix
Table 16: Historical AMC23 final answer-selection accuracy (%). Every row uses the same 32 candidates for each model, and full question coverage. Changes do not alter the models’ sampled Pass@1 or Pass@32.
Table 18: Rectifier inference cost for the unified deployment comparison. Counts cover all eligible responses in each fixed pool.
Figure 14: Selection quality and additional rectifier cost on AMC23. The same frozen rectifier scores TRUER epoch 30 candidates. Output-token counts are sums from the cached judgments, excluding rectifier input tokens and the unchanged response generation. Calls count candidate judgments. All policies retain all 40 questions.
Pass@1 (%)
GPU-hours
Model / task
Seed
N0
N+
T0
N0
N+
T0
Qwen3-1.7B / AMC23
42
56.09
59.14
59.38
9.78
27.65
27.31
Qwen3-1.7B / AIME24
42
20.21
23.75
24.79
19.70
39.39
38.96
Qwen3-4B / AMC23
43
79.69
78.52
82.34
27.10
30.64
30.21
Qwen3-4B / MATH-L5
43
79.82
80.36
80.68
56.45
68.75
67.84
Llama-3.1-8B / AMC23
43
49.45
47.50
60.23
29.26
63.69
62.97
Appendix
Table 19: GPU-hour-matched naive continuation. N0 and T0 are the original naive TTRL and TRUER endpoints; N+ is the extended naive TTRL endpoint. GPU-hours cover adaptation and online rectifier inference, excluding rectifier pretraining, saving, export, and evaluation.
Active groups
Model / task
T0
Target
T+
Added steps
Pass@1 (%)
GPU-hours
Qwen3-1.7B / AMC23
636
930
935
65
61.02
43.46
Qwen3-1.7B / AIME24
530
1,179
1,180
144
27.92
121.80
Qwen3-4B / AMC23
569
651
653
21
81.72
38.00
Qwen3-4B / MATH-L5
1,094
1,197
1,197
24
80.74
76.15
Llama-3.1-8B / AMC23
415
630
630
139
64.53
140.13 †
Appendix
Table 20: Active-group-matched TRUER continuation. T+ is the extended TRUER endpoint. Group counts include the original training; the target is the original naive TTRL count. GPU-hours are cumulative. Seeds and original scores are given in Table 19 .