Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference (NTP), an unsupervised reward-modeling method that constructs preference pairs from the continuation structure of raw web text. NTP uses each prefix's original continuation as the chosen response and nearby continuations as in-batch negatives, without candidate generation or human or model-generated preference annotations. Training on 11M tokens of math-focused web data yields steady gains on RewardBench and RewardBench 2, with gains across model families and sizes. Across mathematical and general-domain web corpora, our method improves overall RewardBench 2 accuracy by 7.6-8.3 percentage points over initialization under the same token budget. Training on math-focused text yields gains of up to +16.1 on in-domain math subsets and consistent improvements on out-of-domain safety and general subsets. These reward models improve best-of-N selection on math, code, safety, and instruction following, and support math policy optimization competitive with supervised reward models of similar size. NTP also improves the label efficiency of subsequent supervised fine-tuning. Using half the labeled pairs from a curated preference dataset, NTP initialization surpasses base initialization trained on the full dataset. These results suggest that web text contains transferable preference signals that can reduce dependence on costly preference annotations.
Figures & tables
Figure 1: Schematic overview of our NTP reward model training workflow from web math text. Raw documents are split into prefix–suffix pairs, where the true continuation is treated as the chosen response and other in-batch continuations serve as implicit negatives. The reward model is trained with a Bradley–Terry objective over these online preference pairs, enabling scalable reward learning without human annotations.
Figure 2: Scalability of NTP with respect to data size. Reward models initialized from pretrained Llama-3.2-3B improve across RewardBench and RewardBench 2 subsets as the token budget increases to 11M.
Method
Budget
RewardBench 2 gain
FineMath NTP (ours)
11M tokens
+7.7
Skywork-80K-v0.2
≈ 9K pairs (11M)
+1.8
Skywork-80K-v0.2
77K pairs (full)
+9.8
RLHFMix1
≈ 22K pairs (11M)
+0.8
RLHFMix1
244K pairs (full)
+8.3
Table 1: Overall RewardBench 2 gains (percentage points) over Llama-3.2-3B initialization. Bold : best gain at the matched 11M-token budget.
Figure 3: Effect of batch size on peak gains and learning trajectory of RewardBench 2.
Figure 4: Effect of data splitting design on peak gains and learning trajectory of RewardBench 2.
Figure 5: Effect of centering loss on RewardBench 2 peak gains, learning trajectory and BoN MAP.
Figure 6: BoN accuracy curve on Llama-3.1-8B-Instruct GSM8K rollouts for RMs trained with or without centering loss ( init –initialized seed).
Training corpus
RewardBench 2 gain
FineMath-4plus
+7.7
MegaMath-Web-Pro-Max
+8.3
FineWeb-Edu
+7.6
Table 2: Peak average RewardBench 2 accuracy gains (percentage points) over the shared initialization for Llama-3.2-3B RMs trained on 11M tokens from each corpus.
Figure 7: BoN scaling curves of RMs across two ID math tasks (GSM8K, MATH500) and three OOD tasks (HumanEval+, Toxigen, IFEval).
Actor: Llama-3.2- 3B-Instruct
Actor: Llama-3.1- 8B-Instruct
MATH
GSM8K
MATH
GSM8K
Before training
0.268
0.789
0.423
0.876
Skywork-Reward-V2-Llama-3.1-8B
0.419
0.833
0.447
0.882
Skywork-Reward-V2-Llama-3.2-3B
0.439
0.819
0.465
0.884
beaver-7b-unified-reward
0.416
0.803
0.430
0.881
FineMath-RM-Qwen-2.5-7B
0.420
0.823
0.437
0.886
Table 3: Peak mean@1 test accuracy within five GRPO epochs. Both actors use instruction-tuned initializations. Bold and underlined values denote the best and second-best results, respectively.
Checkpoint
ID
OOD Safety
OOD others
RewardBench 2
Llama-3.2-3B
+14.5
+22.3
+1.6
+8.3
Llama-3.2-3B-Instruct
+19.4
+22.6
+7.8
+13.1
Finemath-RM-Llama-3.2-3B
+21.4
+24.4
+2.0
+10.4
Table 4: RewardBench 2 gains from training on curated preference dataset RLHFMix1 under different initializations.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
#Pairs N
Prompt length
Preferred length
Rejected length
Tcand
Overhead (in/out)
Tingen
Toutgen
Cost ($)
$/pair
HH-RLHF
115,396
160.4
82.2
73.6
155.8
0 / –
18.51M
17.98M
180.85
0.00157
SHP
93,301
186.2
173.6
88.8
262.4
0 / –
17.37M
24.48M
230.60
0.00247
HelpSteer
37,131
530.0
116.4
89.3
205.7
0 / –
19.68M
7.64M
100.46
0.00271
PKU-SafeRLHF-30K
26,874
21.5
70.4
74.6
145.0
0 / –
0.58M
3.90M
32.33
0.00120
UltraFeedback
340,025
161.5
279.5
211.1
490.6
0 / –
54.91M
166.82M
1444.36
0.00425
UltraInteract
161,927
507.4
396.6
416.7
813.3
0 / –
82.16M
131.70M
1217.89
0.00752
Appendix
Table 5: Estimated candidate generation cost instantiated with a GPT-4–class completion model gpt-4o .
Dataset
#Pairs N
Prompt length
Preferred length
Rejected length
Tcontent
Overhead (in/out)
Tinjudge
Toutjudge
Cost ($)
$/pair
HH-RLHF
115,396
160.4
82.2
73.6
316.2
40 / 40
41.10M
4.62M
148.92
0.00129
SHP
93,301
186.2
173.6
88.8
448.6
40 / 40
45.59M
3.73M
151.29
0.00162
HelpSteer
37,131
530.0
116.4
89.3
735.7
40 / 40
28.79M
1.49M
86.91
0.00234
PKU-SafeRLHF-30K
26,874
21.5
70.4
74.6
166.5
40 / 40
5.55M
1.07M
24.57
0.00091
UltraFeedback
340,025
161.5
279.5
211.1
652.1
40 / 40
235.71M
13.60M
725.83
0.00213
UltraInteract
161,927
507.4
396.6
416.7
1320.7
40 / 40
220.56M
6.48M
616.28
0.00381
Appendix
Table 6: Estimated LLM annotation cost for preference labeling with a single GPT-4–class judge ( gpt-4o pricing). We use Oin=40 input overhead tokens and Oout=40 output tokens per pair.
Backbone
Seed
Optimizer
Learning rate
Batch size
c (center)
Llama-3.2-1B
2025
Adam betas 0.9, 0.95
10−6 , constant 0.05 warmup ratio
32
0.01
Llama-3.2-3B
2025
Adam betas 0.9, 0.95
10−6 , constant 0.05 warmup ratio
32
0.01
Qwen2.5-3B-Inst.
2028
Adam betas 0.9, 0.95
10−6 , constant 0.05 warmup ratio
32
0.01
Qwen2.5-7B-Inst.
2025
Adam betas 0.9, 0.95
10−6 , constant 0.05 warmup ratio
8
0.01
Appendix
Table 7: RM training hyperparameters.
Backbone
Corpus
Optimizer
Learning rate
Blocks per step
Cutoff (tokens)
Epochs
Llama-3.2-3B
FineMath-4plus
AdamW
10−5 peak, cosine, warmup ratio 0.1
1024
2048
1
Appendix
Table 8: Continued-pretraining recipe for the 11M-token control in Table 9 . Blocks per step is the global batch size.
Readout
Untrained
CPT
RM
Frozen probe
30.0
28.9
36.4
Conversion (512 pairs)
32.3
31.2
–
Original RM head
–
–
36.2
Appendix
Table 9: RewardBench 2 scores for Llama-3.2-3B before training and after continued pretraining (CPT) or NTP reward-model training (RM), each on 11M FineMath tokens.
Dataset
#Pairs
Instruction (tokens)
Chosen (tokens)
Rejected (tokens)
Tokens/ pair
Equiv. pairs @ 11M tok.
Skywork-80K-v0.2
77,016
≈ 115
≈ 514
≈ 584
≈ 1,213
≈ 9,070
RLHFMix1
244,216
≈ 224
≈ 149
≈ 120
≈ 493
≈ 22,310
Appendix
Table 10: Per-entry token statistics for curated comparison datasets and the equivalent number of preference pairs under an 11M-token training budget. Token counts are estimated at ∼4 characters per token.
Task (ID/OOD)
Dataset / split
Prompt format
Actor decoding
Accuracy / scoring criterion
MATH500 (ID)
HuggingFaceH4/MATH-500 (test)
Problem: question . Provide your reasoning step by step, then the final answer the form FINAL_ANSWER: answer .
T= 1.0, top- p= 1.0 max tokens = 512
lm-eval-harness math-verify
GSM8K (ID)
openai/gsm8k (test)
Problem: question . Provide your reasoning step by step, then the final answer the form FINAL_ANSWER: answer .
T= 1.0, top- p= 1.0 max tokens = 512
lm-eval-harness math-verify
HumanEval+ (OOD)
evalplus/humanevalplus v0.1.10 (160 of 164 problems whose canonical solutions pass under EvalPlus)
Please provide a self-contained Python script that solves the following problem in a markdown code block: problem
T= 1.0, top- p= 1.0 max tokens = 512
evalplus ; pass iff the extracted program passes both the base and the plus test suites
ToxiGen (OOD)
toxigen/toxigen-data (test)
Is the following statement hateful? Statement: text . Respond with either Yes or No. Response:
T= 1.0, top- p= 1.0 max tokens = 20
lm-eval-harness
IFEval (OOD)
google/IFEval (train)
prompt
T= 1.0, top- p= 1.0 max tok. = 1280
lm-eval-harness prompt level strict
Appendix
Table 11: BoN response generation and evaluation details by task.
Figure 11: Validation reward curve with or without centering loss.
Seed
ID
OOD Safety
OOD Others
RewardBench 2 Avg
2025
+16.1
+5.4
+7.4
+7.7
2027
+17.5
+10.5
+4.7
+8.4
2028
+19.1
+16.1
+0.2
+7.2
Mean
+17.6
+10.7
+4.1
+7.7±0.35
Appendix
Table 12: RewardBench 2 gains in percentage points over each run’s initialization for three independent runs of the default Llama-3.2-3B setup. The final row reports means across seeds; the overall gain is reported as mean ± standard error.
Benchmark
Subset
Gain
RewardBench
Chat
+4.9
Chat Hard
+7.0
Safety
+16.0
Reasoning
+13.1
RewardBench 2
Math
+17.6
Safety
+10.7
Appendix
Table 13: RewardBench and RewardBench 2 subset gains in percentage points over each run’s initialization, averaged across three seeds for NTP training of Llama-3.2-3B on 11M FineMath tokens.
Backbone
RewardBench
RewardBench 2
Score
Rank
Score
Rank
Llama-3.2-1B
60.0
3 of 4
33.2
3 of 4
Llama-3.2-3B
65.6
4 of 9
36.2
9 of 12
Qwen2.5-3B-Inst.
70.0
3 of 9
46.2
8 of 12
Qwen2.5-7B-Inst.
73.8
18 of 47
57.0
5 of 11
Appendix
Table 14: Average RewardBench and RewardBench 2 scores and leaderboard ranks for RMs trained with different initialization backbones on 11M tokens. Ranks are reported as k of n , where n counts all entries in the corresponding comparison set, including our models.
Model
Params (B)
AvgAcc
Type
opencompass/CompassJudger-1-1.5B-Instruct
1.5
73.4
Generative
OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5
1.4
69.5
Seq. Classifier
Finemath-RM-Llama-3.2-1B (ours)
1.0
60.0
Seq. Classifier
Qwen/Qwen1.5-0.5B-Chat
0.5
53.8
DPO
Appendix
Table 15: RewardBench leaderboard RMs in the 0.5–1.5B parameter range, sorted by AvgAcc.
Model
Params (B)
AvgAcc
Type
Skywork/Skywork-Reward-V2-Llama-3.2-1B
1.0
64.376
Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-0.6B
0.6
61.250
Seq. Classifier
Finemath-RM-Llama-3.2-1B (ours)
1.0
33.2
Seq. Classifier
OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5
1.4
26.479
Seq. Classifier
Appendix
Table 16: RewardBench 2 leaderboard RMs in the 0.5–1.5B parameter range, sorted by AvgAcc.
Model
Params (B)
AvgAcc
Type
Ray2333/GRM-llama3.2-3B-rewardmodel-ft
3.0
90.9
Seq. Classifier
stabilityai/stablelm-zephyr-3b
3.0
74.0
DPO
Finemath-RM-Qwen-2.5-3B (ours)
3.0
70.0
Seq. Classifier
Finemath-RM-Llama-3.2-3B (ours)
3.0
65.6
Seq. Classifier
stabilityai/stable-code-instruct-3b
3.0
64.3
DPO
Qwen/Qwen1.5-4B-Chat
4.0
56.0
DPO
Appendix
Table 17: RewardBench leaderboard RMs in the 3-4B parameter range, sorted by AvgAcc.
Model
Params (B)
Avg Acc
Type
Skywork/Skywork-Reward-V2-Qwen3-4B
4.0
75.51
Seq. Classifier
Skywork/Skywork-Reward-V2-Llama-3.2-3B
3.0
74.665
Seq. Classifier
Schrieffer/Llama-SARM-4B
4.0
73.793
Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-1.7B
1.7
68.176
Seq. Classifier
Skywork/Skywork-Reward-V2-Llama-3.2-1B
1.0
64.376
Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-0.6B
0.6
61.25
Seq. Classifier
Appendix
Table 18: RewardBench 2 leaderboard RMs up to 4B parameters, sorted by AvgAcc.
Model
Params (B)
AvgAcc
Type
Skywork/Skywork-VL-Reward-7B
7.0
90.070
Seq. Classifier
R-I-S-E/RISE-Judge-Qwen2.5-7B
7.0
88.191
Generative
internlm/internlm2-7b-reward
7.0
87.593
Seq. Classifier
ZiyiYe/Con-J-Qwen2-7B
7.0
87.120
Generative
opencompass/CompassJudger-1-7B-Instruct
7.0
83.167
Generative
CIR-AMS/BTRM_Qwen2_7b_0613
7.0
83.152
Seq. Classifier
Appendix
Table 19: RewardBench leaderboard RMs of 7B parameter size, sorted by AvgAcc.
Model
Params (B)
Avg Acc
Type
Skywork/Skywork-VL-Reward-7B
7.0
68.847
Seq. Classifier
weqweasdas/RM-Mistral-7B
7.0
59.601
Seq. Classifier
openbmb/Eurus-RM-7b
7.0
58.057
Seq. Classifier
CIR-AMS/BTRM_Qwen2_7b_0613
7.0
57.363
Seq. Classifier
Finemath-RM-Qwen-2.5-7B (ours)
7.0
57.0
Seq. Classifier
internlm/internlm2-7b-reward
7.0
53.348
Seq. Classifier
Appendix
Table 20: RewardBench 2 leaderboard RMs of 7B parameter size, sorted by AvgAcc.
Task
Ep.
Optimizer
LR
λ KL
Rollout K
Max len.
Batch size
Eval metric
GSM8K
5
Adam betas 0.9, 0.95
10−6
0.1
8
512
512
mean@1 test accuracy using math-verify
MATH
5
Adam betas 0.9, 0.95
10−6
0.1
8
512
512
mean@1 test accuracy using math-verify
Appendix
Table 21: Actor training hyperparameters for GRPO experiments. λ denotes the coefficient of the reference regularization term in Eq. ( 11 ); rollout K is the number of sampled completions per prompt.
Dataset
Epochs
Optimizer
LR
Warmup ratio
Cutoff len.
Batch size
RLHFMix1
1
Adam betas 0.9, 0.95
2e – 6
0.03
2048
512
Appendix
Table 22: Hyperparameters for RM training with curated preference dataset RLHFMix1 . Cutoff length indicates the token number limits for prompt and response concatenated sequence.
Figure 12: RewardBench 2 curve across the 1 epoch training course on the curated preference dataset RLHFMix1 for 3 representative initializations Llama-3.2-3B base model, FineMath-RM-Llama-3.2-3B trained from our method, and Llama-3.2-3B-Instruct .
Labeled fraction
Pairs
Base init.
NTP init.
1%
2.4K
23
32
5%
12K
23
29
10%
24K
24
29
25%
61K
31
33
50%
122K
32
44
100%
244K
32
44
Appendix
Table 23: Overall RewardBench 2 scores after training on different fractions of RLHFMix1 , comparing base and NTP initialization.
Reward models play a fundamental role in aligning large language models with human preferences. Existing methods predominantly follow two paradigms: scalar discriminative preference models, which are efficient but lack interpretability, and generative judging models, which offer richer reasoning at the cost of higher computational overhead. We observe that the log-probability margin between verdict tokens strongly correlates with prediction correctness, providing a reliable proxy for instance difficulty without additional inference cost. Building on this insight, we propose CAMEL, a confidence-gated reflection framework that performs a lightweight single-token preference decision first and selectively invokes reflection only for low-confidence instances. To induce effective self-correction, we train the model via reinforcement learning with counterfactual prefix augmentation, which exposes the model to diverse initial verdicts and encourages genuine revision. Empirically, CAMEL achieves state-of-the-art performance on three widely used reward-model benchmarks with 82.9% average accuracy, surpassing the best prior model by 3.2% and outperforming 70B-parameter models using only 14B parameters, while establishing a strictly better accuracy-efficiency Pareto frontier.
Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.
Yunsheng Lu, Zijiang Yang, Licheng Pan +1
College of Computer Science and Technology, Zhejiang University · Department of Statistics, University of Chicago
Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter β decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.
Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA