Next-token Preference: Scaling Unsupervised Reward Modeling

Jingxuan Fan; Yueying Li; Zhenting Qi; Dinghuai Zhang; Kianté Brantley; Sham M. Kakade; Hanlin Zhang

cs.LGFeb 11, 2026

Next-token Preference: Scaling Unsupervised Reward Modeling

Authors: Jingxuan Fan, Yueying Li, Zhenting Qi, Dinghuai Zhang, Kianté Brantley, Sham M. Kakade, Hanlin Zhang

Organizations: Harvard University · Cornell University · Microsoft Research · Kempner Institute

Abstract

Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference (NTP), an unsupervised reward-modeling method that constructs preference pairs from the continuation structure of raw web text. NTP uses each prefix's original continuation as the chosen response and nearby continuations as in-batch negatives, without candidate generation or human or model-generated preference annotations. Training on 11M tokens of math-focused web data yields steady gains on RewardBench and RewardBench 2, with gains across model families and sizes. Across mathematical and general-domain web corpora, our method improves overall RewardBench 2 accuracy by 7.6-8.3 percentage points over initialization under the same token budget. Training on math-focused text yields gains of up to +16.1 on in-domain math subsets and consistent improvements on out-of-domain safety and general subsets. These reward models improve best-of-N selection on math, code, safety, and instruction following, and support math policy optimization competitive with supervised reward models of similar size. NTP also improves the label efficiency of subsequent supervised fine-tuning. Using half the labeled pairs from a curated preference dataset, NTP initialization surpasses base initialization trained on the full dataset. These results suggest that web text contains transferable preference signals that can reduce dependence on costly preference annotations.

Figures & tables

Figure 1: Schematic overview of our NTP reward model training workflow from web math text. Raw documents are split into prefix–suffix pairs, where the true continuation is treated as the chosen response and other in-batch continuations serve as implicit negatives. The reward model is trained with a Bradley–Terry objective over these online preference pairs, enabling scalable reward learning without human annotations.

Figure 2: Scalability of NTP with respect to data size. Reward models initialized from pretrained Llama-3.2-3B improve across RewardBench and RewardBench 2 subsets as the token budget increases to 11M.

Method	Budget	RewardBench 2 gain
FineMath NTP (ours)	11M tokens	+7.7
Skywork-80K-v0.2	$\approx$ 9K pairs (11M)	$+1.8$
Skywork-80K-v0.2	77K pairs (full)	$+9.8$
RLHFMix1	$\approx$ 22K pairs (11M)	$+0.8$
RLHFMix1	244K pairs (full)	$+8.3$

Table 1: Overall RewardBench 2 gains (percentage points) over Llama-3.2-3B initialization. Bold : best gain at the matched 11M-token budget.

Figure 3: Effect of batch size on peak gains and learning trajectory of RewardBench 2.

Figure 4: Effect of data splitting design on peak gains and learning trajectory of RewardBench 2.

Figure 5: Effect of centering loss on RewardBench 2 peak gains, learning trajectory and BoN MAP.

Figure 6: BoN accuracy curve on Llama-3.1-8B-Instruct GSM8K rollouts for RMs trained with or without centering loss ( init –initialized seed).

Training corpus	RewardBench 2 gain
FineMath-4plus	$+7.7$
MegaMath-Web-Pro-Max	$+8.3$
FineWeb-Edu	$+7.6$

Table 2: Peak average RewardBench 2 accuracy gains (percentage points) over the shared initialization for Llama-3.2-3B RMs trained on 11M tokens from each corpus.

Figure 7: BoN scaling curves of RMs across two ID math tasks (GSM8K, MATH500) and three OOD tasks (HumanEval+, Toxigen, IFEval).

	Actor: Llama-3.2- 3B-Instruct		Actor: Llama-3.1- 8B-Instruct
	MATH	GSM8K	MATH	GSM8K
Before training	$0.268$	$0.789$	$0.423$	$0.876$
Skywork-Reward-V2-Llama-3.1-8B	$0.419$	0.833	0.447	$0.882$
Skywork-Reward-V2-Llama-3.2-3B	0.439	$0.819$	0.465	0.884
beaver-7b-unified-reward	$0.416$	$0.803$	$0.430$	$0.881$
FineMath-RM-Qwen-2.5-7B	0.420	0.823	$0.437$	0.886

Table 3: Peak mean@1 test accuracy within five GRPO epochs. Both actors use instruction-tuned initializations. Bold and underlined values denote the best and second-best results, respectively.

Checkpoint	ID	OOD Safety	OOD others	RewardBench 2
Llama-3.2-3B	+14.5	+22.3	+1.6	+8.3
Llama-3.2-3B-Instruct	+19.4	+22.6	+7.8	+13.1
Finemath-RM-Llama-3.2-3B	+21.4	+24.4	+2.0	+10.4

Table 4: RewardBench 2 gains from training on curated preference dataset RLHFMix1 under different initializations.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Pairs $N$	Prompt length	Preferred length	Rejected length	$T_{\text{cand}}$	Overhead (in/out)	$T_{\text{in}}^{\text{gen}}$	$T_{\text{out}}^{\text{gen}}$	Cost ($)	$/pair
HH-RLHF	115,396	160.4	82.2	73.6	155.8	$0$ / –	18.51M	17.98M	180.85	0.00157
SHP	93,301	186.2	173.6	88.8	262.4	$0$ / –	17.37M	24.48M	230.60	0.00247
HelpSteer	37,131	530.0	116.4	89.3	205.7	$0$ / –	19.68M	7.64M	100.46	0.00271
PKU-SafeRLHF-30K	26,874	21.5	70.4	74.6	145.0	$0$ / –	0.58M	3.90M	32.33	0.00120
UltraFeedback	340,025	161.5	279.5	211.1	490.6	$0$ / –	54.91M	166.82M	1444.36	0.00425
UltraInteract	161,927	507.4	396.6	416.7	813.3	$0$ / –	82.16M	131.70M	1217.89	0.00752

Appendix

Table 5: Estimated candidate generation cost instantiated with a GPT-4–class completion model gpt-4o .

Dataset	#Pairs $N$	Prompt length	Preferred length	Rejected length	$T_{\text{content}}$	Overhead (in/out)	$T_{\text{in}}^{\text{judge}}$	$T_{\text{out}}^{\text{judge}}$	Cost ($)	$/pair
HH-RLHF	115,396	160.4	82.2	73.6	316.2	40 / 40	41.10M	4.62M	148.92	0.00129
SHP	93,301	186.2	173.6	88.8	448.6	40 / 40	45.59M	3.73M	151.29	0.00162
HelpSteer	37,131	530.0	116.4	89.3	735.7	40 / 40	28.79M	1.49M	86.91	0.00234
PKU-SafeRLHF-30K	26,874	21.5	70.4	74.6	166.5	40 / 40	5.55M	1.07M	24.57	0.00091
UltraFeedback	340,025	161.5	279.5	211.1	652.1	40 / 40	235.71M	13.60M	725.83	0.00213
UltraInteract	161,927	507.4	396.6	416.7	1320.7	40 / 40	220.56M	6.48M	616.28	0.00381

Appendix

Table 6: Estimated LLM annotation cost for preference labeling with a single GPT-4–class judge ( gpt-4o pricing). We use

O_{\text{in}}=40

input overhead tokens and

O_{\text{out}}=40

output tokens per pair.

Backbone	Seed	Optimizer	Learning rate	Batch size	$c$ (center)
Llama-3.2-1B	2025	Adam betas 0.9, 0.95	$10^{-6}$ , constant 0.05 warmup ratio	32	0.01
Llama-3.2-3B	2025	Adam betas 0.9, 0.95	$10^{-6}$ , constant 0.05 warmup ratio	32	0.01
Qwen2.5-3B-Inst.	2028	Adam betas 0.9, 0.95	$10^{-6}$ , constant 0.05 warmup ratio	32	0.01
Qwen2.5-7B-Inst.	2025	Adam betas 0.9, 0.95	$10^{-6}$ , constant 0.05 warmup ratio	8	0.01

Appendix

Table 7: RM training hyperparameters.

Backbone	Corpus	Optimizer	Learning rate	Blocks per step	Cutoff (tokens)	Epochs
Llama-3.2-3B	FineMath-4plus	AdamW	$10^{-5}$ peak, cosine, warmup ratio 0.1	1024	2048	1

Appendix

Table 8: Continued-pretraining recipe for the 11M-token control in Table 9 . Blocks per step is the global batch size.

Readout	Untrained	CPT	RM
Frozen probe	30.0	28.9	36.4
Conversion (512 pairs)	32.3	31.2	–
Original RM head	–	–	36.2

Appendix

Table 9: RewardBench 2 scores for Llama-3.2-3B before training and after continued pretraining (CPT) or NTP reward-model training (RM), each on 11M FineMath tokens.

Dataset	#Pairs	Instruction (tokens)	Chosen (tokens)	Rejected (tokens)	Tokens/ pair	Equiv. pairs @ 11M tok.
Skywork-80K-v0.2	77,016	$\approx$ 115	$\approx$ 514	$\approx$ 584	$\approx$ 1,213	$\approx$ 9,070
RLHFMix1	244,216	$\approx$ 224	$\approx$ 149	$\approx$ 120	$\approx$ 493	$\approx$ 22,310

Appendix

Table 10: Per-entry token statistics for curated comparison datasets and the equivalent number of preference pairs under an 11M-token training budget. Token counts are estimated at

{\sim}4

characters per token.

Task (ID/OOD)	Dataset / split	Prompt format	Actor decoding	Accuracy / scoring criterion
MATH500 (ID)	HuggingFaceH4/MATH-500 (test)	Problem: question . Provide your reasoning step by step, then the final answer the form FINAL_ANSWER: answer .	$T{=}$ 1.0, top- $p{=}$ 1.0 max tokens ${=}$ 512	lm-eval-harness math-verify
GSM8K (ID)	openai/gsm8k (test)	Problem: question . Provide your reasoning step by step, then the final answer the form FINAL_ANSWER: answer .	$T{=}$ 1.0, top- $p{=}$ 1.0 max tokens ${=}$ 512	lm-eval-harness math-verify
HumanEval+ (OOD)	evalplus/humanevalplus v0.1.10 (160 of 164 problems whose canonical solutions pass under EvalPlus)	Please provide a self-contained Python script that solves the following problem in a markdown code block: problem	$T{=}$ 1.0, top- $p{=}$ 1.0 max tokens ${=}$ 512	evalplus ; pass iff the extracted program passes both the base and the plus test suites
ToxiGen (OOD)	toxigen/toxigen-data (test)	Is the following statement hateful? Statement: text . Respond with either Yes or No. Response:	$T{=}$ 1.0, top- $p{=}$ 1.0 max tokens ${=}$ 20	lm-eval-harness
IFEval (OOD)	google/IFEval (train)	prompt	$T{=}$ 1.0, top- $p{=}$ 1.0 max tok. ${=}$ 1280	lm-eval-harness prompt level strict

Appendix

Table 11: BoN response generation and evaluation details by task.

Figure 11: Validation reward curve with or without centering loss.

Seed	ID	OOD Safety	OOD Others	RewardBench 2 Avg
2025	$+16.1$	$+5.4$	$+7.4$	$+7.7$
2027	$+17.5$	$+10.5$	$+4.7$	$+8.4$
2028	$+19.1$	$+16.1$	$+0.2$	$+7.2$
Mean	$+17.6$	$+10.7$	$+4.1$	$+7.7\pm 0.35$

Appendix

Table 12: RewardBench 2 gains in percentage points over each run’s initialization for three independent runs of the default Llama-3.2-3B setup. The final row reports means across seeds; the overall gain is reported as mean

\pm

standard error.

Benchmark	Subset	Gain
RewardBench	Chat	$+4.9$
	Chat Hard	$+7.0$
	Safety	$+16.0$
	Reasoning	$+13.1$
RewardBench 2	Math	$+17.6$
	Safety	$+10.7$

Appendix

Table 13: RewardBench and RewardBench 2 subset gains in percentage points over each run’s initialization, averaged across three seeds for NTP training of Llama-3.2-3B on 11M FineMath tokens.

Backbone	RewardBench		RewardBench 2
Backbone	Score	Rank	Score	Rank
Llama-3.2-1B	60.0	3 of 4	33.2	3 of 4
Llama-3.2-3B	65.6	4 of 9	36.2	9 of 12
Qwen2.5-3B-Inst.	70.0	3 of 9	46.2	8 of 12
Qwen2.5-7B-Inst.	73.8	18 of 47	57.0	5 of 11

Appendix

Table 14: Average RewardBench and RewardBench 2 scores and leaderboard ranks for RMs trained with different initialization backbones on 11M tokens. Ranks are reported as

k

n

, where

n

counts all entries in the corresponding comparison set, including our models.

Model	Params (B)	AvgAcc	Type
opencompass/CompassJudger-1-1.5B-Instruct	1.5	73.4	Generative
OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5	1.4	69.5	Seq. Classifier
Finemath-RM-Llama-3.2-1B (ours)	1.0	60.0	Seq. Classifier
Qwen/Qwen1.5-0.5B-Chat	0.5	53.8	DPO

Appendix

Table 15: RewardBench leaderboard RMs in the 0.5–1.5B parameter range, sorted by AvgAcc.

Model	Params (B)	AvgAcc	Type
Skywork/Skywork-Reward-V2-Llama-3.2-1B	1.0	64.376	Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-0.6B	0.6	61.250	Seq. Classifier
Finemath-RM-Llama-3.2-1B (ours)	1.0	33.2	Seq. Classifier
OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5	1.4	26.479	Seq. Classifier

Appendix

Table 16: RewardBench 2 leaderboard RMs in the 0.5–1.5B parameter range, sorted by AvgAcc.

Model	Params (B)	AvgAcc	Type
Ray2333/GRM-llama3.2-3B-rewardmodel-ft	3.0	90.9	Seq. Classifier
stabilityai/stablelm-zephyr-3b	3.0	74.0	DPO
Finemath-RM-Qwen-2.5-3B (ours)	3.0	70.0	Seq. Classifier
Finemath-RM-Llama-3.2-3B (ours)	3.0	65.6	Seq. Classifier
stabilityai/stable-code-instruct-3b	3.0	64.3	DPO
Qwen/Qwen1.5-4B-Chat	4.0	56.0	DPO

Appendix

Table 17: RewardBench leaderboard RMs in the 3-4B parameter range, sorted by AvgAcc.

Model	Params (B)	Avg Acc	Type
Skywork/Skywork-Reward-V2-Qwen3-4B	4.0	75.51	Seq. Classifier
Skywork/Skywork-Reward-V2-Llama-3.2-3B	3.0	74.665	Seq. Classifier
Schrieffer/Llama-SARM-4B	4.0	73.793	Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-1.7B	1.7	68.176	Seq. Classifier
Skywork/Skywork-Reward-V2-Llama-3.2-1B	1.0	64.376	Seq. Classifier
Skywork/Skywork-Reward-V2-Qwen3-0.6B	0.6	61.25	Seq. Classifier

Appendix

Table 18: RewardBench 2 leaderboard RMs up to 4B parameters, sorted by AvgAcc.

Model	Params (B)	AvgAcc	Type
Skywork/Skywork-VL-Reward-7B	7.0	90.070	Seq. Classifier
R-I-S-E/RISE-Judge-Qwen2.5-7B	7.0	88.191	Generative
internlm/internlm2-7b-reward	7.0	87.593	Seq. Classifier
ZiyiYe/Con-J-Qwen2-7B	7.0	87.120	Generative
opencompass/CompassJudger-1-7B-Instruct	7.0	83.167	Generative
CIR-AMS/BTRM_Qwen2_7b_0613	7.0	83.152	Seq. Classifier

Appendix

Table 19: RewardBench leaderboard RMs of 7B parameter size, sorted by AvgAcc.

Model	Params (B)	Avg Acc	Type
Skywork/Skywork-VL-Reward-7B	7.0	68.847	Seq. Classifier
weqweasdas/RM-Mistral-7B	7.0	59.601	Seq. Classifier
openbmb/Eurus-RM-7b	7.0	58.057	Seq. Classifier
CIR-AMS/BTRM_Qwen2_7b_0613	7.0	57.363	Seq. Classifier
Finemath-RM-Qwen-2.5-7B (ours)	7.0	57.0	Seq. Classifier
internlm/internlm2-7b-reward	7.0	53.348	Seq. Classifier

Appendix

Table 20: RewardBench 2 leaderboard RMs of 7B parameter size, sorted by AvgAcc.

Task	Ep.	Optimizer	LR	$\bm{\lambda}$ KL	Rollout $K$	Max len.	Batch size	Eval metric
GSM8K	5	Adam betas 0.9, 0.95	$10^{-6}$	0.1	8	512	512	mean@1 test accuracy using math-verify
MATH	5	Adam betas 0.9, 0.95	$10^{-6}$	0.1	8	512	512	mean@1 test accuracy using math-verify

Appendix

Table 21: Actor training hyperparameters for GRPO experiments.

\lambda

denotes the coefficient of the reference regularization term in Eq. ( 11 ); rollout

K

is the number of sampled completions per prompt.

Dataset	Epochs	Optimizer	LR	Warmup ratio	Cutoff len.	Batch size
RLHFMix1	1	Adam betas 0.9, 0.95	$2e$ – $6$	0.03	2048	512

Appendix

Table 22: Hyperparameters for RM training with curated preference dataset RLHFMix1 . Cutoff length indicates the token number limits for prompt and response concatenated sequence.

Figure 12: RewardBench 2 curve across the 1 epoch training course on the curated preference dataset RLHFMix1 for 3 representative initializations Llama-3.2-3B base model, FineMath-RM-Llama-3.2-3B trained from our method, and Llama-3.2-3B-Instruct .

Labeled fraction	Pairs	Base init.	NTP init.
1%	2.4K	23	32
5%	12K	23	29
10%	24K	24	29
25%	61K	31	33
50%	122K	32	44
100%	244K	32	44

Appendix

Table 23: Overall RewardBench 2 scores after training on different fractions of RLHFMix1 , comparing base and NTP initialization.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

CAMEL: Confidence-Gated Reflection for Reward Modeling
Feb 24, 2026Zirui Zhu, Hailun Xu, Yang Luo +4Reward Modeling LLM Alignment
National University of Singapore · TikTok
Robust Reward Modeling for Large Language Models via Causal Decomposition
Apr 15, 2026Yunsheng Lu, Zijiang Yang, Licheng Pan +1Reward Modeling LLM Alignment
College of Computer Science and Technology, Zhejiang University · Department of Statistics, University of Chicago
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA

Feb 24, 2026J/K move · Enter open · S save

CAMEL: Confidence-Gated Reflection for Reward Modeling

Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu+3

National University of Singapore · TikTok

Reward Modeling LLM Alignment

Open paper AskPDF arXiv

Reward models play a fundamental role in aligning large language models with human preferences. Existing methods predominantly follow two paradigms: scalar discriminative preference models, which are efficient but lack interpretability, and generative judging models, which offer richer reasoning at the cost of higher computational overhead. We observe that the log-probability margin between verdict tokens strongly correlates with prediction correctness, providing a reliable proxy for instance difficulty without additional inference cost. Building on this insight, we propose CAMEL, a confidence-gated reflection framework that performs a lightweight single-token preference decision first and selectively invokes reflection only for low-confidence instances. To induce effective self-correction, we train the model via reinforcement learning with counterfactual prefix augmentation, which exposes the model to diverse initial verdicts and encourages genuine revision. Empirically, CAMEL achieves state-of-the-art performance on three widely used reward-model benchmarks with 82.9% average accuracy, surpassing the best prior model by 3.2% and outperforming 70B-parameter models using only 14B parameters, while establishing a strictly better accuracy-efficiency Pareto frontier.

Abstract

Reward Modeling

Explore similar work

CAMEL: Confidence-Gated Reflection for Reward Modeling

Robust Reward Modeling for Large Language Models via Causal Decomposition

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights