Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

Pranjal Aggarwal; Marjan Ghazvininejad; Seungone Kim; Ilia Kulikov; Jack Lanchantin; Xian Li; Tianjian Li; Bo Liu

Figure 1: RL training on the Principia Collection, which requires mathematical-object outputs, improves an LM’s reasoning capability. (a) On our new challenging benchmark, PrincipiaBench , which requires deriving mathematical objects, Qwen3-4B-Base, Qwen2.5-7B-Base, and OctoThinker-8B-Long-Base trained on Principia Collection yield average gains of +18.23%, +10.23%, and +15.16%, respectively. (b) Training on the Principia Collection also improves performance on numerical (AIME-2025) and MCQA (SuperGPQA) benchmarks, demonstrating cross-format reasoning gains. (c) Using a strong model-based verifier (GPT-OSS-120B) and excluding MCQA data provides the best performance.

Figure 2: LMs struggle to solve problems that require mathematical objects as answers. For example, Qwen3-235B can solve this problem from SuperGPQA ( Du et al., 2025 ) in an MCQ setting by using the options as an anchor to perform backward chaining. However, when removing the options, the model starts making false assumptions ( i.e. , that all 1-eigenspaces share a common fixed vector and that the representation cannot be fully trivial) and derives the incorrect solution ( i.e. , concluding

\langle 1_{G},\chi\rangle=1

instead of allowing the valid case

\langle 1_{G},\chi\rangle=2

). Note that yellow-highlighted comments pinpoint and describe the critical flaws included within the model’s chain-of-thought.

Figure 3: Performance consistently drops when removing options in MCQA benchmarks. In the mathematical and engineering subset of SuperGPQA, among instances where answers are expressed as mathematical objects, LMs show a 10–-20% decrease in performance when options are removed. This suggests that MCQA evaluations tend to overestimate true reasoning ability ( i.e. , high MCQA scores do not necessarily translate into strong reasoning over mathematical objects).

Figure 4: Token count distribution comparison between the Principia Collection and other widely used RL post-training datasets. The problem statements and answers of Principia Collection are relatively longer since it requires the derivation of complex mathematical objects and the problem statements based on graduate-level STEM subjects are very detailed.

Figure 5: Annotation UI for constructing the Principia VerifyBench.

Figure 6: Two qualitative examples of instances within the Principia Collection. Each instance consists of a problem statement, an answer, and a subject entity. The subject entities originate from Physics Subject Headings (PhySH) and Mathematics Subject Classification System (MSC2020). The required answer types are either equation, inequality, interval, set, matrix, and piecewise function. Note that for the example on the right side, the detailed setup (definitions of

t_{i},t_{0},t_{G}

, etc.) is not a hint but a necessary part of the problem statement , which is typical in research-level mathematics, specifying how to combine these quantities into the final piecewise formula for

\widetilde{T}

.

Category	Content
Subject Entity	Statistical mechanics, structure of matter $\rightarrow$ Equilibrium statistical mechanics $\rightarrow$ Classical equilibrium statistical mechanics (general)
Strategy Description	Evaluating free-energy differences by exploiting convexity and pair-correlation functions: Applying the convexity of the Helmholtz free energy with respect to an interpolating interaction parameter, using Jensen’s inequality and the definition of the two-particle density (or radial distribution function) to derive rigorous upper and lower bounds on $F-F^{\prime}$ for systems with ordered pair potentials.
Problem Statement	1. Identify a suitable statistical-mechanical inequality — monotonicity of $F$ with respect to $u(r)$ .
Proposal CoT	2. Define the Hamiltonians and show $u^{\prime}(r)\leq u(r)$ implies $Z^{\prime}\geq Z$ .
(Summary)	3. Convert to free energies: $F^{\prime}\leq F$ .
	4. Require a single final inequality as the answer.

Table 1: Example of a subject entity (acquired from PhySH), a strategy description (from step 1), an initial problem statement and its corresponding CoT, and a revised problem statement and its corresponding CoT (from step 2).

Figure 7: Illustration of our majority voting procedure used in dataset construction. Determining the majority vote itself becomes challenging with mathematical object predictions since the same object could be represented in distinct forms. Hence, (a) we assess the equivalence between each pair, (b) check if transitivity is preserved, (c) adopt a conservative rule of flipping judgments if transitivity is not preserved w.r.t. witnessess of each prediction pairs (we adopt a threshold of 0.6), and (d) using the final results, determine the majority vote using the equivalence judgments. Detailed procedure is in Algorithm 1 given in the Appendix.

Figure 8: Answer type counts of the Principia Collection. Unlike other datasets, all 250K instances consist of mathematical objects.

Model	Training Data	PrincipiaBench (Math Obj.)					MCQA		Numerical		Total Avg.
Model	Training Data	RealMath	Physics	ARB	SuperGPQA ∗	Avg.	SuperGPQA ∗	GPQA-Diamond	AIME-24	AIME-25	Total Avg.
Qwen2.5-7B-Base
Qwen2.5-7B-Base	-	11.19	9.32	16.76	13.75	12.75	20.41	27.40	8.02	3.65	13.81
Qwen2.5-7B-Instruct	-	12.95	10.45	19.15	20.05	15.65	32.61	35.29	12.08	7.81	18.80
SimpleRL-7B-Zoo	SimpleZoo-Data.	14.00	10.68	26.86	21.17	18.18	28.49	28.85	15.31	8.75	19.26
General-Reasoner-7B	WebInstruct-Ver.	15.96	12.39	26.86	23.15	19.59	32.95	36.55	10.63	7.71	20.78

Table 2: Evaluation performance on PrincipiaBench involving hard reasoning problems with mathematical objects . Principia models are trained with new synthetic reasoning problems involving mathematical objects (Principia Collection) – which gives improvements both on PrincipiaBench and existing MCQA and numerical benchmarks. Note that for SuperGPQA ∗ , we use problems that require the generation of mathematical objects from the mathematics and engineering subset. LMs are grouped based on the base model it was trained on, and the best scores from each groups are bolded . Note that Reimplemented denotes the baselines we trained ourselves with either DeepScaleR or WebInstruct-Verified datasets, since there are no publicly available checkpoints for some of the base LMs.

Figure 9: Training directly on complex mathematical objects yields substantially better transfer than training on datasets requiring only numerical values or simple mathematical objects. Validation performance across different answer types when training with three RL post-training datasets. DeepScaleR and WebInstruct-Verified contain problems from reliable online sources but mostly require simple mathematical objects or numerical answers. In contrast, Principia Collection is a fully synthetic dataset built around diverse and complex mathematical objects. Across all categories, Principia-trained models exhibit consistently higher and more stable improvements over training steps, demonstrating that exposure to complex answer structures during learning is crucial for generalizing to tasks involving rich mathematical forms.

Figure 10: Response length gradually increases when training on the Principia Collection. Note that the token length is measured on our validation set.

Verifier Type	Verifier	Active Parameters	Principia VerifyBench
Verifier Type	Verifier	Active Parameters	Agreement with Humans (%)	Precision	Recall	F1
Rule-based Verifier	math-verify	-	5.95%	5.38	6.67	5.96
Fine-tuned Verifier	general-verifier	1.5B	82.74%	83.13	93.24	87.90
Fine-tuned Verifier	CompassVerifier	3B	81.55%	90.74	65.33	75.97
Fine-tuned Verifier	CompassVerifier	7B	88.69%	93.75	80.00	86.33
Fine-tuned Verifier	CompassVerifier	32B	91.66%	94.20	86.67	90.28

Table 3: Principia VerifyBench results . We find that model-based verifiers are more effective at determining answer equivalence than rule-based verifiers. Also, o3 and GPT-OSS-120B achieve high agreement rate with ground truth labels, justifying our selection in employing a strong judge in PrincipiaBench and a reward model for RL training with the Principia Collection.

Data	Verifier Type	Verifier	Math Obj. (PrincipiaBench)					MCQA		Numerical		Total Avg.
Data	Verifier Type	Verifier	RealMath	Physics	ARB	SuperGPQA ∗	Avg.	SuperGPQA ∗	GPQA-Diamond	AIME-24	AIME-25	Total Avg.
-	-	-	9.43	5.80	17.81	12.18	11.31	20.46	18.09	12.5	10.21	13.31
DeepScaleR	Rule	math-verify	9.24	20.91	38.3	28.10	24.14	43.38	44.63	23.02	21.77	28.67
DeepScaleR	Model	general-verifier	16.20	18.86	38.03	30.13	25.81	42.35	45.74	25.00	23.13	29.93
DeepScaleR	Model	GPT-OSS-120B	15.55	19.93	39.16	31.04	27.42	45.32	46.14	24.17	22.29	30.45
Principia-Collec.	Rule	math-verify	11.53	10.80	21.54	15.67	14.89	29.06	32.29	5.52	3.96	16.30

Table 4: Training with rule-based verifier vs. model-based verifier : Using a strong model-based verifier becomes highly important when the data is more challenging to verify. While there is little difference in terms of performance when training on DeepScaleR, which consists of numerical value and simple mathematical object answers, the performance varies a lot when training on the Principia Collection with consists of complex mathematical object answers. The best performances across all baselines are bolded .

Figure 11: Example of training instances used in data ablation experiments. We prepare training data that share the same subject entity (“ Information and communication theory, circuits ¿¿ Communication, information ¿¿ Sampling theory in information and communication theory ”) but requires either a mathematical object answer (our Principia Collection data), numerical answers, and multiple choice options. By training on these distinctly formatted datasets ( subsubsection 1.5.2 ), we find that training to generate mathematical objects is effective at improving reasoning capabilities.

Training Data Answer Format			Method	Math Obj. (PrincipiaBench)					MCQA		Numerical		Total Avg.
Math Obj.	Numer.	MCQA.	Method	RealMath	Physics	ARB	SuperGPQA ∗	Avg.	SuperGPQA ∗	GPQA-Diamond	AIME-24	AIME-25	Total Avg.
✗	✗	✗	-	9.43	5.80	17.81	12.18	11.31	20.46	18.09	12.5	10.21	13.31
✓	✗	✗	Single Training	19.28	21.81	43.62	33.92	29.66	45.53	43.56	22.71	23.96	31.80
✗	✓	✗	Single Training	17.96	20.14	40.96	30.28	27.34	46.10	43.50	26.98	23.75	31.19
✗	✗	✓	Single Training	15.84	18.75	37.23	27.87	24.96	34.74	29.29	17.71	19.79	25.15
✓	✓	✗	Joint Training	17.46	19.89	40.96	32.18	27.62	43.75	42.27	22.50	20.31	29.92

Table 5: Unifying mathematical objects and numerical type data using weight merging leads to the best performances. We systematically generate three different types of synthetic data that shares the same subject entities but require different format of answers. Then, we either train on one of them, jointly train them, or merge the weights of single trained checkpoints. The best scores are bolded .

Figure 12: Reinforcement Learning with an LM as Reward Model (RLLM). Comparison of RLHF, RLVR, and RLLM for post-training LLMs. RLHF optimizes a policy against a scalar reward model trained on human-annotated preference data using the Bradley-Terry objective. RLVR trains a policy using a rule-based verifier and hence is restricted to easy-to-verify tasks with ground-truth labels. RLLM (ours) is a special case of RL-from-AI-Feedback (RLAIF) that first trains an LM-as-RM on synthetic judgments using RL and then uses its generative rewards to optimize the policy. An LM-as-RM exploits an LLM’s (1) reasoning capabilities to produce higher-quality reward signals and (2) instruction-following capabilities to allow flexible reward design. Thus, RLLM unifies the post-training paradigm, enabling the policy model to excel across easy-to-verify, hard-to-verify, and non-verifiable tasks. We further show that training the RLLM reward model on-policy (via responses sampled from the policy model) improves results.

Figure 13: Verifiable Tasks (Math)

Figure 16: RL validation rewards for LM-as-RM training (left) and RLLM policy training (center), alongside the downstream AIME24 accuracy of the policy model (right). Centered moving average is shown to highlight the trend.

Method	RM	RM Type	RM Size	Math (Easy-to-Verify)						Hard-to-Verify
Method	RM	RM Type	RM Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average	Physics
Qwen3-1.7B	–	–	–	45.22	36.20	49.78	23.30	21.44	35.20	15.57
Policy Trained on hard-to-verify samples w/o Reference Answer
RLHF	Nexusflow/Athene-RM-8B	Scalar	8B	40.22	32.92	38.95	17.08	19.58	29.75	13.18
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	48.33	36.26	47.91	22.92	24.36	35.96	16.93
RLHF	nvidia/AceMath-7B-RM	Scalar	7B	48.76	38.34	47.69	23.12	22.71	36.12	16.24

Table 6: Reference-free setting : Comparison of different post-trained Qwen3-1.7B (Instruct) models using RLLM or RLHF on easy-to-verify and hard-to-verify reasoning benchmarks. All models are trained on hard-to-verify samples in a reference-free setting. RLHF’ed models are optimized using SOTA scalar RMs. RLLM models are optimized using either prompted LM-as-RM or our trained J1 LM-as-RM. We observe improved RLLM results by scaling up the LM-as-RM, with J1-Qwen3-32B-RM improving AIME24 by 12% on top of a Qwen3-1.7B (Instruct) model.

Method	Verifier	Verifier	Verifier	Math (Easy-to-Verify)						Hard-to-Verify
Method	Verifier	Type	Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average	Physics
Qwen3-1.7B	–	–	–	45.22	36.20	49.78	23.30	21.44	35.20	15.57
Policy Trained on hard-to-verify samples w/ Reference Answer
RLVR	Math-Verify	Rule-based	–	48.96	34.79	47.51	21.24	25.83	35.67	15.95
RLVR	TIGER-Lab/general-verifier	Generative	1.5B	50.42	41.66	49.38	23.55	23.33	37.67	17.39
RLLM	Qwen3-1.7B	Generative	1.7B	52.50	41.46	49.59	33.30	26.66	40.70	16.93

Table 7: Reference-based setting : Comparison of different post-trained Qwen3-1.7B (Instruct) models using RLLM or RLVR on easy-to-verify and hard-to-verify reasoning benchmarks. All models are trained on hard-to-verify examples in a reference-based setting. RLVR models are optimized using either rule-based or model-based verifiers. RLLM models are optimized using either prompted or trained LM-as-RM (functioning as reference-based verifiers). All RLLM variants outperform all RLVR variants.

Method	RM/Verifier	RM Type	RM Size	Math
Method	RM/Verifier	RM Type	RM Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average
Qwen3-1.7B	–	–	–	45.22	36.20	49.78	23.30	21.44	35.20
Policy Trained on easy-to-verify samples w/o Reference Answer
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	48.96	35.21	43.96	33.30	25.20	37.33
RLLM	J1-Qwen-32B-RM	Generative	32B	53.34	45.43	52.08	26.70	27.33	41.18
Policy Trained on easy-to-verify samples w/ Reference Answer

Table 8: Comparison of RLLM, RLHF, and RLVR across different training datasets – easy-to-verify, hard-to-verify, reference-free, and reference-based. RLLM on hard-to-verify data with a strong LM-as-RM outperforms all models trained on easy-to-verify data.

Method	RM	RM Type	RM Size	Datasets
				AlpacaEval 2.0	ArenaHard 2.0	ArenaHard 2.0
				(LCWR / WR)	(Hard Prompts)	(Creative Writing)
Policy Trained on non-verifiable WildChat samples in thinking mode
Qwen3-1.7B	–	–	–	37.5 / 39.2	6.4 (-0.6 / +0.7)	6.9 (-1.0 / +1.1)
RLHF	Nexusflow/Athene-RM-8B	Scalar	8B	38.0 / 42.6	4.3 (-0.6 / +0.7)	15.8 (-2.0 / +1.9)
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	43.0 / 47.2	6.0 (-0.6 / +0.6)	8.4 (-0.9 / +1.0)

Table 9: Comparison of Win Rate (WR) and Length Controlled Win Rate (LCWR) of RLLM and RLHF on non-verifiable instruction-following tasks when training a Qwen3-1.7B policy (either in thinking or non-thinking mode). For AlpacaEval 2.0, we use GPT-4o as the evaluator and for ArenaHard 2.0, we use GPT-4.1 as the evaluator. RLLM matches or outperforms RLHF, obtaining best win rates on hard prompts of ArenaHard 2.0.

Method	RM/Verifier	RM Type	RM Size	Math (Easy-to-Verify)						Hard-to-Verify
Method	RM/Verifier	RM Type	RM Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average	Physics
Qwen3-1.7B	–	–	–	45.22	36.20	49.78	23.30	21.44	35.20	15.57
Policy Trained on hard-to-verify samples w/o Reference Answer
RLLM	Qwen3-1.7B	Generative	1.7B	51.25	37.71	51.67	26.70	26.88	38.84	17.84
RLLM	J1-Qwen3-1.7B	Generative	1.7B	52.29	39.59	45.63	30.00	21.46	37.79	15.11
Policy Trained on hard-to-verify samples w/o Reference Answer

Table 10: Analysis of Generator-Verifier Gap. RLLM post-training of a Qwen3-1.7B policy with a J1-Qwen3-1.7B LM-as-RM does not improve performance over the prompted LM-as-RM baseline while post-training with a stronger J1-Qwen3-32B LM-as-RM improves over the corresponding prompted baseline.

Figure 17: Analysis of Generator-Verifier Gap. (a) Comparison of different LMs-as-RMs in a reference-free setting on a held-out validation set (of correct/incorrect responses). J1 training on top of a weaker Qwen3-1.7B does not lead to further improvements, while the same on top of a stronger Qwen3-32B leads to 10% absolute improvement. Results are averaged across 8 seeds. (b) Corresponding validation reward curves for J1 training across RL steps.

Method	RM/Verifier	RM Training Data	Math
Method	RM/Verifier	RM Training Data	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average
Qwen3-1.7B	–	–	45.22	36.20	49.78	23.30	21.44	35.20
Policy Trained on hard-to-verify samples w/o Reference Answer
RLLM	Qwen3-32B (prompted)	–	54.38	43.53	53.76	26.70	31.02	41.88
RLLM	J1-Qwen3-32B-RM (off-policy-trained)	Llama-3.1-8B-Instruct	57.09	44.37	48.96	26.87	28.12	41.08
RLLM	J1-Qwen3-32B-RM (off-policy-trained)	Qwen3-8B	54.99	45.82	50.00	23.55	27.71	40.41

Table 11: Comparison of RLLM post-training of Qwen3-1.7B with on-policy versus off-policy J1 -trained LMs-as-RMs. On-policy J1-Qwen3-32B-RM is trained on Qwen3-1.7B responses while off-policy models are trained on either weaker Llama responses or stronger Qwen3-8B responses. On-policy trained LM-as-RM outperforms off-policy trained ones.

Method	RM	RM Type	#RM Judgments	Datasets
				AlpacaEval 2.0	ArenaHard 2.0	ArenaHard 2.0
				(LCWR / WR)	(Hard Prompts)	(Creative Writing)
Qwen3-1.7B	–	–	–	37.5 / 39.2	6.4 (-0.6 / +0.7)	6.9 (-1.0 / +1.1)
Policy Trained on WildChat non-verifiable samples in thinking mode
RLLM	Qwen3-1.7B (prompted)	Pointwise	$n$	40.1 / 46.2	5.3 (-0.7 / +0.8)	12.4 (-1.6 / +1.4)
RLLM	Qwen3-1.7B (prompted)	Pairwise with pivot	$2n$	41.0 / 42.8	6.3 (-0.8 / +0.8)	13.9 (-1.3 / +1.2)

Table 12: Effect of scaling up reward modeling compute in RLLM via pointwise, pairwise, pairwise with a pivot rollout, and triplet-based scoring between rollouts.

Figure 18: Comparison of validation reward curves for J1 training on different distributions of data. Data is either sampled from a Llama3 model (off-policy weakest), Qwen3-8B model (off-policy strongest), or Qwen3-1.7B model (on-policy). While all LMs-as-RMs improve on their respective in-distribution validation data, off-policy trained RMs do not lead to downstream task improvements potentially because of lack of OOD generalization ( Table 11 ).

Figure 19: (a) Validation reward curves for a reference-free J1-Qwen3-32B-RM and a reference-based J1-Qwen3-32B-Verifier. (b) Comparison of different LMs-as-RMs evaluated in either a reference-free setting (suffixed with ‘-RM’) or in a reference-based setting (suffixed with ‘-Verifier’). Reference-based LMs-as-RMs are more accurate than reference-free ones (e.g., J1-Qwen3-32B-Verifier is 10% better than J1-Qwen3-32B-RM).

Figure 20: ParaGator Training. Illustration of our parallel thinking scaffolding and our method. We use pass@k optimization ( Chen et al., 2025b ) for optimizing the initial round of responses and pass@1 optimization (standard RLVR) for optimizing the aggregation rollouts, and train end-to-end.

Figure 21: ParaGator Inference. During each round, we sample rollouts from the past aggregation round, pack them into the aggregation prompt, and perform inference to obtain the next pool of rollouts.

Figure 22: Parallel generation + aggregation ( orange ) brings gains across 4 competition math benchmarks (AIME, Brumo, HMMT and IMO-Answerbench) on top of 3 strong models: Kimi-K2-Thinking, Qwen3-4B-Thinking-2507, and Qwen3-4B-Instruct-2507, compared to standard generation ( blue ) and majority voting ( green ).

Figure 23: Self-aggregation performance on Qwen3-4B-Thinking-2507.

	Avg. of HMMT, Brumo, and AIME
Initial Temperature	Initial Pass@1	Initial Pass@k	$\rightarrow$ Aggregation Pass@1
T=0.6	65.8	81.01	74.11
T=0.8	65.8	81.3	74.9
T=1.0	65.7	81.7	75.2

Table 13: Model = Qwen3-4B-Thinking-2507. Effect of initial sampling temperature on decoding performance, averaged over HMMT, Brumo, and AIME. Increasing the initial temperature leaves pass@1 nearly unchanged while improving pass@k, resulting in higher aggregation performance.

Table 14: Comparison of training strategies across the initial and aggregation rounds. Columns show whether model parameters are updated via pass@1 or pass@k optimization, or kept fixed.

Figure 26: Reward by training step of the initial round.

Method	Round	HMMT	Brumo	AIME	Olympiad	Avg.
Qwen3-4B-Base	Initial	3.20/5.15	16.67/29.04	8.20/21.33	33.72/47.41	15.45/25.73
	Agg 1	2.50	14.25	10.30	30.28	14.33
	Agg 2	1.80	14.39	9.20	28.15	13.38
	Agg 3	1.60	11.38	8.03	25.16	11.54
Dr.GRPO ( Liu et al., 2025d )	Initial	9.79/15.94	25.05/35.27	19.27/28.11	45.65 /54.92	24.94/33.56
	Agg 1	11.30	26.35	21.15	47.25	26.51

Table 15: Competition Math evaluation results by method and round, with fine-tuning starting from Qwen3-4B-Base. Best values in each column bolded. Numbers = Pass@1/Pass@4. ParaGator gives the overall best results.

Figure 29: Reward by training step of the initial round.

	Scientific Reasoning (PrincipiaBench)					Competition Math
Model	RealMath	Physics	ARB	SuperGPQA	Avg.	AIME	Brumo	HMMT	Avg.
Experiments on Qwen3-4B-Base
Qwen3-4B-Base	9.11	5.69	17.44	11.01	10.81	8.20	16.67	3.20	9.36
+aggregation	8.35	4.79	18.24	10.68	10.52	10.30	14.25	2.50	9.02
Dr.GRPO	17.40	15.65	35.26	29.3	24.40	20.93	23.19	8.14	17.42
+aggregation	19.32	21.06	44.17	32.28	29.21	25.95	31.07	13.10	23.37

Table 16: Scientific reasoning (PrincipiaBench) and competition math evaluation results. Numbers denote Pass@1. Best values per column and model group are bolded. ParaGator gives the overall best results.

Figure 32: Prompt template for pointwise reference-free LMs-as-RMs

Figure 33: Prompt template for pairwise LMs-as-RMs for non-verifiable tasks. Note that unlike verifiable tasks, here we ask the model to assign a score between 0 to 10 to capture the fine-grained quality.

Figure 34: Prompt template for Pointwise Reference-based LMs-as-RMs, where an additional reference answer is provided compared to the reference-free version. Note that the reference answer is only provided as additional context and the LM-as-RM is still prompted to evaluate the entire response.

Method	RM/Verifier	RM Type	RM Size	Math (Easy-to-Verify)						Hard-to-Verify
Method	RM/Verifier	RM Type	RM Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average	Physics
Policy Trained on hard-to-verify samples w/o Reference Answer
Qwen3-8B	–	–	–	76.00	67.30	68.75	35.41	41.25	57.74	51.70
RLHF	Nexusflow/Athene-RM-8B	Scalar	8B	73.34	60.42	65.00	37.91	38.33	55.00	50.35
RLVR	nvidia/AceMath-7B-RM	Scalar	7B	72.91	65.00	65.41	33.75	42.49	55.91	50.76
RLLM	Qwen3-8B	Generative	8B	75.84	70.00	69.58	50.00	43.33	61.75	54.32

Table 17: Comparison of different post-trained Qwen3-8B (Instruct) models using RLLM, RLVR, or RLHF, showing generalization of our RLLM recipe to a stronger 8B policy model. All models are evaluated in thinking mode.

Method	RM/Verifier	RM Type	RM Size	Math (easy-to-verify)						Hard-to-Verify
Method	RM/Verifier	RM Type	RM Size	AIME24	AIME25	BRUMO25	HMMT24	HMMT25	Average	Physics
Qwen3-1.7B	–	–	–	13.33	10.83	16.04	6.25	5.21	10.33	8.75
Policy Trained on hard-to-verify samples w/o Reference Answer
RLHF	Nexusflow/Athene-RM-8B	Scalar	8B	0 5.84	0 6.25	12.08	0 2.28	0 1.65	0 5.62	7.95
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	21.87	20.21	21.87	12.92	10.00	17.37	10.00
RLHF	nvidia/AceMath-7B-RM	Scalar	7B	20.21	16.87	18.96	0 7.72	0 5.84	13.92	10.45

Table 18: Comparison of different post-trained Qwen3-1.7B (Instruct) models using RLLM, RLVR, or RLHF. All models are evaluated in non-thinking mode. Similar to our conclusions with results in thinking mode, RLLM outperforms RLHF and RLVR.

Method	RM/Verifier	RM Type	RM Size	AIME24	AIME25	HMMT24	HMMT25	BRUMO25	MATH500	Average
OctoThinker-8B-Hybrid-Base	–	–	–	2.49	2.70	4.30	0.00	0.00	52.36	10.32
Policy Trained on hard-to-verify samples w/o Reference Answer
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	7.30	3.74	6.88	3.30	0.00	60.04	13.54
RLLM	Qwen3-1.7B (prompted)	Generative	1.7B	7.93	4.37	9.79	6.70	0.83	60.86	15.08
$\Delta$ w/ OctoThinker-8B-Hybrid-Base	–	–	–	+5.44	+1.67	+5.49	+6.70	+0.83	+8.50	+4.76
Policy Trained on hard-to-verify samples w/ Reference Answer

Table 19: Comparison of different post-trained OctoThinker-8B-Hybrid-Base models using RLLM, RLVR, or RLHF. This shows the generalization of RLLM to Llama-based models.

Method	RM	RM Type	RM Size	Datasets
				AlpacaEval 2.0	ArenaHard 2.0	ArenaHard 2.0
				(LCWR / WR)	(Hard Prompts)	(Creative Writing)
Policy Trained on non-verifiable WildChat samples in thinking mode
Qwen3-8B	–	–	–	63.1 / 65.1	22.4 (-1.5 / +1.7)	33.3 (-2.5 / +2.7)
RLHF	Nexusflow/Athene-RM-8B	Scalar	8B	70.9 / 71.2	25.4 (-1.6 / +1.6)	57.9 (-2.8 / +2.0)
RLHF	Skywork-Reward-V2-Llama-3.1-8B	Scalar	8B	68.6 / 72.3	26.4 (-1.6 / +1.7)	50.9 (-2.9 / +2.9)

Table 20: Comparison of Win Rate (WR) and Length Controlled Win Rate (LCWR) of RLLM and RLHF on non-verifiable instruction-following tasks when training a Qwen3-8B policy. For AlpacaEval 2.0, we use GPT-4o as the evaluator and for ArenaHard 2.0, we use GPT-4.1 as the evaluator. RLLM matches or outperforms RLHF, obtaining best win rates on hard prompts of ArenaHard 2.0.

Figure 35: Example of a synthetically-generated correct training example for LM-as-RM training. Given a (question, reference answer), we sample a response from the policy and check whether the answer matches the reference answer using GPT-OSS-120B.

Figure 36: Example of a synthetically-generated incorrect training example for LM-as-RM training. Given a (question, reference answer), we sample a response sampled from the policy and check whether the answer matches the reference answer using GPT-OSS-120B.

Figure 37: Example of a hard-to-verify math problem for policy training where the reference answer is a mathematical expression.

Figure 38: Example of an easy-to-verify math problem for policy training where the reference answer is an integer.

Figure 39: Qualitative comparison of a weaker LM-as-RM J1-Qwen3-1.7B’s thinking trace with a stronger J1-Qwen3-32B’s thinking trace. The stronger LM-as-RM correctly identifies the response as incorrect and also specifies the correct answer (which matches the reference answer).

Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

Abstract

Figures & tables

Explore similar work

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches