LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator

Yiming Feng; Naihao Deng; Yulong Chen; Rada Mihalcea

cs.CLSep 19, 2026

LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator

Authors: Yiming Feng, Naihao Deng, Yulong Chen, Rada Mihalcea

Organizations: University of Michigan · University of Aberdeen · University of Cambridge

Abstract

Step-level reasoning evaluators are commonly based on autoregressive language models, whose causal attention restricts each step representation to the problem, previous steps, and the current step. Yet, when the complete solution is available, the validity of an earlier step may become clearer only through its downstream consequences. We validate this hypothesis through a controlled 54-run comparison of causal and bidirectional LLaDA evaluators at 1B--3B scale, changing only the self-attention mask, and find bidirectional attention yields consistent improvements. Building on this finding, we introduce \prm{}, an 8B bidirectional evaluator that reaches 88.8 step-level F1 on MR-MATH-invalid and 83.8 on the out-of-distribution MR-GSM8K original-question subset, outperforming ReasonEval-Llemma-34B by 11.3 and 10.3 F1 points, respectively. \prm{} also remains effective when evaluating incomplete reasoning traces in online settings, outperforming the strongest baselines on both benchmarks by a large margin. We further show that \prm{} provides an effective training-data selection signal, improving Mistral-7B performance on MATH-500.

Figures & tables

Figure 1: Causal and bidirectional evaluators have access to different contextual information on the same step.

	1B			2B			3B
Architecture	Acc	F1	AUC	Acc	F1	AUC	Acc	F1	AUC
LLaDA-bidir	45.4	40.1	59.6	46.9	41.1	60.8	46.7	41.1	60.6
LLaDA-causal	45.1	40.4	59.7	45.2	39.9	59.6	44.1	38.7	58.5
$\boldsymbol{\Delta}$ LLaDA-bidir $-$ LLaDA-causal (pp)	$+0.3$	$-0.3$	$-0.1$	$\boldsymbol{+1.7}$	$\boldsymbol{+1.2}$	$\boldsymbol{+1.2}$	$\boldsymbol{+2.6}$	$\boldsymbol{+2.4}$	$\boldsymbol{+2.1}$

Table 1: Step-level test metrics on the 3,000-trace subset, averaged over 3 seeds. Metric values are reported as percentages, and deltas are reported in percentage points. The full 54-run grid is reported in Appendix B .

Model	Params	Sol-F1	Sol-AUC	Step-F1	Step-AUC
MR-MATH-invalid (in-domain)
Math-Shepherd	7B	70.1	77.3	60.0	77.2
ReasonEval-Mistral	7B	78.0	85.1	68.6	85.7
ReasonEval-WizardMath-V1.1	7B	78.6	87.5	73.9	89.5
WizardMath-V1.1 (control, eff_bs=8)	7B	72.5	79.6	64.7	82.1
ReasonEval-Llemma	34B	79.6	90.8	77.5	92.8

Table 2: Performance on the three benchmarks.

Benchmark	Configuration	Sol-F1	Sol-AUC	Step-F1	Step-AUC
MR-MATH-invalid	ReasonEval-WizardMath-V1.1 (7B)	$78.6$	$87.5$	$73.9$	$89.5$
	ReasonEval-Llemma (34B)	$79.6$	$90.8$	$77.5$	$92.8$
	LLaDA-PRM (online)	$\mathbf{89.8}$	$\mathbf{96.4}$	$\mathbf{84.9}$	$\mathbf{94.7}$
	LLaDA-PRM (offline)	$93.0$	$97.2$	$88.8$	$96.4$
	$\Delta$ online $-$ offline (pp)	$-3.2$	$-0.8$	$-3.9$	$-1.7$
MR-GSM8K (OOD)	ReasonEval-WizardMath-V1.1 (7B)	$74.1$	$90.7$	$72.8$	$91.4$

Table 3: Offline and online evaluation of LLaDA-PRM.

Scorer	Filter	N	Acc.	Tok.
LLaDA-PRM	Val + Red	5,222	20.9	280
	Val-only	5,222	19.2	276
	Red-only	5,222	22.5	249
ReasonEval WizardMath-V1.1	Val + Red	4,437	19.2	276
	Val-only	4,437	19.3	267
	Red-only	4,437	19.3	250

Table 4: Data-filtering ablation on Mistral-7B-v0.1, averaged over seeds.

N

is the number of training samples, Acc. is MATH-500 greedy Pass@1 reported as a percentage, and Tok. is mean generation length.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Config	$\boldsymbol{d_{\text{model}}}$	$\boldsymbol{n_{\text{layers}}}$	$\boldsymbol{n_{\text{heads}}}$	MLP
LLaDA-8B-Base (ref.)	4096	32	32	12288
LLaDA-1B	2048	16	16	5504
LLaDA-2B	2560	24	20	6912
LLaDA-3B	2816	26	22	7168

Appendix

Table 5: Scratch-config dimensions for the 1B–3B grid, obtained by downscaling LLaDA-8B-Base along the four reported axes. All other architectural components are held fixed across sizes.

Figure 2: LLaDA-PRM 8B fine-tuning over the single epoch on PRM800K Phase 2 (

\sim

12k optimizer steps). Left: training cross-entropy falls from

3.65

\sim

0.5 and then plateaus. Center: gradient norm (log scale) stays bounded after the warmup spike. Right: the learning rate follows the

64

-step warmup and cosine decay. All curves are taken from the run’s log_history.json .

Size	Data	Seed	Mask	Acc	macro-F1	macro-AUC	Late-AUC
1B	1k	13	Bidir	48.1	41.4	61.8	31.8
1B	1k	13	Causal	42.9	39.4	59.2	18.9
1B	1k	42	Bidir	45.7	39.9	60.9	33.3
1B	1k	42	Causal	46.6	41.3	61.3	24.4
1B	1k	7	Bidir	45.9	41.1	61.8	35.0
1B	1k	7	Causal	45.4	40.9	60.9	26.0

Appendix

Table 6: Per-run step-level test results for all

54

runs in the controlled grid. Each row represents one trained model evaluated on the same held-out 608-trace test set. All metric values are percentages and are rounded to one decimal place for display; paired comparisons use the unrounded values. Bidir and Causal rows with the same model size, training-set size, and seed form one matched pair.

Figure 3: Step-level macro-AUC and macro-F1 versus model size (1B/2B/3B) and training-set size (1k/2k/3k), averaged over the other factors and the three seeds. LLaDA-bidir (blue) is at or above LLaDA-causal (orange) in every panel.

Metric	Wins	Mean $\Delta$ (pp)
Accuracy	$22/27$	$+2.03$
macro-F1	$22/27$	$+0.90$
macro-AUC	$22/27$	$+1.05$

Appendix

Table 7: Descriptive LLaDA-bidir

-

LLaDA-causal contrasts over the

27

seed-matched run pairs. Mean differences are computed from unrounded metrics.

Figure 4: Seed-matched macro-AUC comparison. Each point represents one pair of runs sharing the same model size, training subset, and seed. Points above the dotted diagonal favor the bidirectional mask.

Figure 5: Descriptive macro-AUC difference (LLaDA-bidir

-

LLaDA-causal) by step location, pooled over the

27

seed-matched run pairs. The early bucket contains no negative labels and therefore averages two present classes, whereas the middle and late buckets average three; values across buckets are not directly comparable.

Figure 6: Controlled-grid diagnostics. Left: per-class AUC and F1 for the negative, neutral, and positive step classes, averaged over the grid; both arms remain weak on the rare neutral class. Right: per-cell macro-AUC, mean

\pm

standard deviation over the three seeds, for each (model size, training-set size) design cell.

Panel of Figure 7 — Figure 7: PRM800K neutral-class diagnostics. Top: natural train/test label shares (left) and gold versus predicted test shares (right). Bottom left: the 8B model’s confusion matrix; only $49$ of $18{,}559$ steps are predicted neutral, and most gold-neutral steps are routed to positive. Bottom right: one-vs-rest ROC curves; neutral-class AUC remains $0.745$ despite the collapsed argmax decisions.

Model	Params	Sol-F1	Sol-AUC	Step-F1	Step-AUC
ReasonEval-Mistral	7B	61.2	75.6	61.9	84.9
ReasonEval-WizardMath-V1.1	7B	74.6	87.5	71.1	89.8
WizardMath-V1.1 (control, eff_bs=8)	7B	69.5	79.3	66.2	82.2
ReasonEval-Llemma	34B	77.7	85.5	70.8	85.3
Batch-matched LLaDA-bidir (eff_bs=64)	8B	77.1	86.5	70.3	85.9
LLaDA-PRM (eff_bs=8, offline)	8B	87.9	94.0	78.7	91.6

Appendix

Table 8: Full MR-GSM8K results on all

2{,}999

examples.

Reasoning Evaluation

Reasoning evaluation measures whether AI systems solve multi-step problems reliably, use evidence and valid intermediate steps, and generalize beyond benchmarks.

See all 51 papers

The Bidirectional Process Reward Model
Aug 3, 2025Lingyin Zhang, Jun Gao, Xiaoxue Ren +1Process Reward Models Reasoning Verification
School of Computer Science and Technology, Soochow University · Biomedical Basic Research Center of Jiangsu, Soochow University, Suzhou, Jiangsu 215123, China · School of Software Technology, Zhejiang University +1
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3
Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring
Dec 16, 2025Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1LLM Inference Efficiency Efficient Language Model Reasoning
1IBM Research Europe, Dublin, Ireland · 2Trinity College Dublin, Ireland · 3ADAPT Research Centre, Dublin, Ireland

Aug 3, 2025J/K move · Enter open · S save

The Bidirectional Process Reward Model

Lingyin Zhang, Jun Gao, Xiaoxue Ren, Ziqiang Cao

School of Computer Science and Technology, Soochow University · Biomedical Basic Research Center of Jiangsu, Soochow University, Suzhou, Jiangsu 215123, China · School of Software Technology, Zhejiang University+1

Process Reward Models Reasoning Verification

Open paper AskPDF arXiv

Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promising approach to enhance the reasoning quality of Large Language Models (LLMs). However, most existing PRMs rely on a unidirectional left-to-right (L2R) evaluation scheme, which restricts their utilization of global context. In light of this challenge, we propose a novel bidirectional evaluation paradigm, named Bidirectional Process Reward Model (BiPRM). BiPRM incorporates a parallel right-to-left (R2L) evaluation stream, implemented via prompt reversal, alongside the conventional L2R flow. Then a gating mechanism is introduced to adaptively fuse the reward scores from both streams to yield a holistic quality assessment. Remarkably, compared to the original PRM, BiPRM introduces only a 0.3% parameter increase for the gating module, and the parallel execution of two streams incurs merely 5% inference time latency. Our extensive empirical evaluations spanning diverse benchmarks, LLM backbones, PRM objectives and sampling policies demonstrate that BiPRM consistently surpasses unidirectional baselines, achieving an average relative gain of 10.6% over 54 solution-level configurations and 37.7% in 12 step-level error detection scenarios. Generally, our results highlight the effectiveness, robustness and general applicability of BiPRM, offering a promising new direction for process-based reward modeling.

Abstract

Reasoning Evaluation

Explore similar work

The Bidirectional Process Reward Model

Scaling Evaluation-time Compute with Reasoning Models as Evaluators

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring