Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning

Yan Zhan; Shaobo Liu; Zhijun Gao

cs.AISep 28, 2026

Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning

Authors: Yan Zhan, Shaobo Liu, Zhijun Gao

Organizations: Peking University · Shenzhen University

Abstract

Discrete diffusion language models (dLLMs) expose a denoised solution at every step, which makes process reward model (PRM) guidance look like a way to spend compute at test time. We show that once denoising, PRM scoring, and outcome reward model (ORM) scoring are charged in the same budget of forward passes, its deterministic form loses to a much simpler baseline. Our PRMs score intermediate denoising states and are trained on the correctness of the final answer. On Dream-v0-Instruct-7B with 8 candidates per GSM8K problem, keeping the candidate with the highest PRM score at every scoring step reaches 65.18%, while independent sampling plus an ORM reranker trained for the task reaches 75.13%. The gap grows to 12.69 percentage points (pp) with 32 candidates, and is 9.85 pp on MATH and 12.16 pp on MBPP. We trace it to two separable failures. First, guidance prunes on a weak signal: on GSM8K, PRM ROC-AUC falls from 0.77 to 0.54 as the mask ratio rises, a decay that persists when states are relabeled with fresh rollouts, and pruning lowers the best accuracy reachable from the candidate pool from 81.05% for independent samples to 67.30%. Second, on GSM8K and MATH, the PRM is a poor final judge: a sequential Monte Carlo sampler at the same budget restores that ceiling to 77.89%, yet selecting with the PRM gives 65.48%, on par with deterministic guidance, while a PRM retrained on final states matches the ORM on identical candidates. MBPP separates the two: there the PRM reaches 65.47% when reranking finished programs, on par with the ORM, but 50.88% when it guides denoising. The results point to two targets for dLLM guidance: keep correct partial solutions alive through early denoising, and leave the final choice to a verifier trained on final states. We release the corpus of denoising states with outcome labels and evaluation toolkit for reproducible comparisons at matched compute.

Figures & tables

Figure 1 : Framework overview. (A) Signal decay. Bidirectional PRM ROC-AUC rises from $0.54$ in the most-masked bucket to $0.77$ in the least-masked bucket $[0.0,0.1)$ and $0.78$ on fully decoded states, so guidance decisions made at high mask ratios rely on the weakest signal. (B) Diversity collapse. Deterministic top- $1$ PRM pruning collapses the candidate pool, while ORM Rerank spends the same forward-pass budget on independent trajectories. (C) Readout mismatch. For causal PRM variants, switching from mean pooling to last-token pooling raises ROC-AUC on final states from $0.61$ to $0.73$ but does not close the downstream gap to ORM Rerank.

			GSM8K Accuracy (%)
Method	N/K	Forward passes	N=8 / K=8	N=32 / K=32
Matched-compute comparison
Vanilla	1	128	43.14
Majority@ $N$	$N$	$N{\times}128$	60.05	67.63
PRM Guided ( $b{=}64$ )	$K$	$130K$	65.18	70.02
ORM Rerank@ $N$	$N$	$N{\times}128+N$	75.13	82.71

Table 1: Matched-compute GSM8K accuracy with Dream-v0-Instruct-7B. ORM Rerank leads at both budgets. PRM Guided entries average three independent runs at

K{=}8

and two at

K{=}32

(App. D.1 ). The diagnostic rows isolate causal PRM readout. Bold: best comparable value in each accuracy column.

Task-specific verifiers	ORM Rerank	PRM Rerank	PRM Guided
MATH, $N{=}K{=}8$	30.65	20.80	20.80
MBPP	63.04	65.47	50.88
GSM8K, same candidate pool	ORM Rerank	Final-state PRM	Cross-mask PRM
$N{=}8$	75.13	75.40	42.84
$N{=}32$	82.71	82.79	65.35

Table 2: Controls beyond the GSM8K headline. Top: task-specific verifiers on held-out MATH and MBPP problems. Bottom: terminal scoring on the same GSM8K candidate pool. Splits and full results are in App. B . Bold: best comparable value in each result column.

	Candidate pool		Accuracy of the final choice (%)
Search	Oracle@8 (%)	Unique answers	Cross-mask PRM	ORM	Final-state PRM
Independent samples	81.05	4.31	42.84	75.13	75.40
Top- $1$ guidance	67.30	1.75	65.18	–	–
SMC	77.89	3.95	65.48	–	–
Removal risk (%)	Initial state	Middle state	Final state
Top- $1$ cut	46.06	37.02	19.92

Table 3: Pool quality and final selection on GSM8K at the headline budget (

K{=}N{=}8

). Top: the first two columns measure the candidate pool and the last three the accuracy of the final choice from it. Top-

1

guidance damages the pool; SMC repairs most of it, yet the cross-mask PRM still selects at the top-

1

level, while final-state scorers select well from independent samples. Bottom: how often a PRM cut removes every lineage that later reaches a correct answer, by the stored state at which the cut is applied. Dashes mark configurations that were not run. Full results are in App. B . Bold: best comparable value in each accuracy column.

Candidate	Final computation	Causal PRM score	Correct
$t{=}13$	$2\times\$ 1.50=$30$	$+1.001$	$\times$
$t{=}25$	$(12-2)\times\$ 1.50=$15$	$-0.827$	$\times$
$t{=}3$	$2\times\$ 1.50\times 7=$21$	$-0.908$	$\checkmark$

Table 4: One GSM8K test problem (id

526

), the one sketched in Figure 1 (A): Ada uses

12

kWh a day, adds a device that uses

2

kWh a day, and pays

\

1.50

per kWh; the question asks for the weekly bill difference, which is

$21$ . Three of the independent candidates are shown with their causal PRM scores.

Explore similar work

Jun 1, 2026cs.CL

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels. We propose Chunk-Level Guided Generation, a training-free alternative that uses an off-the-shelf large language model as a process scorer. At each step, a small model samples k fixed-length candidate chunks, while the larger model scores the candidates using likelihoods without generating any text. The selected chunk is committed before the next step, steering generation before errors can propagate. We instantiate this framework with two selection rules: Likelihood-Guided Selection (LGS), which selects the chunk with the highest length-normalized large-model log-probability, and Contrastive-Guided Selection (CGS), which subtracts the small model's log-probability to favor chunks where the large model's preference diverges from the small model's. We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias that persists even after length normalization, and that fixed-length chunks avoid this confound. On GSM8K, MATH, Minerva Math, AMC23, and AIME24 with Qwen2.5-1.5B guided by Qwen2.5-32B and Llama-3.2-1B guided by Llama-3.1-70B, CGS outperforms majority voting by up to 28 pp and, under matched guidance budgets, matches or outperforms Qwen2.5-Math-PRM-72B guided search on most benchmarks without reward-model training. With Qwen2.5-7B guided by Qwen2.5-72B, CGS reaches 81.8% on MATH and 63.6% on Minerva Math at k=16, surpassing majority voting by 4--6 pp. Finally, Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.

Atoosa Chegini, Soheil Feizi

Department of Computer Science, University of Maryland

LLM Reasoning Strategies Mathematical Reasoning Process Reward Model Inference-Time Scoring Training-Free

Ask PDF

Aug 6, 2026cs.CL

Answer First, Reason Later: Commitment Order in Diffusion LLMs

Masked diffusion language models (dLLMs) can commit tokens in any order -- a freedom marketed as their core advantage over autoregressive decoding. We show that on reasoning tasks this freedom is instead the axis of failure. Logging every commitment during decoding of LLaDA-8B on GSM8K, we find that unconstrained (pure) decoding commits the final answer at 15-24% of the trajectory while half the reasoning region is still masked, and collapses to answer-only outputs on up to 90% of problems as the canvas grows. The cause is not the model's termination beliefs -- EOS "pressure" is nearly identical across decoders -- but reachability: whether the sampler may act on those beliefs at distant positions. A 2x2 prompt-decoder design shows that chain-of-thought helps only under ordered commitment (interaction +34.8 percentage points, 95% CI [26.8, 42.8]; without reasoning text the decoders are indistinguishable), an interaction we decompose into a collapse channel and an order channel and replicate on Dream-7B and MATH-500. A single-knob intervention -- frontier-gated commitment -- causally recovers the full gap (0.528 to 0.852) while preserving up to 4x parallel decoding, along a measured frontier whose optimal window flips from w=1 at full refinement to unconstrained at 8 tokens/step. Our results reframe existing window-style samplers, previously motivated by efficiency, as the minimal fix for a reasoning pathology they were never designed to address.

Jewon Yeom, Jaewon Sok, Seonghyeon Park +3

Graduate School of Data Science, Seoul National University · Department of Rural Systems Engineering, Seoul National University · Department of Aerospace Engineering, Seoul National University

Diffusion Language Models LLM Reasoning Strategies Masked Diffusion Language Models Reasoning Skills Commitment Autoregressive Decoding

Ask PDF

Apr 27, 2026cs.LG

DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models

Diffusion language models generate without a fixed left-to-right order, leaving token ordering as a central algorithmic choice. Existing systems mainly use random masking or confidence-driven ordering, which respectively suffer from train--test mismatch and myopic exploration. We introduce DPRM (Doob -transform Process Reward Model), a plug-in token-ordering module that keeps the host architecture, denoising objective and supervision unchanged, and modifies only the ordering policy. DPRM starts from confidence-driven ordering and gradually shifts to process-reward-guided ordering through online estimates. We characterize the exact DPRM policy as a reward-tilted Gibbs reveal law, prove convergence of its stagewise Soft-BoN approximation, show that the online bucketized controller tracks the exact DPRM score at empirical-Bernstein rates, and establish a sample-complexity advantage under tractable optimization assumptions. Across nine hosts covering language reasoning, test-time scaling, protein, single-cell, molecular, DNA, text-to-image generation, and VQA, DPRM order variants improve several language, DNA, and multimodal settings while also identifying boundary cases where confidence-only ordering or task-specific utilities are preferable. Code is available at: https://github.com/DakeBU/DPRM-DLLM

Dake Bu, Wei Huang, Andi Han +4

City University of Hong Kong · CFAR and IHPC, A*STAR · RIKEN AIP +4

Diffusion Language Models Order Matters Plug-And-Play

Ask PDF