ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models

Zhaoyan Jia; Chengwei Zhou; Gourav Datta; Massoud Pedram

cs.CLOct 14, 2026

ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models

Authors: Zhaoyan Jia, Chengwei Zhou, Gourav Datta, Massoud Pedram

Organizations: University of Southern California, Los Angeles, USA · Case Western Reserve University, Cleveland, USA

Abstract

Diffusion large language models (dLLMs) have advanced rapidly, yet strengthening their reasoning through reinforcement learning (RL) remains an open challenge. Any-order generation can suffer from entropy degradation, which narrows the reasoning paths available for exploration. Recent work mitigates this issue by applying standard GRPO with left-to-right generation. However, independently generating every rollout from the input prompt can leave alternative continuations from intermediate states underexplored, while trajectory-level advantages assign uniform credit to all tokens without distinguishing the contributions of individual reasoning segments. We propose ForkGRPO, an RL framework that couples efficient exploration with fine-grained segment-level credit assignment. It samples multiple continuations from high-entropy intermediate states along a reference trajectory, prioritizing textual boundaries and reusing shared prefixes. The continuations provide complementary learning signals: within-state group-relative advantages guide which continuations to reinforce, while Monte Carlo state-value differences indicate which reasoning segments improve expected outcomes. Among the evaluated architecture-preserving RL methods, ForkGRPO achieves the best performance on three of four benchmarks, while improving over the directly comparable JustGRPO by up to 1.2 percentage points and using up to 27.8% fewer generated tokens. ForkGRPO-Fast further achieves a $2.49\times$ training wall-clock speedup with negligible accuracy loss.

Figures & tables

Figure 1: Overview of ForkGRPO . ForkGRPO samples multiple continuations from high-entropy intermediate states along a reference trajectory and uses their terminal rewards to estimate intermediate state values. These samples provide two complementary credit signals for joint policy optimization: within-state relative credit among sibling continuations and segment-level credit derived from the change in value between successive fork states.

Method	GSM8K	MATH-500	HumanEval	MBPP
LLaDA-1.5 ‡ ( Zhu et al., 2026 )	83.3	–	39.6	39.9
LLaDOU † ( Huang et al., 2025 )	88.1	41.1	59.1	51.6
d1 ( Zhao et al., 2025 )	81.1	38.6	–	–
DMPO ( Zhu et al., 2025 )	82.4	38.2	–	–
wd1 ( Tang et al., 2026 )	80.8	34.4	–	–
d-TreeRPO ( Pan et al., 2026 )	81.2	37.7	–	–

Table 1: Main results at generation length 256. We report accuracy (%) on GSM8K and MATH-500 and pass@1 (%) on HumanEval and MBPP.

\dagger

indicates architecture modifications;

\ddagger

indicates training with substantially more private data.

Method	Fork	Credit	Acc. (%)
JustGRPO	No	group	$88.81\pm 0.53$
Fork + Group	Yes	group	$89.69\pm 0.21$
Fork + Segment	Yes	segment	$89.45\pm 0.29$
ForkGRPO	Yes	segment + group	$\mathbf{90.31}\pm 0.06$

Table 2: Component ablation on GSM8K. Forked variants share the same sampling configuration and differ only in their training objective.

Selection	Boundary	Entropy	Acc. (%)
Fixed	No	No	$89.10\pm 0.33$
Random Boundary	Yes	No	$89.25\pm 0.21$
Max Entropy	No	Yes	$89.67\pm 0.16$
Entropy Boundary	Yes	Yes	$\mathbf{90.31}\pm 0.06$

Table 3: Fork-state selection on GSM8K. All variants differ only in how branching positions are selected within each window.

Figure 2: Nonzero rates of group advantages and window segment value differences, averaged over the first 10 training steps on GSM8K with LLaDA-8B-Instruct. Both signals become substantially sparser at later segments. Shading indicates the first four 32-token windows covered by the default ForkGRPO configuration (

K=3

$K$	$M$	Groups	Acc. (%)
0	32	1	$89.14\pm{\mathrm{0.38}}$
1	16	2	$89.36\pm{\mathrm{0.19}}$
3	8	4	$\mathbf{90.31}\pm 0.06$
7	4	8	$89.93\pm{\mathrm{0.11}}$

Table 4: Sampling allocation on GSM8K.

Metric	JustGRPO	ForkGRPO
Training updates	200	100
Scored responses / prompt	16	32
New tokens / prompt	4096	5912–6656
Total token budget	1.0000	$[0.7217,\,0.8125]$
GSM8K accuracy (%)	89.6	90.4

Table 5: Sampling efficiency on GSM8K. Both methods use LoRA. Total token budgets are normalized to JustGRPO-LoRA; ranges denote analytical bounds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Diagnosis of segment-level credit over the first 10 training steps on GSM8K. Left: $P(\Delta V_{j}>0\mid R(q,y^{\mathrm{ref}})=-1)$ , measuring segments that improve continuation value despite an unsuccessful final response. Middle: $P(\Delta V_{j}<0\mid R(q,y^{\mathrm{ref}})=+1)$ , measuring segments that reduce continuation value despite a successful final response. Right: $P(\exists j:\Delta V_{j}\neq 0\mid A_{\mathrm{ref}}^{\mathrm{traj}}\approx 0)$ , measuring segment-level signals missed by trajectory-level credit.

$M$	Value MAE $\downarrow$	$\Delta V$ Agree. $\uparrow$
2	0.29	74.4
4	0.22	84.1
8	0.07	92.3
16	0.06	94.8

Appendix

Table 6: Monte Carlo value-estimation reliability.

Variant	Acc.	Rollout	Replay	Overall
ForkGRPO	90.4	$1.00\times$	$1.00\times$	$1.00\times$
+ Adaptive rollout	90.3	$1.76\times$	$1.00\times$	$1.37\times$
+ Branch replay (1+1)	90.3	$1.00\times$	$2.26\times$	$1.24\times$
+ State subsampling ( $\rho{=}0.25$ )	90.2	$1.00\times$	$2.43\times$	$1.34\times$
ForkGRPO-Fast	90.3	$\mathbf{1.76}\times$	$\mathbf{5.32}\times$	$\mathbf{2.49}\times$

Appendix

Table 7: Ablation of ForkGRPO-Fast on GSM8K. Wall-clock measurements are averaged over five training steps. Rollout, replay, and overall speedups are measured directly relative to full ForkGRPO ; overall speedup is computed from total step time.

Figure 4: Representative ForkGRPO generation on GSM8K. The model correctly propagates the multiplicative relations among the three quantities, obtaining 80 sheep for Charleston and 160 for Toulouse, before producing the correct total of

260

Figure 5: Representative ForkGRPO generation on MBPP. The model constructs a valid implementation by merging the two input lists, sorting the combined sequence, and averaging its two central elements. The resulting function satisfies all provided unit tests.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models
Sep 30, 2026Yue Yu, Bowen Zuo, David Crandall +2RL for Language Model Reasoning Group Relative Policy Optimization
Indiana University · University of California, Riverside
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
Jun 7, 2026Yawen Shao, Jie Xiao, Kai Zhu +6RL for Language Model Reasoning Credit Assignment in RL
University of Science and Technology of China · Tongyi Lab · Northeastern University

May 8, 2026J/K move · Enter open · S save

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade, Lokesh Boominathan+3

The Ohio State University · Apple

Group Relative Policy Optimization Credit Assignment in RL Step-Level Credit Assignment in RL Step-Level Credit Assignment RL for Diffusion Models Diffusion Language Models+1

Open paper AskPDF arXiv

Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.

Abstract

RL for Language Model Reasoning

Explore similar work

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models

Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models