Diffusion large language models (dLLMs) have advanced rapidly, yet strengthening their reasoning through reinforcement learning (RL) remains an open challenge. Any-order generation can suffer from entropy degradation, which narrows the reasoning paths available for exploration. Recent work mitigates this issue by applying standard GRPO with left-to-right generation. However, independently generating every rollout from the input prompt can leave alternative continuations from intermediate states underexplored, while trajectory-level advantages assign uniform credit to all tokens without distinguishing the contributions of individual reasoning segments. We propose ForkGRPO, an RL framework that couples efficient exploration with fine-grained segment-level credit assignment. It samples multiple continuations from high-entropy intermediate states along a reference trajectory, prioritizing textual boundaries and reusing shared prefixes. The continuations provide complementary learning signals: within-state group-relative advantages guide which continuations to reinforce, while Monte Carlo state-value differences indicate which reasoning segments improve expected outcomes. Among the evaluated architecture-preserving RL methods, ForkGRPO achieves the best performance on three of four benchmarks, while improving over the directly comparable JustGRPO by up to 1.2 percentage points and using up to 27.8% fewer generated tokens. ForkGRPO-Fast further achieves a 2.49× training wall-clock speedup with negligible accuracy loss.
Figures & tables
Figure 1: Overview of ForkGRPO . ForkGRPO samples multiple continuations from high-entropy intermediate states along a reference trajectory and uses their terminal rewards to estimate intermediate state values. These samples provide two complementary credit signals for joint policy optimization: within-state relative credit among sibling continuations and segment-level credit derived from the change in value between successive fork states.
Method
GSM8K
MATH-500
HumanEval
MBPP
LLaDA-1.5 ‡ ( Zhu et al., 2026 )
83.3
–
39.6
39.9
LLaDOU † ( Huang et al., 2025 )
88.1
41.1
59.1
51.6
d1 ( Zhao et al., 2025 )
81.1
38.6
–
–
DMPO ( Zhu et al., 2025 )
82.4
38.2
–
–
wd1 ( Tang et al., 2026 )
80.8
34.4
–
–
d-TreeRPO ( Pan et al., 2026 )
81.2
37.7
–
–
Table 1: Main results at generation length 256. We report accuracy (%) on GSM8K and MATH-500 and pass@1 (%) on HumanEval and MBPP. † indicates architecture modifications; ‡ indicates training with substantially more private data.
Method
Fork
Credit
Acc. (%)
JustGRPO
No
group
88.81±0.53
Fork + Group
Yes
group
89.69±0.21
Fork + Segment
Yes
segment
89.45±0.29
ForkGRPO
Yes
segment + group
90.31±0.06
Table 2: Component ablation on GSM8K. Forked variants share the same sampling configuration and differ only in their training objective.
Selection
Boundary
Entropy
Acc. (%)
Fixed
No
No
89.10±0.33
Random Boundary
Yes
No
89.25±0.21
Max Entropy
No
Yes
89.67±0.16
Entropy Boundary
Yes
Yes
90.31±0.06
Table 3: Fork-state selection on GSM8K. All variants differ only in how branching positions are selected within each window.
Figure 2: Nonzero rates of group advantages and window segment value differences, averaged over the first 10 training steps on GSM8K with LLaDA-8B-Instruct. Both signals become substantially sparser at later segments. Shading indicates the first four 32-token windows covered by the default ForkGRPO configuration ( K=3 ).
K
M
Groups
Acc. (%)
0
32
1
89.14±0.38
1
16
2
89.36±0.19
3
8
4
90.31±0.06
7
4
8
89.93±0.11
Table 4: Sampling allocation on GSM8K.
Metric
JustGRPO
ForkGRPO
Training updates
200
100
Scored responses / prompt
16
32
New tokens / prompt
4096
5912–6656
Total token budget
1.0000
[0.7217,0.8125]
GSM8K accuracy (%)
89.6
90.4
Table 5: Sampling efficiency on GSM8K. Both methods use LoRA. Total token budgets are normalized to JustGRPO-LoRA; ranges denote analytical bounds.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Diagnosis of segment-level credit over the first 10 training steps on GSM8K. Left: P(ΔVj>0∣R(q,yref)=−1) , measuring segments that improve continuation value despite an unsuccessful final response. Middle: P(ΔVj<0∣R(q,yref)=+1) , measuring segments that reduce continuation value despite a successful final response. Right: P(∃j:ΔVj=0∣Areftraj≈0) , measuring segment-level signals missed by trajectory-level credit.
M
Value MAE ↓
ΔV Agree. ↑
2
0.29
74.4
4
0.22
84.1
8
0.07
92.3
16
0.06
94.8
Appendix
Table 6: Monte Carlo value-estimation reliability.
Variant
Acc.
Rollout
Replay
Overall
ForkGRPO
90.4
1.00×
1.00×
1.00×
+ Adaptive rollout
90.3
1.76×
1.00×
1.37×
+ Branch replay (1+1)
90.3
1.00×
2.26×
1.24×
+ State subsampling ( ρ=0.25 )
90.2
1.00×
2.43×
1.34×
ForkGRPO-Fast
90.3
1.76×
5.32×
2.49×
Appendix
Table 7: Ablation of ForkGRPO-Fast on GSM8K. Wall-clock measurements are averaged over five training steps. Rollout, replay, and overall speedups are measured directly relative to full ForkGRPO ; overall speedup is computed from total step time.
Figure 4: Representative ForkGRPO generation on GSM8K. The model correctly propagates the multiplicative relations among the three quantities, obtaining 80 sheep for Charleston and 160 for Toulouse, before producing the correct total of 260 .
Figure 5: Representative ForkGRPO generation on MBPP. The model constructs a valid implementation by merging the two input lists, sorting the combined sequence, and averaging its two central elements. The resulting function satisfies all provided unit tests.
Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.
Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings.
Yue Yu, Bowen Zuo, David Crandall +2
Indiana University · University of California, Riverside
Reinforcement learning (RL) holds immense promise for enhancing the reasoning capabilities of diffusion large language models (dLLMs). However, progress is fundamentally constrained by a dual misalignment between authentic generation trajectory and the gradient update process: (i) Process-reward misalignment. Sparse, terminal rewards are indiscriminately assigned to all intermediate steps of the generation process, failing to provide discriminative credit assignment. (ii) State-trajectory misalignment. Policy updates are often diverted toward artificial, out-of-trajectory states, squandering gradients on less informative samples. To address these limitations, we introduce Process Aligned Policy Optimization (PAPO), a novel framework that holistically aligns the RL update with the dLLM's generative trajectory via Step-Aware Process Rewards (SPR) that transform sparse terminal rewards into dense, step-wise credit, and Entropy-Guided Historical Re-enactment (EHR) that replays authentic trajectories at high-uncertainty steps. Extensive experiments on four benchmarks demonstrate that PAPO significantly outperforms baselines, achieving gains of up to 4.5% on GSM8K, 4.8% on MATH500, 42.2% on Countdown and 16.1% on Sudoku.
Yawen Shao, Jie Xiao, Kai Zhu +6
University of Science and Technology of China · Tongyi Lab · Northeastern University