FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

Darshan Thaker; Lachlan Ewen MacDonald; René Vidal

cs.CLSep 28, 2026

FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

Authors: Darshan Thaker, Lachlan Ewen MacDonald, René Vidal

Organizations: University of Pennsylvania

Abstract

Gradient-based reward guidance provides a flexible way to use downstream reward models to control masked diffusion language models at inference time. However, its computational cost remains high as each decoding iteration incurs expensive diffusion model forward passes and reward model backpropagation steps. To address this, we introduce FastGuide, an adaptive hybrid of parallel and autoregressive decoding to accelerate reward guidance for diffusion language models. In analogy to parallel decoding, FastGuide amortizes the cost of reward model backpropagation by computing guidance once per decoding step and reusing it to generate multiple tokens. Within each decoding step, FastGuide makes diffusion forward passes autoregressive by unmasking tokens one at a time while efficiently recomputing token distributions after each unmasking by utilizing KV caching techniques and sparse recomputation of attention. Lastly, to adapt hybrid decoding to the model's confidence, FastGuide defers any token that the model is unconfident about under its recomputed distribution. Experiments on three reward benchmarks demonstrate that FastGuide is up to $4.4\times$ faster than sequential reward-guided decoding while retaining similar generation quality.

Figures & tables

Figure 1: Left: Reward guidance ( Tejaswi et al., 2026 ) can increase incompatibilities across token values proposed during parallel decoding. Above, CPA should expand to Certified Public Accountant . While unguided generation with parallel decoding method Fast-dLLM ( Wu et al., 2025 ) produces the correct token prefix for CPA , Fast-dLLM with guidance fails. Right: Our method is much faster than sequential guided generation while retaining similar generation quality.

Figure 2: Overview of FastGuide. Left: Illustration of one hybrid decoding step of FastGuide with $k=4$ . FastGuide utilizes guidance caching and autoregressive sparse dLLM recomputation. After recomputation, tokens are deferred if token confidence drops below a threshold. Right: Sparse dLLM recomputation of token 7 after committing token 6, with window radius $w=1$ . Attention and FFN outputs are recomputed only at the selected positions, whose queries attend to the full sequence using updated K/V inside the windows and cached K/V elsewhere.

Figure 3: Disentangling Reuse of dLLM Logits and Guidance.

Panel of Figure 4 — Figure 4: Left: (a) Replacing fresh guidance with a cached guidance produces small changes in high-confidence token distributions. Right: (b) Percentage of prompts exhibiting certain number of conflicts between proposed tokens (as measured by PMI being less than $-1$ ).

JudgeBench
		Dream-7B				LLaDA-8B
Method		Top@1	Seq Gap	LMUnit	s/gen	Top@1	Seq Gap	LMUnit	s/gen
	Conf (BoN)	$+1.52_{\pm.16}$	–	$3.97_{\pm.03}$	$10.8_{\pm 4.7}$	$+2.65_{\pm.21}$	–	$4.05_{\pm.03}$	$11.8_{\pm 5.1}$
Seq	Conf	$+2.53_{\pm.16}$	–	$4.05_{\pm.03}$	$37.3_{\pm 15.5}$	$+3.23_{\pm.20}$	–	$4.08_{\pm.03}$	$45.3_{\pm 18.5}$
Parallel	Conf (BoN)	$-3.14_{\pm.20}$	$-4.66$	$3.01_{\pm.05}$	$\phantom{0}2.7_{\pm 1.1}$	$-2.23_{\pm.22}$	$-4.88$	$3.05_{\pm.05}$	$\phantom{0}3.0_{\pm 1.3}$
	Conf	$+0.38_{\pm.20}$	$-2.15$	$3.64_{\pm.04}$	$\phantom{0}9.7_{\pm 3.8}$	$+1.54_{\pm.22}$	$-1.69$	$3.78_{\pm.04}$	$11.8_{\pm 5.1}$

Table 1: Quantitative results for reward-guided generation on JudgeBench, RM-Bench, and Reward-Bench-2. All parallel decoding methods use cached EntRGi reward guidance, unless (BoN) is mentioned, denoting Best-of-N sampling. Subscripts denote the standard error over prompts.

Figure 7: Sequential gap by category (closer to

0

is better). FastGuide has the smallest gap to sequential decoding, especially in reasoning categories such as coding where baselines degrade.

Panel of Figure 8 — Figure 8: Qualitative generations on Math RM-Bench prompt. Red highlighted tokens denote incorrect tokens. This reinforces that guidance can worsen performance of parallel decoders, while FastGuide succeeds.

Panel of Figure 9 — Figure 9: Analysis of sparse dLLM recomputation vs. exact dLLM recomputation.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Method	High-quality regime	High-throughput regime
Confidence	top- $4$ per step	top- $8$ per step
Fast-dLLM	$c=0.8$	$c=0.6$
KLASS	$c=0.4$ , $\epsilon=1.5$ , $h=1$	$c=0.25$ , $\epsilon=1.0$ , $h=1$
EB-Sampler	$\gamma=4$	$\gamma=16$
DAPD	schedule scale $s=4$	schedule scale $s=16$
FastGuide	$k=8$ , $\tau=0.5$ , $w=2$	$k=16$ , $\tau=0.5$ , $w=2$

Appendix

Table 2: Operating points of all parallel decoders. The high-quality regime trades throughput for quality, and the high-throughput regime trades quality for throughput.

Panel of Figure 10 — Figure 10: Prompts given to the LMUnit judge. We also include in the system message the phrase rationale=No , which instructs LMUnit to output only a score for the given unit test without a written rationale as suggested by Saad-Falcon et al. (2025)

JudgeBench
		Dream-7B
Method		Top@1	Seq Gap	LMUnit	s/gen
	Conf	$+2.53_{\pm.16}$	–	$4.05_{\pm.03}$	$37.3_{\pm 15.5}$
	Conf (Expectation)	$+2.61_{\pm.17}$	–	$4.07_{\pm.03}$	$37.3_{\pm 16.4}$
Seq	Conf (APS)	$+2.06_{\pm.16}$	–	$4.04_{\pm.03}$	$37.0_{\pm 15.8}$
Parallel	Conf	$+0.38_{\pm.20}$	$-2.15$	$3.64_{\pm.04}$	$\phantom{0}9.7_{\pm 3.8}$

Appendix

Table 3: Generalization across reward-guidance objectives. FastGuide stays close to sequential decoding under the Expectation and APS objectives (in parentheses), while confidence-based parallel decoding degrades under all three.

JudgeBench , Skywork-Reward-V2-Qwen3-0.6B (smaller)
		Dream-7B
Method		Top@1	Seq Gap	LMUnit	s/gen
	Conf (BoN)	$+1.72_{\pm.13}$	–	$3.90_{\pm.03}$	$11.3_{\pm 5.4}$
	Conf (Expectation)	$+2.86_{\pm.13}$	–	$4.03_{\pm.03}$	$29.7_{\pm 16.8}$
	Conf (APS)	$+2.46_{\pm.12}$	–	$4.02_{\pm.03}$	$30.0_{\pm 16.9}$
Seq	Conf	$+2.66_{\pm.12}$	–	$3.99_{\pm.03}$	$29.9_{\pm 16.3}$

Appendix

Table 4: Robustness to the reward model on JudgeBench.

JudgeBench
		Dream-7B
Method		Top@1	Seq Gap	Avg@4	LMUnit	tok/step	s/gen
	Conf (BoN)	$+1.52_{\pm.16}$	–	$-1.07_{\pm.15}$	$3.97_{\pm.03}$	$1.00$	$10.8_{\pm 4.7}$
Seq	Conf	$+2.53_{\pm.16}$	–	$+0.08_{\pm.15}$	$4.05_{\pm.03}$	$1.00$	$37.3_{\pm 15.5}$
Parallel	Conf (BoN)	$-3.14_{\pm.20}$	$-4.66$	$-6.04_{\pm.17}$	$3.01_{\pm.05}$	$4.00$	$\phantom{0}2.7_{\pm 1.1}$
	Conf	$+0.38_{\pm.20}$	$-2.15$	$-2.64_{\pm.19}$	$3.64_{\pm.04}$	$4.00$	$\phantom{0}9.7_{\pm 3.8}$

Appendix

Table 5: Full results on Dream-7B with Avg@4 and tokens/step added.

JudgeBench
		Dream-7B
Method		Top@1	Seq Gap	LMUnit	tok/step	s/gen
	Conf (BoN)	$+1.52_{\pm.16}$	–	$3.97_{\pm.03}$	$1.00$	$10.8_{\pm 4.7}$
Seq	Conf	$+2.53_{\pm.16}$	–	$4.05_{\pm.03}$	$1.00$	$37.3_{\pm 15.5}$
Parallel	Conf	$-1.91_{\pm.20}$	$-4.45$	$3.04_{\pm.05}$	$8.00$	$\phantom{0}4.8_{\pm 1.9}$
	Fast-dLLM	$-2.13_{\pm.23}$	$-4.66$	$3.04_{\pm.06}$	$8.63$	$\phantom{0}4.9_{\pm 2.5}$

Appendix

Table 6: Results on Dream-7B with higher candidate token set size of

k=16

for FastGuide.

JudgeBench
		LLaDA-8B
Method		Top@1	Seq Gap	Avg@4	LMUnit	tok/step	s/gen
	Conf (BoN)	$+2.65_{\pm.21}$	–	$+0.51_{\pm.20}$	$4.05_{\pm.03}$	$1.00$	$11.8_{\pm 5.1}$
Seq	Conf	$+3.23_{\pm.20}$	–	$+1.31_{\pm.20}$	$4.08_{\pm.03}$	$1.00$	$45.3_{\pm 18.5}$
Parallel	Conf (BoN)	$-2.23_{\pm.22}$	$-4.88$	$-4.87_{\pm.19}$	$3.05_{\pm.05}$	$4.00$	$\phantom{0}3.0_{\pm 1.3}$
	Conf	$+1.54_{\pm.22}$	$-1.69$	$-1.17_{\pm.21}$	$3.78_{\pm.04}$	$4.00$	$11.8_{\pm 5.1}$

Appendix

Table 7: Full results on LLaDA-8B with Avg@4 and tokens/step added.

Figure 11: Sparse dLLM verification benefits more from batching than exact recomputation. Left: inference time per generation; right: speedup of sparse over exact recomputation.

Token selection in FastGuide	Top@1	LMUnit	tok/step	s/gen
Sampled at temperature $0.7$ (default)	$+4.70_{\pm.12}$	$4.04_{\pm.02}$	$5.43$	$6.2_{\pm 1.5}$
Most likely token	$+5.00_{\pm.12}$	$4.09_{\pm.02}$	$5.59$	$6.1_{\pm 1.6}$

Appendix

Table 9: Greedy selection of recomputed tokens slightly improves FastGuide. RM-Bench, Dream-7B,

k=8

Table 20

Panel of Figure 13 — Figure 13: Guided DAPD unmasks an incorrect final answer while most of its derivation is still masked. Decoding trajectories for the prompt of Figure 8 ; rows show the sequence after $25\%$ , $50\%$ , $75\%$ , $100\%$ of each method’s decoding steps, where <M> denotes a masked position.

Panel of Figure 14 — Figure 14: DAPD unmasks “5.” and “ial” around a single masked position, forcing the malformed “5.emonial”. Decoding trajectories on a Chat prompt from RM-Bench; rows show the sequence after $25\%$ , $50\%$ , $75\%$ , $100\%$ of each method’s decoding steps, where <M> denotes a masked position.

Panel of Figure 15 — Figure 15: DAPD unmasks scattered fragments of a numbered list and fills the remaining positions incoherently. Decoding trajectories on a Safety prompt from RM-Bench in the high-throughput regime; rows show the sequence after $25\%$ , $50\%$ , $75\%$ , $100\%$ of each method’s decoding steps, where <M> denotes a masked position.

Panel of Figure 16 — Figure 16: Deferral prevents the same word from being unmasked at three positions. Once “mindful” is unmasked in item 7, its recomputed probability in items 5 and 6 falls below $\tau$ and both positions are deferred (left: unmasked and deferred proposals of one step with $k=8$ ; right: final values of the deferred positions).

Panel of Figure 17 — Figure 17: Deferral lets low-confidence words be revised once their context is unmasked. The proposals “dissolved”, “the”, “desired”, and “cold” are deferred and later become “boiling”, “for”, “small”, and “it” (left: unmasked and deferred proposals of one step with $k=8$ ; right: final values of the deferred positions).

Diffusion Model Guidance

Diffusion model guidance steers denoising toward desired conditions, rewards, constraints, or away from unwanted concepts.

See all 368 papers

Supportive Token Revealing for Fast Diffusion Language Model Decoding
Jun 2, 2026Giries Abu Ayoub, Mario Barbara, Lluís Pastor-Pérez +4Masked Diffusion Models Diffusion Language Model Decoding
Department of Computer Science, University of Haifa · Institute for AI, University of Stuttgart · IMPRS-IS
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA
DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention
Mar 9, 2026Younjoo Lee, Seungkyun Dan, Junghoo Lee +2Diffusion Language Model Inference Diffusion Language Model Decoding
Seoul National University, Seoul, Republic of Korea

Jun 2, 2026J/K move · Enter open · S save

Supportive Token Revealing for Fast Diffusion Language Model Decoding

Giries Abu Ayoub, Mario Barbara, Lluís Pastor-Pérez, Tanja Bien+3

Department of Computer Science, University of Haifa · Institute for AI, University of Stuttgart · IMPRS-IS

Masked Diffusion Models Diffusion Language Model Decoding Diffusion Model Inference Acceleration Parallel Decoding for Diffusion Language Models

Open paper AskPDF arXiv

Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

Abstract

Diffusion Model Guidance

Explore similar work

Supportive Token Revealing for Fast Diffusion Language Model Decoding

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention