RFG: Self-Improving Diffusion Large Language Models with Reward-Free Guidance

Tianlang Chen; Minkai Xu; Jure Leskovec; Stefano Ermon

cs.CLSep 29, 2025

RFG: Self-Improving Diffusion Large Language Models with Reward-Free Guidance

Authors: Tianlang Chen, Minkai Xu, Jure Leskovec, Stefano Ermon

Organizations: Stanford University

Abstract

Diffusion Large Language Models (dLLMs) have shown strong reasoning capabilities, yet further improving them typically requires costly post-training with additional data and supervision. We ask whether a post-trained dLLM can improve itself at inference time without additional training, data, or reward models. This requires a guidance signal from the checkpoints alone that is well-defined on the partially masked intermediate states of dLLMs, which existing methods fail to provide. Here we propose Reward-Free Guidance (RFG), a training-free framework for inference-time self-improvement of dLLMs that derives such a signal from the checkpoints themselves. We theoretically demonstrate that a reward signal for a partially masked state can be parameterized by the log-likelihood ratio between a post-trained policy dLLM and its reference model. In practice, this signal can be obtained using off-the-shelf checkpoints alone. Extensive experiments show that RFG consistently improves state-of-the-art post-trained dLLMs by up to 16.1%. Furthermore, despite being entirely training-free, RFG delivers gains that rival or even surpass resource-intensive reinforcement learning techniques.

Figures & tables

Figure 1: RFG consistently enhances performance over the original post-trained baseline across diverse benchmarks and model variants. Results are averaged across generation sequence lengths.

		GSM8K		MATH-500		HumanEval		MBPP
Method	Train GPU-h	Avg. Acc.	Rel. Gain	Avg. Acc.	Rel. Gain	Avg. Acc.	Rel. Gain	Avg. Acc.	Rel. Gain
LLaDA-Instruct	—	$79.9$	—	$37.8$	—	$39.0$	—	$51.9$	—
+RL (LLaDA-1.5)	$405$	$80.9$	$+1.3\%$	$37.5$	$-0.8\%$	$\underline{39.4}$	$\underline{+1.0\%}$	$\underline{52.3}$	$\underline{+0.8\%}$
+RL (d1-LLaDA)	$110+108$	$\bm{81.8}$	$\bm{+2.4\%}$	$\underline{38.5}$	$\underline{+1.9\%}$	—	—	—	—
+RFG (ours)	$0$	$\underline{81.0}$	$\underline{+1.4\%}$	$\bm{39.3}$	$\bm{+4.0\%}$	$\bm{40.7}$	$\bm{+4.4\%}$	$\bm{54.2}$	$\bm{+4.4\%}$

Table 1: Comparison of RFG (training-free) with RL. Accuracy is averaged across generation lengths. Training time is measured in H100 GPU hours. d1-LLaDA trains a separate checkpoint for each benchmark, so its two numbers are the training time for GSM8K and MATH-500, respectively. Best is in bold and second best is underlined .

Figure 2: Illustration of RFG sampling. By linearly combining logits from the policy and reference models at each denoising step, RFG steers the generation trajectory toward accurate reasoning traces, resulting in improved performance without external reward models.

	Seq Len = 128				Seq Len = 256				Seq Len = 512
Model	Original	CFG	RFG	Rel. Gain	Original	CFG	RFG	Rel. Gain	Original	CFG	RFG	Rel. Gain
Instruction Fine-tuning
LLaDA 8B Instruct	$76.3$	$72.6$	$\bm{77.2}$	$\bm{+1.2\%}$	$79.8$	$80.7$	$\bm{81.3}$	$\bm{+1.9\%}$	$83.7$	$81.5$	$\bm{84.6}$	$\bm{+1.1\%}$
Dream 7B Instruct	$67.9$	$63.6$	$\bm{74.2}$	$\bm{+9.3\%}$	$80.9$	$72.2$	$\bm{82.1}$	$\bm{+1.5\%}$	$84.8$	$80.1$	$\bm{85.1}$	$\bm{+0.4\%}$
Reinforcement Learning
d1-LLaDA	$79.3$	$77.4$	$\bm{79.8}$	$\bm{+0.6\%}$	$82.5$	$81.3$	$\bm{84.7}$	$\bm{+2.7\%}$	$83.6$	$82.2$	$\bm{84.6}$	$\bm{+1.2\%}$

Table 2: Performance on GSM8K across different sequence lengths. Best in bold . Relative gain of RFG over the original post-trained model in green .

	GSM8K				MATH-500
Model	Original	CFG	RFG	Rel. Gain	Original	CFG	RFG	Rel. Gain
Instruction Fine-tuning
LLaDA 8B Instruct	$78.6$	$77.9$	$\bm{80.7}$	$\bm{+2.7\%}$	$34.2$	$28.6$	$\bm{38.0}$	$\bm{+11.1\%}$
Dream 7B Instruct	$68.5$	$57.2$	$\bm{73.7}$	$\bm{+7.6\%}$	$35.4$	$19.4$	$\bm{37.6}$	$\bm{+6.2\%}$
Reinforcement Learning
d1-LLaDA	$79.8$	$78.3$	$\bm{82.0}$	$\bm{+2.8\%}$	$36.6$	$30.0$	$\bm{39.2}$	$\bm{+7.1\%}$

Table 6: Performance on mathematical reasoning tasks with a sequence length of 256. This setting doubles the number of tokens generated per step compared to Table 5 and Table 5 .

Figure 3: Accuracy under varying guidance strength $w$ . GSM8K and MATH-500 use LLaDA-1.5 with sequence length 128, HumanEval and MBPP use Dream-Instruct with sequence length 512. RFG consistently improves performance over a broad range of guidance strength.

	LLaDA 8B Instruct				Dream 7B Instruct
Method	P-strict	P-loose	I-strict	I-loose	P-strict	P-loose	I-strict	I-loose
Original	$55.6$	$60.6$	$65.7$	$69.8$	$55.6$	$59.1$	$67.1$	$70.0$
RFG	$\bm{60.1}$	$\bm{64.1}$	$\bm{68.3}$	$\bm{71.5}$	$\bm{57.7}$	$\bm{60.4}$	$\bm{68.3}$	$\bm{70.6}$

Table 8: IFEval instruction-following accuracy (%). P: prompt-level, I: instruction-level, each with the official strict and loose variants. Best in bold .

	GSM8K			MATH-500
Model	Original	Sharpen	RFG	Original	Sharpen	RFG
LLaDA-Instruct	$83.7$	$82.9$	$\bm{84.6}$	$42.6$	$42.2$	$\bm{44.0}$
Dream-Instruct	$84.8$	$83.0$	$\bm{85.1}$	$50.4$	$49.4$	$\bm{52.6}$
d1-LLaDA	$83.6$	$83.9$	$\bm{84.6}$	$42.6$	$41.8$	$\bm{43.8}$
LLaDA-1.5	$84.8$	$84.4$	$\bm{85.3}$	$42.0$	$42.8$	$\bm{44.0}$

Table 9: Policy-only sharpening control at sequence length 512. Sharpen applies

(1+w)\log\pi_{\theta}

with the same

w

as RFG . Best in bold .

Panel of Figure 4 — Figure 4: Qualitative examples for mathematical reasoning task.

Panel of Figure 5 — Figure 5: Qualitative examples for code generation task.

Explore similar work

GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
May 28, 2026Xiaohang Tang, Keyue Jiang, Che Liu +4Diffusion Language Models Large Language Model Reinforcement Learning
UNIST · University of Basel
Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
Oct 6, 2026Yiming Qin, Ke Wang, Amel Abdelraheem +2Diffusion Language Models Diffusion Models
EPFL, Lausanne, Switzerland
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
Jun 7, 2026Yawen Shao, Jie Xiao, Kai Zhu +6Large Language Model Reinforcement Learning Diffusion Language Models
University of Science and Technology of China · Tongyi Lab · Northeastern University

May 28, 2026, cs.LGJ/K move · Enter open · S save

GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

Xiaohang Tang, Keyue Jiang, Che Liu, Qifang Zhao+3

UNIST · University of Basel

Diffusion Language Models Large Language Model Reinforcement Learning Unsupervised On-Policy Self-Distillation Denoising Diffusion Probabilistic Model Lower Bounds

Open paper AskPDF arXiv

Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of methods replaces the likelihood in standard RL with its evidence lower bound (ELBO), estimated from randomly masked sequences. Despite being well aligned with pre-training, these approaches introduce bias through training--inference mismatch by using the ELBO as a likelihood surrogate, which can degrade performance. In this work, we propose Guided Denoiser Self-Distillation (GDSD) to directly distill the denoiser of dLLMs from an advantage-guided self-teacher, derived from the closed-form optimum of reverse-KL regularized RL. GDSD matches the dLLM's denoiser logits to the teacher's via a normalization-free objective, which reduces RL to likelihood-free self-distillation and thus bypasses the TIM biases. Recent ELBO-based methods emerge as instances of applying different distillation divergences, but with diagnosable pathologies that GDSD avoids. On planning, math, and coding benchmarks with LLaDA-8B and Dream-7B, GDSD consistently outperforms prior state-of-the-art ELBO-based methods with a more stable training reward dynamics, achieving test-accuracy improvements of up to

+19.6\%

. These results suggest that direct denoiser self-distillation, without relying on an ELBO likelihood surrogate, can provide a more stable and effective RL procedure for dLLMs. Code is available at https://github.com/GaryBall/GDSD.

Abstract

Explore similar work

GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights

Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models