CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

Yuji Ren; Chenkai Xu; Zhuocheng Gong; Jianguo Li; Zhijie Deng

cs.LGAug 14, 2026

CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

Authors: Yuji Ren, Chenkai Xu, Zhuocheng Gong, Jianguo Li, Zhijie Deng

Organizations: Shanghai Jiao Tong University · Ant Group

Abstract

Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, leading to errors that can propagate to later stages. To tackle this issue, we present Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages. CForce trains the model on pre-collected self-rollout trajectories, thereby improving training-inference alignment. We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL. We further provide a theoretical analysis for the consistency objective to explain why CForce can approximately minimize the prediction error of early stages. Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding; in the edit-capable case, later token-to-token refinements provide additional supervision for earlier masked-state predictions. Experiments on non-edit and edit-capable LLaDA models show improved speed-quality trade-offs, especially under high-parallelism decoding budgets. Code is available at: https://github.com/inclusionAI/dFactory.

Figures & tables

Figure 1: Speed–quality trade-off: accuracy vs. tokens per forward pass (TPF). Upper-right is better. Consistency Forcing moves both models toward a higher-parallelism trade-off.

Figure 2: Per-task Accuracy Under Parallelism (AUP). AUP integrates the score–parallelism curve into a single metric. CForce yields consistent gains across all four benchmarks.

Figure 3: An illustration of edit-capable dLLM decoding. At each decoding step, high-confidence masked positions are denoised, while already generated tokens may also be revised to correct earlier mistakes.

Figure 4: Overview of Consistency Forcing. Left: self-rollout threshold-decoding trajectories are downsampled into adjacent stage pairs whenever the cumulative number of M2T reveals reaches the stage size $S$ ; right: the earlier low-context state is trained against same-model stop-gradient predictions from the later, more informative state.

Diffusion type	Edit?	Model	GSM8K			MATH500			MBPP			HumanEval
Diffusion type	Edit?	Model	Score	TPF	AUP	Score	TPF	AUP	Score	TPF	AUP	Score	TPF	AUP
Pure diffusion	$\times$	dUltra-coding-b32	81.52 *	8.40	–	35.64	6.72 *	–	37.04	7.29 *	–	35.85	7.23 *	–
		d3LLM-LLaDA	73.09	9.11 *	637.65 *	30.36	5.74	107.64 *	40.60	4.21	88.36	39.63	5.95	96.64
		d3LLM-Dream	81.36	4.94	391.33	38.21 *	3.92	97.50	55.60 *	2.96	141.41 *	57.10 *	3.20	129.48 *
		D2F-LLaDA	74.39	2.88	213.76	28.94	2.66	49.00	39.00	2.13	52.96	40.64	2.69	61.98
Block diffusion	$\times$	LightningRL-8B-b32	90.30	5.58	492.40	63.00	6.28	407.50	58.30	11.10	641.60	72.60	6.30	450.10

Table 1: Main Comparison by Diffusion Type and Edit Capability. Each benchmark group reports Score, TPF, and AUP. Bold and underline mark the best and second-best metrics within each comparable block; stars mark the best pure-diffusion results.

Figure 5: Few-step Generation Results (Score %) on LLaDA2.0 Series with Fixed TPF Budgets. Vanilla, CAP, and Ours represent LLaDA2.0-mini, LLaDA2.0-mini-CAP, and LLaDA2.0-mini-CForce respectively.

Stage size $S$	AVG Score	AVG TPF
4	80.62	5.64
8	80.88	6.42
16	75.05	7.97

Table 2: Ablation on Stage Size for LLaDA2.0-mini-CForce. Full benchmark-level results are in Appendix F . The shaded row denotes the default setting; bold marks the best value in each metric.

Component	Variant	AVG
Component	Variant	Score	TPF
KL divergence	Forward KL	85.75	5.07
	Reverse KL	65.57	8.72
	CAD	80.88	6.42
CE anchor	$\lambda_{\mathrm{CE}}=0$	79.02	6.52
	$\lambda_{\mathrm{CE}}=0.1$	80.88	6.42

Table 4: Ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2.0-mini-CForce.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Decode config	AVG		GSM8K		MATH500		MBPP		HumanEval
Model	Decode config	Score	TPF	Score	TPF	Score	TPF	Score	TPF	Score	TPF
LLaDA2.0-mini	$\tau=0.95$	86.52	3.09	93.78	2.42	84.20	2.99	81.50	3.21	86.59	3.74
LLaDA2.0-mini	$\tau=0.9$	84.32	3.60	93.25	2.85	81.80	3.46	78.69	3.80	83.54	4.28
LLaDA2.0-mini-CAP	$\tau=0.95$	82.70	4.26	92.34	3.40	83.00	4.15	76.81	4.57	78.66	4.93
LLaDA2.0-mini-CAP	$\tau=0.9$	79.51	5.07	91.74	4.09	81.00	4.85	72.13	5.43	73.18	5.89
LLaDA2.0-mini-CForce (Ours)	$\tau=0.95$	83.90	5.32	92.27	4.33	82.60	5.34	79.63	4.87	81.10	6.73

Appendix

Table 5: Benchmark-level Operating Points Used to Compute AUP for the LLaDA-family Rows in Table 1 . Each benchmark group reports the Score/TPF pair used by the two-point AUP frontier. Bold and underline mark the best and second-best metrics within each LLaDA-series block. All evaluations use maximum generation length 4096; LLaDA2.0 models are M2T-only and therefore do not use an edit threshold.

Model	GSM8K TPS	HumanEval TPS	AVG TPS
Non-edit dLLMs
LLaDA2.0-mini	489.87	711.59	600.73
LLaDA2.0-mini-CAP	710.88	891.94	801.41
LLaDA2.0-mini-CForce (Ours)	783.09	932.82	857.96
Edit-capable dLLMs
LLaDA2.1-mini	987.78	1116.10	1051.94

Appendix

Table 6: Wall-clock Throughput on GSM8K and HumanEval. Bold marks the best throughput within each edit-capability group.

Stage size $S$	AVG		GSM8K		MATH500		MBPP		HumanEval
Stage size $S$	Score	TPF	Score	TPF	Score	TPF	Score	TPF	Score	TPF
4	80.62	5.64	91.51	4.56	79.80	5.64	74.94	5.22	76.22	7.14
8	80.88	6.42	91.74	5.36	79.20	6.35	73.30	6.00	79.27	7.97
16	75.05	7.97	90.30	7.61	75.20	9.86	62.76	6.71	71.95	7.70

Appendix

Table 7: Full benchmark-level ablation on stage size for LLaDA2.0-mini-CForce. Each benchmark group reports Score and TPF.

Curriculum	Target type	AVG		GSM8K		MATH500		MBPP		HumanEval
Curriculum	Target type	Score	TPF	Score	TPF	Score	TPF	Score	TPF	Score	TPF
$\checkmark$	Same-model target	80.88	6.42	91.74	5.36	79.20	6.35	73.30	6.00	79.27	7.97
$\checkmark$	Frozen teacher	79.49	5.91	91.21	4.67	79.20	5.57	70.73	5.63	76.83	7.75
$\times$	Same-model target	79.87	5.97	91.66	4.89	77.40	5.67	72.37	5.30	78.05	8.00

Appendix

Table 8: Full benchmark-level ablation on curriculum learning and target type for LLaDA2.0-mini-CForce. The frozen-teacher variant changes only the later-state target predictor, while the no-curriculum row keeps the same-model stop-gradient target and removes the gradual exposure schedule. Each benchmark group reports Score and TPF.

Component	Variant	AVG		GSM8K		MATH500		MBPP		HumanEval
Component	Variant	Score	TPF	Score	TPF	Score	TPF	Score	TPF	Score	TPF
KL divergence	Forward KL	85.75	5.07	92.95	4.00	82.80	4.72	81.26	4.74	85.98	6.81
	Reverse KL	65.57	8.72	87.64	7.02	70.20	8.15	51.99	8.78	52.44	10.92
	CAD	80.88	6.42	91.74	5.36	79.20	6.35	73.30	6.00	79.27	7.97
CE anchor	$\lambda_{\mathrm{CE}}=0$	79.02	6.52	91.58	5.34	78.80	6.36	68.85	6.06	76.83	8.32
	$\lambda_{\mathrm{CE}}=0.1$	80.88	6.42	91.74	5.36	79.20	6.35	73.30	6.00	79.27	7.97

Appendix

Table 9: Full benchmark-level ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2.0-mini-CForce. Each benchmark group reports Score and TPF.

Parallel Decoding for Diffusion Language Models

See all 37 papers

DMax: Aggressive Parallel Decoding for dLLMs
Apr 9, 2026Zigeng Chen, Gongfan Fang, Xinyin Ma +2Diffusion Language Model Training Parallel Decoding for Diffusion Language Models
National University of Singapore
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Parallel Speculative Decoding Masked Diffusion Models
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation
May 29, 2026Zekai Li, Ji Liu, Yiqing Huang +3Adaptive Computation for Diffusion Models Diffusion Language Model Decoding
Advanced Micro Devices, Inc. (AMD)

Apr 9, 2026J/K move · Enter open · S save

DMax: Aggressive Parallel Decoding for dLLMs

Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu+1

National University of Singapore

Diffusion Language Model Training Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration Masked Diffusion Models

Open paper AskPDF arXiv

We present DMax, a new paradigm for efficient diffusion language models (dLLMs). It mitigates error accumulation in parallel decoding, enabling aggressive decoding parallelism while preserving generation quality. Unlike conventional masked dLLMs that decode through a binary mask-to-token transition, DMax reformulates decoding as a progressive self-refinement from mask embeddings to token embeddings. At the core of our approach is On-Policy Uniform Training, a novel training strategy that efficiently unifies masked and uniform dLLMs, equipping the model to recover clean tokens from both masked inputs and its own erroneous predictions. Building on this foundation, we further propose Soft Parallel Decoding. We represent each intermediate decoding state as an interpolation between the predicted token embedding and the mask embedding, enabling iterative self-revising in embedding space. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of DMax. Compared with the original LLaDA-2.0-mini, our method improves TPF on GSM8K from 2.04 to 5.47 while preserving accuracy. On MBPP, it increases TPF from 2.71 to 5.86 while maintaining comparable performance. On two H200 GPUs, our model achieves an average of 1,338 TPS at batch size 1. Code is available at: https://github.com/czg1225/DMax

Abstract

Parallel Decoding for Diffusion Language Models

Explore similar work

DMax: Aggressive Parallel Decoding for dLLMs

PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding

Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation