Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States
Organizations: King’s College London, UK · Centre for AI, Data Science & Artificial Intelligence, BioPharmaceuticals R&D, AstraZeneca, UK · The Chinese University of Hong Kong · MoE Lab, CUHK ·
Abstract
Autoregressive (AR) models remain the dominant paradigm for natural language generation, but their strictly sequential decoding process leads to high inference latency. Recent diffusion-inspired language models, such as LLaDA and Dream, alleviate this issue through parallel generation, yet they still face two key limitations: information loss, since predictive distributions over non-finalised tokens are discarded at each step, and unstable commitment dynamics, where local token decisions are not sufficiently coordinated at the global level. We propose Latent Refinement Decoding (LRD), a two-stage decoding framework consisting of Latent Refinement and a Predictive Feedback Loop. In the first stage, LRD keeps masked positions as distributional mixtures of predicted tokens and the mask embedding, enabling the model to form more globally consistent beliefs before committing tokens. In the second stage, LRD progressively finalises confident tokens while preserving uncertain ones for further iterative refinement, with KL-divergence dynamics serving as a stable criterion for convergence and early stopping. Experiments show that LRD improves performance on both coding tasks, including HumanEval by 6.3 and MBPP by 2.6, and reasoning tasks, including GSM8K by 2.9 and MATH500 by 3.8, while achieving up to 10.6x overall speedup. Moreover, LRD is compatible with single-token decoding, multi-token threshold commitment, and system-level accelerators; when combined with Fast-dLLM, it reaches up to 18.4x speedup over vanilla decoding while further improving accuracy.
Figures & tables
| Model | Len | Commit | Method | HumanEval | MBPP | GSM8K | MATH500 | ||||
| Acc | Speed | Acc | Speed | Acc | Speed | Acc | Speed | ||||
| Dream Base-7B | 256 | Single | Entropy-min | 50.6 | 1.0 | 55.8 | 1.0 | 75.3 | 1.0 | 36.9 | 1.0 |
| Prophet | 51.1 | 1.2 | 55.1 | 2.1 | 75.6 | 1.5 | 36.5 | 1.4 | |||
| LRD | 56.9 +6.3 | 1.2 | 57.6 +1.8 | 2.3 | 78.2 +2.9 | 1.8 | 39.8 +2.9 | 1.4 | |||
| Multi | Conf-thresh | 48.9 | 1.4 | 55.5 | 1.9 | 74.7 | 2.5 | 36.7 | 2.4 | ||
| KLASS | 51.4 | 1.5 | 56.6 | 1.4 | 75.2 | 1.8 | 37.2 | 1.2 | |||
| Method | HumanEval | MBPP | GSM8K | MATH500 | ||||
| Acc | Speed | Acc | Speed | Acc | Speed | Acc | Speed | |
| Baseline | 43.9 | 1.0 | 38.2 | 1.0 | 81.3 | 1.0 | 37.7 | 1.0 |
| SOAR | 46.6 | 1.7 | 39.2 | 1.6 | 82.1 | 1.4 | 39.0 | 1.2 |
| CreditDecoding | 45.6 | 1.3 | 38.9 | 1.5 | 80.7 | 1.9 | 37.6 | 1.5 |
| LRD | 48.4 | 1.3 | 40.6 | 1.5 | 84.5 | 2.0 | 39.8 | 1.6 |
| Method | HE | MBPP | GSM8K | MATH |
| Baseline | 56.1 | 56.7 | 80.2 | 38.6 |
| GN, | 49.2 | 52.2 | 70.4 | 33.4 |
| GN, | 47.0 | 51.4 | 69.8 | 32.4 |
| KL-stop only | 55.8 | 56.9 | 80.4 | 38.3 |
| LRD w/o ent. wt. | 57.7 | 57.9 | 76.8 | 40.2 |
| LRD (full) | 60.9 | 58.8 | 82.7 | 41.8 |
| Method | HumanEval | MBPP | GSM8K | MATH500 | ||||
| Acc | Speed | Acc | Speed | Acc | Speed | Acc | Speed | |
| Baseline | 56.1 | 1.0 | 56.7 | 1.0 | 80.2 | 1.0 | 38.6 | 1.0 |
| LRD (full) | 60.9 | 2.9 | 58.8 | 4.6 | 82.7 | 3.6 | 41.8 | 1.2 |
| w/o latent refinement | 59.9 | 3.1 | 57.8 | 4.9 | 82.2 | 3.8 | 41.0 | 1.2 |
| w/o mixed embeddings | 58.0 | 2.7 | 57.8 | 4.3 | 80.7 | 3.0 | 40.8 | 1.0 |
| w/o early stopping | 61.2 | 1.2 | 58.8 | 1.1 | 82.9 | 1.1 | 41.9 | 1.0 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Len | Method | HumanEval | MBPP | GSM8K | MATH500 | ||||
| Acc | Speed | Acc | Speed | Acc | Speed | Acc | Speed | |||
| Dream-Base-7B | 256 | baseline | 50.6 | 1.0 | 55.8 | 1.0 | 75.3 | 1.0 | 36.9 | 1.0 |
| LRD | 56.9 +6.3 | 1.2 | 57.6 +1.8 | 2.3 | 78.2 +2.9 | 1.8 | 39.8 +2.9 | 1.4 | ||
| 512 | baseline | 54.4 | 1.0 | 55.8 | 1.0 | 76.2 | 1.0 | 37.5 | 1.0 | |
| LRD | 58.8 +4.4 | 2.6 | 58.4 +2.6 | 4.5 | 77.4 +1.2 | 3.4 | 40.8 +3.3 | 1.8 | ||
| 1024 | baseline | 54.8 | 1.0 | 58.0 | 1.0 | 76.8 | 1.0 | 39.1 | 1.0 | |
| HumanEval | MBPP | GSM8K | MATH-500 | |||||
| Method | Acc. | Speed | Acc. | Speed | Acc. | Speed | Acc. | Speed |
| Baseline | 56.1 | 1.0 | 56.7 | 1.0 | 80.2 | 1.0 | 38.6 | 1.0 |
| Top by confidence | 55.1 | 1.0 | 55.0 | 1.0 | 80.7 | 1.0 | 37.4 | 1.0 |
| Top by confidence | 41.9 | 1.9 | 43.0 | 1.9 | 75.9 | 2.1 | 29.0 | 1.9 |
| Top by confidence | 26.7 | 2.9 | 27.2 | 2.8 | 66.1 | 3.1 | 26.2 | 2.9 |
| Top by confidence | 21.6 | 3.8 | 22.2 | 3.9 | 57.7 | 4.1 | 14.0 | 3.9 |
| Length | Method | HumanEval | MBPP | GSM8K | MATH500 |
| 256 | Baseline | 38.7 / 1.0 | 36.9 / 1.0 | 77.4 / 1.0 | 33.8 / 1.0 |
| 256 | SOAR | 40.4 / 1.3 | 37.6 / 1.3 | 78.2 / 1.2 | 34.7 / 1.0 |
| 256 | CreditDecoding | 40.3 / 1.2 | 37.5 / 1.3 | 76.9 / 1.4 | 33.6 / 1.3 |
| 256 | LRD | 43.3 / 1.2 | 40.0 / 1.3 | 78.8 / 1.5 | 35.8 / 1.4 |
| 512 | Baseline | 43.9 / 1.0 | 38.2 / 1.0 | 81.3 / 1.0 | 37.7 / 1.0 |
| 512 | SOAR | 46.6 / 1.7 | 39.2 / 1.6 | 82.1 / 1.4 | 39.0 / 1.2 |
| Method | Benchmark | Method accuracy | LRD accuracy |
| Soft-Masked DLMs | HumanEval | 57.8 | 60.9 |
| Soft-Masked DLMs | MBPP | 56.4 | 58.8 |
| EvoToken-DLM | GSM8K | 84.0 | 84.5 |
| EvoToken-DLM | MATH500 | 40.0 | 39.8 |
| RCD | GSM8K | 79.1 | 84.5 |
| RCD | MATH500 | 37.0 | 39.8 |
| HumanEval | MBPP | GSM8K | MATH500 | |||||
| Method | Acc. | Speed | Acc. | Speed | Acc. | Speed | Acc. | Speed |
| Baseline | 56.1 | 1.0 | 56.7 | 1.0 | 80.2 | 1.0 | 38.6 | 1.0 |
| DAEDAL [ Li et al., 2025a ] | 54.3 | 3.2 | 56.0 | 7.7 | 79.1 | 8.4 | 38.2 | 1.2 |
| DAEDAL+LRD | 58.8 | 3.3 | 58.4 | 7.6 | 81.9 | 8.6 | 41.0 | 1.2 |
| LRD (ours) | 60.9 | 2.9 | 58.8 | 4.6 | 82.7 | 3.6 | 41.8 | 1.2 |
| Len | Method | RACE | TruthfulQA | ||||
| Acc | Speed | bleu_acc | rouge1_acc | rouge2_acc | Speed | ||
| 256 | baseline | 81.1 | 1.0 | 40.3 | 45.3 | 32.8 | 1.0 |
| Ours | 82.2 +1.1 | 1.3 | 42.5 +2.2 | 47.6 +2.3 | 34.2 +1.4 | 2.1 | |
| 512 | baseline | 80.9 | 1.0 | 41.9 | 45.4 | 33.1 | 1.0 |
| Ours | 82.4 +1.5 | 1.4 | 44.3 +2.4 | 48.3 +2.9 | 35.1 +2.0 | 2.6 | |
| 1024 | baseline | 81.2 | 1.0 | 41.7 | 45.5 | 33.6 | 1.0 |
| Len | Method | CNN/DailyMail | NQ-Open | ||||
| BERTScore | ROUGE-1 | ROUGE-L | Speed | EM | Speed | ||
| 256 | baseline | 85.6 | 21.7 | 15.0 | 1.0 | 12.0 | 1.0 |
| Ours | 85.7 +0.1 | 22.3 +0.6 | 15.5 +0.5 | 1.3 | 15.0 +3.0 | 2.2 | |
| 512 | baseline | 85.5 | 22.6 | 15.6 | 1.0 | 12.8 | 1.0 |
| Ours | 85.6 +0.1 | 23.0 +0.4 | 15.9 +0.3 | 1.4 | 14.4 +1.6 | 2.6 | |
| 1024 | baseline | 85.7 | 22.8 | 15.7 | 1.0 | 12.8 | 1.0 |
| Benchmark | Length | Native accuracy | LRD accuracy | LRD speedup |
| GPQA | 256 | 34.2 | 36.8 | 1.2 |
| GPQA | 512 | 35.5 | 38.4 | 1.5 |
| BBH | 256 | 59.1 | 61.6 | 1.5 |
| BBH | 512 | 60.4 | 63.2 | 2.9 |
| Model | Len | Method | HumanEval | MBPP | GSM8K | MATH500 | ||||
| Acc | Speed | Acc | Speed | Acc | Speed | Acc | Speed | |||
| LLaDA2.0- mini-preview | 256 | baseline | 5.5 | 1.0× | 19.2 | 1.0× | 63.8 | 1.0× | 16.2 | 1.0× |
| Ours | 7.3 +1.8 | 1.0× | 22.2 +3.0 | 1.1× | 65.6 +1.8 | 1.2× | 17.5 +1.3 | 1.0× | ||
| 512 | baseline | 54.3 | 1.0× | 54.7 | 1.0× | 86.5 | 1.0× | 44.6 | 1.0× | |
| Ours | 56.9 +2.6 | 1.1× | 58.2 +3.5 | 1.6× | 88.1 +1.6 | 2.1× | 46.3 +1.7 | 1.1× | ||
| 1024 | baseline | 74.2 | 1.0× | 63.2 | 1.0× | 87.7 | 1.0× | 61.2 | 1.0× | |
| Method | HumanEval | MBPP | GSM8K | MATH500 |
| Nemotron-Labs-Diffusion-8B-Instruct | ||||
| Native diffusion | 69.8 / 1.0 | 70.9 / 1.0 | 87.4 / 1.0 | 65.6 / 1.0 |
| +LRD | 72.4 / 1.3 | 73.6 / 1.5 | 89.0 / 1.2 | 68.0 / 1.1 |
| DiffusionGemma-26B-A4B-it | ||||
| Native EB + adaptive stopping | 76.3 / 1.0 | 73.8 / 1.0 | 89.6 / 1.0 | 68.2 / 1.0 |
| +LRD-style adaptation | 77.5 / 1.2 | 75.3 / 1.1 | 90.3 / 1.2 | 69.6 / 1.1 |
| Per-Step Cost | FLOPs (G) | Latency | Remark / |
| Baseline (Standard) | 14.14 | 81.8 ms | N/A |
| Backbone | 14.14 | 68.6 ms | MatMul dominated |
| System Overhead | N/A | 13.2 ms | Data dispatch |
| LRD (Proposed) | 14.14 | 94.3 ms | N/A |
| Backbone | 14.14 | 68.6 ms | Identical to baseline |
| KL / Mixing / Aux | 12.6 ms | Memory-bound overhead |
| LRD | Fast-dLLM+LRD | ||||
| Benchmark | Baseline | (change) | (no stop) | (change) | (no stop) |
| HumanEval | 90.4 | 84.6 ( ) | 1.08 | 98.6 ( ) | 1.11 |
| MBPP | 60.5 | 57.2 ( ) | 1.06 | 49.4 ( ) | 1.19 |
| GSM8K | 135.5 | 123.7 ( ) | 1.07 | 115.3 ( ) | 1.13 |
| MATH500 | 482.3 | 437.3 ( ) | 1.09 | 377.2 ( ) | 1.21 |
| Model | Steps | Method | HumanEval | MBPP | GSM8K | MATH500 |
| Dream-Base-7B | 64 | Baseline | 43.2 / 1.0 | 50.2 / 1.0 | 33.2 / 1.0 | 10.9 / 1.0 |
| 64 | LRD | 44.4 / 0.9 | 50.6 / 1.0 | 33.7 / 1.1 | 11.6 / 0.9 | |
| 128 | Baseline | 47.2 / 1.0 | 54.4 / 1.0 | 63.4 / 1.0 | 28.9 / 1.0 | |
| 128 | LRD | 49.0 / 1.1 | 55.4 / 1.1 | 64.6 / 1.3 | 30.4 / 1.1 | |
| LLaDA-Ins-8B | 64 | Baseline | 31.2 / 1.0 | 36.0 / 1.0 | 61.0 / 1.0 | 24.3 / 1.0 |
| 64 | LRD | 31.8 / 0.9 | 36.7 / 0.9 | 61.5 / 1.0 | 24.8 / 0.9 |
| HumanEval | MBPP | GSM8K | MATH-500 | |||||
| Baseline | LRD | Baseline | LRD | Baseline | LRD | Baseline | LRD | |
| 1 | 37.8 | 42.7 | 36.4 | 39.7 | 77.2 | 78.6 | 33.9 | 35.9 |
| 2 | 45.7 | 50.1 | 44.1 | 47.1 | 86.5 | 87.7 | 41.5 | 43.4 |
| 5 | 52.8 | 56.5 | 52.6 | 54.7 | 92.4 | 93.0 | 50.3 | 52.0 |
| 10 | 56.3 | 58.7 | 58.1 | 60.2 | 94.2 | 95.2 | 54.9 | 56.7 |
| Length | Method | Accuracy | Speed | ||||||||||
| HumanEval | MBPP | GSM8K | MATH500 | HumanEval | MBPP | GSM8K | MATH500 | HumanEval | MBPP | GSM8K | MATH500 | ||
| 256 | baseline | 55.4 | 57.4 | 80.8 | 37.9 | 1.0 | 1.0 | 1.0 | 1.0 | 117.2 | 53.5 | 132.4 | 228.4 |
| Ours | 61.6 +6.2 | 59.4 +2.0 | 83.0 +2.2 | 40.6 +2.7 | 1.4 +0.4× | 2.4 +1.4× | 1.4 +0.4× | 1.1 +0.1× | 108.4 -8.8 | 49.2 -4.3 | 128.6 -5.8 | 226.0 -2.4 | |
| w/o latent refinement | 60.1 -1.5 | 58.6 -0.8 | 82.3 -0.7 | 39.4 -1.2 | 1.5 +0.1× | 2.5 +0.1× | 1.5 +0.1× | 1.1 +0.0× | 108.7 +0.3 | 50.4 +1.2 | 129.9 +1.3 | 226.8 +0.8 | |
| w/o mix embed | 59.5 -2.1 | 58.8 -0.6 | 82.7 -0.3 | 38.9 -1.7 | 1.3 -0.1× | 2.2 -0.2× | 1.5 +0.1× | 1.0 -0.1× | 117.2 +8.8 | 49.5 +0.3 | 129.4 +0.8 | 228.4 +2.4 | |
| w/o early stop | 61.8 +0.2 | 59.4 +0.0 | 83.2 +0.2 | 40.6 +0.0 | 1.0 -0.4× | 1.1 -1.3× | 1.1 -0.3× | 1.0 -0.1× | 109.7 +1.3 | 51.4 +2.2 | 129.9 +1.3 | 228.0 +2.0 | |
| Final outcome | Benchmark | Positions | Hard | Soft | [95% CI] | Rank: Hard Soft |
| Baseline wrong / LRD correct | HumanEval | 73 | 0.24 | 0.37 | [0.08, 0.18] | |
| Baseline wrong / LRD correct | GSM8K | 186 | 0.11 | 0.23 | [0.08, 0.16] | |
| Baseline wrong / LRD wrong | HumanEval | 108 | 0.17 | 0.21 | [0.01, 0.07] | |
| Baseline wrong / LRD wrong | GSM8K | 327 | 0.09 | 0.15 | [0.03, 0.09] |
| Method | HumanEval | MBPP | GSM8K | MATH500 |
| Baseline | 56.1 | 56.7 | 80.2 | 38.6 |
| Ours | 60.9 +4.8 | 58.8 +2.1 | 82.7 +2.5 | 41.8 +3.2 |
| Auto | 59.6 +3.5 | 57.7 +1.0 | 81.5 +1.3 | 41.4 +2.8 |
| LF 1 | 60.4 +4.3 | 57.8 +1.1 | 81.6 +1.4 | 40.2 +1.6 |
| LF 2 | 58.3 +2.2 | 57.2 +0.5 | 81.2 +1.0 | 40.2 +1.6 |
| LF 3 | 57.9 +1.8 | 57.8 +1.1 | 81.8 +1.6 | 39.0 +0.4 |
| Hyperparam | Value | HumanEval | MBPP | GSM8K | MATH-500 | Avg Speed |
| 0.05 | 60.8 | 58.8 | 82.4 | 41.9 | 2.6 | |
| 0.10 (default) | 60.9 | 58.8 | 82.7 | 41.8 | 2.8 | |
| 0.15 | 60.8 | 58.6 | 82.6 | 41.6 | 2.9 | |
| 0.05 | 61.0 | 58.8 | 82.6 | 41.5 | 2.3 | |
| 0.10 (default) | 60.9 | 58.8 | 82.7 | 41.8 | 2.8 | |
| 0.15 | 60.7 | 58.7 | 82.6 | 41.6 | 3.0 |
| Parameter | Value | HumanEval | MBPP | GSM8K | MATH500 | Avg. speed |
| 0.05 | 61.5 | 59.3 | 82.8 | 40.7 | 1.4 | |
| 0.10 | 61.6 | 59.4 | 83.0 | 40.6 | 1.5 | |
| 0.15 | 61.4 | 59.2 | 82.9 | 40.4 | 1.6 | |
| 0.05 | 61.7 | 59.4 | 83.1 | 40.7 | 1.3 | |
| 0.10 | 61.6 | 59.4 | 83.0 | 40.6 | 1.5 | |
| 0.15 | 61.3 | 59.2 | 82.8 | 40.3 | 1.7 |
| Parameter | Value | HumanEval | MBPP | GSM8K | MATH500 | Avg. speed |
| 0.05 | 60.9 | 58.9 | 83.3 | 44.0 | 5.1 | |
| 0.10 | 61.0 | 59.0 | 83.5 | 43.9 | 5.5 | |
| 0.15 | 60.8 | 58.8 | 83.4 | 43.7 | 5.8 | |
| 0.05 | 61.1 | 58.9 | 83.6 | 44.0 | 4.6 | |
| 0.10 | 61.0 | 59.0 | 83.5 | 43.9 | 5.5 | |
| 0.15 | 60.7 | 58.8 | 83.2 | 43.6 | 6.1 |
| Operator | : mild strongest | CR: mild strongest | Accuracy at strongest |
| LRD | 0.14 | ||
| SM-style | 0.19 | ||
| Evo-style | 0.19 | ||
| RCD-style | 0.00 |
Diffusion Language Model Decoding
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.
See all 264 papers