ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
Organizations: JetBrains Research · Constructor University · JetBrains
Abstract
Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.
Figures & tables
| Variant | GSM8K | MATH500 |
|---|---|---|
| ReMaRL | 50.04 | 20.00 |
| No position embedding | 48.98 | 19.73 |
| No hidden states | 45.08 | 18.20 |
| No remasking | 38.26 | 15.00 |
| Position and mask biases only | 29.16 | 14.07 |
| Random init., no added confidence | 47.54 | 18.20 |
| Math | Code | |||||||
|---|---|---|---|---|---|---|---|---|
| 8 calls | 16 calls | 8 calls | 16 calls | |||||
| P2 ′ temperature | GSM8K | MATH500 | GSM8K | MATH500 | HumanEval | MBPP | HumanEval | MBPP |
| 23.50 | 13.60 | 42.68 | 19.00 | 14.02 | 3.00 | 17.68 | 11.40 | |
| 29.19 | 14.80 | 41.57 | 16.87 | 13.01 | 4.60 | 18.70 | 9.33 | |
| 34.04 | 16.00 | 46.25 | 18.00 | 13.41 | 9.27 | 18.90 | 17.73 | |
| 42.30 | 17.87 | 56.25 | 20.93 | 14.23 | 21.47 | 22.56 | 29.33 | |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Mean-NFE budget | ||||||||
|---|---|---|---|---|---|---|---|---|
| Method | Blocks | 4 | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | ||||||||
| ReMaRL (ours) | 36.14 | 53.60 | 65.13 | 72.81 | 79.71 | — | — | |
| ReMaRL (ours), trained at NFE64 | — | — | — | — | 79.71 | 80.39 | 82.49 | |
| P2 ′ (ours, ) | 27.29 | 42.30 | 56.25 | 63.79 | 66.87 | 69.12 | 69.42 | |
| P2 ′ (ours, ) | 16.17 | 28.76 | 43.59 | 59.19 | 68.11 | 70.58 | 73.16 | |
| Mean-NFE budget | ||||||||
|---|---|---|---|---|---|---|---|---|
| Method | Blocks | 4 | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | ||||||||
| ReMaRL (ours) | 17.07 | 19.00 | 24.87 | 27.27 | 31.07 | — | — | |
| ReMaRL (ours), trained at NFE64 | — | — | — | — | 31.07 | 32.93 | 33.27 | |
| P2 ′ (ours, ) | 13.73 | 17.87 | 20.93 | 22.60 | 24.67 | 26.27 | 26.07 | |
| P2 ′ (ours, ) | 11.07 | 13.73 | 17.73 | 21.27 | 24.40 | 27.07 | 27.20 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 20.73 | 29.47 | 31.71 | — | — | — | |
| ReMaRL (ours), trained at NFE32 | — | — | 31.71 | 38.41 | 38.62 | 39.23 | |
| P2 ′ (ours, ) | 14.23 | 22.56 | 24.19 | 23.17 | 25.20 | 22.97 | |
| P2 ′ (ours, ) | 11.59 | 14.84 | 19.92 | 26.22 | 22.36 | 24.80 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 25.40 | 32.40 | 36.47 | — | — | — | |
| ReMaRL (ours), trained at NFE32 | — | — | 36.47 | 39.53 | 41.33 | 43.53 | |
| P2 ′ (ours, ) | 21.47 | 29.33 | 33.67 | 33.73 | 37.00 | 35.73 | |
| P2 ′ (ours, ) | 15.73 | 23.33 | 28.07 | 32.27 | 32.47 | 35.93 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 47.86 | 58.15 | 65.48 | — | — | — | |
| P2 ′ (ours, ) | 20.29 | 38.51 | 50.72 | 57.42 | 60.73 | 60.58 | |
| P2 ′ (ours, greedy) | 2.20 | 9.63 | 27.29 | 38.82 | 39.58 | 35.71 | |
| Confidence | 1.59 | 8.57 | 32.30 | 42.99 | 33.59 | 35.78 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 18.60 | 24.33 | 27.40 | — | — | — | |
| P2 ′ (ours, ) | 8.73 | 16.00 | 20.67 | 22.80 | 22.27 | 21.53 | |
| P2 ′ (ours, greedy) | 0.60 | 3.80 | 12.80 | 19.80 | 19.20 | 21.60 | |
| Confidence | 1.00 | 3.80 | 15.00 | 21.00 | 17.20 | 21.20 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 24.39 | 38.01 | 46.95 | — | — | — | |
| P2 ′ (ours, ) | 10.16 | 19.31 | 26.22 | 30.69 | 31.71 | 33.94 | |
| P2 ′ (ours, greedy) | 7.32 | 14.02 | 18.90 | 28.05 | 29.27 | 33.54 | |
| Confidence | 9.15 | 11.59 | 21.34 | 28.66 | 28.66 | 26.22 | |
| Mean-NFE budget | |||||||
|---|---|---|---|---|---|---|---|
| Method | Blocks | 8 | 16 | 32 | 64 | 128 | 256 |
| Fixed-budget decoding | |||||||
| ReMaRL (ours) | 29.27 | 40.13 | 51.87 | — | — | — | |
| P2 ′ (ours, ) | 23.60 | 34.07 | 41.87 | 45.93 | 47.33 | 49.53 | |
| P2 ′ (ours, greedy) | 0.60 | 9.20 | 23.40 | 39.80 | 45.40 | 45.60 | |
| Confidence | 1.40 | 6.60 | 22.80 | 44.40 | 45.80 | 47.60 | |
| Method | Remasking events | Replacements | Replaced (%) |
|---|---|---|---|
| ReMaRL | 673 | 33 | 4.9 |
| P2 ′ , | 1,158 | 27 | 2.3 |
| P2 ′ , | 1,227 | 84 | 6.8 |
| Position temperature | GSM8K accuracy (%) |
|---|---|
| 0 | 33.36 |
| 0.10 | 34.47 |
| 0.25 | 39.20 |
| 0.50 | 48.19 |
| 0.75 | 51.98 |
| 0.85 | 53.53 |
| Run | Attempted groups | Accepted (%) |
|---|---|---|
| Math, NFE4 | 74,845 | 43.8 |
| Math, NFE8 | 60,539 | 54.1 |
| Math, NFE16 | 61,974 | 52.9 |
| Math, NFE32 | 65,803 | 49.8 |
| Math, NFE64 | 75,748 | 43.3 |
| Code, NFE8 | 99,632 | 32.9 |
| Comparison (math, NFE8) | GSM8K | MATH500 | Average |
|---|---|---|---|
| ReMaRL (32k) vs. P2 ′ , | +24.84 [22.87, 26.81] | +5.27 [3.00, 7.60] | +15.05 [13.55, 16.59] |
| ReMaRL (32k) vs. calibrated P2 ′ | +11.30 [9.48, 13.12] | +1.13 [-0.93, 3.27] | +6.21 [4.88, 7.61] |
| ReMaRL (16k) vs. calibrated P2 ′ | +7.73 [5.86, 9.60] | +2.13 [0.00, 4.33] | +4.93 [3.53, 6.35] |
| ReMaRL (32k) vs. remasking-matched P2 ′ | +25.30 [23.35, 27.27] | +4.47 [2.33, 6.60] | +14.88 [13.44, 16.33] |
| ReMaRL (16k) vs. no remasking | +11.78 [9.98, 13.57] | +5.00 [2.80, 7.20] | +8.39 [6.96, 9.83] |
| ReMaRL (16k) vs. no hidden states | +4.95 [3.21, 6.72] | +1.80 [-0.07, 3.73] | +3.38 [2.08, 4.66] |
Diffusion Language Model Inference
Diffusion language model inference studies how to generate, guide, and serve text efficiently through iterative denoising and parallel token updates.
See all 201 papers