Large language models (LLMs) are often used in environments where facts evolve, yet factual knowledge updates via fine-tuning on unstructured plain text often suffer from 1) reliance on paraphrasing augmentation, without which training does not generalize to downstream QA tasks, and 2) the reversal curse. Recent evidence of the sample efficiency of masked diffusion large language models (dLLMs) and the bidirectional information flow in their demasking training objective suggests that the demasking objective may facilitate knowledge learning from raw knowledge text. We test this hypothesis in controlled knowledge fine-tuning experiments on dLLMs and autoregressive LLMs (arLLMs) and find that while arLLMs rely on paraphrase augmentation to generalize knowledge text into question-answering (QA) capability, dLLMs do not require paraphrases to achieve high QA accuracy. We find that giving arLLMs a demasking objective, where the LLM must reconstruct the original text from a masked copy given in context, improves the efficacy of knowledge injection, removing the need for paraphrases and mitigating the reversal curse, recovering most of the gap between arLLMs and dLLMs. We also provide preliminary evidence of the broader applicability of using the demasking objective in arLLM fine-tuning on a large-scale knowledge-intensive dataset.
Figures & tables
Figure 1: A schematic summary of the results. First row: autoregressive LLM requires paraphrases to generalize knowledge from the fine-tuning text to QA tasks, and suffers from reversal curse (i.e. fail to answer backward questions). Second row: masked diffusion LLM can easily generalize fine-tuning text to QA tasks in both forward and backward styles. Third row: demasking fine-tuning of an arLLM recovers most of the fine-tuning gap between autoregressive LLMs and masked diffusion LLMs.
NameDescription
Biography
Wiki
Metric
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Before fine-tuning
ROUGE
0.027 ± 0.013
0.023 ± 0.009
0.019 ± 0.027
0.001 ± 0.001
0.169 ± 0.062
0.199 ± 0.044
Judge
0.000 ± 0.000
0.000 ± 0.000
0.000 ± 0.000
0.000 ± 0.000
0.065 ± 0.041
0.016 ± 0.013
w/o paraphrases
ROUGE
0.257 ± 0.169
0.025 ± 0.008
0.106 ± 0.125
0.001 ± 0.001
0.478 ± 0.154
0.321 ± 0.095
Judge
0.231 ± 0.173
0.009 ± 0.009
0.067 ± 0.090
0.000 ± 0.000
0.401 ± 0.159
0.165 ± 0.071
w/ paraphrases
ROUGE
0.925 ± 0.047
0.042 ± 0.013
0.881 ± 0.070
0.002 ± 0.002
0.630 ± 0.063
0.379 ± 0.022
Table 1: arLLM fine-tuning accuracy at the fixed training-compute budget (Appendix A.5 ), averaged across four models (Llama-3.1-8B, Llama-3.2-3B, Qwen2.5-7B, Qwen3-4B), under both accuracy measures: ROUGE-1 recall (ROUGE) and the LLM judge (Section 3 ). Values are mean ± std across models; Wiki paraphrases are the same-order set. (Per-model: Appendix Tables 9 , 14 ; peak protocol: Appendix A.6 .)
NameDescription
Biography
Wiki
Metric
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Before fine-tuning
ROUGE
0.015
0.014
0.030
0.000
0.156
0.210
Judge
0.000
0.000
0.001
0.000
0.078
0.050
w/o paraphrases
ROUGE
0.888
0.769
0.847
0.345
0.922
0.694
Judge
0.868
0.744
0.605
0.214
0.862
0.612
w/ paraphrases
ROUGE
0.948
0.927
0.979
0.683
0.899
0.755
Table 2: dLLM (LLaDA-8B-Instruct) fine-tuning accuracy at the same compute budgets as Table 1 , under both accuracy measures. (Per-model: Appendix Table 9 .)
Figure 3: Summary of QA accuracy at the fixed training-compute budget (Appendix A.5 ; equal total FLOPs across methods) across all models (Llama-3.1-8B, Llama-3.2-3B, Qwen2.5-7B, Qwen3-4B), averaged per method. Error bars indicate standard deviation across models (dLLM: single model). Hatched bars denote training with paraphrase augmentation; bars show ROUGE-1 recall and white diamonds the LLM-judge accuracy at the same checkpoints (knowledge-editing baselines: ROUGE-1 only; their judge values are likewise ≈0 ). Colors denote the training method: arLLM (red), dLLM (blue), demasking arLLM (teal) and knowledge-editing baselines (gray). (Per-model: Appendix Table 9 ; peak protocol: Appendix A.6 .)
NameDescription
Biography
Wiki
Method
Metric
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Demasking (ours)
ROUGE
0.895 ± 0.047
0.919 ± 0.056
0.898 ± 0.098
0.643 ± 0.106
0.778 ± 0.061
0.762 ± 0.072
Judge
0.879 ± 0.057
0.914 ± 0.061
0.874 ± 0.108
0.736 ± 0.127
0.741 ± 0.082
0.693 ± 0.058
AR fine-tuning
ROUGE
0.257 ± 0.169
0.025 ± 0.008
0.106 ± 0.125
0.001 ± 0.001
0.478 ± 0.154
0.321 ± 0.095
Judge
0.231 ± 0.173
0.009 ± 0.009
0.067 ± 0.090
0.000 ± 0.000
0.401 ± 0.159
0.165 ± 0.071
Reverse training
ROUGE
0.231 ± 0.143
0.049 ± 0.039
0.146 ± 0.069
0.006 ± 0.005
0.472 ± 0.096
0.379 ± 0.076
Table 3: Demasking fine-tuning accuracy (without paraphrases) with baselines on the three datasets, at the fixed training-compute budget (Appendix A.5 ). Values are mean ± std across the four arLLMs; the dLLM is a single model (LLaDA-8B-Instruct) and therefore has no across-model spread. (Knowledge-editing baselines: Appendix A.10 .)
Figure 4: Peak QA accuracy on NameDescription for different fixed mask ratios ( t ). Accuracy is the average of forward and backward accuracy, taken at the best training epoch. Dashed lines show the random- t baseline U(0.05,0.95) . The dLLM has no trainable loss at t=0 ; its value there shows the base model accuracy before fine-tuning.
Setting
Condition
Δ MMLU
Δ GSM8K
Δ IFEval
NameDescription w/o para
AR
-0.03 ± 0.04
-0.20 ± 0.29
-0.26 ± 0.24
NameDescription w/o para
Demask
-0.03 ± 0.03
-0.11 ± 0.07
-0.25 ± 0.05
NameDescription w/ para
AR
-0.09 ± 0.06
-0.36 ± 0.33
-0.49 ± 0.07
NameDescription w/ para
Demask
-0.02 ± 0.01
-0.12 ± 0.07
-0.17 ± 0.02
Biography w/o para
AR
-0.06 ± 0.07
-0.24 ± 0.35
-0.10 ± 0.14
Biography w/o para
Demask
-0.02 ± 0.01
-0.19 ± 0.04
-0.20 ± 0.05
Table 4: General-capability change relative to the base model ( Δ = fine-tuned − base, mean ± std over the four arLLMs), measured at the same checkpoints that produce the main results. (Protocol: Appendix A.5 .)
Val (selection)
Test [95% CI]
Full-set peak
Qwen3-4B (base)
0.212
0.273 [0.192, 0.364]
0.242
CPT AR
0.263
0.293 [0.212, 0.384]
0.278
CPT demasking-AR
0.293
0.333 [0.242, 0.424]
0.313
SFT AR
0.394
0.323 [0.232, 0.424]
0.359
SFT demasking-AR
0.424
0.394 [0.293, 0.485]
0.409
Table 5: Large-scale fine-tuning on Webscale-RL (1.1M samples), Qwen3-4B on GPQA-diamond with a 99/99 validation/test split. Checkpoints are selected on the validation half only; we report test accuracy with 95% bootstrap confidence intervals, and the peak accuracy on the full benchmark for comparability with the peak-reporting protocol of Appendix A.6 .
MMLU
IFEval
BBH
Qwen3-4B (base)
0.706
0.691
0.307
CPT AR
0.710
0.483
0.698
CPT demasking-AR
0.711
0.655
0.617
SFT AR
0.716
0.595
0.455
SFT demasking-AR
0.710
0.691
0.488
Table 6: General-capability benchmarks of the Webscale-RL fine-tuned models at their validation-selected checkpoints (MMLU 0-shot acc; IFEval instruction-level strict; BBH 3-shot exact match).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Learning rate sweep of Llama-3.1-8B-instruct. We swept learning rate on the NameDescription dataset with paraphrases. We picked optimal learning rate which induces fast convergence and with no overfitting and minimal fluctuation: 5e-6 for arLLM; 1e-5 for dLLM; 3e-6 for demasking arLLM.
NameDescription
Biography
Wiki
Condition
FLOPs
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
AR w/o para
F/2
0.197 ± 0.102
0.018 ± 0.005
0.165 ± 0.113
0.002 ± 0.002
0.464 ± 0.123
0.323 ± 0.088
AR w/o para
F
0.257 ± 0.169
0.025 ± 0.008
0.106 ± 0.125
0.001 ± 0.001
0.478 ± 0.154
0.321 ± 0.095
Demasking w/o para
F
0.895 ± 0.047
0.919 ± 0.056
0.898 ± 0.098
0.643 ± 0.106
0.778 ± 0.061
0.762 ± 0.072
Demasking w/o para
2F
0.810 ± 0.110
0.915 ± 0.063
0.984 ± 0.013
0.664 ± 0.109
0.878 ± 0.026
0.849 ± 0.015
AR w/ para
F/2
0.883 ± 0.035
0.038 ± 0.007
0.696 ± 0.199
0.002 ± 0.001
0.615 ± 0.061
0.379 ± 0.054
Appendix
Table 7: QA accuracy at the pre-declared training budgets (ROUGE-1 recall, mean ± std over the four arLLMs; re-trained runs, seed 0 masks). F is each dataset’s total training-FLOP budget; because a demasking epoch costs ≈2× a standard epoch, the same budget buys demasking half as many epochs. Rows at F are the checkpoints reported in the main tables; F/2 is standard fine-tuning at half the budget, and 2F demasking at standard fine-tuning’s epoch count.
end of grid
peak
Condition
Fwd
Bwd
Fwd
Bwd
NameDescription (end of grid: 3840 document-passes)
arLLM w/o paraphrases
0.167 ± 0.333
0.005 ± 0.011
0.263 ± 0.274
0.029 ± 0.013
arLLM w/ same-order paraphrases
0.823 ± 0.220
0.038 ± 0.021
0.954 ± 0.025
0.030 ± 0.008
dLLM w/o paraphrases
0.761
0.819
0.925
0.753
dLLM w/ same-order paraphrases
0.975
0.989
0.975
0.989
Appendix
Table 8: End-of-grid versus peak reporting, across all four autoregressive models and the diffusion model. Peak is the value at the best-total-accuracy checkpoint, computed over the shared budget grid from the seed-0 run; end-of-grid is the value at one pre-declared budget per dataset, identical across every condition and model, with no selection. (Because the peak entries use a single seed, the dLLM values for NameDescription and Biography differ slightly from Tables 2 and 9 , which average the dLLM over four mask-sampling seeds; all arLLM values, including demasking, come from single-seed runs.) Budgets count passes over the original documents, so paraphrase-augmented runs (30, 5 and 10 samples per document on NameDescription, Biography and Wiki) are compared at equal training cost; the evaluation grids coincide exactly, so no interpolation is used. arLLM entries are mean ± std over the four models; the dLLM is a single model.
NameDescription
Biography
Wiki
N2D-fwd
N2D-bwd
D2N-fwd
D2N-bwd
Fwd
Bwd
Fwd
Bwd
LLaDA before fine-tuning
0.030
0.000
0.000
0.028
0.030
0.000
0.156
0.210
LLaDA w/o paraphrases
0.884
0.893
0.893
0.645
0.847
0.345
0.922
0.694
LLaDA w paraphrases
0.902
0.997
0.994
0.857
0.979
0.683
0.899
0.755
Llama 8B before fine-tuning
0.072
0.000
0.000
0.054
0.001
0.000
0.127
0.164
Llama 8B reverse training
0.637
0.125
0.113
0.088
0.181
0.011
0.554
0.350
Appendix
Table 9: Fine-tuning performance of the dLLM and arLLMs across all three datasets at the fixed training-compute budget (Appendix A.5 ; demasking, whose per-step cost is ≈2× , at half the passes so total FLOPs are equal); arLLM and demasking values are from training runs with checkpoints at the budget; dLLM and reverse-training values are read from their training curves at the budget epochs. “Demasking” denotes the demasking fine-tuning paradigm of Section 4.3 ; “Reverse training” follows Golovneva et al. (2024) ; Wiki paraphrases are the same-order set. (LLM-judge counterpart: Appendix Table 14 .)
Figure 6: arLLM fine-tuning on the Wiki dataset, averaged across four models (Llama-3.1-8B, Llama-3.2-3B, Qwen2.5-7B, Qwen3-4B). Error bars indicate standard deviation across models.
NameDescription
Biography
Wiki
Method
Model
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
ROME
Llama 8B
0.019
0.000
0.003
0.000
0.095
0.156
Llama 3B
0.077
0.000
0.001
0.002
0.109
0.154
Qwen 7B
0.041
0.000
0.025
0.000
0.193
0.242
Qwen 4B
0.030
0.000
0.060
0.001
0.241
0.239
MEMIT
Llama 8B
0.019
0.000
0.003
0.000
0.095
0.156
Appendix
Table 10: Per-model knowledge-editing results (ROUGE-1 recall). ROME and MEMIT produce identical predictions for three of the four models; this is a property of the runs, not a transcription error. (Method details: Appendix A.10 .)
Figure 7: Training dynamics of arLLM (Llama 8B), dLLM (LLaDA), and demasking arLLM (Llama 8B). For the NameDescription dataset, forward and backward accuracy are the average of N2D and D2N types. Paraphrases used in the Wiki dataset are the same-order paraphrases set. Due to the randomness of sampling the masks, we average across 4 random seeds for the dLLM and demasking arLLM on NameDescription and Biography Datasets. The horizontal axis counts passes over the original document set, so paraphrase-augmented runs (which see 30, 5 and 10 samples per document) are placed on the same training-cost axis as unaugmented runs. (Other arLLMs: Figures 9 – 11 ; per-seed curves: Figures 12 – 13 .)
Figure 8: Total accuracy (macro average of forward and backward accuracy) of experiments on Llama-3.1-8B-Instruct. Main tables report accuracy at the fixed training-compute budget F (Appendix A.5 ); the total accuracy shown here is also used to pick the best checkpoints reported under the peak protocol (Appendix A.6 ).
Figure 9: Learning dynamics of Llama-3.2-3B-Instruct.
Figure 10: Learning dynamics of Qwen/Qwen2.5-7B-Instruct.
Figure 11: Learning dynamics of Qwen/Qwen3-4B-Instruct-2507.
Figure 12: Random seed effects in LLaDA. Random seed determines the sampling of mask ratio and masked tokens. Each line represents a random seed.
Figure 13: Random seed effects in demasking Llama-3.1-8B. Random seed determines the sampling of mask ratio and masked tokens. We found slightly larger variability across the seed in demasking arLLM than dLLM, though the general trend and peak accuracy do not vary much.
Figure 14: To verify the advantage of demasking fine-tuning of arLLMs is not simply due to “data augmentation” (i.e. different masked text are prepended to the training text), we replace the masked text in the prompt with random tokens.
NameDescription
Biography
Wiki
Forward
Backward
Forward
Backward
Forward
Backward
B/4
B
B/4
B
B/4
B
B/4
B
B/4
B
B/4
B
AR w/o paraphrases
.192
.441
.013
.032
.137
.000
.000
.001
.486
.577
.278
.308
AR w/ paraphrases
.931
.956
.008
.049
.673
.875
.000
.000
.574
.661
.371
.392
dLLM w/o paraphrases
.591
.888
.322
.769
.420
.847
.169
.345
.795
.922
.651
.694
dLLM w/ paraphrases
.585
.948
.563
.927
.716
.979
.283
.683
.811
.899
.680
.755
Appendix
Table 11: Accuracy after a fixed number of passes over the document set, at a quarter of the training budget ( B/4 ) and at the full budget ( B ), where B=960 passes for NameDescription and 1280 for Biography and Wiki (Appendix A.5 ). Paraphrase-augmented runs are placed on the same document-pass axis as unaugmented runs. The arLLM rows are Llama-3.1-8B-Instruct (single runs) and the dLLM rows are LLaDA-8B-Instruct; dLLM values on NameDescription and Biography are means over four mask-sampling seeds, and single runs elsewhere.
Figure 15: Training dynamics of dLLM and demasking arLLM fine-tuning with fixed mask ratios ( t ) on the NameDescription dataset. Solid lines: without paraphrases; dashed lines: with paraphrases.
NameDescription
Biography
Wiki
Model
Para
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Llama 8B
w/o para
+0.02
+0.02
−0.05
+0.01
+0.09
+0.11
w/ para
−0.03
−0.01
−0.02
+0.01
+0.14
+0.05
Llama 3B
w/o para
+0.04
+0.00
+0.15
+0.01
+0.07
+0.12
w/ para
+0.00
−0.01
−0.03
+0.01
+0.03
+0.07
Qwen 7B
w/o para
+0.16
+0.01
+0.27
+0.01
−0.09
+0.01
Appendix
Table 12: Change in QA accuracy when using chat template vs. pre-training format for arLLM fine-tuning ( Δ=accchat−accpretrain ). Averaged across all models and datasets, the mean effect is +0.038 (mean ± std across 28 configurations, the 24 shown plus Wiki with permute-order paraphrases for each model: 0.461±0.186 with chat, 0.423±0.205 without).
NameDescription
Biography
Wiki
Method
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
AR w/o para
0.37±0.30
0.02±0.02
0.09±0.06
0.00±0.00
0.35±0.05
0.25±0.04
AR w/ para
0.89±0.13
0.05±0.02
0.56±0.05
0.00±0.00
0.48±0.05
0.41±0.12
dLLM w/o para
0.74
0.90
0.83
0.60
0.71
0.78
dLLM w/ para
0.90
0.97
0.90
0.75
0.75
0.74
Demasking AR w/o para
0.61±0.24
0.86±0.14
0.86±0.13
0.40±0.18
0.69±0.01
0.77±0.05
Appendix
Table 13: Few-shot (2-shot) QA evaluation results. For AR and demasking AR: mean ± std across four arLLMs. dLLM results are from a single model (LLaDA).
NameDescription
Biography
Wiki
Model
Condition
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
LLaDA
dLLM w/o para
0.868
0.744
0.605
0.214
0.862
0.612
LLaDA
dLLM w/ para
0.930
0.908
0.960
0.720
0.853
0.663
Llama 8B
AR w/o para
0.408
0.013
0.000
0.000
0.500
0.174
Llama 8B
AR w/ para
0.958
0.029
0.848
0.000
0.621
0.267
Llama 8B
Demasking w/o para
0.938
0.946
0.973
0.880
0.707
0.632
Appendix
Table 14: Per-model LLM-judge accuracy at the pre-declared budget checkpoints. (ROUGE counterpart: Appendix Table 9 .)
Llama 8B
LLaDA
Demasking Llama 8B
w/o paraphrases (1)
w paraphrases (2)
reverse training (3)
w/o paraphrases (4)
w/o paraphrases (5, ours )
Data
paraphrase compute
NA
0.1M tokens (o3-mini)
NA
NA
NA
Training
FLOPs theoretical
T
T
T
T
2T+c
FLOPs empirical (TFLOPs/step)
91.9
92.7
91.9
101.6
199.3
wall time (s/step)
2.32
2.54
2.84
2.52
2.76
peak memory (GB)
37.9
37.9
37.9
37.9
51.1
Appendix
Table 15: Comparison of data preparation, training and inference computational costs among different model architecture and training methods on Wiki dataset. Bold indicates best single performance and underline indicates best tied performance.