The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aware masking schemes can improve downstream transfer. We study masking as a task-adaptive component of the fine-tuning pipeline and introduce Typhoon, a masking strategy that uses the gradient of the MLM loss with respect to one-hot token inputs to estimate online how much each token type contributes to the objective. Typhoon maintains an exponential moving average of per-token-type saliency and calibrates these scores into a masking distribution whose expected masking rate matches a target budget under a token-independence approximation. We formalize the method and evaluate it against random masking and whole-word masking on two GLUE tasks, MRPC and CoLA, across three BERT-family backbones (TinyBERT, DistilBERT, and BERT-base) and five random seeds per configuration (90 training runs in total). Once seed variance is accounted for, no masking strategy is reliably better than the others on these tasks: on MRPC the gap between Typhoon and the best baseline stays within 0.004F1, across all twelve Typhoon comparisons no paired test reaches significance, and every 95% confidence interval contains zero. Inspecting Typhoon's learned distribution explains why: budget-preserving min-max calibration compresses the gradient ranking into a narrow band whose Kullback-Leibler divergence from uniform random masking is 7×10−4 nats per token, and widening the calibration spread by an order of magnitude in KL still leaves downstream accuracy unchanged within seed noise. Typhoon's apparent advantage in single-run experiments does not survive multi-seed evaluation, showing that gradient-based task-adaptive masking is competitive with, but does not outperform, uniform random masking at this scale.
Figures & tables
Figure 1: Intermediate-phase MLM training loss versus optimizer step for BERT-base (logged every ten steps, seed 0 ). Random and Typhoon masking are nearly indistinguishable; whole-word masking holds a higher loss throughout because it hides every subword of a masked word, yet this does not change downstream accuracy in Table 1 .
Backbone
Masking
MRPC Acc.
MRPC F1
CoLA Matthews
CoLA Acc.
TinyBERT
Random masking
0.833 ± 0.006
0.885 ± 0.004
0.148 ± 0.020
0.697 ± 0.006
Whole-word masking
0.832 ± 0.010
0.885 ± 0.006
0.162 ± 0.014
0.700 ± 0.002
Typhoon (ours)
0.833 ± 0.008
0.885 ± 0.005
0.148 ± 0.023
0.695 ± 0.008
DistilBERT-base
Random masking
0.845 ± 0.004
0.892 ± 0.003
0.512 ± 0.020
0.799 ± 0.010
Whole-word masking
0.852 ± 0.003
0.899 ± 0.003
0.504 ± 0.011
0.798 ± 0.004
Typhoon (ours)
0.849 ± 0.003
0.895 ± 0.002
0.509 ± 0.009
0.798 ± 0.006
Table 1: Downstream results on the GLUE validation sets across three backbones, reported as mean ± standard deviation over five seeds. Within each backbone–task block no strategy is reliably best on the primary metric; in particular Typhoon stays within 0.004F1 of the best baseline on MRPC.
Figure 2: Primary downstream metric by masking strategy and backbone (MRPC F1 , CoLA Matthews correlation), mean over five seeds with error bars showing one standard deviation. The strategies are within noise of each other at every scale.
Backbone
Random
Whole-word
Typhoon
Overhead
(ms/step)
(ms/step)
(ms/step)
( × random)
TinyBERT
8.8
8.8
12.1
1.38
DistilBERT
13.5
13.5
18.4
1.37
BERT-base
20.5
20.5
26.3
1.28
Table 2: Median wall-clock time per optimizer step during the intermediate MLM phase (NVIDIA H100, batch size 32 , fp16), averaged over MRPC and CoLA. The final column is Typhoon’s per-step overhead relative to random masking.
Figure 3: Frequency-weighted distribution of Typhoon’s calibrated per-type masking probability πv after the intermediate MLM phase on BERT-base (MRPC, seed 0 , default spread δ=0.4 ). Almost all token mass sits in a narrow band of width ≈0.02 around the budget p=0.3 , and the per-token KL divergence against uniform random masking is 7×10−4 nats—five orders of magnitude below log2 . The budget-preserving calibration squashes the gradient ranking into a distribution operationally indistinguishable from random.
Spread δ
Rate range
KL (nats)
MRPC Acc.
MRPC F1
0.4 (default)
[0.22, 0.37]
7.0×10−4
0.864 ± 0.020
0.905 ± 0.014
0.8
[0.10, 0.48]
4.6×10−3
0.851 ± 0.015
0.897 ± 0.010
0.99
[0.01, 0.59]
1.2×10−2
0.860 ± 0.016
0.903 ± 0.011
Table 3: Spread ablation on BERT-base / MRPC: varying Typhoon’s spread δ moves the calibrated rate range and KL by more than an order of magnitude, but downstream metrics stay flat within seed noise (three seeds per row).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Comparison
Δ
95% CI
p
peq
MRPC (F1)
TinyBERT
Typhoon − Random
+0.0001
[−0.0037,+0.0038]
0.949
0.001 ✓
TinyBERT
Typhoon − Whole-word
−0.0003
[−0.0100,+0.0094]
0.940
0.025 ✓
DistilBERT
Typhoon − Random
+0.0030
[−0.0009,+0.0070]
0.102
0.004 ✓
DistilBERT
Typhoon − Whole-word
−0.0033
[−0.0095,+0.0028]
0.206
0.020 ✓
BERT-base
Typhoon − Random
+0.0017
[−0.0035,+0.0070]
0.405
0.006 ✓
Appendix
Table 4: Paired significance and equivalence tests for Typhoon against each baseline on the primary metric (five seeds per row).
Most-masked tokens
Least-masked tokens
Rank
Token
π
Rank
Token
π
1
neill
0.369
1
##ecure
0.221
2
vc
0.345
2
implications
0.229
3
##dorf
0.343
3
transfer
0.235
4
##lvis
0.341
4
##pac
0.245
5
anatolia
0.329
5
fine
0.246
Appendix
Table 5: Typhoon’s top and bottom token-type masking probabilities on BERT-base / MRPC after three epochs of intermediate MLM training.
Hyperparameter
Value
Backbones
TinyBERT (4L/312d), DistilBERT-base, BERT-base
Masking budget p
0.3
EMA decay λ (weight on new estimate)
0.8
Probability spread δ
0.4
Optimizer
AdamW
Learning rate (MLM / fine-tune)
1×10−4 / 2×10−5
Appendix
Table 6: Hyperparameters for the intermediate MLM phase and fine-tuning.
Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the input are masked, and the model learns to predict them using the surrounding context. This process enables the model to capture both syntactic and semantic properties of language. Conventionally, the tokens selected for masking are chosen at random, which may not always yield the most effective learning signals. In this work, we examine a token masking strategy based on entropy distribution. We use the model's entropy over token predictions to identify which tokens should be masked. This method aims to target tokens that are more informative and uncertain to improve the training efficacy. We also propose a novel self-masking approach that enhances training efficiency without relying on an external reference model. Experimental results demonstrate that our method achieves an average performance improvement of 5% in GLUE scores compared to the baseline. Further, we experiment with combining knowledge distillation with entropy masking, resulting in the best overall results.
Gokul Srinivasagan, Kai Hartung, Munir Georges
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Supervised fine-tuning of discrete diffusion language models masks some response tokens and trains the model to recover their original values from the visible context. The masking pattern therefore determines both the context available to the model and the tokens it learns to predict. Uniform random masking does not explicitly account for the interaction between these choices. We introduce GoldiMask, which selects tokens to reveal as context by approximately maximizing a submodular objective. This objective uses model signals to balance the benefit of revealing tokens against their value as prediction targets. GoldiMask then weights the remaining targets according to how they benefit from the selected context and their remaining learning potential. Across three backbones and three training datasets, GoldiMask achieves the highest average accuracy in most evaluated settings, demonstrating gains on both reasoning and code generation. Component ablations show that both context selection and target weighting contribute to the gains. GoldiMask also reduces decoding iterations on GSM8K and MATH-500 under confidence-threshold parallel decoding, while maintaining comparable accuracy at higher confidence thresholds.
Loay Mualem, Lluís Pastor-Pérez, Vinh Tong +4
IMPRS-IS · Institute for AI, University of Stuttgart · Bitdefender
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.
Haran Raajesh, Kulin Shah, Adam Klivans +1
Department of Computer Science The University of Texas at Austin