Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

Bruce Changlong Xu; Muhammed Shahir Abdurrahman; Hashem Elezabi

cs.CLMar 27, 2023

Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

Authors: Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem Elezabi

Organizations: Department of Computer Science, Stanford University

Abstract

The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aware masking schemes can improve downstream transfer. We study masking as a task-adaptive component of the fine-tuning pipeline and introduce Typhoon, a masking strategy that uses the gradient of the MLM loss with respect to one-hot token inputs to estimate online how much each token type contributes to the objective. Typhoon maintains an exponential moving average of per-token-type saliency and calibrates these scores into a masking distribution whose expected masking rate matches a target budget under a token-independence approximation. We formalize the method and evaluate it against random masking and whole-word masking on two GLUE tasks, MRPC and CoLA, across three BERT-family backbones (TinyBERT, DistilBERT, and BERT-base) and five random seeds per configuration ( $90$ training runs in total). Once seed variance is accounted for, no masking strategy is reliably better than the others on these tasks: on MRPC the gap between Typhoon and the best baseline stays within $0.004$ $F_1$ , across all twelve Typhoon comparisons no paired test reaches significance, and every $95\%$ confidence interval contains zero. Inspecting Typhoon's learned distribution explains why: budget-preserving min-max calibration compresses the gradient ranking into a narrow band whose Kullback-Leibler divergence from uniform random masking is $7\times 10^{-4}$ nats per token, and widening the calibration spread by an order of magnitude in KL still leaves downstream accuracy unchanged within seed noise. Typhoon's apparent advantage in single-run experiments does not survive multi-seed evaluation, showing that gradient-based task-adaptive masking is competitive with, but does not outperform, uniform random masking at this scale.

Figures & tables

Figure 1: Intermediate-phase MLM training loss versus optimizer step for BERT-base (logged every ten steps, seed

0

). Random and Typhoon masking are nearly indistinguishable; whole-word masking holds a higher loss throughout because it hides every subword of a masked word, yet this does not change downstream accuracy in Table 1 .

Backbone	Masking	MRPC Acc.	MRPC F1	CoLA Matthews	CoLA Acc.
TinyBERT	Random masking	0.833 $\pm$ 0.006	0.885 $\pm$ 0.004	0.148 $\pm$ 0.020	0.697 $\pm$ 0.006
	Whole-word masking	0.832 $\pm$ 0.010	0.885 $\pm$ 0.006	0.162 $\pm$ 0.014	0.700 $\pm$ 0.002
	Typhoon (ours)	0.833 $\pm$ 0.008	0.885 $\pm$ 0.005	0.148 $\pm$ 0.023	0.695 $\pm$ 0.008
DistilBERT-base	Random masking	0.845 $\pm$ 0.004	0.892 $\pm$ 0.003	0.512 $\pm$ 0.020	0.799 $\pm$ 0.010
	Whole-word masking	0.852 $\pm$ 0.003	0.899 $\pm$ 0.003	0.504 $\pm$ 0.011	0.798 $\pm$ 0.004
	Typhoon (ours)	0.849 $\pm$ 0.003	0.895 $\pm$ 0.002	0.509 $\pm$ 0.009	0.798 $\pm$ 0.006

Table 1: Downstream results on the GLUE validation sets across three backbones, reported as mean

\pm

standard deviation over five seeds. Within each backbone–task block no strategy is reliably best on the primary metric; in particular Typhoon stays within

0.004

F_{1}

of the best baseline on MRPC.

Figure 2: Primary downstream metric by masking strategy and backbone (MRPC

F_{1}

, CoLA Matthews correlation), mean over five seeds with error bars showing one standard deviation. The strategies are within noise of each other at every scale.

Backbone	Random	Whole-word	Typhoon	Overhead
	(ms/step)	(ms/step)	(ms/step)	( $\times$ random)
TinyBERT	8.8	8.8	12.1	1.38
DistilBERT	13.5	13.5	18.4	1.37
BERT-base	20.5	20.5	26.3	1.28

Table 2: Median wall-clock time per optimizer step during the intermediate MLM phase (NVIDIA H100, batch size

32

, fp16), averaged over MRPC and CoLA. The final column is Typhoon’s per-step overhead relative to random masking.

Figure 3: Frequency-weighted distribution of Typhoon’s calibrated per-type masking probability

\pi_{v}

after the intermediate MLM phase on BERT-base (MRPC, seed

0

, default spread

\delta=0.4

). Almost all token mass sits in a narrow band of width

\approx 0.02

around the budget

p=0.3

, and the per-token KL divergence against uniform random masking is

7\!\times\!10^{-4}

nats—five orders of magnitude below

\log 2

. The budget-preserving calibration squashes the gradient ranking into a distribution operationally indistinguishable from random.

Spread $\delta$	Rate range	KL (nats)	MRPC Acc.	MRPC $F_{1}$
$0.4$ (default)	[0.22, 0.37]	$7.0{\times}10^{-4}$	0.864 $\pm$ 0.020	0.905 $\pm$ 0.014
$0.8$	[0.10, 0.48]	$4.6{\times}10^{-3}$	0.851 $\pm$ 0.015	0.897 $\pm$ 0.010
$0.99$	[0.01, 0.59]	$1.2{\times}10^{-2}$	0.860 $\pm$ 0.016	0.903 $\pm$ 0.011

Table 3: Spread ablation on BERT-base / MRPC: varying Typhoon’s spread

\delta

moves the calibrated rate range and KL by more than an order of magnitude, but downstream metrics stay flat within seed noise (three seeds per row).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

MRPC (F1)
Backbone	Comparison	$\Delta$	95% CI	$p$	$p_{\text{eq}}$
TinyBERT	Typhoon $-$ Random	$+0.0001$	$[-0.0037,+0.0038]$	$0.949$	$0.001$ ✓
TinyBERT	Typhoon $-$ Whole-word	$-0.0003$	$[-0.0100,+0.0094]$	$0.940$	$0.025$ ✓
DistilBERT	Typhoon $-$ Random	$+0.0030$	$[-0.0009,+0.0070]$	$0.102$	$0.004$ ✓
DistilBERT	Typhoon $-$ Whole-word	$-0.0033$	$[-0.0095,+0.0028]$	$0.206$	$0.020$ ✓
BERT-base	Typhoon $-$ Random	$+0.0017$	$[-0.0035,+0.0070]$	$0.405$	$0.006$ ✓

Appendix

Table 4: Paired significance and equivalence tests for Typhoon against each baseline on the primary metric (five seeds per row).

Rank	Token	$\pi$	Rank	Token	$\pi$
	Most-masked tokens			Least-masked tokens
1	neill	0.369	1	##ecure	0.221
2	vc	0.345	2	implications	0.229
3	##dorf	0.343	3	transfer	0.235
4	##lvis	0.341	4	##pac	0.245
5	anatolia	0.329	5	fine	0.246

Appendix

Table 5: Typhoon’s top and bottom token-type masking probabilities on BERT-base / MRPC after three epochs of intermediate MLM training.

Hyperparameter	Value
Backbones	TinyBERT (4L/312d), DistilBERT-base, BERT-base
Masking budget $p$	$0.3$
EMA decay $\lambda$ (weight on new estimate)	$0.8$
Probability spread $\delta$	$0.4$
Optimizer	AdamW
Learning rate (MLM / fine-tune)	$1\times 10^{-4}$ / $2\times 10^{-5}$

Appendix

Table 6: Hyperparameters for the intermediate MLM phase and fine-tuning.

Masked Language Modeling

Masked language modeling trains text encoders to infer hidden tokens, producing contextual representations for language tasks and specialized domains.

See all 47 papers

Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Language Modeling Self-Supervised Pre-Training
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Fine-Tuning Diffusion Language Models with Context Selection and Target Weighting
Sep 29, 2026Loay Mualem, Lluís Pastor-Pérez, Vinh Tong +4Masked Diffusion Models Diffusion Model Fine-Tuning
IMPRS-IS · Institute for AI, University of Stuttgart · Bitdefender
Mask-Aware Policy Gradients for Diffusion Language Models
Jul 16, 2026Haran Raajesh, Kulin Shah, Adam Klivans +1Masked Diffusion Models RL for Language Model Reasoning
Department of Computer Science The University of Texas at Austin

May 27, 2026J/K move · Enter open · S save

Entropy-aware Masking for Masked Language Modeling

Gokul Srinivasagan, Kai Hartung, Munir Georges

AImotion Bavaria, Technische Hochschule Ingolstadt, Germany

Masked Language Modeling Self-Supervised Pre-Training

Open paper AskPDF arXiv

Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the input are masked, and the model learns to predict them using the surrounding context. This process enables the model to capture both syntactic and semantic properties of language. Conventionally, the tokens selected for masking are chosen at random, which may not always yield the most effective learning signals. In this work, we examine a token masking strategy based on entropy distribution. We use the model's entropy over token predictions to identify which tokens should be masked. This method aims to target tokens that are more informative and uncertain to improve the training efficacy. We also propose a novel self-masking approach that enhances training efficiency without relying on an external reference model. Experimental results demonstrate that our method achieves an average performance improvement of 5% in GLUE scores compared to the baseline. Further, we experiment with combining knowledge distillation with entropy masking, resulting in the best overall results.

Abstract

Masked Language Modeling

Explore similar work

Entropy-aware Masking for Masked Language Modeling

Fine-Tuning Diffusion Language Models with Context Selection and Target Weighting

Mask-Aware Policy Gradients for Diffusion Language Models