Modals such as may and must can take on different senses depending on discourse context, alternating between root interpretations (e.g., permission, obligation) and epistemic interpretations (e.g., knowledge, belief). While modals are known to be notoriously difficult for children to acquire, limited research exists on whether language models (LMs) can reliably distinguish between different modal senses. We evaluate seven LMs that vary in architecture, size, and training data type using vector-based similarity and surprisal measures across four datasets. Specifically, we investigate whether LMs can leverage contextual information to disambiguate modal senses and whether their performance reflects the distribution of modal senses in their training data. We show that larger LMs trained on diverse text successfully leverage additional contextual information when available. In the absence of context, they tend to favor the dominant sense based on corpus statistics. Smaller developmentally plausible LMs, including those trained on child-directed language (CDL), fail to show consistent sensitivity to modal sense distinctions and do not benefit from context to the same degree.
Figures & tables
Figure 1: An example of modal sense disambiguation.
Figure 2: Example data pair from the validated dataset.
May
Must
Root
Epistemic
All
Root
Epistemic
All
Masked LMs
RoBERTa (110M)
EPOS
0.95
0.99
0.98
0.95
0.87
0.90
Validated
1.00
1.00
1.00
0.93
0.95
0.94
Naturalistic
0.72
1.00
0.85
0.86
0.86
0.86
Naturalistic (with context)
0.67
1.00
0.82
0.89
0.96
0.94
CHILDES RoBERTa (12.7M)
EPOS
0.91
0.86
0.86
0.70
0.80
0.76
Table 1: Modal sense disambiguation accuracy by different LMs, based on the vector similarity method (cf. Section 4.1 ). The red-green color gradient represents performance from low (red) to high (green), with the 0.50 midpoint marking random chance levels. Parameter counts are indicated in indicated in parentheses for each LM. See Appendix C for more detailed representations of the results.
Figure 3: Surprisal score distributions for the modals in validated data. Lower surprisal values for a given sense are interpreted as a preference for that sense.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Data
Flavor
Context
Target sequence
#
EPOS 000
Root
i will now , if i may , read out the key points of the recommendations we adopted today .
134
Epistemic
this wording may sound a little far-fetched and need some explanation .
875
Root
we must be prepared for a long haul .
417
Epistemic
there must be a lower back injury as well .
637
Validated 000000
Root
as of now , snow white lives in the castle . and the princess assures
that the dwarfs may too .
40
Epistemic
in direct sunlight , trolls turn into stone . but the wizard worries
that the dwarfs may too .
40
Appendix
Table 2: Samples of modal expressions with root and epistemic interpretations in different datasets. EPOS and the validated data are released under a CC BY-SA 4.0 license and the naturalistic (CHILDES) data is released under CC BY-NC-SA 3.0. Across all datasets, the average sequence length is 15 tokens, with the longest sequence reaching 64 tokens.
Expectations
Corpus
Modal
Dominant sense
Source
#
Non-CDL-models
CDL-models
General (AmE)
must
root (84%)
Collins (2007)
930
root > epistemic
may
epistemic (83%)
Hacquard and Wellwood (2012)
138
epistemic > root
CHILDES (BrE) 0000
must
epistemic (65%)
van Dooren et al. (2022)
452
epistemic > root
may
root (56%)
van Dooren et al. (2022)
39
root > epistemic
Appendix
Table 3: Corpus statistics and performance expectations for the dominant senses for modals may and must .
Figure 7
Figure 4: LM accuracy results based on vector similarity scores. Error bars represent binomial 95% CIs across five cross-validation folds.
Data
Flavor
Context
Target sequence
#
Validated 000000
Root
as of now , snow white lives in the castle . and the princess assures
that the dwarfs may too .
40
Epistemic
in direct sunlight , trolls turn into stone . but the wizard worries
that the dwarfs may too .
40
Root
as of now , snow white lives in the castle . and the princess declares
that the dwarfs must too .
40
Epistemic
in direct sunlight , trolls turn into stone . but the wizard figures
that the dwarfs must too .
40
Validated (no context) 0
Root
the princess assures
that the dwarfs may too .
40
Epistemic
the wizard worries
that the dwarfs may too .
40
Appendix
Table 4: Example data points from different validated dataset versions.
May
Must
Root
Epistemic
All
Root
Epistemic
All
Masked LMs
RoBERTa
Validated
1.00
1.00
1.00
0.93
0.95
0.94
Validated (no context)
0.80
0.93
0.86
0.95
0.90
0.93
Validated (monoclausal)
0.85
0.93
0.89
0.88
0.85
0.86
CHILDES RoBERTa
Validated
0.42
0.55
0.49
0.55
0.55
0.55
Validated (no context)
0.50
0.55
0.53
0.62
0.40
0.51
Appendix
Table 5: LM accuracy results on variations of validated data.
Figure 5: Surprisal score distributions for the modals in validated data. (Figure 3 in the main body.)
Figure 6: Surprisal score distributions for the modals in validated data (no context).
Figure 7: Surprisal score distributions for the modals in validated data (monoclausal).