Previous work examining the Uniform Information Density (UID) hypothesis has shown that while information as measured by surprisal metrics is distributed more or less evenly across documents overall, local discrepancies can arise due to functional pressures corresponding to syntactic and discourse structural constraints. However, work thus far has largely disregarded the relative salience of discourse participants. We fill this gap by studying how overall salience of entities in discourse relates to surprisal using 70K manually annotated mentions across 16 genres of English and a novel minimal-pair prompting method. Our results show that globally salient entities exhibit significantly higher surprisal than non-salient ones, even controlling for position, length, and nesting confounds. Moreover, salient entities systematically reduce surprisal for surrounding content when used as prompts, enhancing document-level predictability. This effect varies by genre, appearing strongest in topic-coherent texts and weakest in conversational contexts. Our findings refine the UID competing pressures framework by identifying global entity salience as a mechanism shaping information distribution in discourse.
Figures & tables
Genre
Docs
Tokens
Modality
Mentions
Entities
% Top1
% Top2
% Top3
% Salient
Academic writing
18
17,169
Written
5,055
3,001
1.90
3.35
5.00
15.65
Biographies
20
18,213
Written
5,772
3,052
2.32
3.71
5.57
16.02
Vlog
15
16,864
Spoken
4,499
1,256
1.63
3.44
5.61
14.83
Conversations
15
17,932
Spoken
4,531
1,166
1.12
2.49
4.64
16.24
Courtroom transcripts
9
11,148
Spoken
2,938
1,053
1.80
3.46
5.82
14.68
Essays
9
10,842
Written
3,061
1,672
0.93
1.74
2.89
10.88
Table 1: Overview of GUM-SAGE genres with document counts, token counts, modality, mention and entity counts, and proportion of salient entities by salience level (Top1: score=5, Top2: score ⩾ 4, Top3: score ⩾ 3). Percentages are calculated over all entities per genre.
Figure 1: Change in mean surprisal for salience scores 1–5 relative to score 0 in Experiment 1.
Predictor
Coef.
P -value
95% CI
Δ AIC
Intercept
0.8205
[0.764, 0.877]
–
Position
0.0002
[0.000, 0.000]
2.68
Salience score
-0.0471
[-0.070, -0.025]
14.86
Mention length
-0.0535
[-0.059, -0.048]
352.11
Is Nesting
-0.8493
[-0.903, -0.796]
886.27
Table 2: Results from an OLS regression model Surprisal ~ Salience score + Mention length + Is Nested + Position , with predictors ordered by Δ AIC from single-term deletions (least to most informative): larger Δ AIC indicates a larger degradation in model fit when that predictor is removed. Significance codes: ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 .
Figure 2: Decrease in mean surprisal relative to salience score 0 in Experiment 2. Magnitude of surprisal reduction largely grows with salience, indicating that more salient entities facilitate sentence-level predictability.
Figure 3: Mean surprisal scores across genres for Experiment 2 (salience score>0 in orange, score=0 in green).
Figure 4: Decrease in mean surprisal relative to salience score 0 in Experiment 3.
Figure 5: Mean surprisal scores across genres for Experiment 3 (salience > 0 in orange, salience=0 in green).
Figure 6: Mean surprisal (z-score) for salient vs. non-salient entities across genres. Bars show mean surprisal for salient (orange) and non-salient (blue) entities, with genres sorted by surprisal difference. Error bars represent 95% confidence intervals. Asterisks indicate significant differences based on adjusted t-tests (* p < .05, ** p < .01, *** p < .001). Genres shaded gray show no significant difference between salient and non-salient entities.
Figure 7: Surprisal contour in different genres. Blue lines represent the surprisal contour of the sentence followed by salient entities; Red lines represent the surprisal contour of the sentence followed by non salient entities. Each stimulus is distinguished by different markers.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Change in mean surprisal for salience scores 1–5 relative to score 0 in Experiment 1 using GPT2-Small .
Figure 9: Change in mean surprisal for salience scores 1–5 relative to score 0 in Experiment 2 using GPT2-Small .
Figure 10: Change in mean surprisal for salience scores 1–5 relative to score 0 in Experiment 3 using GPT2-Small .
Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.
Samuel Kiegeland, Vésteinn Snæbjarnarson, Tim Vieira +1
The Uniform Information Density (UID) hypothesis posits that speakers are subject to a communicative pressure to distribute information evenly within utterances, minimising surprisal variance. While this hypothesis has been tested empirically, prior studies are limited exclusively to text-only inputs, abstracting away from the perceptual context in which utterances are produced. In this work, we present the first computational study of UID in visually grounded settings. We estimate surprisal using multilingual vision-and-language models over image-caption data in 30 languages and visual storytelling data in 13 languages, together spanning 11 families. We find that grounding on perception consistently smooths the distribution of information, increasing both global and local uniformity across typologically diverse languages compared to text-only settings. In visual narratives, grounding in both image and discourse contexts has additional effects, with the strongest surprisal reductions occurring at the onset of discourse units. Overall, this study takes a first step towards modelling the temporal dynamics of information flow in ecologically plausible, multimodal language use, and finds that grounded language exhibits greater information uniformity, supporting a context-sensitive formulation of UID.
Matteo Gay, Coleman Haley, Mario Giulianelli +1
KU Leuven · University of Edinburgh · University College London
Surprisal, the negative log-probability a language model assigns to a word given its preceding context, reliably predicts adult reading times. Does it contribute as much to explaining when children acquire individual words? Frequency reflects a learner's cumulative exposure to a word, whereas surprisal reflects how predictable a single occurrence is given its context. We investigate this question across two corpus-based studies of Spanish. In Study 1, we modeled age of acquisition (AoA) for 225 Spanish nouns using lexical frequency and contextual diversity from child-directed speech, plus surprisal from three language models differing in architecture and training language (BETO, BERTIN, mGPT). Frequency strongly predicted AoA (r=-.597, p<.001); surprisal added little beyond frequency and word length, including in a naturalistic-context analysis. In Study 2, we modeled adult fixation durations in the Chilean Spanish subsample of the Multilingual Eye-movement Corpus (MECO Wave 2), using mGPT surprisal alongside two independent frequency measures. Surprisal robustly predicted longer fixation durations after controlling for frequency and word length, consistent across both frequency sources. A matched word-type-level comparison showed the surprisal-behavior association was stronger in reading than in acquisition (z=3.63, p<.001). The findings suggest cumulative lexical exposure and contextual predictability play different roles across the language trajectory: frequency is particularly informative about when early lexical representations are acquired, whereas surprisal captures moment-to-moment processing difficulty in an already-established linguistic system. We discuss this pattern in relation to usage-based and entrenchment-based accounts of lexical development and to the evaluation of language models as models of human language behavior.