Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models
Authors: Wentian Li, Oscar Fontanelli
Organizations: Department of Applied Mathematics and Statistics, Stony Brook University, Stony Brook, NY, USA · The Robert S. Boas Center for Genomics and Human Genetics, The Feinstein Institutes for Medical Research, Northwell Healths, Manhasset, NY, USA · Methods Lab, Facultad Latinoamericana de Ciencias Sociales, México
Stopwords and function words are relatively less informative for the content of a language and more often play a structural role in a sentence. Stopwords are ubiquitous words and may contain verbs, adjectives and adverbs. On the other hand, function words are strictly prepositions, conjunctions, pronouns, determiners, qualifiers, articles, interrogatives, and a limited number of auxiliary verbs. In contrast to the well known Zipf's law for rank-frequency plot for all words, the rank-frequency plots for stopwords or function words are best fitted by the Beta Rank Function (BRF). On the other hand, the rank-frequency plots of non-stopwords or non-function-words also deviate from the Zipf's law, but are better described by a quadratic function of log-token-count over log-rank than by BRF. Based on the observed rank of stopwords or function words in the full word list, we propose a stopword/function word/subset selection model that the probability for being selected, as a function of the word's rank r, is a decreasing Hill's function (1/(1+(r/rmid)γ)); whereas the probability for not being selected is the standard Hill's function (1/(1+(rmid/r)γ)). We validate this selection probability model by a direct estimation from an independent collection of texts. We also show analytically that this model leads to a BRF rank-frequency distribution for stopwords or function words when the original full word list follows the Zipf's law, as well as explaining the quadratic fitting function for the non-stopwords or non-function-words. A corollary of these results is that Zipf's law is not expected to be true for telegraphic speech in early childhood language learners or in agrammatism patients.
Figures & tables
Figure 1: Rank-frequency plot of all words (black) from Moby Dick (top row) and Brown corpus (bottom row), and stopwords (red) for NLTK (left column, n= 122 after removing the contracted words and/or their components), stopwords for spaCy (middle column, n= 302) lists, and 228 function words (right column). The fitting lines for all words are power-law (Zipf) function, and those for stopwords or function words (pink lines) are BRFs. Note that there are six plots for stopwords or function words because the number of combinations between SFW lists and source texts is 6.
Figure 2: Subset sampling model. A rank- r may be selected (subset selection) with a new rank rnew within the subset. (A) rnew/r as a function of r in 4 combinations of 2 text sources ( Moby Dick and Brown corpus) and 3 stopword/function-word lists. Six cumulative sums of the decreasing Hill function Eq.( 7 )) are shown that best fit the six sets of data. (B) Rank-frequency plot of a simulated dataset from the subset selection model. The red line is a BRF fitting function.
Figure 3: We use 30 books to estimate the probability of a rank- r word to be a stopword. Each dot shows the statistics of rank- r words: the y is the proportion of 30 rank- r words that are stopwords, and x is the rank r . Circles represent the geometric mean of dots with the same y value. A nonlinear function fitting of Eq.( 7 ) leads to the estimation of parameters: γ=1.78 and rmid=74.9 .
Figure 4: Rank-frequency plots for non-stopwords and non-function-words. Top (bottom) row is for Moby Dick ( Brown corpus ), and left (middle,right) column is for NLTK stopwords excluding contracted words (spaCy stopwords, function words). The fitting function by power-law (Zipf’s) function is in black, and those by quadratic function, log(T)∼−αlog(r)−κ(log(r))2 (Eq.( 4 )), are in red.
function
MD/common
MD/spaCy
MD/funW
Brown/common
Brown/spaCy
Brown/funW
Zipf
0.967
0.973
0.970
0.941
0.930
0.931
BRF
0.972
0.978
0.975
0.953
0.944
0.944
Mandelbrot
0.995
0.992
0.993
0.981
0.976
0.979
quadratic
0.998
0.997
0.998
0.994
0.993
0.995
Table 1: Adjusted R2 of four fitting functions of rank-frequency plot of NSFWs, on six combinations of text sources and SFW lists. The four fitting functions are power-law (Zipf’s law) Eq.( 3 ), Beta Rank Function (BRF) Eq.( 5 ), Mandelbrot function Eq.( 6 ), and quadratic regression Eq.( 4 ). The fitting is not carried out over all data points, but on a log-evenly-distributed points. The best performing function is marked with bold.
Figure 5: The r and r′new are the rank of a word that does not belong to a stopword or a function word list (NSFW) before and after stopwords/function-words are removed. (A) The ratio of these two ranks r′new/r for the six text source/SFW list combinations. The fitting lines are the cumulative sum from Eq.( 13 ) with the best fitting parameters. (B) Rank-frequency plot of an artificial dataset produced by picking a word whenever its r′new according to Eq.( 13 ) reaches a new integer. The three fitting lines are BRF (grey), Mandelbrot function (pink), and quadratic function (red).
Figure 6: Schematic illustration of two methods to produce a dataset with BRF starting from a Zipf’s law following dataset. The rank-frequency plot of the original dataset (grey line) is a straight line in log-log scale. (A) The subset selection method discussed in Sec. 3.3 . In this method, only a subset of the original dataset is selected. (B) The regressive modulation method discussed in ( Fontanelli and Li, 2026 ) . In this method, a sample’s value is reduced regressively (the lower the value, the more reduction).
Unsupervised term discovery involves segmenting unlabelled speech into word- or syllable-like units and clustering these into a lexicon of candidate types. True lexicons follow a Zipfian distribution, yet the dominant centre-based clustering approach -- K-means -- produces a more uniform distribution due to an inductive bias toward spherical clusters. In this paper we revisit graph-based clustering as a bottom-up alternative, where segment embeddings are connected by pairwise similarity and partitioned using the Leiden algorithm. We show that graph clustering substantially outperforms centre-based approaches (K-means, GMM, BIRCH) in both word- and syllable-level lexicon discovery across three languages, producing more Zipf-like distributions. Another bottom-up approach, agglomerative clustering with average linkage, also performs well, although it is computationally less efficient and allows for less control over the resulting distribution. Our work calls into question the dominance of centre-based clustering for term discovery, and promotes graph clustering as an attractive alternative.
Danel Slabbert, Simon Malan, Herman Kamper
Electrical and Electronic Engineering, Stellenbosch University, South Africa
We inspect the deductive connection between the neural scaling law and Zipf's law -- two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate of a foundation model -- such as a large language model -- changes with respect to the amount of training tokens, parameters, and compute. By contrast, Zipf's law posits that the distribution of tokens exhibits a power law tail. Whereas similar claims have been made in more specific settings, we show that the neural scaling law is a consequence of Zipf's law under certain broad assumptions that we reveal systematically. The derivation steps are as follows: We derive Heaps' law on the vocabulary growth from Zipf's law, Hilberg's hypothesis on the entropy scaling from Heaps' law, and the neural scaling from Hilberg's hypothesis. We illustrate these inference steps by a toy example of the Santa Fe process that satisfies all four statistical laws.
Łukasz Dębowski
Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland
We survey the theory of vocabulary growth founded in the setting of stochastic processes. In particular, we model the expected number of types through Bernstein functions and Hausdorff sequences. These classes of mathematical objects, defined by alternating signs of their derivatives or differences, can be related to continuous-time Poisson point processes and discrete-time IID processes, respectively. Building on previous accounts of the vocabulary growth, we integrate the broader theories of Bernstein functions and Hausdorff sequences and connect them with recently developed hapax rate models. In particular, we prove that the logistic hapax rate model has a non-negative spectrum and hence it defines a Bernstein function, thereby solving an earlier posed problem. We also analyze the limitations of the Bernstein--Hausdorff theory of the vocabulary growth by considering its generalizations under stationary and Weibull renewal processes.
Łukasz Dębowski
Institute of Computer Science, Polish Academy of Sciences ul. Jana Kazimierza 5, 01-248 Warszawa, Poland