Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models

Wentian Li; Oscar Fontanelli

cs.CLMar 5, 2026

Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models

Authors: Wentian Li, Oscar Fontanelli

Organizations: Department of Applied Mathematics and Statistics, Stony Brook University, Stony Brook, NY, USA · The Robert S. Boas Center for Genomics and Human Genetics, The Feinstein Institutes for Medical Research, Northwell Healths, Manhasset, NY, USA · Methods Lab, Facultad Latinoamericana de Ciencias Sociales, México

Abstract

Stopwords and function words are relatively less informative for the content of a language and more often play a structural role in a sentence. Stopwords are ubiquitous words and may contain verbs, adjectives and adverbs. On the other hand, function words are strictly prepositions, conjunctions, pronouns, determiners, qualifiers, articles, interrogatives, and a limited number of auxiliary verbs. In contrast to the well known Zipf's law for rank-frequency plot for all words, the rank-frequency plots for stopwords or function words are best fitted by the Beta Rank Function (BRF). On the other hand, the rank-frequency plots of non-stopwords or non-function-words also deviate from the Zipf's law, but are better described by a quadratic function of log-token-count over log-rank than by BRF. Based on the observed rank of stopwords or function words in the full word list, we propose a stopword/function word/subset selection model that the probability for being selected, as a function of the word's rank $r$ , is a decreasing Hill's function ( $1/(1+(r/r_{mid})^γ)$ ); whereas the probability for not being selected is the standard Hill's function ( $1/(1+(r_{mid}/r)^γ)$ ). We validate this selection probability model by a direct estimation from an independent collection of texts. We also show analytically that this model leads to a BRF rank-frequency distribution for stopwords or function words when the original full word list follows the Zipf's law, as well as explaining the quadratic fitting function for the non-stopwords or non-function-words. A corollary of these results is that Zipf's law is not expected to be true for telegraphic speech in early childhood language learners or in agrammatism patients.

Figures & tables

Figure 1: Rank-frequency plot of all words (black) from Moby Dick (top row) and Brown corpus (bottom row), and stopwords (red) for NLTK (left column, $n=$ 122 after removing the contracted words and/or their components), stopwords for spaCy (middle column, $n=$ 302) lists, and 228 function words (right column). The fitting lines for all words are power-law (Zipf) function, and those for stopwords or function words (pink lines) are BRFs. Note that there are six plots for stopwords or function words because the number of combinations between SFW lists and source texts is 6.

Figure 2: Subset sampling model. A rank-

r

may be selected (subset selection) with a new rank

r_{new}

within the subset. (A)

r_{new}/r

as a function of

r

in 4 combinations of 2 text sources ( Moby Dick and Brown corpus) and 3 stopword/function-word lists. Six cumulative sums of the decreasing Hill function Eq.( 7 )) are shown that best fit the six sets of data. (B) Rank-frequency plot of a simulated dataset from the subset selection model. The red line is a BRF fitting function.

Figure 3: We use 30 books to estimate the probability of a rank-

r

word to be a stopword. Each dot shows the statistics of rank-

r

words: the

y

is the proportion of 30 rank-

r

words that are stopwords, and

x

is the rank

r

. Circles represent the geometric mean of dots with the same

y

value. A nonlinear function fitting of Eq.( 7 ) leads to the estimation of parameters:

\gamma=1.78

and

r_{mid}=74.9

Figure 4: Rank-frequency plots for non-stopwords and non-function-words. Top (bottom) row is for Moby Dick ( Brown corpus ), and left (middle,right) column is for NLTK stopwords excluding contracted words (spaCy stopwords, function words). The fitting function by power-law (Zipf’s) function is in black, and those by quadratic function, $\log(T)\sim-\alpha\log(r)-\kappa(\log(r))^{2}$ (Eq.( 4 )), are in red.

function	MD/common	MD/spaCy	MD/funW	Brown/common	Brown/spaCy	Brown/funW
Zipf	0.967	0.973	0.970	0.941	0.930	0.931
BRF	0.972	0.978	0.975	0.953	0.944	0.944
Mandelbrot	0.995	0.992	0.993	0.981	0.976	0.979
quadratic	0.998	0.997	0.998	0.994	0.993	0.995

Table 1: Adjusted

R^{2}

of four fitting functions of rank-frequency plot of NSFWs, on six combinations of text sources and SFW lists. The four fitting functions are power-law (Zipf’s law) Eq.( 3 ), Beta Rank Function (BRF) Eq.( 5 ), Mandelbrot function Eq.( 6 ), and quadratic regression Eq.( 4 ). The fitting is not carried out over all data points, but on a log-evenly-distributed points. The best performing function is marked with bold.

Figure 5: The $r$ and ${r^{\prime}}_{new}$ are the rank of a word that does not belong to a stopword or a function word list (NSFW) before and after stopwords/function-words are removed. (A) The ratio of these two ranks ${r^{\prime}}_{new}/r$ for the six text source/SFW list combinations. The fitting lines are the cumulative sum from Eq.( 13 ) with the best fitting parameters. (B) Rank-frequency plot of an artificial dataset produced by picking a word whenever its ${r^{\prime}}_{new}$ according to Eq.( 13 ) reaches a new integer. The three fitting lines are BRF (grey), Mandelbrot function (pink), and quadratic function (red).

Zipf's Law

See all 8 papers

Recovering the Zipfian Distribution in Unsupervised Term Discovery
Jun 9, 2026Danel Slabbert, Simon Malan, Herman KamperZipf's Law Clustering
Electrical and Electronic Engineering, Stellenbosch University, South Africa
From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
Dec 15, 2025Łukasz DębowskiZipf's Law Neural Scaling Laws
Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland
Vocabulary Growth Fundamentals: Bernstein Functions and Hausdorff Sequences
Aug 29, 2026Łukasz Dębowski
Institute of Computer Science, Polish Academy of Sciences ul. Jana Kazimierza 5, 01-248 Warszawa, Poland

Jun 9, 2026J/K move · Enter open · S save

Recovering the Zipfian Distribution in Unsupervised Term Discovery

Danel Slabbert, Simon Malan, Herman Kamper

Electrical and Electronic Engineering, Stellenbosch University, South Africa

Zipf's Law Clustering Hierarchical Clustering Graph Clustering

Open paper AskPDF arXiv

Unsupervised term discovery involves segmenting unlabelled speech into word- or syllable-like units and clustering these into a lexicon of candidate types. True lexicons follow a Zipfian distribution, yet the dominant centre-based clustering approach -- K-means -- produces a more uniform distribution due to an inductive bias toward spherical clusters. In this paper we revisit graph-based clustering as a bottom-up alternative, where segment embeddings are connected by pairwise similarity and partitioned using the Leiden algorithm. We show that graph clustering substantially outperforms centre-based approaches (K-means, GMM, BIRCH) in both word- and syllable-level lexicon discovery across three languages, producing more Zipf-like distributions. Another bottom-up approach, agglomerative clustering with average linkage, also performs well, although it is computationally less efficient and allows for less control over the resulting distribution. Our work calls into question the dominance of centre-based clustering for term discovery, and promotes graph clustering as an attractive alternative.

Abstract

Zipf's Law

Explore similar work

Recovering the Zipfian Distribution in Unsupervised Term Discovery

From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

Vocabulary Growth Fundamentals: Bernstein Functions and Hausdorff Sequences