Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness
Authors: Zeyad Ahmed, Paul Sheridan, Michael McIsaac, Aitazaz A. Farooque
Organizations: SCHOOL OF MATHEMATICAL AND COMPUTATIONAL SCIENCES, UNIVERSITY OF PRINCE EDWARD ISLAND · CANADIAN CENTRE FOR CLIMATE CHANGE AND ADAPTATION, UNIVERSITY OF PRINCE EDWARD ISLAND · FACULTY OF SUSTAINABLE DESIGN ENGINEERING, UNIVERSITY OF PRINCE EDWARD ISLAND
TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks. This paper provides insights into TF-IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.
Figures & tables
Symbol / Expr.
Name (Abbr.)
Description
C
Collection
Generic symbol for a collection of d>0 documents composed of m>0 distinct terms.
d>0
Collection size
Number of documents in the collection.
m>0
Vocabulary size
Number of distinct terms out of which collection documents are composed.
ti ( 1≤i≤m )
i ’th term
Generic symbol for the i ’th term in the vocabulary.
dj ( 1≤j≤d )
j ’th document
Generic symbol for the j ’th document in the collection.
bij
Binary term frequency (BTF)
Indicator function defined as 1 if the i ’th term occurs in the j ’th document, and 0 otherwise.
Table 1: Notation for the bag-of-words model representation of a collection of documents.
Fig 1: Beta-binomial samples of target term ti counts in documents with fixed αi/α0i=0.3 and increasing precision α0i∈{0.1,1.0,10,100} . Each point represents the count of ti in a single document of length nj=100 .
Fig 2: Comparison of the approximate beta-binomial and penalized beta-binomial language models on a simulated document collection of size d=40 , with all documents having equal length nj=50(∀j) , and true parameter values αi=0.10 and α¬i=49.90 . The log-likelihood contour (upper left) and surface (upper right) plots for the beta-binomial approximation show a pronounced ridge structure. In contrast, the corresponding plots for the penalized beta-binomial approximation (bottom left and bottom right) show that the gamma penalty term breaks the ridge by regularizing the objective function. Gamma parameter values of μ=50 and σ2=4 were used to generate the plots. The choice σ2=4 was made to make the breaking of the ridge visually apparent. However, alternative values of σ2 produce the same qualitative effect, differing only in the steepness of the surface gradient.
Fig 3: Scatterplot of the relationship between the likelihood–ratio test score λi and the total TF–IDF weight of term ti across all collection documents. Point diameter is proportional to Dirichlet concentration parameter αi magnitude.
Fig 4: Empirical distributions of the fitted beta-binomial parameters αi and α¬i for the 20 Newsgroups dataset vocabulary, shown on a natural logarithmic scale. The pronounced concentrations of mass at αi≪1 and α¬i≫1 support the assumptions of Claim 1 .
TF–IDF
S(λij)
Document Class
Precision
Recall
F1
Precision
Recall
F1
alt.atheism
0.82
0.82
0.82
0.79
0.76
0.78
comp.graphics
0.61
0.76
0.68
0.74
0.69
0.71
comp.os.ms-windows.misc
0.86
0.10
0.17
0.84
0.45
0.58
comp.sys.ibm.pc.hardware
0.52
0.78
0.62
0.62
0.76
0.69
comp.sys.mac.hardware
0.73
0.85
0.78
0.86
0.75
0.80
Table 2: Multinomial Naive Bayes classification results on the 20 Newsgroups dataset using TF–IDF and S(λij) features.
Fig 5: Empirical distributions of the fitted beta-binomial parameters αi and α¬i for the R8 dataset vocabulary, shown on a natural logarithmic scale. The pronounced concentrations of mass at αi≪1 and α¬i≫1 support the assumptions of Claim 1 .
TF–IDF
S(λij)
Document Class
Precision
Recall
F1
Precision
Recall
F1
acq
0.98
0.91
0.94
0.95
0.99
0.97
crude
0.90
0.93
0.92
0.91
0.87
0.89
earn
0.95
0.97
0.96
0.99
0.98
0.98
grain
0.91
1.00
0.95
1.00
0.30
0.46
interest
0.87
0.74
0.80
0.98
0.52
0.68
Table 3: Multinomial Naive Bayes classification results on the R8 dataset using TF–IDF and S(λij) features.
Fig A1: Sensitivity of classification accuracy for a Naive Bayes classifier using S(λij) features, with respect to the hyperparameters μ∈{30,60,90,120,150,180} and σ2∈{0.5,1.0,2.0,3.0,4.0,5.0} , evaluated on the 20 Newsgroups dataset.
Fig A2: Sensitivity of classification accuracy for a Naive Bayes classifier using S(λij) features, with respect to the hyperparameters μ∈{30,60,90,120,150,180} and σ2∈{0.5,1.0,2.0,3.0,4.0,5.0} , evaluated on the R8 dataset.
Statistical hypothesis testing develops methods for assessing evidence against a stated null while controlling false-positive and false-negative rates.
TF-IDF and BM25 are two of the most widely used methods for scoring query-document relevance, yet neither has a standard probabilistic derivation that justifies it as a statistical method within a unified framework. We address this gap by showing that both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models. We treat the BM25 variant that includes the plus 1 correction in the IDF term, which is the one used in practice, and also discuss the original BM25 formulation without that correction. The resulting framework provides a common theoretical basis for TF-IDF and BM25, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods rather than only experimentally.
Measuring the breadth of a word's meaning, or its spread across contexts, has become feasible with contextualized token embeddings. A word type can be represented as a cloud of token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding "statistically significant" outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear algebra operations. Empirically, our alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23x speedup over the CPU baseline.
Flesch Reading Ease (FRE) and the Flesch-Kincaid Grade Level (FKGL) are widely used readability scores for English computed from the same two document statistics, yet their stability on long documents need not imply invariance to lexical composition. Surprisingly, under a topic model with an explicit sentence-boundary token, both scores converge almost surely to deterministic functions of the document topic distribution through just two scalar rates: in the long-text limit, all score variation is mediated by topical composition rather than any residual readability signal. The theory covers both formulae, while the experiments evaluate FKGL. In a fixed admixture with rank[1, q, s] = 3, fibres through interior topic vectors are locally (K-3)-dimensional, whereas regular iso-score level sets are locally (K-2)-dimensional and curved. In out-of-fold evaluation on two balanced corpora, Brown and the written BNC, a topic vector inferred from one document half's content words predicts the other half's FKGL at r = 0.779 and 0.884, respectively. On Brown, adding the topic prediction to genre and mean content-word syllable count yields ΔR2 = 0.002, with a confidence interval spanning zero; on the BNC, the corresponding split-half increment is 0.024, positive in four of five K = 100 fits (median 0.021). Because inferred topics may also absorb genre, register, and style, we do not interpret these results as evidence about human readability or causal effects.