Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness

Zeyad Ahmed; Paul Sheridan; Michael McIsaac; Aitazaz A. Farooque

cs.CLApr 1, 2026

Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness

Authors: Zeyad Ahmed, Paul Sheridan, Michael McIsaac, Aitazaz A. Farooque

Organizations: SCHOOL OF MATHEMATICAL AND COMPUTATIONAL SCIENCES, UNIVERSITY OF PRINCE EDWARD ISLAND · CANADIAN CENTRE FOR CLIMATE CHANGE AND ADAPTATION, UNIVERSITY OF PRINCE EDWARD ISLAND · FACULTY OF SUSTAINABLE DESIGN ENGINEERING, UNIVERSITY OF PRINCE EDWARD ISLAND

Abstract

TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks. This paper provides insights into TF-IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.

Figures & tables

Symbol / Expr.	Name (Abbr.)	Description
$\mathcal{C}$	Collection	Generic symbol for a collection of $d>0$ documents composed of $m~>~0$ distinct terms.
$d>0$	Collection size	Number of documents in the collection.
$m>0$	Vocabulary size	Number of distinct terms out of which collection documents are composed.
$t_{i}$ ( $1\leq i\leq m$ )	$i$ ’th term	Generic symbol for the $i$ ’th term in the vocabulary.
$d_{j}$ ( $1\leq j\leq d$ )	$j$ ’th document	Generic symbol for the $j$ ’th document in the collection.
$b_{ij}$	Binary term frequency (BTF)	Indicator function defined as $1$ if the $i$ ’th term occurs in the $j$ ’th document, and $0$ otherwise.

Table 1: Notation for the bag-of-words model representation of a collection of documents.

Fig 1: Beta-binomial samples of target term

t_{i}

counts in documents with fixed

\alpha_{i}/\alpha_{0i}=0.3

and increasing precision

\alpha_{0i}~\in~\{0.1,1.0,10,100\}

. Each point represents the count of

t_{i}

in a single document of length

n_{j}=100

Fig 2: Comparison of the approximate beta-binomial and penalized beta-binomial language models on a simulated document collection of size

d=40

, with all documents having equal length

n_{j}=50

(\forall j)

, and true parameter values

\alpha_{i}=0.10

and

\alpha_{\neg i}=49.90

. The log-likelihood contour (upper left) and surface (upper right) plots for the beta-binomial approximation show a pronounced ridge structure. In contrast, the corresponding plots for the penalized beta-binomial approximation (bottom left and bottom right) show that the gamma penalty term breaks the ridge by regularizing the objective function. Gamma parameter values of

\mu=50

and

\sigma^{2}=4

were used to generate the plots. The choice

\sigma^{2}=4

was made to make the breaking of the ridge visually apparent. However, alternative values of

\sigma^{2}

produce the same qualitative effect, differing only in the steepness of the surface gradient.

Fig 3: Scatterplot of the relationship between the likelihood–ratio test score

\lambda_{i}

and the total TF–IDF weight of term

t_{i}

across all collection documents. Point diameter is proportional to Dirichlet concentration parameter

\alpha_{i}

magnitude.

Fig 4: Empirical distributions of the fitted beta-binomial parameters

\alpha_{i}

and

\alpha_{\neg i}

for the 20 Newsgroups dataset vocabulary, shown on a natural logarithmic scale. The pronounced concentrations of mass at

\alpha_{i}\ll 1

and

\alpha_{\neg i}\gg 1

support the assumptions of Claim 1 .

	TF–IDF			$\boldsymbol{\mathcal{S}(\lambda_{ij})}$
Document Class	Precision	Recall	F1	Precision	Recall	F1
alt.atheism	0.82	0.82	0.82	0.79	0.76	0.78
comp.graphics	0.61	0.76	0.68	0.74	0.69	0.71
comp.os.ms-windows.misc	0.86	0.10	0.17	0.84	0.45	0.58
comp.sys.ibm.pc.hardware	0.52	0.78	0.62	0.62	0.76	0.69
comp.sys.mac.hardware	0.73	0.85	0.78	0.86	0.75	0.80

Table 2: Multinomial Naive Bayes classification results on the 20 Newsgroups dataset using TF–IDF and

\mathcal{S}(\lambda_{ij})

features.

Fig 5: Empirical distributions of the fitted beta-binomial parameters

\alpha_{i}

and

\alpha_{\neg i}

for the R8 dataset vocabulary, shown on a natural logarithmic scale. The pronounced concentrations of mass at

\alpha_{i}\ll 1

and

\alpha_{\neg i}\gg 1

support the assumptions of Claim 1 .

	TF–IDF			$\boldsymbol{\mathcal{S}(\lambda_{ij})}$
Document Class	Precision	Recall	F1	Precision	Recall	F1
acq	0.98	0.91	0.94	0.95	0.99	0.97
crude	0.90	0.93	0.92	0.91	0.87	0.89
earn	0.95	0.97	0.96	0.99	0.98	0.98
grain	0.91	1.00	0.95	1.00	0.30	0.46
interest	0.87	0.74	0.80	0.98	0.52	0.68

Table 3: Multinomial Naive Bayes classification results on the R8 dataset using TF–IDF and

\mathcal{S}(\lambda_{ij})

features.

Fig A1: Sensitivity of classification accuracy for a Naive Bayes classifier using

\mathcal{S}(\lambda_{ij})

features, with respect to the hyperparameters

\mu\in\{30,60,90,120,150,180\}

and

\sigma^{2}\in\{0.5,1.0,2.0,3.0,4.0,5.0\}

, evaluated on the 20 Newsgroups dataset.

Fig A2: Sensitivity of classification accuracy for a Naive Bayes classifier using

\mathcal{S}(\lambda_{ij})

features, with respect to the hyperparameters

\mu\in\{30,60,90,120,150,180\}

and

\sigma^{2}\in\{0.5,1.0,2.0,3.0,4.0,5.0\}

, evaluated on the R8 dataset.

Statistical Hypothesis Testing

Statistical hypothesis testing develops methods for assessing evidence against a stated null while controlling false-positive and false-negative rates.

See all 101 papers

TF-IDF and BM25 Are Exact KL Divergences
Sep 12, 2026Ivan SilajevKullback-Leibler Divergence Information Retrieval
Independent Researcher
Accurate and Efficient Statistical Testing for Word Semantic Breadth
May 8, 2026Yo EharaTwo-Sample Testing Permutation Testing
Tokyo Gakugei University
Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models
Aug 24, 2026Yo Ehara
Tokyo Gakugei University Tokyo, Japan

Sep 12, 2026J/K move · Enter open · S save

TF-IDF and BM25 Are Exact KL Divergences

Ivan Silajev

Independent Researcher

Kullback-Leibler Divergence Information Retrieval Lexical Retrieval

Open paper AskPDF arXiv

TF-IDF and BM25 are two of the most widely used methods for scoring query-document relevance, yet neither has a standard probabilistic derivation that justifies it as a statistical method within a unified framework. We address this gap by showing that both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models. We treat the BM25 variant that includes the plus 1 correction in the IDF term, which is the one used in practice, and also discuss the original BM25 formulation without that correction. The resulting framework provides a common theoretical basis for TF-IDF and BM25, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods rather than only experimentally.

Abstract

Statistical Hypothesis Testing

Explore similar work

TF-IDF and BM25 Are Exact KL Divergences

Accurate and Efficient Statistical Testing for Word Semantic Breadth

Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models