How Do LLM-Generated Texts Impact Term-Based Retrieval Models?

Wei Huang; Keping Bi; Yinqiong Cai; Wei Chen; Jiafeng Guo; Xueqi Cheng

cs.IRAug 25, 2025

How Do LLM-Generated Texts Impact Term-Based Retrieval Models?

Authors: Wei Huang, Keping Bi, Yinqiong Cai, Wei Chen, Jiafeng Guo, Xueqi Cheng

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China

Abstract

As more content generated by large language models (LLMs) floods into the Internet, information retrieval (IR) systems now face the challenge of distinguishing and handling a blend of human-authored and machine-generated texts. Recent studies suggest that neural retrievers may exhibit a preferential inclination toward LLM-generated content, while classic term-based retrievers like BM25 tend to favor human-written documents. This paper investigates the influence of LLM-generated content on term-based retrieval models, which are valued for their efficiency and robust generalization across domains. Our linguistic analysis reveals that LLM-generated texts exhibit smoother high-frequency and steeper low-frequency Zipf slopes, higher term specificity, and greater document-level diversity. These traits are aligned with LLMs being trained to optimize reader experience through diverse and precise expressions. Our study further explores whether term-based retrieval models demonstrate source bias, concluding that these models prioritize documents whose term distributions closely correspond to those of the queries, rather than displaying an inherent source bias. This work provides a foundation for understanding and addressing potential biases in term-based IR systems managing mixed-source content.

Figures & tables

Dataset	# Query	# Corpus	Avg. D/Q	Avg. Word Count
Dataset	# Query	# Corpus	Avg. D/Q	H-Q	L-Q	H-D	L-D 0.2	L-D 0.7	L-D 1.0
ANTIQUE	2,426	403,666	11.3	9.2	10.0	40.2	41.2	42.0	43.1
FiQA-2018	648	57,450	2.6	10.8	12.7	133.2	105.6	106.0	106.8
MS MARCO	6,979	542,203	1.1	5.9	6.9	58.1	57.3	58.1	59.1
NQ	3,446	104,194	1.2	9.2	9.9	86.9	83.4	84.0	84.9
HotpotQA	7,405	111,107	2.0	15.7	16.0	67.9	69.4	69.8	70.6

Table 1. Statistics of all 8 datasets. Avg. D/Q denotes the average number of relevant documents per query. H-Q/L-Q and H-D/L-D represent human/LLM-written queries and documents, respectively. L-D values are reported under different temperature settings (0.2, 0.7, 1.0).

Figure 1. Multi-level interpretation of Zipf’s law: from statistical patterns in core and extended vocabulary, to functional implications for communication efficiency and expression richness, to the principle of least effort as a balance of speaker’s and listener’s economy. A conceptual diagram illustrating a multi-level interpretation of Zipf’s law. The top section presents statistical patterns, distinguishing core vocabulary and extended vocabulary, with a schematic Zipf curve showing two slopes. The middle section shows functional implications, linking communication efficiency and expression richness. The bottom section depicts theoretical foundations based on the principle of least effort, balancing speaker’s economy and listener’s economy, and indicating an equilibrium point. Arrows indicate relationships across the three levels.

Dataset	Human		Llama 3
Dataset	$\alpha_{1}$	$\alpha_{2}$	$\alpha_{1}$	$\alpha_{2}$
ANTIQUE	1.0643	1.6479	0.9516 $\downarrow$	1.8511 $\uparrow$
FiQA-2018	1.0726	1.7473	0.9766 $\downarrow$	1.9049 $\uparrow$
MS MARCO	0.8873	1.6531	0.8454 $\downarrow$	1.7722 $\uparrow$
NQ	0.8657	1.4902	0.8278 $\downarrow$	1.5493 $\uparrow$
HotpotQA	0.9211	1.4231	0.8964 $\downarrow$	1.4793 $\uparrow$

Table 2. Zipf’s law slopes for human and Llama 3 texts (

\alpha_{1}

: core vocabulary,

\alpha_{2}

: extended vocabulary). An upward arrow (

\uparrow

) indicates the value for Llama 3 is larger than for the human corpus, while a downward arrow (

\downarrow

) indicates it is smaller.

Dataset	Llama 2		Qwen2.5
Dataset	$\alpha_{1}$	$\alpha_{2}$	$\alpha_{1}$	$\alpha_{2}$
ANTIQUE	1.0378 $\downarrow$	1.9683 $\uparrow$	0.9233 $\downarrow$	1.9183 $\uparrow$
MS MARCO	0.8622 $\downarrow$	1.7670 $\uparrow$	0.8359 $\downarrow$	1.7885 $\uparrow$
SCIDOCS	0.9076 $\uparrow$	1.8071 $\uparrow$	0.8937 $\uparrow$	1.8082 $\uparrow$

Table 3. Zipf’s law slopes for Llama 2 and Qwen2.5. Arrows indicate the change relative to the corresponding human values reported in Table 2 (

\uparrow

increase,

\downarrow

decrease)

Figure 2. Impact of sampling temperature on the two Zipf’s-law exponents. (a) Core-vocabulary slope

\alpha_{1}

; (b) extended-vocabulary slope

\alpha_{2}

. Two grouped bar charts comparing Zipf’s-law exponents across datasets and sampling temperatures. The left subplot shows the core-vocabulary exponent

\alpha_{1}

and the right subplot shows the extended-vocabulary exponent

\alpha_{2}

. Bars correspond to human queries and LLM-generated queries with different sampling temperatures. The horizontal axis lists the datasets (ANTIQUE, MS MARCO, and SCIDOCS), and the vertical axis shows the estimated exponent values.

Figure 3. IDF distributions in ANTIQUE. (a) Core vocabulary and (b) extended vocabulary. Histograms represent distributions (blue: Human, red: LLM); vertical dashed lines show respective mean values. Two histogram plots showing inverse document frequency (IDF) distributions in the ANTIQUE dataset. The left subplot presents the core vocabulary and the right subplot presents the extended vocabulary. Blue bars correspond to human queries and red bars correspond to LLM-generated queries. The horizontal axis shows IDF values and the vertical axis shows probability. Vertical dashed lines indicate the mean IDF value for each query type.

Figure 4. Human vs. LLM lexical diversity. (a) Average document-level TTR values. (b) Average WordNet-based synonym cluster sizes. Two grouped bar charts comparing lexical diversity between human and LLM queries across three datasets. The left subplot shows average document-level type–token ratio (TTR) values, and the right subplot shows average WordNet-based synonym cluster sizes. Blue bars represent human queries and red bars represent LLM-generated queries. The horizontal axis lists the datasets (ANTIQUE, MS MARCO, and SCIDOCS), and the vertical axis shows the corresponding average values.

Rank	Source	Relevance	Source-based	Relevance-based
1	Human	Unknown	+1	0
2	Human	Unknown	+1	0
3	LLM	Relevant	+1	+0.50
4	LLM	Relevant	+1	+0.43
5	Human	Irrelevant	+1	0
6	Human	Relevant	+1	+0.36

Table 4. Example ranking scenario demonstrating misalignment between source distribution in ranking results and relevance-based measurement which uses Discounted Cumulative Gain (DCG) calculations. Despite humans occupying 6 of 10 positions, relevance-based metrics misleadingly suggest an LLM advantage. Blue cells indicate Human contribution, red cells indicate LLM contribution.

Dataset	Query Type	$\Delta$ MASR
Dataset	Query Type	TF-IDF	BM25	QL	DFR
ANTIQUE	Human	0.381*	0.366*	0.419*	0.376*
ANTIQUE	LLM	0.018*	0.014*	0.040*	0.008
FiQA	Human	0.069*	0.179*	0.196*	0.082*
FiQA	LLM	-0.110*	0.012	0.015	-0.099*
MS MARCO	Human	0.361*	0.346*	0.342*	0.351*

Table 5. Source preference results based on MASR across different datasets and retrieval methods (top 200 per query).

\Delta

MASR is computed using the normalized difference defined in Equation 13 . Blue cells indicate preference towards human-written documents, while red cells indicate preference towards LLM-generated content. * indicates statistical significance (

p<0.05

Figure 5. KL-divergence differences (

\text{KL}_{\text{LLaMA}}-\text{KL}_{\text{human}}

) versus

\Delta

MASR for four retrieval methods: (a) TF-IDF (b) BM25, (c) QL, and (d) DFR. Blue and red points represent human and LLM queries respectively, with each shape denoting a different dataset. Four scatter plots comparing the relationship between KL-divergence difference and changes in MASR for different retrieval methods. Each subplot corresponds to a retrieval model: TF-IDF, BM25, QL, and DFR. The horizontal axis shows the KL-divergence difference between LLaMA-generated and human-written documents, and the vertical axis shows the change in MASR. Blue points represent human queries and red points represent LLM-generated queries, while different marker shapes indicate different datasets.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Official Website Link	License
ANTIQUE	https://ciir.cs.umass.edu/downloads/Antique/	Research-only use (no formal license)
FiQA-2018	https://sites.google.com/view/fiqa/	Research-only use (no formal license)
MS MARCO	https://microsoft.github.io/msmarco/	MIT License
NQ	https://ai.google.com/research/NaturalQuestions	CC BY-SA 3.0 license
HotpotQA	https://hotpotqa.github.io	CC BY-SA 4.0 license
FEVER	http://fever.ai	CC BY-SA 3.0 license

Appendix

Table 6. Website links and licenses for the benchmarked datasets used in this paper.

Dataset	# Query	# Corpus	Avg. D/Q	Avg. Word Count
Dataset	# Query	# Corpus	Avg. D/Q	H-Q	L-Q	H-D	L-D Temp0.2	L-D Temp0.7	L-D Temp1.0
ANTIQUE	2426	403,666	11.3	9.2	10.0	40.2	41.2	42.0	43.1
FiQA-2018	648	57,450	2.6	10.8	12.7	133.2	105.6	106.0	106.8
MS MARCO	6,979	542,203	1.1	5.9	6.9	58.1	57.3	58.1	59.1
NQ	3,446	104,194	1.2	9.2	9.9	86.9	83.4	84.0	84.9
HotpotQA	7,405	111,107	2.0	15.7	16.0	67.9	69.4	69.8	70.6

Appendix

Table 7. Statistics of all 8 datasets. Avg. D/Q denotes the average number of relevant documents per query. H/L-Q and H/L-D represent human/LLM-written queries and documents respectively, with L-D under different temperature settings (0.2/0.7/1.0).

Domain	# Questions	# Human Answers	# ChatGPT Answers	Source
reddit_eli5	17,112	51,336	16,660	ELI5 dataset ( Fan et al., 2019 )
open_qa	1,187	1,187	3,561	WikiQA dataset ( Yang et al., 2015 )
wiki_csai	842	842	842	Crawled Wikipedia
medicine	1,248	1,248	1,337	Medical Dialog dataset ( Zeng et al., 2020 )
finance	3,933	3,933	4,503	FiQA dataset ( Maia et al., 2018 )
Total	24,322	58,546	26,903	—

Appendix

Table 8. Distribution of questions and answers in the HC3-English corpus, including data sources (from Guo et al., 2023).

Dataset	Jaccard Similarity			Overlap Similarity
Dataset	T=0.2	T=0.7	T=1.0	T=0.2	T=0.7	T=1.0
ANTIQUE	0.325 ( $\pm$ 0.107)	0.312 ( $\pm$ 0.101)	0.297 ( $\pm$ 0.094)	0.518 ( $\pm$ 0.158)	0.506 ( $\pm$ 0.157)	0.493 ( $\pm$ 0.157)
FiQA	0.374 ( $\pm$ 0.093)	0.357 ( $\pm$ 0.088)	0.335 ( $\pm$ 0.082)	0.534 ( $\pm$ 0.107)	0.519 ( $\pm$ 0.105)	0.500 ( $\pm$ 0.104)
MS MARCO	0.450 ( $\pm$ 0.111)	0.432 ( $\pm$ 0.107)	0.410 ( $\pm$ 0.104)	0.643 ( $\pm$ 0.122)	0.632 ( $\pm$ 0.122)	0.616 ( $\pm$ 0.122)
NQ	0.494 ( $\pm$ 0.109)	0.472 ( $\pm$ 0.106)	0.445 ( $\pm$ 0.103)	0.663 ( $\pm$ 0.109)	0.648 ( $\pm$ 0.110)	0.628 ( $\pm$ 0.112)
HotpotQA	0.580 ( $\pm$ 0.104)	0.561 ( $\pm$ 0.103)	0.536 ( $\pm$ 0.102)	0.763 ( $\pm$ 0.099)	0.751 ( $\pm$ 0.101)	0.736 ( $\pm$ 0.103)

Appendix

Table 9. Jaccard and Overlap similarity (mean

\pm

std) across datasets and LLM temperatures.

Dataset	Matching Pairs			Random Pairs
Dataset	T=0.2	T=0.7	T=1.0	T=0.2	T=0.7	T=1.0
ANTIQUE	0.936 ( $\pm$ 0.029)	0.933 ( $\pm$ 0.029)	0.930 ( $\pm$ 0.029)	0.704 ( $\pm$ 0.027)	0.704 ( $\pm$ 0.027)	0.704 ( $\pm$ 0.027)
FiQA	0.960 ( $\pm$ 0.019)	0.958 ( $\pm$ 0.020)	0.955 ( $\pm$ 0.021)	0.721 ( $\pm$ 0.032)	0.722 ( $\pm$ 0.032)	0.722 ( $\pm$ 0.032)
MS MARCO	0.970 ( $\pm$ 0.021)	0.969 ( $\pm$ 0.022)	0.966 ( $\pm$ 0.022)	0.675 ( $\pm$ 0.028)	0.675 ( $\pm$ 0.028)	0.675 ( $\pm$ 0.028)
NQ	0.977 ( $\pm$ 0.018)	0.976 ( $\pm$ 0.019)	0.973 ( $\pm$ 0.019)	0.698 ( $\pm$ 0.028)	0.698 ( $\pm$ 0.028)	0.699 ( $\pm$ 0.028)
HotpotQA	0.981 ( $\pm$ 0.016)	0.979 ( $\pm$ 0.016)	0.977 ( $\pm$ 0.018)	0.705 ( $\pm$ 0.028)	0.705 ( $\pm$ 0.028)	0.705 ( $\pm$ 0.028)

Appendix

Table 10. Cosine similarity (mean

\pm

std) for matching and random pairs across datasets and LLM temperatures.

Dataset

Temp.

\Delta

P@10

\Delta

NDCG@10

\Delta

MAP

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

ANTIQUE

0.2

0.629*

0.530*

0.634*

0.604*

0.696*

0.601*

0.719*

0.678*

0.730*

0.627*

0.736*

0.713*

0.7

0.630*

0.523*

0.635*

0.623*

0.696*

0.591*

0.690*

0.687*

0.747*

0.640*

0.746*

0.738*

1.0

0.700*

0.574*

0.698*

0.693*

0.769*

0.650*

0.757*

0.759*

0.792*

0.696*

0.790*

0.787*

FiQA-2018

0.2

0.082

0.168*

0.172*

0.109*

0.091*

0.223*

0.243*

0.112*

0.289*

0.287*

0.133*

0.7

0.104

0.188*

0.180*

0.151*

0.165*

0.286*

0.265*

0.168*

0.178*

0.318*

0.312*

0.160*

Appendix

Table 11. Preference analysis results across different datasets, temperatures, and retrieval methods. Temperature (Temp.) refers to the sampling temperature used for generating LLM-generated documents. For all retrieval methods, we retrieved the top 200 documents. blue cells indicate preference towards human-written documents, while red cells indicate preference towards LLM-generated content.

\Delta

P@10,

\Delta

NDCG@10, and

\Delta

MAP represent the preference metrics based on Precision@10, NDCG@10, and MAP respectively. * indicates statistical significance (

p<0.05

Range	Largest Frequency Increases	Largest Frequency Decreases
Top 2000	situated, significant, individuals, characterized, typically	important, biggest, mostly, answer, say
Top 2000-10%	additionally, crucial, furthermore, notably, boasts	wanted, sometime, knows, dies, sort
10%-20%	interestingly, noting, encompassing, staggering, firstly	10000, u.k, catalyzes, tremendous, practically
20%-50%	rewritten, boasting, commences, showcasing, delve	dredge, manmade, noncoding, vä, o.s
50%-100%	paraphrased, necessitate, chilliest, necessitates, thirdly	humiliated, cubicle, pov, karats, elites

Appendix

Table 12. Words with the largest relative frequency changes between human-written and LLM-generated texts in the MS MARCO dataset.

Figure 7. Distribution of word usage in the “important” synonym cluster for human-written and LLM-generated texts.

Symbol	Description
$V$	vocabulary.
$\mathcal{Q}=\{q_{i}\}$	Query set, where $q\sim\mathrm{Unif}(\mathcal{Q})$ .
$\mathcal{D}=\{d_{i}\}$	Document set, where $d\sim\mathrm{Unif}(\mathcal{D})$ .
$\|q\|,\|d\|$	Length of a query / document.
$L_{q},L_{d}$	Average lengths: $L_{q}=\mathbb{E}_{q}[\|q\|],L_{d}=\mathbb{E}_{d}[\|d\|]$ .
$\mathrm{tf}(w,q),\mathrm{tf}(w,d)$	Term frequencies. The expected values are: $\mathbb{E}_{q}[\mathrm{tf}(w,q)]=L_{q}P_{Q}(w)$ , $\mathbb{E}_{d}[\mathrm{tf}(w,d)]=L_{d}P_{D}(w)$ .

Appendix

Table 13. Description of notations.

Dataset

Query Type

\Delta

SR@10

\Delta

NDSR@10

\Delta

MASR

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

ANTIQUE

Human

0.428*

0.382*

0.450*

0.424*

0.460*

0.414*

0.474*

0.456*

0.376*

0.349*

0.412*

0.371*

LLM

0.008

-0.025*

-0.006

-0.005

0.014

-0.025*

-0.013

-0.001

0.035*

0.011

0.039*

0.024*

FiQA

Human

0.080*

0.201*

0.183*

0.096*

0.090*

0.217*

0.188*

0.101*

0.072*

0.172*

0.179*

0.084*

LLM

-0.153*

-0.018

-0.079*

-0.151*

-0.160*

-0.022

-0.086*

-0.156*

-0.113*

-0.004

-0.027*

-0.104*

MS MARCO

Human

0.411*

0.390*

0.364*

0.396*

0.449*

0.428*

0.396*

0.434*

0.336*

0.322*

0.310*

0.327*

Appendix

Table 14. Preference analysis results across different datasets and retrieval methods Porter stemming and stopwords removal (top 200 per query).

\Delta

SR@10,

\Delta

NDSR@10, and

\Delta

MASR represent the preference metrics based on SR@10, NDSR@10, and MASR respectively. blue cells indicate preference towards human-written documents, while red cells indicate preference towards LLM-generated content. * indicates statistical significance (

p<0.05

Dataset

Query Type

\Delta

SR@10

\Delta

NDSR@10

\Delta

MASR

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

TF-IDF

BM25

DFR

ANTIQUE

Human

0.436*

0.400*

0.449*

0.427*

0.463*

0.421*

0.469*

0.454*

0.381*

0.366*

0.419*

0.376*

LLM

-0.017

-0.028*

-0.018

-0.031*

-0.015

-0.033*

-0.028*

-0.030*

0.018*

0.014*

0.040*

0.008

FiQA

Human

0.053*

0.190*

0.195*

0.062*

0.059*

0.201*

0.197*

0.069*

0.179*

0.196*

0.082*

LLM

-0.173*

-0.027

-0.039*

-0.165*

-0.184*

-0.034*

-0.054*

-0.174*

-0.110*

0.012

0.015

-0.099*

MS MARCO

Human

0.433*

0.408*

0.391*

0.418*

0.464*

0.440*

0.413*

0.450*

0.361*

0.346*

0.342*

0.351*

Appendix

Table 15. Preference analysis results across different datasets and retrieval methods without stemming and keeping stopwords (top 200 per query).

\Delta

SR@10,

\Delta

NDSR@10, and

\Delta

p<0.05

Figure 8. KL-divergence differences (

\text{KL}_{\text{LLaMA}}-\text{KL}_{\text{human}}

) versus

\Delta

MASR with Porter stemming and stopwords removal for four retrieval methods: (a) TF-IDF, (b) BM25, (c) QL, and (d) DFR. Blue and red points represent human and LLM queries respectively, with each shape denoting a different dataset.

Figure 9. KL-divergence differences versus preference metrics with Porter stemming and stopwords removal for four retrieval methods. Rows correspond to

\Delta

SR@10,

\Delta

NDSR@10, and

\Delta

MASR; columns correspond to TF-IDF, BM25, QL, and DFR. Blue points represent human queries, red points represent LLM queries, with each point representing a dataset.

Figure 10. KL-divergence differences versus preference metrics without stemming and keeping stopwords for four retrieval methods. Rows correspond to

\Delta

SR@10,

\Delta

NDSR@10, and

\Delta

MASR; columns correspond to TF-IDF, BM25, QL, and DFR. Blue points represent human queries, red points represent LLM queries, with each point representing a dataset.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 471 papers

In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations
Feb 17, 2026Mohammad Aflah Khan, Mahsa Amani, Soumi Das +5Language Model Bias Evaluation
Max Planck Institute for Software Systems · Microsoft, Hyderabad
Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation
Apr 17, 2026Nicolò Pagan, Christopher Barrie, Chris Andrew Bail +1LLM Prompting Social Bias in Language Models
University of Zurich, Department of Informatics, Zurich, CH · New York University, Department of Sociology, New York, USA · University of Oxford, Department of Sociology, Oxford, UK +2
Scientific Knowledge Discovery in the Age of Large Language Models
Jul 29, 2026Eleni Adamidi, Serafeim Chatzopoulos, Thanasis VergoulisAI-Assisted Scientific Research Information Retrieval
IMSI, ATHENA RC

Feb 17, 2026J/K move · Enter open · S save

In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations

Mohammad Aflah Khan, Mahsa Amani, Soumi Das, Bishwamittra Ghosh+4

Max Planck Institute for Software Systems · Microsoft, Hyderabad

Language Model Bias Evaluation

Open paper AskPDF arXiv

Agents based on Large Language Models (LLMs) are increasingly being deployed as interfaces to information on online platforms. These agents filter, prioritize, and synthesize information retrieved from the platforms' back-end databases or via web search. In these scenarios, LLM agents govern the information users receive, by drawing users' attention to particular instances of retrieved information at the expense of others. While much prior work has focused on biases in the information LLMs themselves generate, less attention has been paid to the factors that influence what information LLMs select and present to users. We hypothesize that when information is attributed to specific sources (e.g., particular publishers, journals, or platforms), current LLMs exhibit systematic latent source preferences- that is, they prioritize information from some sources over others. Through controlled experiments on twelve LLMs from six model providers, spanning both synthetic and real-world tasks, we find that several models consistently exhibit strong and predictable source preferences. These preferences are sensitive to contextual framing, can outweigh the influence of content itself, and persist despite explicit prompting to avoid them. They also help explain phenomena such as the observed left-leaning skew in news recommendations in prior work. Our findings advocate for deeper investigation into the origins of these preferences, as well as for mechanisms that provide users with transparency and control over the biases guiding LLM-powered agents.

Abstract

Information Retrieval

Explore similar work

In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations

Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

Scientific Knowledge Discovery in the Age of Large Language Models