The Effect of Document Selection on Query-focused Text Analysis

Sandesh S Rangreji; Mian Zhong; Anjalie Field

cs.IRApr 13, 2026

The Effect of Document Selection on Query-focused Text Analysis

Authors: Sandesh S Rangreji, Mian Zhong, Anjalie Field

Organizations: Johns Hopkins University

Abstract

Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.

Figures & tables

Method	TREC-COVID				Doctor-Reviews
Method	TopicGPT	BERTopic	LDA	HiCode	TopicGPT	BERTopic	LDA	HiCode
Random Uniform	.156 ±.035	.232 ±.030	.266 ±.043	.648 ±.115	.224 ±.077	.293 ±.079	.397 ±.106	.580 ±.051
Keyword Search	.326 ±.072	.325 ±.066	.575 ±.082	.679 ±.105	.341 ±.106	.404 ±.071	.543 ±.082	.628 ±.033
SBERT	.398 ±.078	.458 ±.028	.596 ±.065	.704 ±.083	.402 ±.097	.446 ±.055	.501 ±.081	.627 ±.044
Direct Retrieval	.435 ±.135	.407 ±.038	.606 ±.071	.676 ±.092	.324 ±.116	.471 ±.120	.577 ±.076	.624 ±.042
Dir. Retr. + MMR	.265 ±.065	.436 ±.119	.594 ±.044	.680 ±.088	.288 ±.055	.397 ±.064	.513 ±.101	.619 ±.045

Table 1: Average topic-query relevance scores (mean

\pm

std across queries) for all topic models and datasets. Bold indicates highest relevance in each column. Semantic retrieval methods consistently achieve higher relevance than lexical methods across TopicGPT, BERTopic, and (to a lesser extent) LDA. HiCode shows compressed ranges due to query-aware topic generation. Doctor-Reviews shows weaker differentiation and higher variance.

Figure 1: Information retrieval metrics across all 50 TREC-COVID queries showing precision@20 and recall@1,000. Hybrid fusion methods achieve the highest precision, while MMR diversity reranking substantially reduces precision despite maintaining recall.

Figure 2: Topic-query relevance versus overall semantic diversity for TopicGPT on TREC-COVID. Ellipses show mean $\pm$ 1 standard deviation across 15 queries. Methods cluster along a negative relationship: higher relevance associates with lower diversity. Random Uniform occupies the low-relevance, high-diversity quadrant; Direct Retrieval, Query Expansion, and SBERT occupy the high-relevance, low-diversity quadrant.

Panel of Figure 3 — Figure 3: Semantic diversity among query-relevant topics only (similarity $>$ 0.5) for TopicGPT (left) and BERTopic (right) on TREC-COVID. Error bars show $\pm$ 1 standard deviation across 15 queries. Both models show minimal variation across selection methods, contrasting with the large differences in overall diversity in Fig. 2 . This indicates diversity is preserved where it matters: among topics semantically related to the query.

Panel of Figure 4 — Figure 4: Pairwise relevant topic coverage at semantic similarity threshold 0.7 for TopicGPT (left) and BERTopic (right) on TREC-COVID. Cell (i,j) shows the fraction of method j’s relevant topics covered by method i. Reading guide: Dark row = comprehensive (covers many others’ topics). Light row = Low coverage. Dark column = redundant (widely covered by others). Light column = unique contribution.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Relevance-diversity trade-off on TREC-COVID (15 queries). Topic-query similarity versus overall semantic diversity for TopicGPT (top-left), BERTopic (top-right), LDA (bottom-left), and HiCode (bottom-right). All models show negative correlation: higher relevance → lower diversity. Random Uniform achieves highest diversity but lowest relevance. Semantic methods (SBERT, Direct Retrieval, Query Expansion) cluster in high-relevance, moderate-diversity region. TopicGPT and BERTopic show most pronounced trade-offs with large diversity reduction (TopicGPT: 51%, BERTopic: 49%). LDA shows steeper diversity reduction (49%) across narrow alignment range. HiCode shows compressed relevance range due to query-aware generation.

Panel of Figure 6 — Figure 6: Relevance-diversity trade-off on Doctor-Reviews (11 queries). TopicGPT (top-left), BERTopic (top-right), LDA (bottom-left), and HiCode (bottom-right) all show negative correlation but with attenuated differentiation compared to TREC-COVID. Higher variance and overlapping error bars reflect weaker query quality (App. F ). HiCode maintains compressed ranges consistent with query-aware generation.

Panel of Figure 7 — Figure 7: Diversity among query-relevant topics only on TREC-COVID (15 queries). TopicGPT (top-left), BERTopic (top-right), LDA (bottom-left), and HiCode (bottom-right). Error bars show $\pm$ 1 standard deviation. All models show overlapping values across methods: TopicGPT 0.416–0.476, BERTopic 0.386–0.484, LDA 0.447–0.481, HiCode 0.523–0.558. This demonstrates that while overall diversity decreases with alignment (Fig. 5 ), diversity is preserved among topics semantically related to the query. High overall diversity in Random Uniform primarily reflects off-topic content rather than richer query-relevant coverage.

Panel of Figure 8 — Figure 8: Diversity among query-relevant topics only on Doctor-Reviews (11 queries). All four models, TopicGPT (top-left), BERTopic (top-right), LDA (bottom-left), and HiCode (bottom-right), show overlapping ranges with minimal differentiation across selection methods. Pattern consistent with TREC-COVID (Fig. 7 ): retrieval preserves diversity among query-relevant topics while filtering off-topic content. Slightly higher variance reflects weaker query quality (App. F ).

Topic ID	Docs	Description & Top Words
Violence-Related Topics
T2*	270	Victimization : violence, domestic violence, mental health, women
T0*	27	Crime and Public Safety : crime, impact, cities, pandemics
T3*	35	Social Compliance : social distancing, behaviour, social norms
T6*	6	Cybercrime : cybercrime, internet, attacks, inflicted violence
Non-Violence Topics

Appendix

Table 2: Direct Retrieval topics for Query 43. *Relevant topics (similarity

\geq 0.5

). T2 merges domestic violence, mental health, and women’s issues into a single large topic.

Topic ID	Docs	Description & Top Words
Violence-Related Topics
T1*	145	Family Violence : violence, domestic violence, women, IPV
T0*	29	Crime and Justice : crime, justice, police, burglary, violence
T4*	11	Cybercrime : cybercrime, crimes, conspiracy, fraud, violence
T5*	3	Gun Violence : gun violence, firearms, Los Angeles, Chicago
Non-Violence Topics

Appendix

Table 3: Query Expansion topics for Query 43. *Relevant topics. Query Expansion uniquely captures gun violence despite “gun” not appearing in query.

Selection	TopicGPT	BERTopic	LDA	HiCode
Random Uniform	0.198	0.312	0.389	0.521
Keyword Search	0.267	0.401	0.421	0.558
SBERT	0.289	0.423	0.438	0.567
Direct Retrieval	0.304	0.448	0.447	0.573
DR + MMR	0.291	0.415	0.429	0.564
Query Expansion	0.298	0.431	0.441	0.569

Appendix

Table 4: Query 3 relevance scores by selection strategies. All methods show weak differentiation and near-random performance. Even top semantic methods (Direct Retrieval, Query Expansion) achieve only 0.289–0.573 alignment. Small ranges (TopicGPT: 0.106, BERTopic: 0.136, LDA: 0.058, HiCode: 0.052) indicate minimal impact of selection strategy when query quality is poor.

Panel of Figure 9 — Figure 9: Pairwise relevant topic coverage at threshold 0.7 on TREC-COVID (15 queries). The plots top-to- bottom left-to-right are TopicGPT, BERTopic, LDA, HiCode. Cell $(i,j)$ shows fraction of method $j$ ’s relevant topics covered by method $i$ ’s relevant topics. Semantic methods (SBERT, Direct Retrieval, Query Expansion) show dark rows (0.52–0.68 coverage), indicating convergence on similar query-relevant topics. Random Uniform shows very light rows (0.04–0.09), discovering predominantly off-topic content. LDA (bottom-left) shows substantially higher coverage across all methods (0.65–0.83 among semantic methods), reflecting structural convergence discussed in Sec. 6 . HiCode (bottom-right) shows high symmetric coverage (0.70–0.85) due to query-aware generation.

Panel of Figure 10 — Figure 10: Pairwise relevant topic coverage at threshold 0.7 on Doctor-Reviews (11 queries). Cell $(i,j)$ shows fraction of method $j$ ’s relevant topics covered by method $i$ . All four models, TopicGPT (top-left), BERTopic (top-right), LDA (bottom-left), and HiCode (bottom-right), show semantic methods with darker rows than Random Uniform, indicating convergence on similar relevant topics. Patterns consistent with TREC-COVID (Fig. 9 ) but with higher variance reflecting query quality heterogeneity (App. F ).

Topic ID	Docs	Description & Top Words
Violence-Related Topics (7 relevant)
T0*	48	Domestic Violence : domestic, violence, domestic violence, abuse, family
T9*	45	Violence Against Women : violence, women, IPV, intimate, partner
T2*	15	Crime Trends : crime, prison, burglary, prisons, frequency
T3*	13	Violence Against Health Workers : workers, kind warfare, healthcare workers, violence, law
T1*	5	Gun Violence : gun, guns, firearm, firearms, gun violence

Appendix

Table 5: SBERT topics for Query 43. *Relevant topics. SBERT uniquely separates domestic violence from violence against women, and discovers violence against health workers and youth violence exposure as distinct topics.

Topic ID	Docs	Description & Top Words
Violence-Related Topics (5 relevant)
T2*	106	Intimate Partner Violence : violence, domestic violence, IPV, women
T0*	36	Crime and Safety : crime, violence, police, burglary
T11*	16	Police Violence : violence, women, police violence, law
T5*	10	Cybercrime : cybercrime, cyber, internet, violence
T29*	6	Race and Violence : race, violence, African Americans, racism

Appendix

Table 6: Keyword Search topics for Query 43. *Relevant topics. Medical specialty topics resulted from “society” (query) matching medical society names (e.g., “surgical society”, “American Society of”). These

\sim

300 documents represent

\sim

30% of the sampled data.

ID	Query Text	TopicGPT	BERTopic	LDA	HiCode	Status
6	Patients’ dislikes about doctors	0.602	0.571	0.611	0.627	Strong
2	Specialist referral experiences	0.493	0.498	0.684	0.630	Strong
7	Asthma follow-up care recommendations	0.314	0.771	0.591	0.701	Strong
4	Asthma patient management	0.368	0.566	0.628	0.703	Moderate
8	Likes about treatment recommendations	0.528	0.486	0.592	0.635	Moderate
1	Finding and choosing doctors	0.462	0.457	0.648	0.649	Moderate

Appendix

Table 7: Per-query alignment analysis for Doctor-Reviews (11 queries). Alignment scores averaged across all seven selection methods (Random, Keyword Search, SBERT, Direct Retrieval, DR+MMR, Query Expansion, Retrieval Random) for each topic modeling framework. Query quality varies: strong queries (6, 2, 7) have specific, actionable focus (“dislikes”, “referrals”, “follow-up care”); moderate queries (4, 8, 1, 5, 9) balance specificity with scope; weak queries (11, 10, 3) use generic or compound phrasing and show lower alignment, particularly Query 3 which suffers from compound structure (“problems AND treatment”) and vague terminology. Full query texts can be found in App. A.2

Data Selection

Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.

See all 82 papers

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
Jul 26, 2026Madhav S BaidyaModel Selection Text Embedding Evaluation
Indian Institute of Technology (BHU) Varanasi
From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents
Apr 2, 2026Meftun Akarsu, Recep Kaan Karaman, Christopher MierbachRAG Evaluation Hybrid Retrieval
Technische Hochschule Ingolstadt · Uludag University · Radiate
Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget
Oct 24, 2025Xiyang Zhang, Hongzhi Wang, Yuanhe Tian +1Training Data Selection Data Valuation
Faculty of Computing, Harbin Institute of Technology

Jul 26, 2026J/K move · Enter open · S save

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework

Madhav S Baidya

Indian Institute of Technology (BHU) Varanasi

Model Selection Text Embedding Evaluation Text Embedding Models IR Evaluation

Open paper AskPDF arXiv

Choosing the right text embedding model is one of the most consequential -- and most frequently under-examined -- decisions in building a retrieval or search system, yet the model that tops a leaderboard is rarely the best choice for a given deployment. This report develops a practical, evidence-based framework for embedding model selection, built on a benchmarking study that evaluates T3EM (Text 3 Embedding Model), a commercial API-based embedding model, against a broad set of open-source alternatives on English-language retrieval tasks, and situates these findings within the wider Massive Text Embedding Benchmark (MTEB) landscape spanning classification, clustering, semantic similarity, reranking, pair classification, bitext mining, and summarization. Beyond raw benchmark scores, the report traces the full path from embedding model to retrieved result -- how embeddings are produced, how they are indexed and searched at scale, and how document chunking strategy shapes retrieval quality -- so that model choice can be reasoned about as one decision within a complete retrieval pipeline rather than in isolation. The result is a consolidated set of practical recommendations for selecting an embedding model according to task, latency, cost, and deployment constraints.

Abstract

Data Selection

Explore similar work

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework

From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents

Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget