Benchmarking Literature Retrieval for a Model Organism: A Dictyostelium Case Study

Yun Wang; Gad Shaulsky; Tomaž Curk; Blaž Zupan

cs.IROct 2, 2026

Benchmarking Literature Retrieval for a Model Organism: A Dictyostelium Case Study

Authors: Yun Wang, Gad Shaulsky, Tomaž Curk, Blaž Zupan

Organizations: University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, 1000 Ljubljana, Slovenia · Department of Molecular and Human Genetics, Baylor College of Medicine, Baylor Plaza, Houston, TX 77030, USA

Abstract

Biological literature retrieval systems are often developed and evaluated using broad biomedical corpora and general-purpose search tasks. However, many curated knowledge bases operate in narrower model-organism domains, where the literature is sparse and terminology is organism-specific. We introduce a retrieval benchmark from dictyBase for Dictyostelium, a model organism in cell and developmental biology. The benchmark consists of curator-generated biological queries linked to PubMed-indexed articles, together with structured gene annotations. Using this benchmark, we study three factors in niche biological retrieval: cross-encoder reranking, gene-aware query expansion, and abstract-only versus full-text retrieval. We report that reranking and gene-aware query expansion improve retrieval selectively: reranking is most useful when the model is well suited to biological evidence matching, whereas curated annotations help clarify compact biological queries by reducing vocabulary mismatch. Full-text chunks substantially improve retrieval when abstracts omit supporting evidence, increasing both candidate recall and top-rank performance, although these cases are harder than queries supported by abstracts. Data and code are publicly available at https://github.com/fulaibaowang/dictycite, and the benchmark dataset is additionally archived on Zenodo.

Figures & tables

Figure 1: Construction of a retrieval instance from a dictyBase gene page. (A) The source page for gene pkaC ( http://dictybase.org/gene/DDB_G0283907 ) contains structured gene annotations and curator-generated notes with inline citations. (B) A curator-generated claim is extracted as the query, and its inline citation is mapped to a PubMed-indexed article used as the cited relevance judgment. Auxiliary fields are used in later experiments: gene annotations support gene-aware query expansion, and the LLM-assisted evidence label supports the full-text analysis by marking whether the cited abstract is sufficient for the claim or whether full-text evidence may be needed.

Figure 2: Two-stage retrieval and reranking pipeline. Stage 1 builds a recall-oriented candidate pool by fusing BM25 and dense retrieval with RRF. Stage 2 reranks the fused candidates with a cross-encoder and combines the reranker order with the original retrieval-fusion order through a second RRF step to produce the final ranking.

System	Description	Model size	MRR@10	$\Delta$ vs. BM25
BM25	Lexical baseline	–	0.593	–
BM25+Dense	Lexical+dense RRF	–	0.568	$-2.6$ [ $-4.1$ , $-1.0$ ]
MiniLM-L12	Lightweight MS MARCO CE	33M	0.541	$-5.2$ [ $-7.0$ , $-3.4$ ]
BGE-m3	Strong multilingual CE	0.6B	0.605	$+1.2$ [ $-0.5$ , $+2.8$ ]
MedCPT	PubMed-trained biomedical CE	$\sim$ 110M	0.650	$+5.7$ [ $+4.2$ , $+7.2$ ]
BGE-Gemma	LLM-based CE reranker	3B	0.707	$+11.4$ [ $+9.8$ , $+13.0$ ]

Table 1: Ranker comparison on the full dataset (1,656 queries). Model size denotes the approximate number of parameters for neural rerankers; “–” indicates systems without a neural reranker. MRR@10 is the mean reciprocal rank at cutoff 10, where higher values indicate better top-rank performance.

\Delta

vs. BM25 reports the difference in MRR@10 from the BM25 baseline in percentage points; positive values indicate improvement over BM25, and brackets show 95% paired-bootstrap confidence intervals.

Figure 3: Effect of gene-aware query expansion. Three query variants are compared: the original query (the curator claim), the claim plus gene synonyms, and the claim plus both gene synonyms and product descriptions. (a) BM25 Recall@K, (b) dense Recall@K, and (c) mean MRR@K after BGE-m3 reranking, each plotted against the cutoff rank K on the x-axis. (d) Change in MRR@10 (y-axis, in MRR@10 units) relative to the original-query baseline for BGE-m3, MedCPT, and BGE-Gemma; the two bars per reranker are the two expansion variants, and positive values indicate that expansion improved the final ranking. Paired-bootstrap confidence intervals for the MRR@10 deltas are reported in Table S1 .

Figure 4: Retrieval and reranking by abstract-evidence level on the has-PDF subset. The upper row shows first-stage Recall@K from BM25+Dense RRF retrieval. The lower row shows post-rerank MRR@K using BGE-m3. Columns correspond to abstract supports detail (n=707), abstract supports core (n=585), and abstract insufficient (n=350). Solid lines show the abstract-only baseline; dashed lines show the corpus augmented with full-text chunks.

Figure S1: Per-query reranking effect relative to BM25. $\Delta$ MRR@10 is computed as reranker minus BM25 and grouped as hurt, helped, or near tie using thresholds of $\pm 0.025$ . Similar aggregate scores mask different query-level behavior: MiniLM-L12 is mostly harmful, BGE-m3 is balanced, and MedCPT and BGE-Gemma shift more queries toward positive deltas.

Reranker	Original	+ Synonyms	+ Synonyms and products
BGE-m3	0.576	0.623 ( $+4.7$ , [ $+2.6$ , $+6.8$ ])	0.622 ( $+4.6$ , [ $+2.2$ , $+7.1$ ])
MedCPT	0.624	0.682 ( $+5.8$ , [ $+3.7$ , $+7.9$ ])	0.673 ( $+5.0$ , [ $+2.7$ , $+7.2$ ])
BGE-Gemma	0.694	0.703 ( $+0.9$ , [ $-0.5$ , $+2.3$ ]) †	0.701 ( $+0.7$ , [ $-0.6$ , $+2.1$ ]) †

Table S1: Effect of gene-aware query expansion on the query-expansion subset. Each cell reports MRR@10. For expanded-query variants, parentheses report the paired-bootstrap difference from the original-query baseline for the same reranker, in percentage points, with 95% confidence intervals.

	has-PDF subset					full dataset
		R@1000		MRR@10			R@1000		MRR@10
Evidence level	$n$	Abs	+Chk	Abs	+Chk	$n$	Abs	+Chk	Abs	+Chk
Abstract supports detail	707	0.993	0.997	0.802	0.843	816	0.993	0.997	0.803	0.829
Abstract supports core	585	0.986	0.998	0.505	0.636	654	0.982	0.992	0.503	0.608
Abstract insufficient	350	0.784	0.972	0.161	0.385	387	0.788	0.949	0.164	0.357

Table S2: First-stage Recall@1000 and post-rerank MRR@10 (BGE-m3) by abstract-evidence level, shown for the has-PDF subset (Fig. 4 ) and for the full 1,656-query dataset with partial full-text coverage. Full-dataset gains are diluted because some cited PMIDs lack chunked-corpus coverage, but the qualitative pattern is unchanged.

Scientific Document Retrieval

See all 27 papers

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies
Jan 18, 2026Ming Zhang, Jiabao Zhuang, Wenqing Jing +15AI Agent Benchmarks Deep Research Agents
Fudan University · Hunyuan Team, Tencent
OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
Sep 8, 2026Xiao Yu Cindy Zhang, Wyeth Wasserman, Jian ZhuInformation Retrieval Biomedical IR
Graduate Program in Bioinformatics, The University of British Columbia · Centre for Molecular Medicine and Therapeutics, BC Children’s Hospital Research Institute · Department of Medical Genetics, The University of British Columbia +1
Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
May 4, 2026Devi Prasad Bal, Subhashree PuhanCross-Encoder Reranking Retrieval-Augmented Generation
The University of Texas at Austin

Jan 18, 2026J/K move · Enter open · S save

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan+14

Fudan University · Hunyuan Team, Tencent

AI Agent Benchmarks Deep Research Agents IR Evaluation Taxonomy Induction

Open paper AskPDF arXiv

Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize those papers into paper-grounded taxonomies. We introduce TaxoBench, a benchmark built from 72 highly cited LLM surveys, 3,815 cited papers, and their expert-authored taxonomies. TaxoBench evaluates systems in two settings: Deep Research mode measures end-to-end retrieval and organization from a topic, while Bottom-Up mode provides the expert paper set and isolates organization. We evaluate leaf-level assignments with ARI and V-Measure and hierarchy-level structure with US-TED, US-NTED, and Sem-Path. Across 7 Deep Research Agents and 16 LLM configurations, the best agent retrieves only 20.92% of expert-cited papers, and none of 70 standard Bottom-Up runs reaches the experts' average depth of 4.86. A controlled probe shows that models which match this depth do so by fragmenting the taxonomy, reducing alignment with the expert reference. We further find that raw Sem-Path remains near a no-organization floor even when a newer model generation gains 3.68 pp ARI; after depth matching, humans lead on all 10 matched surveys by 13.27 pp. These results identify retrieval and hierarchical organization as separate bottlenecks and show why hierarchy metrics must be calibrated before they are used to compare models.

Abstract

Scientific Document Retrieval

Explore similar work

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study