RankEvolve: Automating the Discovery of Retrieval Algorithms via LLM-Driven Evolution

Jinming Nian; Fangchen Li; Dae Hoon Park; Yi Fang

cs.IRFeb 18, 2026

RankEvolve: Automating the Discovery of Retrieval Algorithms via LLM-Driven Evolution

Authors: Jinming Nian, Fangchen Li, Dae Hoon Park, Yi Fang

Organizations: Santa Clara University, Santa Clara, CA, USA · Independent Researcher, Bothell, WA, USA · Walmart Global Tech, Sunnyvale, CA, USA

Abstract

Retrieval algorithms like BM25 and query likelihood with Dirichlet smoothing remain strong and efficient first-stage rankers, yet improvements have mostly relied on parameter tuning and human intuition. We investigate whether a large language model, guided by an evaluator and evolutionary search, can automatically discover improved lexical retrieval algorithms. We introduce RankEvolve, a program evolution setup based on AlphaEvolve, in which candidate ranking algorithms are represented as executable code and iteratively mutated, recombined, and selected based on retrieval performance across 12 IR datasets from BEIR and BRIGHT. RankEvolve starts from two seed programs: BM25 and query likelihood with Dirichlet smoothing. The evolved algorithms are novel, effective, and show promising transfer to the full BEIR and BRIGHT benchmarks as well as TREC DL 19 and 20. Our results suggest that evaluator-guided LLM program evolution is a practical path towards automatic discovery of novel ranking algorithms.

Figures & tables

Figure 1. Combined score over evolution steps for two seed programs. The combined score is the optimization target, defined as

0.8\times\text{Avg Recall@100}+0.2\times\text{Avg nDCG@10}

, averaged across 12 IR datasets 2 2 2 BEIR: ArguAna, FiQA, NFCorpus, SciFact, SciDocs, TREC-COVID. BRIGHT: Biology, Earth Science, Economics, Pony, StackOverflow, TheoremQA. .

	BRIGHT		BEIR		TREC DL
Method	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100
BM25 ( Robertson and Zaragoza, 2009 )	10.55	32.11	48.16	70.95	62.16	46.02
BM25+ ( Lv and Zhai, 2011b )	9.72	31.54	45.71	69.93	61.07	46.69
BM25-adpt ( Lv and Zhai, 2011a )	11.05	34.67	44.62	69.55	60.90	48.23
BM25 ⋆	11.79 †	37.51 †	47.90	72.43 †	64.57 †	47.10
QL-Dir ( Zhai and Lafferty, 2004 )	9.52	32.48	44.15	68.72	57.03	46.69

Table 1. Macro-averaged results on unseen BRIGHT (6), BEIR (8), and TREC DL 19/20 datasets. Best per-group scores bolded . ⋆ Evolved with RankEvolve. † Significant over the respective seed (per-query paired

t

-test,

p<0.05

Figure 2. Evolution trajectories for the BM25 seed (left) and Dirichlet seed (right). Recall@100 improves nearly monotonically in both runs, while nDCG@10 occasionally regresses at the same steps, reflecting deliberate trades made by the evolutionary process to maximize the optimization target.

	BRIGHT		BEIR		TREC DL		Optimization Target
Code Structure	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	Seen (12 datasets)	Unseen (16 datasets)
Original (step 0)	10.55	32.11	48.16	70.95	62.16	46.02	40.30	49.77
Constrained (step 115/200)	11.36	34.68	48.27	70.75	61.86	45.44	42.87	50.47
Composable (step 185/200)	11.97	37.27	46.95	70.30	63.07	48.62	43.31	51.33
Freeform (step 177/200)	11.39	36.92	46.87	71.27	61.17	45.40	44.35	51.20
Freeform (step 293/300)	11.79	37.51	47.90	72.43	64.57	47.10	44.58	52.22

Table 2. Ablation on code structure design (BM25 family). Each row reports the best program found by RankEvolve under a given level of structural freedom; evolve step and total search budget are shown in parentheses. The optimization target is

0.8\times\text{R@100}+0.2\times\text{nDCG@10}

, macro-averaged over the 12 datasets used during evolution; unseen evaluation covers 16 held-out datasets (see Section 4.2 ).

Method	Indexing (ms/doc)	Query (ms/query)
BM25	1.79	56.72
BM25 ⋆ Constrained	1.85	58.50
BM25 ⋆ Composable	1.77	111.49
BM25 ⋆ Freeform (step 177)	2.37	171.52
BM25 ⋆ Freeform (step 293)	2.81	648.89
QL-Dir	2.02	178.26

Table 3. Average per-document indexing latency and per-query retrieval latency across all 28 datasets. Lowest values are bolded , second lowest are underlined . ⋆ Evolved with RankEvolve.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 454 papers

EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines
Aug 30, 2026Rayhan Patel, Shabaz PatelLearning to Rank Automated Algorithm Discovery
University of Maryland, College Park, MD, USA · Independent Researcher
RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models
Sep 30, 2026Zheng Chen, Linfeng Liu, Hong Li +1Learning to Rank LLM Agent Orchestration
Meta
Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
Oct 8, 2025Tiancheng Xing, Jerry Li, Yixuan Du +1Adversarial Attacks on LLMs LLM Reranking
National University of Singapore · University of Southern California · Georgetown University +1

Aug 30, 2026J/K move · Enter open · S save

EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines

Rayhan Patel, Shabaz Patel

University of Maryland, College Park, MD, USA · Independent Researcher

Learning to Rank Automated Algorithm Discovery Large Language Model-Guided Optimization Multi-Objective Evolutionary Optimization

Open paper AskPDF arXiv

We present EvoRank, an open autonomous ranking engineer: an LLM-guided evolutionary loop that discovers complete Learning-to-Rank pipelines (features, models, losses, ensembles) for multi-objective e-commerce search. On the Expedia ICDM 2013 dataset, with relevance, conversion, and revenue as competing objectives, three independent runs each converge within 50 iterations (about ten dollars) on interpretable pipelines that beat an Optuna-tuned LambdaMART on 60k held-out queries, an advantage that persists at full data scale and places in the top 6 percent of the original competition. A first campaign, evolving only training objectives, builds the central design rule: it appeared to work on its selection fold (the small dataset it uses to pick winners) while a transfer audit, re-scoring winners on held-out data, showed the gains were almost entirely fitness noise (the randomness of its own scoring), and neither seeded domain knowledge nor richer diagnostic feedback changed what transferred. The deciding quantity is measurable in advance: search-space headroom relative to fitness noise. We package this as a headroom gate that predicts, before any LLM spend, whether the loop will pay off, and we release the system, the auditing tools, and a catalog of failure modes with their guardrails, so teams can apply the procedure to their own ranking stacks.

Abstract

Information Retrieval

Explore similar work

EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization