RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Zheng Chen; Linfeng Liu; Hong Li; Hong Yan

cs.AISep 30, 2026

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Authors: Zheng Chen, Linfeng Liu, Hong Li, Hong Yan

Organizations: Meta

Abstract

Auto-research agents, LLM systems that propose, implement, train, and evaluate model changes across iterations, promise to automate applied ML's experimental loop. Over long horizons, execution accuracy is a binding constraint: a change can silently leak held-out data, omit normalization, disconnect a gradient, or leave a train/eval flag unwired, invalidating expensive runs and compounding error across iterations. We present RankEvolve, an auto-research framework for evolving generative ranking models. An Executable Operating Protocol (EOP) declares phases, gates, branches, and loops, and the runtime enforces the compiled state machine. A meta-meta-harness composes complete black-box coding-agent products, including Claude Code and Codex, as execution-graph nodes that review and repair one another's work. In a budget-matched evaluation, heterogeneous composition raises all-oracle execution accuracy from the best single-product baseline of 45.8 percent to 62.5 percent (paired +16.7 points, 95 percent CI [6.6, 26.7]) while achieving a 10.4 percent silent critical-defect rate. An implemented knowledge layer carries findings, including negative results, across iterations. In a twelve-iteration deployment on the open-source HSTU recommender, RankEvolve reported NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48 percent over the published anchor) and 0.1948 on BASE (+2.80 percent). ExecML-HSTU, seeded by incidents from that deployment, provides the oracle benchmark for the execution-accuracy evaluation. A pre-specified LitGPT transfer split replicates the heterogeneous-composition effect beyond recommendation (+12.5 points, 95 percent CI [3.0, 22.0]), and a paired ablation isolates per-step from full-protocol instruction injection. These results characterize when runtime-controlled composition of coding-agent products improves execution accuracy.

Figures & tables

Panel of Figure 1 — Figure 1: Anatomy of RankEvolve : two orthogonal axes. Horizontally , the EOP drives the long horizon—a compiled phase sequence whose promoted results loop back for the next iteration (Section 2.1 ). Vertically , the meta-meta-harness supplies depth and diversity: coding and review nodes bind complete products, while routing, aggregation, and gates remain deterministic; a persistent state/artifact store spans the run. The diagram describes the deployed integration boundary, not a novel graph formalism.

Panel of Figure 2 — Figure 2: Per-iteration process-faithfulness violations across the ten full-context playbook runs (no runtime enforcement). Deviations are rare in iteration 1 and proliferate from iteration 2 as accumulated context crowds out the current-step instruction. Runs reaching each iteration: $n{=}10,10,7$ ; a point counts runs exhibiting that violation at that iteration, so a run may contribute to several (per-run totals in Table 10 ). The pilot lacks a runtime-controlled arm, so it motivates rather than measures the EOP’s effect; the controlled test is Section 4.3 .

Panel of Figure 3 — Figure 3: Execution patterns instantiated by the runtime. Circles denote work nodes, which may bind to a coding-agent product, a deterministic function, or a narrow model call. Disabling the diamond’s aggregator yields a tree; disabling its breakdown yields a chain. These topologies are standard; our object of study is the product-agent adapter boundary and its measured reliability.

Stage I — compose known components ( SYNAPSE ); one objective bet
Iter.	Intervention	Outcome	Finding and claim status
0	Published HSTU anchor	0.2098	Published comparison point
1–2	SSD-style input compression (IC) $+$ PRISM user conditioning	0.2140	$+2.0\%$ ; measured historical endpoint
3	$+$ multi-token scoring head	0.2161	SYNAPSE , $+3.0\%$ : two pooled facets (recent intent, long-term taste), one ANN call; measured historical endpoint
4	Preference optimization (DPO/IPO/SimPO)	DPO: $-35.9\%$ vs. its 0.2191 reference	All three used target-derived negatives (pre-fix mining) and hurt as preference accuracy approached 1; reward over-optimization remains a hypothesis. A leak-free re-derivation mining only internal positions is neutral ( $\Delta{\approx}0$ ). Rejected
Stage II — refine (diminishing returns)

Table 1: Twelve-iteration case history on ML-20M

\times

LARGE. An iteration is one or more propose

\rightarrow

implement

\rightarrow

evaluate turns; iterations are grouped by theme into three stages and numbered by stage, not strictly in run order. NDCG@10 values are reported historical endpoints (best checkpoints, not independent seeds); the final column scopes each claim. Bold marks the stage milestones. Relative gains are over the published anchor (0.2098; 0.1895 for the BASE result below) unless a row names another reference (MPTA: the 0.2140 stack; DPO: its own reference, Appendix H.4 ). DPO’s 0.2191 reference is not a Stage I endpoint; iteration 4 is placed in Stage I by theme. Background recipe sweeps are not listed; among them, the one clean BASE win, dropout

0.2\rightarrow 0.1

in Stage II, gave 0.1948 (

+2.80\%

; Table 2 ), and a

500\rightarrow 700

context extension lost accuracy (Appendix H.4 ).

Dimension	Reported	Reference	Relative $\Delta$
ML-20M $\times$ BASE	0.1948	0.1895 p	$+2.80\%$
ML-20M $\times$ LARGE	0.2192	0.2098 p	$+4.48\%$
ML-32M $\times$ BASE	0.1562	0.1488 i	$+4.97\%$
ML-32M $\times$ LARGE	0.1726	0.1660 i	$+3.98\%$

Table 2: Reported best-checkpoint endpoints from the case history. The ML-32M BASE endpoint is a five-component stack (PRISM

{+}

{+}

time-decay

{+}

dropout

=

0.1

{+}

hard negatives) and the ML-32M LARGE endpoint a PRISM

{+}

{+}

time-decay stack; neither appears in Table 1 , which covers ML-20M only. p Published HSTU anchor; i internal reproduction, not a published baseline.

Dataset	Vanilla	$+$ SYNAPSE	${+}{+}$ UDK
Foursquare-TKY	0.0181	0.0200 ( $+10\%$ )	0.0202 ( $+12\%$ )
Foursquare-NYC	0.0156	0.0180 ( $+15\%$ )	0.0195 ( $+25\%$ )
Gowalla	0.0441	0.0460 ( $+4.3\%$ )	0.0467 ( $+5.9\%$ )
Yelp	0.0321	0.0359 ( $+11.8\%$ )	0.0382 ( $+19.0\%$ )

Table 3: Cross-dataset transfer of the deployment’s discovered recipes (NDCG@10, canonical full-test, one run per cell; columns are cumulative—

{+}{+}

UDK adds the Stage III side-feature on top of

+

SYNAPSE ; percentages over vanilla).

Condition	HSTU EA $\uparrow$	HSTU CDR $\downarrow$	LitGPT EA $\uparrow$	LitGPT CDR $\downarrow$	Cost/task $\downarrow$	Tool actions $\downarrow$
CC, one pass	$22.9$ $\,\pm 10.5$	$35.4$ $\,\pm 12.0$	$20.8$ $\,\pm 10.1$	$37.5$ $\,\pm 12.1$	$0.41$	$37$
CC, extended budget $B$	$33.3$ $\,\pm 11.8$	$27.1$ $\,\pm 11.1$	$29.2$ $\,\pm 11.4$	$29.2$ $\,\pm 11.4$	$0.98$	$94$
CC best-of- $N$ + blind selection	$43.8$ $\,\pm 12.4$	$18.8$ $\,\pm 9.8$	$39.6$ $\,\pm 12.2$	$20.8$ $\,\pm 10.1$	$0.95$	$90$
CC $\rightarrow$ CC $\rightarrow$ CC	$45.8$ $\,\pm 12.5$	$16.7$ $\,\pm 9.3$	$43.8$ $\,\pm 12.4$	$18.8$ $\,\pm 9.8$	$0.97$	$103$
CC $\rightarrow$ Codex $\rightarrow$ CC	$\mathbf{62.5}$ $\,\pm 12.1$	$\mathbf{10.4}$ $\,\pm 7.6$	$\mathbf{56.2}$ $\,\pm 12.4$	$\mathbf{12.5}$ $\,\pm 8.3$	$1.02$	$109$
Codex $\rightarrow$ CC $\rightarrow$ Codex	$56.2$ $\,\pm 12.4$	$12.5$ $\,\pm 8.3$	$50.0$ $\,\pm 12.5$	$14.6$ $\,\pm 8.8$	$1.00$	$106$

Table 4: Primary execution accuracy by condition (

n{=}96

tasks per repository); stage-1 rows 2–6 are resource-matched to envelope

B

, with one-pass CC as an unmatched cheap reference. HSTU columns carry the primary claim, LitGPT columns the pre-specified transfer contrast. Cells: task-level mean % with 95% task-clustered intervals; cost USD/task; tool actions per-task medians. Bold: best stage-1 condition on both EA and CDR at parity of budget. The rule-separated final row is the deployed plan-merge flow, specified after stage-1 unblinding and excluded from the stage-1 contrast and Holm family; † it runs at natural (uncapped) cost, unlike the stage-1 rows (Section 4.2 ).

Panel of Figure 4 — Figure 4: The deployed plan-merge topology (second-stage condition), one planning lane bound to Claude Code and the other to Codex: each lane drafts and iteratively revises its plan while reading the other lane’s latest draft (dashed; ${\leq}k$ rounds, per-lane stop votes), a single aggregation integrates a winner ( merge ), and a review–follow-up-plan dual then an implement–review dual finish.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

System	Control representation	Node granularity	Primary objective/setting	Relation to this work
StateFlow ( Wu et al., 2024 )	Programmed finite-state workflow	LLM instructions, tools, and functions	Task success/cost on interactive SQL, Bash, and ALFWorld	Establishes state-driven control and state-specific instructions; EOP does not claim either abstraction
GPTSwarm ( Zhuge et al., 2024 )	Optimizable computational graph	Operations and recursively composed agent graphs	Prompt/connectivity optimization on reasoning tasks	Establishes agents-as-graphs and diversity precedent; motivates a stricter correctness/decorrelation test
ADAS / AFlow ( Hu et al., 2025 ; Zhang et al., 2025 )	Agent code or code-represented workflow search	Generated agents, prompts, operators, and edges	Automatic agent/workflow discovery across benchmark tasks	RankEvolve uses a fixed human-authored flow and makes no automatic-design claim
LangGraph ( LangChain, 2024 )	Stateful graph runtime	Arbitrary callables, subgraphs, or wrapped agents	Durable production orchestration	Provides overlapping runtime primitives; EOP is an ML-evolution authoring/adapter instantiation
RankEvolve (this work)	Semi-structured EOP compiled to runtime state	Black-box coding products at coding/review nodes; functions elsewhere	Long-horizon ML evolution and executable patch correctness	Claimed delta: product boundary, domain specialization, and deployment-seeded, budget-controlled evidence with a pre-specified transfer split

Appendix

Table 5: Taxonomy of the closest control and workflow systems. The table locates the contribution; it is not a feature-count claim.

Parameter	BASE	LARGE
Blocks	4	16
Heads	4	8
$d_{qk}=d_{v}$	64	32
$d_{emb}$	256	256
Seq. length	200 $\rightarrow$ 500	200 $\rightarrow$ 500
Negatives	128	128

Appendix

Table 6: HSTU configurations (verified against the source and ML-20M gin configs). ∗ Dropout reduced to 0.1 for the PRISM-additive BASE endpoint (Stage II background sweep; Appendix H.2 ). The

200\rightarrow 500

entry is the baseline training schedule; a separate experiment that further extended the window to

700

lost accuracy and is reported as a negative result (Appendix H.4 ).

Construct	Compile/static check	Runtime behavior	Not guaranteed
Phase dependency	Referenced phases exist; DAG outside explicit loops	Successors remain unavailable until predecessor completion	That the predecessor’s artifact is semantically correct
Human gate	Gate has prompt and legal continuations	Transition blocks until a recorded approve/reject event	That the human decision is correct
Branch/join	Source variable, cardinality, and join policy are declared	Supported node backend instantiates one branch per item and records join state	Fan-out on an untested backend or useful diversity across branches
Loop/budget	Target phase exists; exit and hard bound are present	Attempt and spend counters stop further dispatch at the bound	That an agent chooses the best stopping point
Tool requirement	Tool name/schema resolve before launch	Calls outside the allowlist are rejected and recorded	Correct tool arguments or scientifically valid interpretation
Checkpoint/restart	Serializable state and artifact references	Resume begins at the last committed node boundary	Recovery inside an uncommitted third-party agent invocation

Appendix

Table 7: EOP enforcement contract. The final column is as important as the middle columns: runtime control constrains admissible transitions but cannot make an agent’s scientific reasoning or code correct. Branch and recovery claims apply only to the tested backend paths reported with the artifact.

Repository	Domain	Impl.	Repair	Dev.	Private
HSTU codebase	Recommendation	$48$	$48$	$32$	$96$
LitGPT	Language-model training	$48$	$48$	$32$	$96$
Total	Two ML domains	$96$	$96$	$64$	$192$

Appendix

Table 8: ExecML composition as frozen before the private evaluation: 96 private tasks per repository. Every oracle passed independent validation and a development-only paired power analysis. The statistical unit is a task. Repeated agent attempts characterize stochasticity but do not create new independent tasks.

Ordered pair	Repository	$p_{a}$	$p_{b}$	$\rho_{ab}$	$D_{a\leftarrow b}$	$G_{a\leftarrow b}$	Rescue / harm
CC $\leftarrow$ CC	HSTU	$0.54$	$0.54$	$0.58$	$0.104$	$-0.01$	$0.04\,/\,0.05$
CC $\leftarrow$ Codex	HSTU	$0.54$	$0.58$	$0.21$	$0.175$	$0.061$	$0.09\,/\,0.03$
CC $\leftarrow$ Codex	LitGPT	$0.58$	$0.60$	$0.26$	$0.169$	$0.050$	$0.08\,/\,0.03$
Codex $\leftarrow$ CC	HSTU	$0.58$	$0.54$	$0.21$	$0.164$	$0.048$	$0.08\,/\,0.03$
Codex $\leftarrow$ Codex	HSTU	$0.58$	$0.58$	$0.55$	$0.112$	$0.004$	$0.05\,/\,0.05$

Appendix

Table 9: Independent-error complementarity and realized review–repair gain. Rows are descriptive; inference resamples source tasks jointly and does not treat overlapping rows as independent observations.

Process-faithfulness violation	Runs
Ran only a subset of selected proposals	10
Stopped early despite self-recommending continue	4
Skipped a prescribed user-confirmation gate	4
Called a tool with off-spec arguments	3
Redundant re-investigation on loop-back	3
Merged the codebase and data investigation phases	2

Appendix

Table 10: Historical motivating pilot, not a treatment comparison. Process-faithfulness audit of a model-evolution procedure delivered as a single full-context playbook. Ten runs under the logged coding-agent configuration were scored against the prescribed phases, gates, branches, and loop; counts are runs affected (out of ten), and a run may incur several violation types, so the column does not sum to ten. The pilot lacks a runtime-controlled arm and cannot identify the effect of context scoping.

Injection policy	Early EA	Middle EA	Late EA	CDR $\downarrow$	Wrong decision $\downarrow$	Input tokens $\downarrow$	Cost $\downarrow$
Current-step only	$64.6$	$58.3$	$52.1$	$11.5$	$8.3$	$4.9$ k	$0.71$
Full protocol	$62.5$	$52.1$	$41.7$	$17.7$	$16.7$	$9.8$ k	$1.28$
Paired difference	$+2.1$	$+6.2$	$+10.4$	$-6.2$	$-8.3$	$-4.9$ k	$-0.57$

Appendix

Table 11: Runtime-controlled context-scope ablation. Cells report task-level means. Both conditions use the same compiled runtime; only inactive EOP content is toggled.

Panel of Figure 5 — Figure 5: The centralized–distributed knowledge layer. Distributed local memory stores each learning as a note in the code folder it describes (a naturally tree-structured, human- and agent-navigable store); a global wiki abstracts transferable knowledge and cites those notes, with bidirectional document-level links (dashed). An entity graph over models, features, metrics, and defects adds entity-level links (dotted) spanning both tiers. A proactive write path distills modeling and execution learnings; a selective read path retrieves the few relevant to the current EOP phase by hybrid search plus graph/tree traversal.

Variant	$\alpha$	NDCG@10	$\Delta_{p}$
Paper baseline	—	0.2098	—
PRISM+IC stack (ref)	—	0.2140	+2.0%
Genre FT (leak-free)	0.05	0.2192	+4.48%
Genre FT (leak-free)	0.10	0.2192	+4.48%
Genre FT (leak-free)	0.20	0.2191	+4.43%
Genre from-scratch ∗	0.10	0.1911	—

Appendix

Table 12: Genre side-feature on ML-20M

\times

LARGE. FT = fine-tuning from the 0.2140 stacked checkpoint. ∗ unfinished run, last evaluated at epoch 26; not an endpoint.

Method	NDCG@10	$\Delta$ vs. ref
Reference (no pref.)	0.2191	—
DPO	0.1405	$-35.9\%$
IPO	0.2012	$-8.2\%$
SimPO	0.1798	$-17.9\%$

Appendix

Table 13: Preference optimization with target-derived negatives (pre-fix mining) on ML-20M

\times

LARGE: all methods hurt downstream NDCG@10 despite near-perfect preference accuracy.

Explore similar work

Sep 1, 2026cs.SE

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

Agent Harness

Ask PDF

Jul 31, 2026cs.IR

RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems

Optimizing modern recommender models still depends heavily on engineers manually iterating over architectural, objective, and training-strategy changes. While LLM-based agents can automate this trial-and-error process, allowing the LLM to both select modification directions and generate concrete hypotheses often leads to unstable search under limited experiment budgets. Inspired by the above challenge, we propose RecHarness, a Bandit-Routed Agentic Harness for automated recommender model optimization. RecHarness separates the optimization process into two steps: a bandit router selects the next modification direction according to historical validation feedback, while the LLM generates a concrete optimization hypothesis and executable code edit within the selected direction. To sustain long-horizon exploration, RecHarness uses a jump-basin mechanism to activate a structural-jump arm when local edits stagnate. Across multiple recommendation tasks, datasets, and model backbones, RecHarness achieves more stable performance improvements and uses limited trial budgets more effectively than LLM-reasoning search. During a 7-day online A/B test on a large-scale short-video advertising platform, the selected candidate improves ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559%. Code is available at https://github.com/6lyc/RecHarness.

Haoran Ling, Yuecheng Li, Zeyu Song +5

Georgia Institute of Technology · Kuaishou Technology, China

Prompt Optimization Bandits

Ask PDF

Jul 19, 2026cs.MA

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Research Automation Auto Research Reusable Knowledge Feedback Materials

Ask PDF