Coupled Harness--Model Evolution for Scientific Equation Discovery

Pengze Li; Jiaquan Zhang; Encheng Su; Yunbo Long; Zihang Zeng; Xinping Liu; Han Zhang; Junchi Yu

cs.AIOct 13, 2026

Coupled Harness--Model Evolution for Scientific Equation Discovery

Authors: Pengze Li, Jiaquan Zhang, Encheng Su, Yunbo Long, Zihang Zeng, Xinping Liu, Han Zhang, Junchi Yu,

Organizations: Fudan University · Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · University of Cambridge ·

Abstract

Large language models (LLMs) are increasingly used to discover scientific equations from experimental data. Such a discovery system has two parts that must work together: a library of executable computational tools and the model's tool-use policy. The tools supply the solving capability, from diagnosing structure in the data to fitting candidate equations; the policy plans and orchestrates their use. Yet existing systems adapt only one part, keeping either the toolset or the model fixed, which leaves the corresponding failures uncorrectable. We develop a coupled evolution framework that improves both parts from the same verifiable fit-error signal. In the tool-evolution phase, an agent grows an executable tool library from a small set of generic numerical operations by diagnosing its own failures: it authors a new tool when a capability is missing and revises its selection rules when the right tool went unused. In the policy-evolution phase, a small open model (roughly 3B active parameters) is fine-tuned on successful tool-use trajectories collected with the frozen library, on training formulas screened to be disjoint from all evaluation benchmarks. Ablations show the two phases are complementary: with no tool library at all the same model solves almost nothing, the tools account for most of the gain, and trajectory training adds a further gain that is largest on the most difficult problems. The full system solves 79.1/71.3% of LSR-Synth problems at fit tolerances $10^{-2}$ / $10^{-3}$ , reaches 63.8% pooled accuracy on the distributionally different SRSD-Feynman panel, and obtains a normalized CP3-28 aggregate of $-7.27$ . These results suggest a broader recipe for scientific agents: let the model and its instruments improve each other under verifiable feedback.

Figures & tables

Figure 1: Overview of the coupled evolution. Phase A (left) starts from a generic tool set and evolves the harness by diagnosing failed attempts: a missing capability adds a tool, a wrong choice revises the interaction or selection rules. Phase B (right) holds the frozen harness fixed and fine-tunes an open Qwen controller on successful tool-use trajectories, yielding the reported model + tools system.

Figure 2: CP3-28 cosmology results over 28 problems. Bars show negated mean

\log_{10}

NMSE (higher is better); GPT-5.5 direct is included as a tool-free reference. Appendix A.1 gives the aggregation procedure and per-problem comparisons.

Traditional SR systems (worst-5% trimmed):
System	Aggregate	Material	Chemistry	Biology	Physics	LSRT111
PySR	29.5/14.5	53.3/22.7	25.9/11.1	16.7/7.0	25.8/16.7	56.76/–
uDSR	29.5/12.4	36.0/8.0	50.0/25.0	29.2/8.3	9.1/6.8	–
LaSR	16.0/10.1	13.3/6.7	16.7/9.3	7.0/7.0	9.9/6.8	50.45/–
External LLM-based (worst-5% trimmed):
SR-Scientist (GPT-OSS-120B)	63.6/49.4	74.7/60.0	81.5 /64.8	66.7 /43.1	40.9/34.1	–

Table 1: LSR-Synth (129 problems) and LSR-Transform-111 accuracy (%). All entries are percentages; slash-separated cells report the first/second accuracy threshold. LSR-Synth cells use Acc@

10^{-2}

/Acc@

10^{-3}

, while the LSRT111 column uses Acc@

10^{-1}

/Acc@

10^{-2}

. Bold marks the highest displayed value for each benchmark and threshold within the reported conventions. “–” marks cells the source does not report.

	SRSD standard			SRSD dummy vars.			Hard	Pooled
Method	Easy	Med.	Hard	Easy	Med.	Hard	(merged)	(240)
uDSR	100.0	75.0	20.0	53.3	37.5	12.0	16.0	44.6
PySR	66.7	45.0	38.0	20.0	10.0	2.0	20.0	28.3
DSR	63.3	45.0	28.0	76.7	45.0	22.0	25.0	42.9
Model + tools (ours)	83.3	67.5	58.0	86.7	60.0	44.0	51.0	63.8

Table 2: SRSD-Feynman-240 against published systems, as accuracy (%) at

R^{2}>0.999

per difficulty tier, on the standard and dummy-variable variants, the merged hard tiers, and the pool over all 240. Bold marks the best value per column.

Tools only (initial)	–	–	–	–	41.1/30.2	6.3/3.6	29.6	18.0	$-4.76$
	Tools	Model		BIC	LSR-Synth	LSRT111	SRSD-240	SRSD hard	CP3-28
Condition	evolved	used	trained		Acc@ $10^{-2}$ /Acc@ $10^{-3}$	Acc@ $10^{-1}$ /Acc@ $10^{-2}$	pooled	merged	$\log_{10}$ NMSE $\,\downarrow$
	–	–	–	✓	43.4/31.0	6.3/3.6	33.3	23.0	$-5.29$
Tools only (evolved)	✓	–	–	–	61.2/50.4	53.2/49.5	51.3	29.0	$-6.86$
	✓	–	–	✓	77.5/69.8	77.5 / 75.7	62.1	45.0	$-6.91$
Untrained + tools	✓	✓	–	–	54.3/46.5	49.5/45.0	52.5	30.0	$-6.69$

Table 3: Ablations over the two components the method evolves: the tool library (Phase A) and the model that controls it (Phase B). Tools evolved distinguishes the final evolved library (✓) from the initial library before evolution; Model used/trained distinguish deterministic traversal (no model), the untrained controller, and the trajectory-trained controller. Rows without BIC report raw candidate quality before post-hoc reselection. Slash-separated entries report the first/second threshold: LSR-Synth uses Acc@

10^{-2}

/Acc@

10^{-3}

and LSRT111 uses Acc@

10^{-1}

/Acc@

10^{-2}

. CP3-28 reports mean

\log_{10}

NMSE

\downarrow

, where lower is better.

Figure 3: Capability growth on LSR-Synth-129 under deterministic tool traversal, scored with SR-Scientist’s trimmed-5% convention so the reference line is like-for-like. Curves show in-domain and shifted-input Acc@ $10^{-2}$ for epochs 1–9; diamonds mark the epoch-10 no-BIC point and stars the BIC-selected endpoint. The top axis counts the executable analysis tools per snapshot. The horizontal line marks SR-Scientist (GPT-OSS-120B), which trains the model on a fixed toolset.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

vs.	mean $\log_{10}$ NMSE $\downarrow$	Tools only	Model + tools
QLattice	$-6.20$	30.8 / 34.6 / 34.6	34.6 / 42.3 / 23.1
ITEA	$-6.16$	25.9 / 40.7 / 33.3	44.4 / 25.9 / 29.6
GPG	$-4.45$	40.7 / 40.7 / 18.5	48.1 / 37.0 / 14.8
GPZGD	$-4.59$	60.7 / 25.0 / 14.3	64.3 / 21.4 / 14.3
Operon	$-2.66$	64.3 / 28.6 / 0 7.1	67.9 / 25.0 / 0 7.1
GE	$-3.66$	70.4 / 25.9 / 0 3.7	70.4 / 25.9 / 0 3.7

Appendix

Table 4: CP3-28 (cosmology) against all 12 published methods. First column: the method’s mean

\log_{10}

NMSE

\downarrow

(procedure above; lower is better; after candidate selection, our reference values are

-7.27

for the trained controller with the library and

-6.91

for deterministic traversal). Remaining columns: per-problem head-to-head as win % / tie % / loss % of paired problems, ordered by increasing trained-controller win rate. The pairing is at order-of-magnitude grain (equal

\lfloor\log_{10}\text{MSE}\rfloor

counts as a tie), which is conservative for us; denominators vary (

22

–

28

) because the source omits some per-problem cells.

Large Language Model-Guided Scientific Discovery

Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.

See all 275 papers

Dictionaries, Not Darwin: Set-Level Selection Beats LLM Evolution in Scientific Equation Discovery
Jul 5, 2026Pan LiEvolutionary Optimization Scientific Discovery
Independent Researcher
The Little Scientist: LLM Agent-Driven Discovery via the Scientific Method
Aug 16, 2026Travis SmithAI Agents for Scientific Discovery Automated Algorithm Discovery
Independent Researcher, S˜ao Miguel, Azores, Portugal
ExpEvolve: Structuring and Reusing Experience for Open-Ended Discovery
Oct 21, 2026Yongqiang Chen, Chenxi Liu, Deyu Zou +4AI Agents for Scientific Discovery Large Language Model-Guided Scientific Discovery
MBZUAI · Carnegie Mellon University · TMLR Group, Hong Kong Baptist University +2

Jul 5, 2026J/K move · Enter open · S save

Dictionaries, Not Darwin: Set-Level Selection Beats LLM Evolution in Scientific Equation Discovery

Pan Li

Independent Researcher

Evolutionary Optimization Scientific Discovery Large Language Model-Guided Scientific Discovery Symbolic Regression

Open paper AskPDF arXiv

Large language models are increasingly used as evolutionary engines for scientific discovery: generate candidates, select winners, feed them back as parents, and repeat. We audit whether this loop actually compounds discovery in scientific equation discovery, a setting where finite samples make structure underdetermined and interpolation easy. Under matched LLM-call budgets, parent-conditioned evolution is indistinguishable from fresh independent sampling: median OOD NMSE is 0.045 vs. 0.049, instructed multi-parent crossover is worse, final success is predicted by initial proposal quality, and multiple iteration schemes fail to add solved problems. Operationally, the loop reduces to what it produces: a dictionary of candidate terms. We turn that diagnosis into PTB-Search, a one-generation method for componentized scientific discovery. PTB-Search samples independent LLM proposals once, extracts reusable terms into a per-problem dictionary, and performs train-only set-level sparse selection with least-squares coefficients. Its central principle is that underdetermined data identifies the joint behavior of term sets, not reliable per-term credit. On identical dictionaries and zero additional LLM calls, set-level selectors solve 165--169 of 717 cells, while single-term reductions solve only 74--78. On the official 239-problem LLM-SRBench split, PTB-Search reaches 73.2% Acc0.1 with Llama-3.1-8B and 77.0% with a single-seed DeepSeek-V4 anchor, versus 49.2% for the best reported baseline, using one tenth of the standardized call budget. A program-domain stress test gives a scoped boundary: generation count remains unreliable, while retained external state can help in harder non-linear spaces. Across these results, LLMs are best understood as material suppliers; discovery is carried by external set-level selection over reusable components.

Abstract

Large Language Model-Guided Scientific Discovery

Explore similar work

Dictionaries, Not Darwin: Set-Level Selection Beats LLM Evolution in Scientific Equation Discovery

The Little Scientist: LLM Agent-Driven Discovery via the Scientific Method

ExpEvolve: Structuring and Reusing Experience for Open-Ended Discovery