L2G-SR: Learning to Guide LLM-based Symbolic Regression

Letian Huang; Xiangyu Wang; Haowei Lin; Di He

cs.AIOct 14, 2026

L2G-SR: Learning to Guide LLM-based Symbolic Regression

Authors: Letian Huang, Xiangyu Wang, Haowei Lin, Di He

Organizations: Peking University

Abstract

Symbolic regression searches for an equation that explains observed data. Numerical error is commonly used to guide the search, but it alone can favor an accurate surrogate over a candidate with the right structure. We propose L2G-SR, a framework for LLM-based symbolic regression that guides search with a learned structural verifier. We construct structural supervision signal using tree edit distance (TED), accounting for commutative operators and alternative expression forms. A verifier first aligns numerical observations with the formulas, then learns to score a candidate's structural similarity to the target formula. At search time, the target formula is hidden and the verifier complements numerical error in parent selection. An analyst then uses verifier feedback and diagnostic tools to propose revisions, which a generator converts into refined expressions. On LLM-SRBench, L2G-SR achieves state-of-the-art numerical fit, equation level accuracy and TED across the evaluated LLM backbones and domains. L2G-SR demonstrates the value of learned structural guidance for LLM-based symbolic regression.

Figures & tables

Figure 1: Overview of L2G-SR. (A) A verifier learns TED with two-stage training. (B) During search, numerical error and predicted TED guide candidate selection and retention.

Figure 2: Examples of structural similarity under the modified TED. (a) The two linear expressions map to the same abstract tree. (b) Scores are normalized by the ground-truth tree size. In these examples, changes in coefficient values, operand order, and numeric exponent values yield zero distance, whereas operator and variable label changes incur a cost.

Llama3.1-8B-Instruct
	Transform		Physics		Materials		Chemistry		Biology
Method	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$
LLMDirect	2.95e-1	34.23	9.95e-3	0.00	8.16e-2	24.00	1.22e0	2.78	1.29e-1	4.17
LLM-SR	2.42e-1	34.23	3.00e-3	6.82	2.16e-1	60.00	5.24e-2	16.67	1.76e-2	12.50
LaSR	2.62e-1	33.33	6.07e-3	9.09	9.47e-4	32.00	1.82e-3	8.33	6.40e-3	0.00
SGA	3.52e-1	0.90	1.55e-1	2.27	4.35e-2	12.00	4.58e-2	8.33	2.42e-1	0.00

Table 1: Performance on five LLM-SRBench domains with Llama3.1-8B-Instruct and Qwen3-4B-Instruct-2507. Columns report NMSE (

\downarrow

) and

\mathrm{Acc}_{0.01}

(

\uparrow

, %). LLMDirect, LLM-SR, LaSR, SGA, and DE results are from DE ( Pang et al., 2026 ) ; A-SR results are from Zhao et al. (2026) . L2G-SR shows mean

\pm

sample standard deviation over three seeds. Bold and underlined means mark the best and second-best displayed values.

Figure 3: True TED ( $\downarrow$ ) of the final formulas from Table 1 .

Table 5

Table 6

Figure 4: PO11 case study. (a) Structural feedback selects slot 47 despite its higher validation NMSE. (b) Ground truth, candidate structures, and the child generated from slot 47;

c_{i}

denote fitted constants. (c) Numerical and structural outcomes.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

	Candidate $f$	Reference $g$	$v_{0}$	$v_{1}$	$v_{2}$	TED
(a)	$x+(y+z)$	$(x+y)+z$	$0/4$	$0/4$	$0/4$	$0$
(b)	$\sin(x+y)$	$\cos(x+y)$	$1/4$	$1/4$	$1/4$	$1/4$
(c)	$x(y+z)$	$xy+xz$	$5/7$	$5/7$	$0/7$	$0$
(d)	$(x+y)^{2}$	$x^{2}+2xy+y^{2}$	$9/11$	$9/11$	$0/11$	$0$
(e)	$x+\sin(yz+w)$	$x$	$6.6/1$	$6.6/1$	$6.6/1$	$6.6$

Appendix

Table 6: Worked TED calculations. Each view reports the raw edit cost divided by its reference-tree size. The letters match the tree diagrams.

Figure 5: Selected abstract trees for examples (a)–(c). Input formulas appear above each tree; the diagrams show the trees after the indicated preprocessing. Flattening removes grouping differences in (a), the highlighted function labels require one edit in (b), and product expansion gives identical trees in (c).

Figure 6: Selected abstract trees for examples (d)–(e). In (d), expanding the square yields identical trees;

N

denotes an abstract numeric leaf. In (e), the dashed outline contains the six-node branch deleted at discounted cost 5.6; the highlighted Add root incurs another unit deletion cost. Only the variable

x

is retained.

	Candidate $f$	Reference $g$	$v_{0}$	$v_{1}$	$v_{2}$	TED
(f)	$x^{2}\sin y$	$\sin y$	$4/2$	$4/2$	$4/2$	$2$
(g)	$(x+y)(x+z)$	$xy+yz$	$4/7$	$4/7$	$6/7$	$4/7$
(h)	$(x+y)^{3}$	$x^{3}+y^{3}$	$5/7$	$5/7$	$11.2/7$	$5/7$
(i)	$\sqrt{x+y}$	$\sqrt{x}$	$2/3$	$2/3$	$2/3$	$2/3$
(j)	$(x+y)(x+z)$	$x(x+y)$	$2/5$	$2/5$	$6/7$	$2/5$
(k)	$(x+y)^{2}$	$(x+y)^{3}$	$0/5$	$0/5$	$7.6/19$	$0$

Appendix

Table 7: Additional worked TED calculations. Each view reports the raw edit cost divided by its reference-tree size. All six pairs select

v_{0}

, using the first view in a tie.

Figure 7: Selected

v_{0}

trees for (f)–(g). Orange marks nodes edited along a minimum-cost path. In (f), the extra power subtree and Mul root are deleted. In (g), four relabelings transform the product of sums into the reference sum of products.

Figure 8: Selected

v_{0}

trees for (h)–(i). In (h), orange marks three relabelings and two inserted nodes; the original exponent leaf is retained as the reference’s rightmost

N

. In (i), deleting the orange Add and

y

nodes leaves the reference square-root tree. Numeric exponents are represented by

N

Figure 9: Selected

v_{0}

trees for (j)–(k). In (j), commutative matching preserves the common additive factor; deleting the orange Add and

z

nodes supplies the remaining

x

factor. In (k), both exponent values become the same numeric leaf

N

, making the abstract trees identical.

Quantity	Outcome	Probability	Quantity	Outcome	Probability
$N$	1–2	0.001000000	Variables	1	0.127767302
	3–7	0.030000000		2	0.352240995
	8–15	0.321712885		3	0.282177813
	16–30	0.440506600		4–5	0.196282613
	31–50	0.184595015		6–8	0.041531276
	51–70	0.018782916		9–10	0

Appendix

Table 8: Effective size, variable-count, and root-family proposals before feasibility constraints. The variable and root columns are separate categorical distributions.

Binary operator	Probability	Unary operator	Probability
Add	0.219895438	Neg	0.034145627
Sub	0.114883686	Inv	0.015434137
Mul	0.509631078	Abs	0.012666583
Div	0.135253958	Sqrt	0.117837361
Pow	0.020335839	Pow2	0.278049997
		Pow3	0.073586740

Appendix

Table 9: Base operator probabilities after the unary floor and before forced assignments or contextual exclusions. Neg , Inv , Pow2 , and Pow3 denote negation, reciprocal, square, and cube.

$\|c\|$ interval	Probability	$\|c\|$ interval	Probability
$0$	0.000248984	$[10^{-1},1)$	0.297092815
$(0,10^{-12})$	0.037271548	$[1,10)$	0.313650602
$[10^{-12},10^{-6})$	0.062787186	$[10,10^{3})$	0.059584268
$[10^{-6},10^{-3})$	0.046988624	$[10^{3},10^{6})$	0.011093352
$[10^{-3},10^{-1})$	0.171282619	$[10^{6},\infty)$	0

Appendix

Table 10: Constant-magnitude proposals. The smallest nonzero bin has the finite sampling range

[10^{-100},10^{-12})

. The original mass above

10^{6}

is redistributed proportionally between the two smallest nonzero bins.

Family	Probability	Count values: probabilities
Point relabeling	0.20	$1,2,3,4$ : $(0.15,0.40,0.30,0.15)$
Unary insertion/deletion	0.15	$1,2,3,4$ : $(0.20,0.40,0.25,0.15)$
Branch insertion/deletion	0.05	$1,2,3$ : $(0.25,0.45,0.30)$
Subtree replacement	0.15	$1,2,3$ : $(0.25,0.45,0.30)$
Crossover/promotion	0.05	$1,2$ : $(0.60,0.40)$
Mixed families	0.40	$2,3,4$ : $(0.10,0.30,0.60)$

Appendix

Table 11: Edit-family probabilities and count proposals conditional on an edited pair. Short-target caps are applied after these proposals.

Family	Maximum count	$r$
Point / unary	$1$ for $L\leq 5$ ; $2$ for $6\leq L\leq 20$ ; $3$ for $21\leq L\leq 50$ ; $4$ otherwise	0.30
Branch / subtree	$1$ for $L\leq 7$ ; $2$ for $8\leq L\leq 30$ ; $3$ otherwise	0.35
Crossover/promotion	$1$ for $L\leq 20$ ; $2$ otherwise	0.20
Mixed	$2$ for $L\leq 5$ ; $3$ for $6\leq L\leq 15$ ; $4$ otherwise	0.40

Appendix

Table 12: Edit-count caps by target length

L

and cumulative affected-node budget ratio

r

. Each piecewise list gives the largest permitted count.

TED interval	Edited count $E_{i}$	Random count $R_{i}$	$q_{i}$ (edit)	$1-q_{i}$ (random)
$[0,0.1)$	1,562,156	106,540	1	0
$[0.1,0.2)$	2,578,355	148,986	1	0
$[0.2,0.3)$	3,802,093	478,678	1	0
$[0.3,0.4)$	3,536,928	1,073,557	0.908795814	0.091204186
$[0.4,0.5)$	2,699,071	2,321,306	0.778598241	0.221401759
$[0.5,0.6)$	2,159,613	6,491,726	0.501534776	0.498465224

Appendix

Table 13: TED-bin probabilities, calibrated source choices, and the counts used to determine them. Every bin has training probability

1/12

; counts describe the August 28 calibration scan.

Setting	Stage 1	Stage 2
Global batch size	256	256
Updates	55,000	100,000
Learning rate	$10^{-4}$ peak	$2{\times}10^{-5}$ to $2{\times}10^{-6}$
Schedule	Warmup–stable–decay	Warmup + cosine decay
Warmup updates	2,000	2,000
Stable / decay updates	48,000 / 5,000	—

Appendix

Table 14: Completed verifier training configuration. Checkpoints were selected on synthetic validation, not benchmark test scores.

Figure 10: Stage 1 training and validation loss.

Figure 11: Stage 2 training and validation loss.

Setting	Value
Backbone for both roles / temperature	Llama3.1-8B-Instruct / 0.8
Islands / capacity / M and T quotas	4 / 64 / 32 each
Initialization / rounds / children	40 / 90 / 4
M rounds / T rounds	45 / 45
Tournament sizes, M / T	3 / 2, with replacement
Credibility threshold / exploration cap	Validation NMSE $\leq 1$ / 8

Appendix

Table 15: Search settings shared by TED and matched numerical-only conditions.

Llama3.1-8B-Instruct
	Transform		Physics		Materials		Chemistry		Biology
Method	NMSE SD	$\mathrm{Acc}_{0.01}$ SD	NMSE SD	$\mathrm{Acc}_{0.01}$ SD	NMSE SD	$\mathrm{Acc}_{0.01}$ SD	NMSE SD	$\mathrm{Acc}_{0.01}$ SD	NMSE SD	$\mathrm{Acc}_{0.01}$ SD
RESTART	7.06e-3	1.88	3.47e-3	4.73	6.91e-5	6.11	1.23e-6	3.21	1.91e-2	6.36
SR-Scientist	1.94e-3	1.38	3.18e-3	1.31	1.49e-3	0.00	7.47e-5	2.78	4.53e-2	4.17
Qwen3-4B-Instruct-2507
RESTART	1.08e2	1.04	5.48e-2	2.62	1.14e-3	4.00	3.81e-2	8.49	2.61e0	2.41

Appendix

Table 16: Sample standard deviations for the RESTART and SR-Scientist means in Table 1 . Accuracy standard deviations are in percentage points.

Method	Successful calls per problem	Input tokens per problem	Output tokens per problem
RESTART (reproduced)	422.7	715,609	103,528
SR-Scientist (reproduced)	519.2	1,517,007	63,266
L2G-SR	189.2	847,822	48,130

Appendix

Table 17: Successful LLM calls and tokens per problem for the Llama3.1-8B-Instruct runs shared by the three methods.

	Oscillation 1		Oscillation 2
Method	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}^{\mathrm{pt}}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}^{\mathrm{pt}}$ $\uparrow$
Llama3.1-8B-Instruct
LLMDirect	6.38e-7 ±4.34e-7	100.00 ±0.00	2.31e-5 ±1.89e-5	88.79 ±9.37
LLM-SR	1.21e-5 ±1.67e-5	94.31 ±9.86	4.51e-4 ±7.17e-4	63.97 ±34.95
LaSR	9.61e-4 ±3.66e-4	16.26 ±3.35	8.36e-4 ±3.27e-4	31.37 ±9.05
SGA	2.23e-4 ±3.76e-4	68.78 ±44.14	4.74e-3 ±2.15e-3	7.96 ±1.71

Appendix

Table 18: ID results on Oscillation 1 and 2.

Llama3.1-8B-Instruct
	Physics		Materials		Chemistry		Biology
Method	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}$ $\uparrow$
A-SR	6.55e3	25.00	6.90e-4	100.00	2.06e0	25.00	1.08e0	45.83
RESTART	2.13e2 ±1.89e2	10.08 ±4.84	6.65e-3 ±5.61e-3	90.28 ±4.81	7.79e-1 ±2.44e-1	41.90 ±4.36	2.10e1 ±2.26e1	18.84 ±6.64
SR-Scientist	7.70e1 ±7.36e1	21.71 ±2.69	9.09e-2 ±7.82e-2	75.00 ±11.02	2.33e0 ±2.53e0	35.24 ±4.36	7.49e3 ±1.30e4	23.19 ±2.51
L2G-SR	2.13e-3 ±1.42e-3	90.70 ±2.33	2.27e-5 ±2.21e-5	100.00 ±0.00	1.92e-1 ±8.96e-2	69.22 ±7.27	1.99e-1 ±1.56e-1	66.67 ±5.02

Appendix

Table 19: OOD evaluation on LSR-Synth. Published baseline rows are retained as reported. Our reproduced rows show mean

\pm

standard deviation across three seeds.

	Oscillation 1		Oscillation 2
Method	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}^{\mathrm{pt}}$ $\uparrow$	NMSE $\downarrow$	$\mathrm{Acc}_{0.01}^{\mathrm{pt}}$ $\uparrow$
Llama3.1-8B-Instruct
LLMDirect	1.03e-2 ±0.88e-2	45.06 ±18.89	3.83e-3 ±3.01e-3	52.51 ±14.82
LLM-SR	3.95e-2 ±4.49e-2	23.52 ±33.64	6.28e-3 ±0.76e-3	29.80 ±18.29
LaSR	1.98e-1 ±2.44e-1	4.60 ±4.91	1.75e-2 ±1.56e-2	13.99 ±5.65
SGA	3.19e-1 ±2.78e-1	37.22 ±50.88	2.30e-2 ±1.46e-2	7.12 ±3.86

Appendix

Table 20: OOD results on Oscillation 1 and 2. NMSE (

\downarrow

) and pointwise

\mathrm{Acc}_{0.01}^{\mathrm{pt}}

(

\uparrow

, %) are computed after 5% relative-error trimming. Entries report mean

\pm

sample standard deviation over three seeds. Bold and underlined means indicate the best and second-best displayed values, respectively.

Setting	Value
Backbone / temperature	Llama3.1-8B-Instruct / 0.8
Training / internal-validation fractions	0.8 / 0.2
Islands / generations / slots per generation	10 / 100 / 4
LLM-generated candidate budget	400 slots per problem
Refinement and memory	E2E-assisted refinement / concept library
Maximum fit rows / constants	4,096 / 10

Appendix

Table 21: Settings for our RESTART reproduction.

Method	Counted LLM search budget	Additional search or optimization
LLMDirect	5 generated candidates	BFGS constant refinement for each hypothesis.
LLM-SR	1,000 generated candidates	Island evolution and BFGS constant refinement; these operations act on the counted candidates.
LaSR	1,000 LLM-generated candidates	More than 453,000 random non-LLM mutations, in addition to PySR fitting and selection.
SGA	1,000 generated candidates	Adam-based inner-loop optimization of continuous parameters.
DE	At most 400 generated candidates ( $100\times 4$ )	Diagnostic tools, BFGS constant fitting, and reflective-memory updates are outside the candidate count.
RESTART (ours)	400 proposal slots ( $100\times 4$ )	Numerical/E2E initialization and E2E calls do not consume proposal slots; constant fitting follows the shared BFGS limits.

Appendix

Table 22: Per-problem search-budget comparison. Candidate or proposal counts follow the accounting used by DE and our implementations; the final column records method-specific computation outside those counts.

Symbolic Regression

Symbolic regression searches data for compact mathematical expressions that balance predictive accuracy with interpretability.

See all 148 papers

DIGEST: Discovery-Guided LLM Tuning for Symbolic Regression
Oct 24, 2026Kaifeng Jin, Ruizhong Qiu, Zihao Li +5Symbolic Regression Large Language Model-Guided Optimization
University of Illinois Urbana-Champaign · Meta
InsightSR: Refining Symbolic Regression Search Spaces via Parallel Semantic and Structural LLM Guidance
Aug 26, 2026Yating Ling, Wenjing Cun, Zhitang ChenSymbolic Regression Large Language Model-Guided Scientific Discovery
FunctionEvolve: Structure-Guided Symbolic Regression with LLMs
Jun 5, 2026Zeyu Xia, Jun Zhu, Dong YanGenetic Programming Large Language Model-Guided Scientific Discovery
Bosch Center for Artificial Intelligence · Department of Computer Science and Technology, Tsinghua University · Department of Computer Science and Technology +1

Oct 24, 2026J/K move · Enter open · S save

DIGEST: Discovery-Guided LLM Tuning for Symbolic Regression

Kaifeng Jin, Ruizhong Qiu, Zihao Li, Xiao Lin+4

University of Illinois Urbana-Champaign · Meta

Symbolic Regression Large Language Model-Guided Optimization Language Model-Guided Search Heuristics Genetic Programming

Open paper AskPDF arXiv

Symbolic regression (SR) seeks to discover mathematical expressions directly from data, without requiring task-specific scientific descriptions. Despite tremendous progress in both conventional search-based methods and recent LLM-based methods, two critical limitations largely remain. First (Limited Generation Flexibility), conventional search-based methods rely on either fixed search rules or specialized learned generators, severely limiting adaptive candidate generation across diverse expression structures. Second (Underutilized Search Discoveries), most LLM-based methods cannot reliably recover ground-truth expressions because they fall short in effectively leveraging the LLM itself to thoroughly digest past expressions or adaptively guide future search. To address these limitations, we propose DIGEST. To render greater flexibility in generation, DIGEST repurposes a pretrained decoder block from a lightweight LLM into a symbolic policy for Genetic Programming (GP) population seeding, thereby leveraging the broader modeling capacity of the LLM. To more thoroughly utilize search discoveries, high-quality expressions discovered by GP are fed back to update the LLM-derived policy online, thereby enabling the LLM to progressively internalize past search discoveries and translate them into improved subsequent population generation. Experiments show that on two ground-truth benchmarks, DIGEST outperforms the strongest competitor by up to 2.59% in recovery. Among LLM-based baselines, it improves recovery by at least 54.91% while using at most 17.4% of the strongest competitor's mean search time. On 122 BlackBox datasets, DIGEST achieves the highest accuracy while maintaining a moderate model complexity.

Abstract

Symbolic Regression

Explore similar work

DIGEST: Discovery-Guided LLM Tuning for Symbolic Regression

InsightSR: Refining Symbolic Regression Search Spaces via Parallel Semantic and Structural LLM Guidance

FunctionEvolve: Structure-Guided Symbolic Regression with LLMs