Neural Structure Embedding for Symbolic Regression via Continuous Structure Search and Coefficient Optimization

Fateme Memar; Tao Zhe; Dongjie Wang

cs.LGMar 23, 2026

Neural Structure Embedding for Symbolic Regression via Continuous Structure Search and Coefficient Optimization

Authors: Fateme Memar, Tao Zhe, Dongjie Wang

Organizations: Department of Electrical Engineering and Computer Science, University of Kansas USA

Abstract

Symbolic regression aims to discover human-interpretable equations that explain observational data. However, existing approaches rely heavily on discrete structure search (e.g., genetic programming), which often leads to high computational cost, unstable performance, and limited scalability to large equation spaces. To address these challenges, we propose SRCO, a unified embedding-driven framework for symbolic regression that transforms symbolic structures into a continuous, optimizable representation space. The framework consists of three key components: (1) structure embedding: we first generate a large pool of exploratory equations using traditional symbolic regression algorithms and train a Transformer model to compress symbolic structures into a continuous embedding space; (2) continuous structure search: the embedding space enables efficient exploration using gradient-based or sampling-based optimization, significantly reducing the cost of navigating the combinatorial structure space; and (3) coefficient optimization: for each discovered structure, we treat symbolic coefficients as learnable parameters and apply gradient optimization to obtain accurate numerical values. Experiments on synthetic and real-world datasets show that our approach consistently outperforms state-of-the-art methods in equation accuracy, robustness, and search efficiency. This work introduces a new paradigm for symbolic regression by bridging symbolic equation discovery with continuous embedding learning and optimization.

Figures & tables

Figure 1. Framework Overview of SRCO . Structure Embedding : A GP-based SR system generates diverse candidate equations, which are converted into postfix sequences with abstracted coefficients ( COF ) to train a Transformer-based structural prior; Continuous Structure Search : The learned prior guides constrained sampling in postfix space, followed by syntactic, semantic, and complexity filtering to obtain valid symbolic templates; Coefficient Optimization : For each selected template, COF tokens are instantiated as learnable parameters and optimized via gradient-based regression to produce the final symbolic equation.

	Easy			Medium			Hard
Model	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )
DSO	0.58108	0.76836	45.0992	-3.63397	0.57411	67.6205	$-2.37841\times 10^{5}$	0.55307	67.0060
gplearn	-0.25600	0.41807	45.0917	0.30200	0.42552	67.6205	$-1.84\times 10^{13}$	0.39730	67.2514
FFX	0.45998	0.57570	45.0911	0.27657	0.61437	67.9408	0.26691	0.67575	68.4654
EFS	-6.79227	0.06667	45.0915	-21.75270	0.00000	67.6205	-10.32860	-0.12000	70.4763
SRCO	0.99821	0.96556	30.4281	0.99555	0.99786	67.14446	0.93708	0.94424	65.8075

Table 1. Overall performance on the Feynman–synthetic dataset. Average test performance over easy, medium, and hard tiers for SRCO and four baselines. Larger is better for

R^{2}

and Pearson correlation

\rho

; smaller is better for

\log(\text{MSE})

	Easy			Medium			Hard
Model	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )
DSO	0.59775	0.82824	45.0910	-0.60127	0.59395	67.6205	-6.17885	0.62223	66.9276
gplearn	0.13200	0.49617	45.0915	-0.30200	0.44105	67.6205	$-3.69\times 10^{5}$	0.41335	67.0701
FFX	0.48157	0.61232	43.5123	0.25864	0.60612	67.9408	0.24350	0.64633	68.4654
EFS	-0.64597	0.06667	45.0915	-10.18850	0.00000	67.6205	-38.91480	-0.16000	70.4763
SRCO	0.95045	0.96984	39.7806	0.99936	0.99940	67.1605	0.94961	0.96152	65.8739

Table 2. Overall performance on the Feynman–real-world dataset. Average test performance over easy, medium, and hard tiers for SRCO and four baselines. Larger is better for

R^{2}

and Pearson correlation

\rho

; smaller is better for

\log(\text{MSE})

Figure 2. Ablation of coefficient optimization on Feynman-bonus.1. We compare SRCO’s gradient-based coefficient fitting ( model+ ) to stochastic hill-climbing (random search; model- ) while keeping the template, train/test split, and optimization budget fixed. Bars report held-out test-set performance (higher is better for

R^{2}

and

\rho

, lower is better for MSE).

Noise (%)	$\log(\mathrm{MSE})$ $\downarrow$	$R^{2}$ $\uparrow$	Pearson $\rho$ $\uparrow$
0	-24.5128	0.9990	0.9990
10	-23.6677	0.9973	0.9976
20	-23.2118	0.9941	0.9995
30	-22.8896	0.9893	0.9916
40	-21.6436	0.9828	0.9868
50	-20.4409	0.9745	0.9802

Table 3. Robustness to feature noise. We add Gaussian noise to the test input features (scaled by per-feature standard deviations computed on the training split) and evaluate a fixed equation on the held-out test set. We report

\log(\mathrm{MSE})

for readability, along with

R^{2}

and Pearson correlation

\rho

Figure 3. Average per-equation equation-evaluation time on the test split (seconds; lower is better), averaged over six settings (2 benchmarks

\times

3 tiers: Feynman–synthetic/real-world

\times

easy/medium/hard). SRCO achieves the fastest evaluation (0.00649 s), essentially tied with EFS (0.00651 s) and outperforming DSO (2.6

\times

slower), FFX (6.1

\times

), and gplearn (38.5

\times

), while maintaining strong accuracy (Tables 1 – 2 ).

Figure 4. Pearson correlation

\rho

for max_term . Accuracy improves monotonically and saturates around 18–22 terms.

Figure 5.

R^{2}

for max_term . Results mirror Pearson correlation

\rho

, with diminishing returns after 18–22 terms.

Figure 6. Pearson correlation

\rho

for max_trig_vars . Most gains accrue by about 3–4 trig variables; larger budgets provide marginal change.

Figure 7.

R^{2}

for max_trig_vars . Performance improves quickly at small budgets and largely saturates by about 3–4, with only incremental differences through 6–10.

Explore similar work

MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery
Sep 17, 2026Peiyi Zheng, Yanming Kang, Hans De Sterck +1Symbolic Regression Scientific Discovery
University of Waterloo
GENSR: Symbolic Regression Based in Equation Generative Space
Feb 24, 2026Qian Li, Yuxiao Hu, Juncheng Liu +1Symbolic Regression Latent Space Optimization
Shanghai Jiao Tong University, Shanghai, China · Eastern Institute of Technology, Ningbo, China · The Hong Kong Polytechnic University, Hong Kong, China +1
When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization
May 22, 2026Boxiao Wang, Kai Li, Zhiwei Chen +5Scientific Discovery Symbolic Regression
Institute of Automation, Chinese Academy of Sciences

Sep 17, 2026J/K move · Enter open · S save

MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery

Peiyi Zheng, Yanming Kang, Hans De Sterck, Giang Tran

University of Waterloo

Symbolic Regression Scientific Discovery

Open paper AskPDF arXiv

Symbolic regression aims to recover closed-form equations from observations, providing interpretable models for scientific discovery. Existing approaches struggle to combine flexible structural search with efficient inference. Search-based methods can refine expression structure but often rely on costly combinatorial optimization with random initialization. Pretrained neural models generate formulas almost instantly, but their predictions often contain symbolic errors. We introduce MOSAIC-SR, which uses a pretrained Transformer to propose multiple initial sketches. These sketches initialize searches in several promising regions, avoiding random starts in the vast expression space. Each search jointly recovers structure and constants through scale-aware constant optimization and local symbolic repair. We evaluate MOSAIC-SR on the SRSD-Feynman dataset with and without dummy variables and on six additional benchmarks. MOSAIC-SR obtains the highest symbolic solution rate on every dataset while ranking among the top two methods in predictive accuracy. This advantage persists in the presence of irrelevant dummy inputs. The results show that learned priors can focus search on promising equation structures, and that numerical optimization and symbolic repair are important for recovery.

Abstract

Explore similar work

MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery

GENSR: Symbolic Regression Based in Equation Generative Space

When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization