Neural Structure Embedding for Symbolic Regression via Continuous Structure Search and Coefficient Optimization

Fateme Memar; Tao Zhe; Dongjie Wang

cs.LGMar 23, 2026

Neural Structure Embedding for Symbolic Regression via Continuous Structure Search and Coefficient Optimization

Authors: Fateme Memar, Tao Zhe, Dongjie Wang

Organizations: Department of Electrical Engineering and Computer Science, University of Kansas USA

Abstract

Symbolic regression aims to discover human-interpretable equations that explain observational data. However, existing approaches rely heavily on discrete structure search (e.g., genetic programming), which often leads to high computational cost, unstable performance, and limited scalability to large equation spaces. To address these challenges, we propose SRCO, a unified embedding-driven framework for symbolic regression that transforms symbolic structures into a continuous, optimizable representation space. The framework consists of three key components: (1) structure embedding: we first generate a large pool of exploratory equations using traditional symbolic regression algorithms and train a Transformer model to compress symbolic structures into a continuous embedding space; (2) continuous structure search: the embedding space enables efficient exploration using gradient-based or sampling-based optimization, significantly reducing the cost of navigating the combinatorial structure space; and (3) coefficient optimization: for each discovered structure, we treat symbolic coefficients as learnable parameters and apply gradient optimization to obtain accurate numerical values. Experiments on synthetic and real-world datasets show that our approach consistently outperforms state-of-the-art methods in equation accuracy, robustness, and search efficiency. This work introduces a new paradigm for symbolic regression by bridging symbolic equation discovery with continuous embedding learning and optimization.

Figures & tables

Figure 1. Framework Overview of SRCO . Structure Embedding : A GP-based SR system generates diverse candidate equations, which are converted into postfix sequences with abstracted coefficients ( COF ) to train a Transformer-based structural prior; Continuous Structure Search : The learned prior guides constrained sampling in postfix space, followed by syntactic, semantic, and complexity filtering to obtain valid symbolic templates; Coefficient Optimization : For each selected template, COF tokens are instantiated as learnable parameters and optimized via gradient-based regression to produce the final symbolic equation.

	Easy			Medium			Hard
Model	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )
DSO	0.58108	0.76836	45.0992	-3.63397	0.57411	67.6205	$-2.37841\times 10^{5}$	0.55307	67.0060
gplearn	-0.25600	0.41807	45.0917	0.30200	0.42552	67.6205	$-1.84\times 10^{13}$	0.39730	67.2514
FFX	0.45998	0.57570	45.0911	0.27657	0.61437	67.9408	0.26691	0.67575	68.4654
EFS	-6.79227	0.06667	45.0915	-21.75270	0.00000	67.6205	-10.32860	-0.12000	70.4763
SRCO	0.99821	0.96556	30.4281	0.99555	0.99786	67.14446	0.93708	0.94424	65.8075

Table 1. Overall performance on the Feynman–synthetic dataset. Average test performance over easy, medium, and hard tiers for SRCO and four baselines. Larger is better for

R^{2}

and Pearson correlation

\rho

; smaller is better for

\log(\text{MSE})

	Easy			Medium			Hard
Model	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	Pearson $\rho$ ( $\uparrow$ )	$\log(\text{MSE})$ ( $\downarrow$ )
DSO	0.59775	0.82824	45.0910	-0.60127	0.59395	67.6205	-6.17885	0.62223	66.9276
gplearn	0.13200	0.49617	45.0915	-0.30200	0.44105	67.6205	$-3.69\times 10^{5}$	0.41335	67.0701
FFX	0.48157	0.61232	43.5123	0.25864	0.60612	67.9408	0.24350	0.64633	68.4654
EFS	-0.64597	0.06667	45.0915	-10.18850	0.00000	67.6205	-38.91480	-0.16000	70.4763
SRCO	0.95045	0.96984	39.7806	0.99936	0.99940	67.1605	0.94961	0.96152	65.8739

Table 2. Overall performance on the Feynman–real-world dataset. Average test performance over easy, medium, and hard tiers for SRCO and four baselines. Larger is better for

R^{2}

and Pearson correlation

\rho

; smaller is better for

\log(\text{MSE})

Figure 2. Ablation of coefficient optimization on Feynman-bonus.1. We compare SRCO’s gradient-based coefficient fitting ( model+ ) to stochastic hill-climbing (random search; model- ) while keeping the template, train/test split, and optimization budget fixed. Bars report held-out test-set performance (higher is better for

R^{2}

and

\rho

, lower is better for MSE).

Noise (%)	$\log(\mathrm{MSE})$ $\downarrow$	$R^{2}$ $\uparrow$	Pearson $\rho$ $\uparrow$
0	-24.5128	0.9990	0.9990
10	-23.6677	0.9973	0.9976
20	-23.2118	0.9941	0.9995
30	-22.8896	0.9893	0.9916
40	-21.6436	0.9828	0.9868
50	-20.4409	0.9745	0.9802

Table 3. Robustness to feature noise. We add Gaussian noise to the test input features (scaled by per-feature standard deviations computed on the training split) and evaluate a fixed equation on the held-out test set. We report

\log(\mathrm{MSE})

for readability, along with

R^{2}

and Pearson correlation

\rho

Figure 3. Average per-equation equation-evaluation time on the test split (seconds; lower is better), averaged over six settings (2 benchmarks

\times

3 tiers: Feynman–synthetic/real-world

\times

easy/medium/hard). SRCO achieves the fastest evaluation (0.00649 s), essentially tied with EFS (0.00651 s) and outperforming DSO (2.6

\times

slower), FFX (6.1

\times

), and gplearn (38.5

\times

), while maintaining strong accuracy (Tables 1 – 2 ).

Figure 4. Pearson correlation

\rho

for max_term . Accuracy improves monotonically and saturates around 18–22 terms.

Figure 5.

R^{2}

for max_term . Results mirror Pearson correlation

\rho

, with diminishing returns after 18–22 terms.

Figure 6. Pearson correlation

\rho

for max_trig_vars . Most gains accrue by about 3–4 trig variables; larger budgets provide marginal change.

Figure 7.

R^{2}

for max_trig_vars . Performance improves quickly at small budgets and largely saturates by about 3–4, with only incremental differences through 6–10.

Explore similar work

Sep 17, 2026

MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery

Symbolic regression aims to recover closed-form equations from observations, providing interpretable models for scientific discovery. Existing approaches struggle to combine flexible structural search with efficient inference. Search-based methods can refine expression structure but often rely on costly combinatorial optimization with random initialization. Pretrained neural models generate formulas almost instantly, but their predictions often contain symbolic errors. We introduce MOSAIC-SR, which uses a pretrained Transformer to propose multiple initial sketches. These sketches initialize searches in several promising regions, avoiding random starts in the vast expression space. Each search jointly recovers structure and constants through scale-aware constant optimization and local symbolic repair. We evaluate MOSAIC-SR on the SRSD-Feynman dataset with and without dummy variables and on six additional benchmarks. MOSAIC-SR obtains the highest symbolic solution rate on every dataset while ranking among the top two methods in predictive accuracy. This advantage persists in the presence of irrelevant dummy inputs. The results show that learned priors can focus search on promising equation structures, and that numerical optimization and symbolic repair are important for recovery.

Peiyi Zheng, Yanming Kang, Hans De Sterck +1

University of Waterloo

Symbolic Regression Scientific Discovery

Ask PDF

Feb 24, 2026

GENSR: Symbolic Regression Based in Equation Generative Space

Symbolic Regression (SR) tries to reveal the hidden equations behind observed data. However, most methods search within a discrete equation space, where the structural modifications of equations rarely align with their numerical behavior, leaving fitting error feedback too noisy to guide exploration. To address this challenge, we propose GenSR, a generative latent space-based SR framework following the

map construction -> coarse localization -> fine search'' paradigm. Specifically, GenSR first pretrains a dual-branch Conditional Variational Autoencoder (CVAE) to reparameterize symbolic equations into a generative latent space with symbolic continuity and local numerical smoothness. This space can be regarded as a well-structured

map'' of the equation space, providing directional signals for search. At inference, the CVAE coarsely localizes the input data to promising regions in the latent space. Then, a modified CMA-ES refines the candidate region, leveraging smooth latent gradients. From a Bayesian perspective, GenSR reframes the SR task as maximizing the conditional distribution

p(\mathrm{Equ.} \mid \mathrm{Num.})

, with CVAE training achieving this objective through the Evidence Lower Bound (ELBO). This new perspective provides a theoretical guarantee for the effectiveness of GenSR. Extensive experiments show that GenSR jointly optimizes predictive accuracy, expression simplicity, and computational efficiency, while remaining robust under noise.

Qian Li, Yuxiao Hu, Juncheng Liu +1

Shanghai Jiao Tong University, Shanghai, China · Eastern Institute of Technology, Ningbo, China · The Hong Kong Polytechnic University, Hong Kong, China +1

Symbolic Regression Latent Space Optimization

Ask PDF

May 22, 2026

When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization

Symbolic Regression (SR) plays a central role in scientific knowledge discovery by distilling mathematical equations from observational data. Most existing SR methods function within a bi-level optimization framework: an outer loop that searches for the discrete equation structure, and an inner loop that optimizes the continuous parameters of that structure. Crucially, parameter-fitting quality directly determines a structure's score and thus the outer-loop search. However, nonlinear operators make the inner loop highly non-convex, and budget-driven reliance on fast local solvers (e.g., BFGS) often yields poor local minima and underestimated scores for correct structures. This ``Good Structure, Bad Score'' phenomenon becomes a key bottleneck, degrading efficiency and misguiding the search away from the true equation. To resolve this, we propose SAGE-Fit (Structure-Aware and Semantics-Guided Evaluator for Symbolic Regression), an SR-native fitting framework that exploits the dual native priors of symbolic expressions. By capitalizing on the structural and semantic priors unique to SR, we design tailored modules for each property, thereby effectively mitigating this optimization bottleneck. Extensive experiments demonstrate that our approach, as a plug-and-play module, significantly enhances evaluation fidelity and universally improves the performance of various SR systems.

Boxiao Wang, Kai Li, Zhiwei Chen +5

Institute of Automation, Chinese Academy of Sciences

Scientific Discovery Symbolic Regression

Ask PDF