Authors: Shuwei Wang, Vadim Bulitko, Michael Youngblood, Ramon Lawrence, William Yeoh, Shinichi Nakagawa, Matthew R. G. Brown, Yu Wang
Organizations: Department of Computing Science, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada · Filuta AI, 1606 Headway Circle, Austin, TX 78754, USA · Department of Computer Science, Data Science, University of British Columbia, 6200 University Blvd, Vancouver, BC Canada V6T 1Z4, Canada · McKelvey School of Engineering, Washington University in St. Louis, 1 Brookings Drive, St. Louis, MO 63130-4899, USA · Department of Biology, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada
Finding a compact formula that fits a set of input-output pairs and predicts outputs on unseen inputs is a fundamental problem in science. Symbolic regression automates the search for such formulae: search-based methods explore the space of possible formulae directly, while transformers pre-trained on synthetic data produce formulae of comparable quality substantially faster. Existing transformers, however, are prone to overfitting --- they find formulae that fit the training data well but do not extrapolate to input ranges unseen during training. We address this by shaping the set of formulae used to train a transformer, and show that the resulting formulae extrapolate substantially better. Fine-tuning the transformer on data with noise-corrupted target values further makes the synthesized formulae robust to noise in the observations. On SRBench and LLM-SRBench our transformer synthesizes a formula in about ten seconds and extrapolates better than all evaluated methods at a comparable budget. Search-based methods surpass our accuracy only when given one to three orders of magnitude more time.
Figures & tables
Figure 1: Solution rate ρk versus extrapolation constant k on the 99 SRBench formulae and on the 111 LSR-Transform formulae.
Figure 2: Average per formula synthesis time and average synthesized formula size on the 99 SRBench formulae and on the 111 LSR-Transform formulae. The median formula size and standard deviation of the ground-truth formulae are shown by the dashed vertical line and the gray region.
Figure 3: In-distribution and out-of-distribution solution rate on the 129 LSR-Synth formulae.
Figure 4: The ablation study results on affine transformation and simplification.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Binary
Unary (continued)
S+S
+
1
S−−
- -
0.35
S×S
1
∣S∣
abs
0.2
S−S
-
0.8
S3
pow3
0.2
S/S
/
0.6
tan(S)
tan
0.12
SS
pow
0.1
arcsin(S)
arcsin
0.1
Unary
arctan(S)
arctan
0.1
Appendix
Table 1: Unnormalized sampling weight of each operator. The probability of drawing an operator is its weight divided by the sum of the weights in its group (binary or unary). pow2 is identical to sqr and is kept only for compatibility with the token vocabulary.
Parameter
Value
Architecture
hidden layers
128,128,64,64
hidden activation
tanh
output layer
linear, one unit
weights
128m+29,121 for m variables
Input and target scaling
Appendix
Table 2: Configuration of the oracle network used by the divide-and-conquer strategy.
Figure 5: Left: solution rate ρk versus extrapolation constant k on the 111 LSR-Transform formulae. Right: mean per-formula synthesis time and mean synthesized formula size on the same formulae, with the median size and ±1 standard deviation of the ground-truth formulae shown by the dashed vertical line and the gray region. The LLM-SR with Gemini curve is compared against the single best seed of each other method in the noise-free setting.
Figure 6: Solution rate ρk versus extrapolation constant k on the 99 SRBench formulae and the 111 LSR-Transform formulae, with every formula-trained arm drawn.
Figure 7: Time per formula and predicted formula complexity for SRBench and LSR-Transform.
Figure 8: Solution rate ρk versus extrapolation constant k on the 99 SRBench formulae and the 111 LSR-Transform formulae, with every LLM arm drawn.
Figure 9: Time per formula and predicted formula complexity for SRBench and LSR-Transform.
Symbol
Value
Meaning
Training data generation (Section 4.1 )
Vmax
10
largest number of variables in a formula
l
1
smallest number of nodes in a formula
r
40
largest number of nodes in a formula
γ
0.99
decay of the formula size distribution
N
U{50,…,400}
data points per training instance
Appendix
Table 3: The control parameters from the main text of the paper and their values.
Symbolic regression aims to recover closed-form equations from observations, providing interpretable models for scientific discovery. Existing approaches struggle to combine flexible structural search with efficient inference. Search-based methods can refine expression structure but often rely on costly combinatorial optimization with random initialization. Pretrained neural models generate formulas almost instantly, but their predictions often contain symbolic errors. We introduce MOSAIC-SR, which uses a pretrained Transformer to propose multiple initial sketches. These sketches initialize searches in several promising regions, avoiding random starts in the vast expression space. Each search jointly recovers structure and constants through scale-aware constant optimization and local symbolic repair. We evaluate MOSAIC-SR on the SRSD-Feynman dataset with and without dummy variables and on six additional benchmarks. MOSAIC-SR obtains the highest symbolic solution rate on every dataset while ranking among the top two methods in predictive accuracy. This advantage persists in the presence of irrelevant dummy inputs. The results show that learned priors can focus search on promising equation structures, and that numerical optimization and symbolic repair are important for recovery.
Scientific equation discovery must combine broad domain priors with strict numerical testing. Symbolic regression supplies numerical grounding but faces a combinatorial search space, whereas many language-model systems ask the model to propose or select formulas directly. We test a different division of labour. We compare role specifications in which the language model acts as equation author, candidate decider or search controller, alongside end-to-end language-model and purely numerical baselines. In the controller setting we propose here, implemented as LLM-PySR, language models specify variables, operators, transformations and search depth; symbolic regression enumerates and fits expressions; and deterministic metrics govern retention. Across 74 AI-Feynman equations and seven complex formula-recovery tasks, search control achieved the strongest observed balance of accuracy, complexity, stability and cost. On an independent battery dataset, LLM-PySR identified a compact piecewise-linear relation between early voltage-curve displacement and cycle life. The results suggest that language models should shape hypothesis exploration rather than decide which equations survive.
Zikai Xie, Wenmei Li, Man Luo +2
State Key Laboratory of Precision and Intelligent Chemistry, Hefei National Research Center for Physical Sciences at the Microscale, School of Chemistry and Materials Science, University of Science and Technology of China, Hefei, China. · Center for Scientific Intelligence Innovation, Hefei, China · School of Chemistry, School of Computer Science, University of Birmingham, Birmingham, UK
Symbolic regression combines structural search with numerical fitting, but aggregate fit scores do not describe how the remaining error varies across inputs. We introduce RISR, a residual-informed method that uses these error patterns to guide formula discovery and learn which corrections are worth fitting. A residual encoder compresses aligned inputs, targets, current predictions, and residuals into continuous tokens that condition a language model to propose formulas. For subsequent refinement, a dual-view relational encoder uses additive and regularized multiplicative residuals to predict the post-fit utility of candidate corrections. We evaluate RISR on scientific tasks from the LLM-SRBench. RISR achieves 63.57% and 38.50% ID accuracy at the 1% and 0.1% pointwise relative-error tolerances, respectively. The corresponding OOD accuracies are 56.07% and 38.24%. RISR outperforms the reported baselines using the same backbone. The results show that our residual-informed approach can improve numerical equation recovery.
Haobo Li, Wenshuo Zhang, Wenxiao Zhao +6
Shanghai AI Laboratory · The Hong Kong University of Science and Technology · University of California, Los Angeles