Synthesizing Physics Formulae with Transformers

Shuwei Wang; Vadim Bulitko; Michael Youngblood; Ramon Lawrence; William Yeoh; Shinichi Nakagawa; Matthew R. G. Brown; Yu Wang

cs.LGOct 2, 2026

Synthesizing Physics Formulae with Transformers

Authors: Shuwei Wang, Vadim Bulitko, Michael Youngblood, Ramon Lawrence, William Yeoh, Shinichi Nakagawa, Matthew R. G. Brown, Yu Wang

Organizations: Department of Computing Science, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada · Filuta AI, 1606 Headway Circle, Austin, TX 78754, USA · Department of Computer Science, Data Science, University of British Columbia, 6200 University Blvd, Vancouver, BC Canada V6T 1Z4, Canada · McKelvey School of Engineering, Washington University in St. Louis, 1 Brookings Drive, St. Louis, MO 63130-4899, USA · Department of Biology, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada

Abstract

Finding a compact formula that fits a set of input-output pairs and predicts outputs on unseen inputs is a fundamental problem in science. Symbolic regression automates the search for such formulae: search-based methods explore the space of possible formulae directly, while transformers pre-trained on synthetic data produce formulae of comparable quality substantially faster. Existing transformers, however, are prone to overfitting --- they find formulae that fit the training data well but do not extrapolate to input ranges unseen during training. We address this by shaping the set of formulae used to train a transformer, and show that the resulting formulae extrapolate substantially better. Fine-tuning the transformer on data with noise-corrupted target values further makes the synthesized formulae robust to noise in the observations. On SRBench and LLM-SRBench our transformer synthesizes a formula in about ten seconds and extrapolates better than all evaluated methods at a comparable budget. Search-based methods surpass our accuracy only when given one to three orders of magnitude more time.

Figures & tables

Figure 1: Solution rate

\rho_{k}

versus extrapolation constant

k

on the

99

SRBench formulae and on the

111

LSR-Transform formulae.

Figure 2: Average per formula synthesis time and average synthesized formula size on the

99

SRBench formulae and on the

111

LSR-Transform formulae. The median formula size and standard deviation of the ground-truth formulae are shown by the dashed vertical line and the gray region.

Figure 3: In-distribution and out-of-distribution solution rate on the

129

LSR-Synth formulae.

Figure 4: The ablation study results on affine transformation and simplification.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Binary			Unary (continued)
$S+S$	+	$1$	$S-\!-$	- -	$0.35$
$S\times S$		$1$	$\|S\|$	abs	$0.2$
$S-S$	-	$0.8$	$S^{3}$	pow3	$0.2$
$S/S$	/	$0.6$	$\tan(S)$	tan	$0.12$
$S^{S}$	pow	$0.1$	$\arcsin(S)$	arcsin	$0.1$
Unary			$\arctan(S)$	arctan	$0.1$

Appendix

Table 1: Unnormalized sampling weight of each operator. The probability of drawing an operator is its weight divided by the sum of the weights in its group (binary or unary). pow2 is identical to sqr and is kept only for compatibility with the token vocabulary.

Architecture
Parameter	Value
hidden layers	$128,128,64,64$
hidden activation	$\tanh$
output layer	linear, one unit
weights	$128m+29{,}121$ for $m$ variables
Input and target scaling

Appendix

Table 2: Configuration of the oracle network used by the divide-and-conquer strategy.

Figure 5: Left: solution rate

\rho_{k}

versus extrapolation constant

k

on the

111

LSR-Transform formulae. Right: mean per-formula synthesis time and mean synthesized formula size on the same formulae, with the median size and

\pm 1

standard deviation of the ground-truth formulae shown by the dashed vertical line and the gray region. The LLM-SR with Gemini curve is compared against the single best seed of each other method in the noise-free setting.

Figure 6: Solution rate

\rho_{k}

versus extrapolation constant

k

on the

99

SRBench formulae and the

111

LSR-Transform formulae, with every formula-trained arm drawn.

Figure 7: Time per formula and predicted formula complexity for SRBench and LSR-Transform.

Figure 8: Solution rate

\rho_{k}

versus extrapolation constant

k

on the

99

SRBench formulae and the

111

LSR-Transform formulae, with every LLM arm drawn.

Figure 9: Time per formula and predicted formula complexity for SRBench and LSR-Transform.

Symbol	Value	Meaning
Training data generation (Section 4.1 )
$V_{\text{max}}$	$10$	largest number of variables in a formula
$l$	$1$	smallest number of nodes in a formula
$r$	$40$	largest number of nodes in a formula
$\gamma$	$0.99$	decay of the formula size distribution
$N$	$\mathcal{U}\{50,\ldots,400\}$	data points per training instance

Appendix

Table 3: The control parameters from the main text of the paper and their values.

Explore similar work

MorphingNet: Symbolic Regression Where the Network Is the Formula
Oct 14, 2026Jincheng Gao, Jiahe Yi, Shuo SunSymbolic Regression Feedforward Neural Networks
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery
Sep 17, 2026Peiyi Zheng, Yanming Kang, Hans De Sterck +1Symbolic Regression Scientific Discovery
University of Waterloo
Language models guide symbolic equation discovery by controlling search
Jul 5, 2026Zikai Xie, Wenmei Li, Man Luo +2Large Language Model-Guided Scientific Discovery Symbolic Regression
State Key Laboratory of Precision and Intelligent Chemistry, Hefei National Research Center for Physical Sciences at the Microscale, School of Chemistry and Materials Science, University of Science and Technology of China, Hefei, China. · Center for Scientific Intelligence Innovation, Hefei, China · School of Chemistry, School of Computer Science, University of Birmingham, Birmingham, UK

Oct 14, 2026J/K move · Enter open · S save

MorphingNet: Symbolic Regression Where the Network Is the Formula

Jincheng Gao, Jiahe Yi, Shuo Sun

Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Symbolic Regression Feedforward Neural Networks

Open paper AskPDF arXiv

We introduce MorphingNet, a network for symbolic regression (SR) whose units change their function continuously. Instead of searching over expressions built from a fixed operator library, MorphingNet trains the form of each function as a parameter, so SR becomes one continuous optimization problem. Its units are either pole-free rational functions or complex exponential--logarithm functions, which move between exponentials, sinusoids, damped oscillations, power laws and logarithms as their parameters change. The network is a readable formula at every step of training, and no operator library is needed to train it; a structure-recovery step afterwards rewrites the trained formula into short elementary expressions. Training is dense linear algebra, so it is GPU-friendly and scales to large data. MorphingNet achieves state-of-the-art accuracy on SRBench, recovers most generating formulas on synthetic benchmarks, and on large data is up to two orders of magnitude faster than the strongest search method. By turning the search for a formula into the training of a network, MorphingNet offers a new paradigm for SR and for scientific discovery from data.

Abstract

Explore similar work

MorphingNet: Symbolic Regression Where the Network Is the Formula

MOSAIC-SR: Transformer-Guided Symbolic Regression for Scientific Equation Recovery

Language models guide symbolic equation discovery by controlling search