Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is central to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical embeddings, reliably distinguish the effects of local edits that the original alignment cannot, and transfer to external SR corpora.
Figures & tables
Figure 1: Overview of SNIP++. A controlled edit replaces the subtree s=sin(x) with s~=cos(x) , producing the perturbed parent expression f~ . From f and s , the symbolic encoder yields the complete-expression embedding gS , the subtree span pooled in context c , and the standalone subtree embedding o ; g~S , c~ , and o~ denote the corresponding representations of f~ and s~ . The numerical encoder similarly maps their full-expression and subtree behaviors on shared inputs X to gV,g~V and p,p~ . Original and perturbed pairs are aligned symmetrically through the local losses Cτs(C,P) and Cτs(O,P) , and the expanded global loss Cτg(GS,+,GV,+) , where + denotes augmenting SNIP’s original complete-expression objective with perturbed parent–behavior pairs. Tree-structural position encoding (inset) : Each token records its nearest L branch choices (first vs. second child) on the path to the root ( L=2 here). In channel c , choice k ( k=0 nearest) is weighted by learned ρck , with ∣ρc∣<1 ; all codes are concatenated so that 2CL=dmodel .
Figure 2: Joint symbolic–numerical embedding geometry. PCA projections of symbolic ( f ) and numerical ( y ) representations for 800 held-out expression–behavior pairs. SNIP separates the two modalities, whereas SNIP++ substantially interleaves them and reduces the modality gap from approximately 1.2 to 0.3 .
Edit type
Example
Seen in training
Unary swap
sin(a)→cos(a)
Variable substitution
x0→x1
Binary swap
a+b→a×b
Held out
Tree reshaping
(a−b)−c→a−(b−c)
Table 2: Controlled symbolic edits. The first three edit types are used during training; the final three test generalization to unseen transformations. All evaluation expressions and individual edits are held out.
Figure 3: Fine-grained symbolic discrimination. Percentile rank of the correct complete-expression pairing across six edit types and both alignment directions. The red dashed line marks chance ( .5 ); shaded rows average the training-represented and held-out edits.
Figure 4: Out-of-domain generalization. Percentile rank of the correct pairing on Feynman, Classic, Strogatz and Pooled over all three for (a) within-corpus global symbolic–numerical alignment and (b) expression-level fine-grained discrimination, evaluated in both alignment directions. The dashed line marks chance ( .5 ).
Figure 5: Functional structure in the learned space. (a) continuous functional change, (b) behavior-preserving versus behavior-changing edits, (c) equivalent rewritings versus near-miss alternatives, and (d) cross-modal similarity to Taylor approximations.
Figure 6: Cumulative ablation from SNIP to SNIP++ . (A) global symbolic–numerical alignment, measured by nDCG@10 in both directions; (B) modality gap, measured as the distance between the symbolic and numerical embedding centroids; (C) fine-grained expression-level discrimination; and (D) fine-grained subtree-level discrimination, both measured by percentile rank of the correct pairing. The largest gains come from adding fine-grained subtree supervision, while the later stages further improve balance across directions and recover the full SNIP++ performance.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Node
Root-to-node path
Node-to-root slots
sin
()
(∅,∅,…)
+
(1)
(1,∅,…)
x0
(1,1)
(1,1,∅,…)
×
(1,2)
(2,1,∅,…)
2.1
(1,2,1)
(1,2,1,∅,…)
x1
(1,2,2)
(2,2,1,∅,…)
Appendix
Table 3: Expression tree and corresponding root-to-node and node-to-root child-slot paths for f(x0,x1)=sin(x0+2.1x1) .
Figure 7: Construction of perturbed training examples. Given a parent expression fi , we select a subtree si0 and construct variants si1,…,siKi . Each variant is inserted at the original subtree location to obtain fik=fi[si0←sik] . The parent expression and all of its variants are evaluated on the same input set Xi . Colored nodes indicate the modified subtree.
Edit family
Targets
Mean
Median
Unary swap
81
2.1
2
Variable substitution
119
26.5
27
Binary swap
127
8.5
9
Tree reshaping
127
10.7
11
Operand reordering
90
1.8
2
Full-subtree rewriting
127
57.6
59
Appendix
Table 4: Complete-expression controlled-edit pool. Targets denotes the number of held-out expressions with at least one retained variant. Mean and median give the number of retained variants per contributing target.
Figure 8: Contextual-subtree discrimination. Percentile rank of the correct subtree–behavior pairing across the six controlled edit families. The red dashed line marks chance ( .5 ), and shaded rows report averages over the three training-represented and three held-out edit families. Error bars show 95% bootstrap confidence intervals over parent expressions.
Evaluation set
Complete expressions
Subtree parents
Feynman
81
79
Classic
35
30
Strogatz
13
12
Pooled union
129
121
Appendix
Table 5: Out-of-domain evaluation pools. Counts are reported after validity filtering.
Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.
Benjamin Léger, Kazem Meidani, Christian Gagné
IID / Mila, Université Laval, Québec, Canada · Department of Mechanical Engineering, Carnegie Mellon University Canada-CIFAR AI Chair
Symbolic regression is the problem of finding an expression that fits a dataset. Solving it requires searching a combinatorial space of candidate expressions. Neural symbolic regression can amortize that search by training a model once on synthetic data and then predicting expressions for a new dataset. But existing models still underperform genetic programming on real, noisy data. We argue that the deficit lies in the training rather than in the paradigm, and present RASR (Reinforced Amortized Symbolic Regression), a method with two key changes to the training recipe. First, we design a synthetic prior that narrows the gap with real-world data: inputs come from the prior of a tabular foundation model; constants are sampled bottom-up, conditioned on the data; targets carry heteroscedastic and structured noise; and everything is standardized. Second, we split training into two stages. Supervised pre-training maximizes the token-level likelihood of a reference expression. This is followed by reinforcement-learning post-training that aligns the model with its inference-time use by rewarding held-out fit and penalizing expression size. On the black-box track of SRBench 2.0, RASR achieves the best mean test R2 rank of all evaluated methods. Three RASR variants, trained with different size penalties, make up most of the accuracy-complexity Pareto front. On Strogatz, our method recovers more ground-truth expressions than any baseline, and on Feynman it is competitive with the best-performing methods.
David Otte, Jörg K. H. Franke, Frank Hutter
LMU Munich · Munich Center for Machine Learning (MCML) · University of Freiburg +3
Symbolic regression (SR) seeks closed-form mathematical expressions that fit observed data. Neural SR methods amortize the search by training an encoder to map observations directly to expressions in a single pass, but this amortized inference leaves a residual amortization gap between its one-shot prediction and the true posterior. We propose Latent Equation Embedding (LEE), a framework that closes this gap through iterative amortized inference in a functionally grounded latent space. LEE learns a shared latent space Z equipped with three components: an encoder f_theta that jointly embeds symbolic tokens and numerical observations into a single latent vector z; an expression decoder g_expr that reconstructs formulas from z; and an evaluation decoder g_eval that predicts function values from z, explicitly grounding the latent space in functional behavior. At inference, LEE performs iterative refinement by re-encoding decoded expressions jointly with observations, progressively improving the latent estimate. LEE uses the encoder itself as a learned inference optimizer: each re-encoding step implicitly computes the mismatch between the candidate and the data. Because g_eval is differentiable in z, we additionally interleave continuous gradient descent with discrete re-encoding, yielding a hybrid iterative and gradient refinement procedure. On SRBench across three noise levels, against 19 baselines spanning genetic programming, symbolic-neural hybrids, and pre-trained Transformers, LEE produces expressions 2--10x simpler than the strongest accuracy-oriented baselines, including Operon, GP-GOMEA, TPSR, RAG-SR, and GenSR, with complexity 8--11 versus 20--90. These results advance the low-complexity region of the accuracy-complexity Pareto frontier and show graceful degradation as noise increases.