SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression

Benjamin Léger; Shubham Gupta; Samy Mammeri; Kazem Meidani; Cem Subakan; Christian Gagné

cs.NESep 25, 2026

SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression

Authors: Benjamin Léger, Shubham Gupta, Samy Mammeri, Kazem Meidani, Cem Subakan, Christian Gagné

Organizations: Université Laval · Mila · Department of Mechanical Engineering, Carnegie Mellon University · Canada CIFAR AI Chair

Abstract

Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is central to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical embeddings, reliably distinguish the effects of local edits that the original alignment cannot, and transfer to external SR corpora.

Figures & tables

Figure 1: Overview of SNIP++. A controlled edit replaces the subtree

s=\sin(x)

with

\tilde{s}=\cos(x)

, producing the perturbed parent expression

\tilde{f}

. From

f

and

s

, the symbolic encoder yields the complete-expression embedding

g^{S}

, the subtree span pooled in context

c

, and the standalone subtree embedding

o

;

\tilde{g}^{S}

\tilde{c}

, and

\tilde{o}

denote the corresponding representations of

\tilde{f}

and

\tilde{s}

. The numerical encoder similarly maps their full-expression and subtree behaviors on shared inputs

X

g^{V},\tilde{g}^{V}

and

p,\tilde{p}

. Original and perturbed pairs are aligned symmetrically through the local losses

\mathcal{C}_{\tau_{s}}(C,P)

and

\mathcal{C}_{\tau_{s}}(O,P)

, and the expanded global loss

\mathcal{C}_{\tau_{g}}(G^{S,+},G^{V,+})

, where

+

denotes augmenting SNIP’s original complete-expression objective with perturbed parent–behavior pairs. Tree-structural position encoding (inset) : Each token records its nearest

L

branch choices (first vs. second child) on the path to the root (

L=2

here). In channel

c

, choice

k

(

k=0

nearest) is weighted by learned

\rho_{c}^{k}

, with

|\rho_{c}|<1

; all codes are concatenated so that

2CL=d_{\mathrm{model}}

Figure 2: Joint symbolic–numerical embedding geometry. PCA projections of symbolic (

f

) and numerical (

y

) representations for

800

held-out expression–behavior pairs. SNIP separates the two modalities, whereas SNIP++ substantially interleaves them and reduces the modality gap from approximately

1.2

0.3

Edit type	Example
Seen in training
Unary swap	$\sin(a)\!\rightarrow\!\cos(a)$
Variable substitution	$x_{0}\!\rightarrow\!x_{1}$
Binary swap	$a+b\!\rightarrow\!a\times b$
Held out
Tree reshaping	$(a-b)-c\!\rightarrow\!a-(b-c)$

Table 2: Controlled symbolic edits. The first three edit types are used during training; the final three test generalization to unseen transformations. All evaluation expressions and individual edits are held out.

Figure 3: Fine-grained symbolic discrimination. Percentile rank of the correct complete-expression pairing across six edit types and both alignment directions. The red dashed line marks chance (

.5

); shaded rows average the training-represented and held-out edits.

Figure 4: Out-of-domain generalization. Percentile rank of the correct pairing on Feynman, Classic, Strogatz and Pooled over all three for (a) within-corpus global symbolic–numerical alignment and (b) expression-level fine-grained discrimination, evaluated in both alignment directions. The dashed line marks chance (

.5

Figure 5: Functional structure in the learned space. (a) continuous functional change, (b) behavior-preserving versus behavior-changing edits, (c) equivalent rewritings versus near-miss alternatives, and (d) cross-modal similarity to Taylor approximations.

Figure 6: Cumulative ablation from SNIP to SNIP++ . (A) global symbolic–numerical alignment, measured by nDCG@10 in both directions; (B) modality gap, measured as the distance between the symbolic and numerical embedding centroids; (C) fine-grained expression-level discrimination; and (D) fine-grained subtree-level discrimination, both measured by percentile rank of the correct pairing. The largest gains come from adding fine-grained subtree supervision, while the later stages further improve balance across directions and recover the full SNIP++ performance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Node	Root-to-node path	Node-to-root slots
$\sin$	$()$	$(\varnothing,\varnothing,\ldots)$
$+$	$(1)$	$(1,\varnothing,\ldots)$
$x_{0}$	$(1,1)$	$(1,1,\varnothing,\ldots)$
$\times$	$(1,2)$	$(2,1,\varnothing,\ldots)$
$2.1$	$(1,2,1)$	$(1,2,1,\varnothing,\ldots)$
$x_{1}$	$(1,2,2)$	$(2,2,1,\varnothing,\ldots)$

Appendix

Table 3: Expression tree and corresponding root-to-node and node-to-root child-slot paths for

f(x_{0},x_{1})=\sin(x_{0}+2.1\,x_{1})

Figure 7: Construction of perturbed training examples. Given a parent expression

f_{i}

, we select a subtree

s_{i0}

and construct variants

s_{i1},\ldots,s_{iK_{i}}

. Each variant is inserted at the original subtree location to obtain

f_{ik}=f_{i}[s_{i0}\leftarrow s_{ik}]

. The parent expression and all of its variants are evaluated on the same input set

X_{i}

. Colored nodes indicate the modified subtree.

Edit family	Targets	Mean	Median
Unary swap	81	2.1	2
Variable substitution	119	26.5	27
Binary swap	127	8.5	9
Tree reshaping	127	10.7	11
Operand reordering	90	1.8	2
Full-subtree rewriting	127	57.6	59

Appendix

Table 4: Complete-expression controlled-edit pool. Targets denotes the number of held-out expressions with at least one retained variant. Mean and median give the number of retained variants per contributing target.

Figure 8: Contextual-subtree discrimination. Percentile rank of the correct subtree–behavior pairing across the six controlled edit families. The red dashed line marks chance (

.5

), and shaded rows report averages over the three training-represented and three held-out edit families. Error bars show

95\%

bootstrap confidence intervals over parent expressions.

Evaluation set	Complete expressions	Subtree parents
Feynman	81	79
Classic	35	30
Strogatz	13	12
Pooled union	129	121

Appendix

Table 5: Out-of-domain evaluation pools. Counts are reported after validity filtering.

Explore similar work

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
Apr 9, 2026Benjamin Léger, Kazem Meidani, Christian GagnéLatent Space Optimization Symbolic Regression
IID / Mila, Université Laval, Québec, Canada · Department of Mechanical Engineering, Carnegie Mellon University Canada-CIFAR AI Chair
Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment
Oct 13, 2026David Otte, Jörg K. H. Franke, Frank HutterSymbolic Regression Synthetic Data Pretraining
LMU Munich · Munich Center for Machine Learning (MCML) · University of Freiburg +3
Symbolic Regression via Latent Iterative Refinement
May 26, 2026Xieting Chu, Sriram Vishwanath, Vijay GaneshLatent Variable Models Amortized Inference
Georgia Institute of Technology

Apr 9, 2026J/K move · Enter open · S save

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization

Benjamin Léger, Kazem Meidani, Christian Gagné

IID / Mila, Université Laval, Québec, Canada · Department of Mechanical Engineering, Carnegie Mellon University Canada-CIFAR AI Chair

Latent Space Optimization Symbolic Regression Cross-Modal Alignment Genetic Programming Multimodal Contrastive Learning

Open paper AskPDF arXiv

Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.

Abstract

Explore similar work

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization

Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment

Symbolic Regression via Latent Iterative Refinement