cs.NEApr 9, 2026

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization

Authors: Benjamin Léger, Kazem Meidani, Christian Gagné

Organizations: IID / Mila, Université Laval, Québec, Canada · Department of Mechanical Engineering, Carnegie Mellon University Canada-CIFAR AI Chair

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.

Figures & tables

Figure 1: Overview of Latent Space Optimization and multi-modal alignment. a) Traditional LSO-GP framework : a continuous optimizer searches the latent space of a symbolic autoencoder. b) SNIP’s [ 19 ] contrastive pre-training: symbolic and numerical encoders are trained to align embeddings of matched expression pairs with batch. c) Illustration of alignment granularity limitations : given a base expression f⋆=sin⁡(x+1)f^{\star}=\sin(x+1) and its numeric embedding ZV⋆Z_{V}^{\star} (labeled N⋆N^{\star} in the figure) encoded from observations (X,y⋆)(X,y^{\star}) , the model should assign highest similarity between ZV⋆Z_{V}^{\star} and the symbolic embedding ZS⋆Z_{S}^{\star} (labeled S⋆S^{\star} ) over embeddings of perturbed variants ( S1,S2,S3S_{1},S_{2},S_{3} ). In practice, the learned alignment often fails to make this distinction.
Figure 2: Evolution of (a) R2R^{2} fitness (left) and (b) cross-modal alignment (right) during LSO across all Feynman and Strogatz equations, averaged over all equations. While fitness improves consistently, alignment remains flat or decreases, indicating that optimization does not exploit cross-modal alignment to produce better symbolic solutions. Iteration 0 corresponds to the model’s one-shot prediction, iteration 1 to the best individual after initialization and BFGS constant refinement.
TypeDescriptionExamples
Unary operator swapReplace unary operator withsin⁡↔cos⁡↔tan⁡\sin\leftrightarrow\cos\leftrightarrow\tan
 semantically related alternativeexp⁡↔log⁡\exp\leftrightarrow\log ; ⋅↔(⋅)2↔(⋅)3\sqrt{\cdot}\leftrightarrow(\cdot)^{2}\leftrightarrow(\cdot)^{3}
Binary operator swapReplace binary operator with+↔−+\leftrightarrow- ; ×↔÷\times\leftrightarrow\div
 any of the other three+↔×+\leftrightarrow\times ; −↔÷-\leftrightarrow\div
Constant changeModify a numeric constant2→32\to 3 ; π→e\pi\to e ; c→1.2cc\to 1.2c
Variable substitutionReplace one variable with anotherx0→x1x_{0}\to x_{1}
Table 1: Perturbation types used to generate candidate expressions. Each perturbation modifies a single element of the expression tree while preserving overall structure.
Table 4
Figure 3: Distribution of ranks for the correct expression across 82 test cases of Feynman set. The correct expression most frequently ranks second (39 cases), with only 15 cases achieving rank 1.
Figure 4: Sensitivity analysis excluding constant change perturbations on Feynman. (a) Accuracy comparison showing that even without constant changes, model accuracy (23.2%) remains below random baseline (32.1%). (b) Rank distribution comparison showing improved but still suboptimal performance when constant changes are excluded.

Explore similar work

CardsList

  1. SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression

    Sep 25, 2026Benjamin Léger, Shubham Gupta, Samy Mammeri +3Symbolic RegressionCross-Modal Alignment

    Université Laval · Mila · Department of Mechanical Engineering, Carnegie Mellon University +1

  2. InsightSR: Refining Symbolic Regression Search Spaces via Parallel Semantic and Structural LLM Guidance

    Aug 26, 2026Yating Ling, Wenjing Cun, Zhitang ChenSymbolic RegressionLarge Language Model-Guided Scientific Discovery

  3. GENSR: Symbolic Regression Based in Equation Generative Space

    Feb 24, 2026Qian Li, Yuxiao Hu, Juncheng Liu +1Symbolic RegressionLatent Space Optimization

    Shanghai Jiao Tong University, Shanghai, China · Eastern Institute of Technology, Ningbo, China · The Hong Kong Polytechnic University, Hong Kong, China +1