Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.
Figures & tables
Figure 1: Overview of Latent Space Optimization and multi-modal alignment. a) Traditional LSO-GP framework : a continuous optimizer searches the latent space of a symbolic autoencoder. b) SNIP’s [ 19 ] contrastive pre-training: symbolic and numerical encoders are trained to align embeddings of matched expression pairs with batch. c) Illustration of alignment granularity limitations : given a base expression f⋆=sin(x+1) and its numeric embedding ZV⋆ (labeled N⋆ in the figure) encoded from observations (X,y⋆) , the model should assign highest similarity between ZV⋆ and the symbolic embedding ZS⋆ (labeled S⋆ ) over embeddings of perturbed variants ( S1,S2,S3 ). In practice, the learned alignment often fails to make this distinction.
Figure 2: Evolution of (a) R2 fitness (left) and (b) cross-modal alignment (right) during LSO across all Feynman and Strogatz equations, averaged over all equations. While fitness improves consistently, alignment remains flat or decreases, indicating that optimization does not exploit cross-modal alignment to produce better symbolic solutions. Iteration 0 corresponds to the model’s one-shot prediction, iteration 1 to the best individual after initialization and BFGS constant refinement.
Type
Description
Examples
Unary operator swap
Replace unary operator with
sin↔cos↔tan
semantically related alternative
exp↔log ; ⋅↔(⋅)2↔(⋅)3
Binary operator swap
Replace binary operator with
+↔− ; ×↔÷
any of the other three
+↔× ; −↔÷
Constant change
Modify a numeric constant
2→3 ; π→e ; c→1.2c
Variable substitution
Replace one variable with another
x0→x1
Table 1: Perturbation types used to generate candidate expressions. Each perturbation modifies a single element of the expression tree while preserving overall structure.
Table 4
Figure 3: Distribution of ranks for the correct expression across 82 test cases of Feynman set. The correct expression most frequently ranks second (39 cases), with only 15 cases achieving rank 1.
Figure 4: Sensitivity analysis excluding constant change perturbations on Feynman. (a) Accuracy comparison showing that even without constant changes, model accuracy (23.2%) remains below random baseline (32.1%). (b) Rank distribution comparison showing improved but still suboptimal performance when constant changes are excluded.
Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is central to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical embeddings, reliably distinguish the effects of local edits that the original alignment cannot, and transfer to external SR corpora.
Benjamin Léger, Shubham Gupta, Samy Mammeri +3
Université Laval · Mila · Department of Mechanical Engineering, Carnegie Mellon University +1
Symbolic regression (SR) seeks to discover parsimonious mathematical laws from observational data, yet conventional approaches often struggle with the vast combinatorial search space of physically meaningful expressions. We present InsightSR, a framework that embeds Large Language Models (LLMs) as a guiding layer around the PySR genetic programming engine. Rather than relying on LLMs to generate expressions directly, InsightSR uses LLMs to progressively transform the search space itself through two complementary pathways: a Semantic Seed Pathway that proposes dimensionally consistent functional skeletons, and a Structural Feature Pathway that recommends nonlinear feature transformations. These transformations accumulate over iterations, broadening the input space and shifting the symbolic search from constructing deep expression trees over raw variables to assembling shallow trees over a rich, semantically informed feature set. A post-generation feedback loop evaluates candidates, categorizes features by their empirical utility, and refines the guidance for the next iteration, transforming the discovery process from open-ended generation into iterative, self-correcting refinement. Across three benchmarks, InsightSR achieves a 95% exact recovery rate on the Feynman benchmark and 80.18% accuracy on the LLM-SRBench LSR-Transform task, substantially outperforming state-of-the-art genetic programming and neural-symbolic methods while maintaining strong out-of-distribution generalization on real-world datasets.
Symbolic Regression (SR) tries to reveal the hidden equations behind observed data. However, most methods search within a discrete equation space, where the structural modifications of equations rarely align with their numerical behavior, leaving fitting error feedback too noisy to guide exploration. To address this challenge, we propose GenSR, a generative latent space-based SR framework following the map construction -> coarse localization -> fine search'' paradigm. Specifically, GenSR first pretrains a dual-branch Conditional Variational Autoencoder (CVAE) to reparameterize symbolic equations into a generative latent space with symbolic continuity and local numerical smoothness. This space can be regarded as a well-structured map'' of the equation space, providing directional signals for search. At inference, the CVAE coarsely localizes the input data to promising regions in the latent space. Then, a modified CMA-ES refines the candidate region, leveraging smooth latent gradients. From a Bayesian perspective, GenSR reframes the SR task as maximizing the conditional distribution p(Equ.∣Num.), with CVAE training achieving this objective through the Evidence Lower Bound (ELBO). This new perspective provides a theoretical guarantee for the effectiveness of GenSR. Extensive experiments show that GenSR jointly optimizes predictive accuracy, expression simplicity, and computational efficiency, while remaining robust under noise.
Qian Li, Yuxiao Hu, Juncheng Liu +1
Shanghai Jiao Tong University, Shanghai, China · Eastern Institute of Technology, Ningbo, China · The Hong Kong Polytechnic University, Hong Kong, China +1