Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

Charles Brum; Edward Finkelstein

cs.LGJul 23, 2026

Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

Authors: Charles Brum, Edward Finkelstein

Abstract

We investigate whether symbolic regression can discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules are represented as fixed-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44.47% over the improved cases. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger-scale validation.

Explore similar work

Neural Symbollic Regression Using Deep Learning and Sparse Modelling
Sep 1, 2026Ravi Kumar U, Sumitra SSymbolic Regression Neuro-Symbolic Framework
Bayesian Symbolic Regression with Entropic Reinforcement Learning
Aug 10, 2026Oussama Boussif, Mohammed Mahfoud, Younesse Kaddar +6Symbolic Regression Entropy Regularized Reinforcement Learning
Deep Variational Inference Symbolic Regression
May 1, 2026James Butterworth, Gevik Grigorian, Alejandro DiazDelaOSymbolic Regression Variational Inference

Sep 1, 2026 · cs.LGJ/K move · Enter open · S save

Neural Symbollic Regression Using Deep Learning and Sparse Modelling

Ravi Kumar U, Sumitra S

Symbolic Regression (SR) seeks to find succinct mathematical expressions that represent the fundamental relationships within data, providing interpretability and scientific understanding that exceeds that of black-box models. Nevertheless, traditional methods like Genetic Programming face challenges with scalability and are highly sensitive to noise, while sparse regression techniques such as SINDy rely significantly on predetermined feature libraries. In this work, we present a Neural Symbolic Regression (NSR) framework that treats neural networks as functional preconditioners for symbolic discovery. Our approach uses a decoupled pipeline: a neural network first learns a smooth, noise-robust approximation of the target function in an interaction- aware nonlinear feature space. LASSO is then applied to extract sparse, interpretable closed-form expressions. To improve predictive accuracy and symbolic fidelity by integrating distributed hyperparameter optimization with Ray Tune and ASHA scheduling. Experiments on the Nguyen benchmark suite show that our approach consistently outperforms SINDy and non-tuned neural baselines in RMSE, noise robustness, and out-of-distribution generalization. Ablation studies confirm the significance of feature interactions, neural depth, and tuning strategies. In general, this study presents a scalable and understandable neural-symbolic framework, creating a solid link between neural approximation and the discovery of sparse equations for scientific machine learning.

Symbolic Regression Neuro-Symbolic Framework Machine Learning Deep Learning Spectral Preconditioning

Open paper Ask about it PDF

Abstract

Explore similar work

Neural Symbollic Regression Using Deep Learning and Sparse Modelling

Bayesian Symbolic Regression with Entropic Reinforcement Learning

Deep Variational Inference Symbolic Regression