Random Symbolic Boosting: A Geometric Approach to Symbolic Regression
Organizations: Tecnológico Nacional de México/Instituto Tecnológico de Tijuana, Blvd. Limón Padilla (Industrial) y Ave. ITR Tijuana s/n, Mesa de Otay, 22500, Tijuana, Baja California, México · Universidad Autónoma de Baja California, Unidad Valle de las Palmas, Calzada Universidad 14418, Parque Industrial Internacional Tijuana, 22427, Tijuana, Baja California, México · LASIGE, Department of Informatics, Faculty of Sciences, University of Lisbon, Campo Grande, 1749-016, Lisbon, Portugal
Abstract
Symbolic models offer interpretability, but their construction through generational evolution can be computationally expensive. This article presents Random Symbolic Boosting (RSB), a method that constructs explicit additive symbolic models via boosting over populations of randomly generated syntax trees, eliminating the need for generational evolution. The idea combines two observations: high-dimensional spaces can be spanned by enough linearly independent vectors, and the semantics of syntax trees can be directly manipulated as such vectors. It is shown that, for Ordinary Least Squares, the boosting process guarantees a monotonic reduction of the training error for both regression and classification. Furthermore, an adaptive geometric early stopping mechanism is introduced. It is based on the angle between the prediction and target vectors, is independent of the problems scale, and requires no adjustments across tasks. Evaluated on a standard benchmark for symbolic regression, the method is located at the Pareto-optimal point between predictive accuracy and training time, ranking first on the real-world datasets. Additionally, it was tested on a large tabular classification benchmark with 200 problems, achieving competitive results.
Figures & tables
| Component | Label | Stork Category | Technical Description |
| Init_Single_Rand | Single candidate (random) | Random selection of a single initial candidate within the valid search space. | |
| Init_Single_Know | Single candidate (knowledge) | Selection of an initial candidate guided by empirical domain knowledge or a priori evaluation. | |
| Initialization | Init_Set_Rand | Set of candidates (random) | Initialization of a candidate set through uniform random sampling to favor spatial exploration. |
| Init_Set_Know_DOE | Set of candidates (knowledge/DOE) | Initialization of a candidate set employing design of experiments (DOE) or prior knowledge of the problem. | |
| Init_Comp_Rand | Algorithm components (random) | Initial random selection of search operators or low-level heuristics. | |
| Init_Comp_Pre | Algorithm components (predefined) | Initial configuration based on a predefined algorithm architecture and fixed search operators. |
| Algorithm | Initialization | Generation | Selection | Control |
|---|---|---|---|---|
| GP | Random population of syntax trees ( Init_Set_Rand ). | Crossover and mutation ( Gen_Pop_Set ). | Tournament selection ( Sel_Func_Prob ). | Fixed population size and operator rates ( Ctrl_Basic ). |
| MRGP | Random population of syntax trees ( Init_Set_Rand ). | Evolution of trees with internal linear regression ( Gen_Pop_Set ). | Tournament selection ( Sel_Func_Prob ). | Fixed evolutionary parameters ( Ctrl_Basic ). |
| MEGP | Population mapped to MAP-Elites grid ( Init_Set_Rand ). | Evolutionary operators on grid elites ( Gen_Pop_Set ). | Elitist replacement per grid cell ( Sel_Imp_Best ). | KPCA and cosine similarity mapping ( Ctrl_Soph_Pop ). |
| GSGP | Random population of syntax trees ( Init_Set_Rand ). | Geometric semantic crossover and mutation ( Gen_Pop_Set ). | Tournament selection ( Sel_Func_Prob ). | Fixed mutation steps and rates ( Ctrl_Basic ). |
| SBP-GP | Random population of syntax trees ( Init_Set_Rand ). | Semantic backpropagation variation (e.g., RDO) ( Gen_Pop_Set ). | Tournament selection ( Sel_Func_Prob ). | Fixed population size, generations, and operator rates ( Ctrl_Basic ). |
| FFX | Expansion of basis functions ( Init_Set_Know_DOE ). | Pathwise Regularized Learning ( Gen_MultiStage ). | Nondominated filtering for Pareto-optimal set ( Sel_Func_Det ). | Fixed regularization path logic ( Ctrl_Basic ). |
| Dataset | Features | Samples | Description |
|---|---|---|---|
| Concrete | 8 | 1030 | High-performance concrete strength |
| EECooling | 8 | 768 | Building energy cooling load |
| EEHeating | 8 | 768 | Building energy heating load |
| Housing | 13 | 506 | Housing prices and air quality |
| Tower | 25 | 4999 | Unknown real-world relationship |
| Yacht | 6 | 308 | Yacht hull design and performance |
| Parameter | Values |
|---|---|
| Basic Function Set | , , , AQ |
| Regularization | None ( ) |
| Parameter | Pearson ( ) | Spearman ( ) |
|---|---|---|
| Algorithm | Configuration |
|---|---|
| RSB | , , , |
| RF | , Library defaults |
| XGB | , Library defaults |
| LGBM | , Library defaults |
| CatBoost | , Library defaults |
| Variant | Nodes ( ) | Time (s) | |
|---|---|---|---|
| OLS | 0.429 | 157.1 | 3.74 |
| Ridge | 0.973 | 151.8 | 0.69 |
| Lasso | 0.965 | 41.9 | 1.20 |
| ENet | 0.953 | 61.3 | 5.82 |
| OLS+ES (angle) | 0.902 | 35.7 | 0.92 |
| Ridge+ES (angle) | 0.970 | 53.6 | 0.33 |
| Reg. (wins/losses/ns) | Angle-based ES vs. no ES | RMSE-based ES vs. no ES | Angle-based ES vs. RMSE-based ES | |||||||||
| Method | Nodes | Time | Nodes | Time | Nodes | Time | Nodes | Time | ||||
| OLS | 0/9/9 | 1/17/0 | 7/11/0 | 3/0/3 | 6/0/0 | 6/0/0 | 3/0/3 | 6/0/0 | 6/0/0 | 0/0/6 | 0/0/6 | 0/0/6 |
| Ridge | 9/0/9 | 5/13/0 | 17/1/0 | 0/2/4 | 5/0/1 | 5/0/1 | 0/5/1 | 6/0/0 | 6/0/0 | 2/0/4 | 0/4/2 | 0/2/4 |
| Lasso | 4/4/10 | 18/0/0 | 10/7/1 | 0/4/2 | 6/0/0 | 6/0/0 | 0/5/1 | 6/0/0 | 6/0/0 | 3/0/3 | 0/3/3 | 0/4/2 |
| ENet | 4/4/10 | 12/6/0 | 1/16/1 | 0/4/2 | 6/0/0 | 6/0/0 | 0/5/1 | 6/0/0 | 6/0/0 | 3/0/3 | 0/3/3 | 0/5/1 |
| Hyperparameter | Evaluated Values |
|---|---|
| , | |
| Operators |
| Hyperparameter | Binary Classification | Multiclass Classification |
|---|---|---|
| , | , | |
| Operators |
Symbolic Regression
Symbolic regression searches data for compact mathematical expressions that balance predictive accuracy with interpretability.
See all 157 papers