SymTorch: Symbolic Distillation of Neural Networks

Elizabeth S. Z. Tan; Adil Soubki; Miles Cranmer

cs.LGFeb 24, 2026

SymTorch: Symbolic Distillation of Neural Networks

Authors: Elizabeth S. Z. Tan, Adil Soubki, Miles Cranmer

Organizations: Department of Applied Mathematics and Theoretical Physics University of Cambridge

Abstract

What mathematical functions do neural network components learn? Symbolic distillation addresses this question by expressing neural network components with interpretable, closed-form mathematical expressions that expose the functional structure learned during training. We develop symbolic distillation as a systematic, architecture-agnostic methodology, and release our approach as the open-source SymTorch package - a PySR-powered library built natively for the PyTorch ecosystem. Applying this methodology across diverse architectures, we find that SymTorch is successful in the automated discovery of physical laws. Specifically, our approach (1) recovers pairwise interaction forces from graph neural networks trained on empirical $n$ -body observations, (2) distills the exact closed-form PDE/ODE solutions of multiple physical systems, including the value of constants, from physics-informed neural networks trained on sparse data, and (3) uncovers the chaotic dynamics of the Lorenz system from high-dimensional data, ultimately outperforming the base neural network on downstream prediction tasks. We further demonstrate the utility of our framework for model interpretability by providing an optimized implementation of SLIME - a symbolic extension to the LIME explainability method. SLIME consistently outperforms LIME across predictive metrics across eight popular classification and regression benchmarks, while still providing an interpretable local symbolic model. Lastly, we investigate replacing transformer MLP layers with symbolic surrogates: replacing 1-7 layers with symbolic approximations yields 2-19% throughput improvements and up to 18.7% VRAM reduction, with the resulting hybrid models lying on the Pareto front of throughput versus perplexity among open-source LLMs of comparable scale.

Figures & tables

Figure 1: Overview of symbolic distillation. For a trained PyTorch model, any NN component is wrapped and its inputs and outputs (I/O) are collected during a forward pass over sample data. Symbolic regression is performed on the I/O to produce expressions approximating the component’s behavior at different levels of complexity. The user can then select an equation from the Pareto front and replace the component with this expression, producing a hybrid neural-symbolic model.

Figure 2: Approximating local model behavior with SLIME. For a complex non-linear model, we choose the point of interest $\mathbf{x}^{*}$ . We sample points around this region and fit a symbolic model to these points.

Figure 3: The model pipeline is trained to predict $\dot{x}$ from $x$ , where $x$ comprises high-dimensional observations whose underlying dynamics are low-dimensional. The linear encoder projects $x$ into a latent space in which an MLP learns these simple dynamics, and a linear decoder maps the output back to produce $\dot{x}$ . We apply SymTorch to distill the MLP into symbolic equations and show that the discovered equations trace out Lorenz-like attractor dynamics after an linear arbitrary transform. Replacing the MLP with these equations and fine-tuning the full model yields a hybrid that outperforms the original NN on out-of-domain chaotic trajectories, as shown in the plots on the right (MSE in $x$ -space per timestep, averaged over 200 trajectories; shaded regions indicate standard error on the mean). (b) is the same plot as (a) with the NN results removed to better distinguish between the symbolic and fine-tuned model performance.

Panel of Figure 4 — Figure 4: Comparison of SLIME and LIME local surrogate fidelity across eight tabular datasets. Results are averaged over five random local points per dataset; error bars show $\pm$ 1 $\sigma$ .

Operation	Expected Equation	LLM-Learned Operation
Addition	$x_{0}+x_{1}$	$x_{1}\cdot((\mathrm{inv}(x_{0}-70.16)+1.07)+$ $(\mathrm{inv}(\sin(x_{1})+0.80)+x_{1})\cdot(-\epsilon))\,x_{0}$
Multiplication	$x_{0}x_{1}$	$x_{1}(\mathrm{inv}(1.66-31.3\sin(x_{0}))+x_{0})$
Counting	$x_{0}+\cdots+x_{5}$	$(-0.11x_{0}+1.67)(x_{1}+(x_{1}(x_{0}+$ $1.43)-2.50)(x_{2}-0.53)(x_{0}+x_{3}-2.15)+x_{2})$
Temperature Conversion	$\frac{9}{5}x+32$	$\mathrm{inv}(x_{0}-168.86)\mathrm{inv}(x_{0}-129.65)+\mathrm{inv}(x_{0}-51.87)x_{0}+33.79$

Table 1: Symbolically distilled LLM-learned operations. The best equation, as determined by Equation 3 , is shown here. The

\epsilon

denotes a small (

\ll 1

) term.

Panel of Figure 5 — Figure 5: Symbolic-hybrid models replace the feed-forward layers of Qwen2.5-1.5B-Instruct with symbolic expressions. (a) shows the perplexity and efficiency trade-offs as more layers are replaced. (b) places the resulting models in context against standard open-source LLMs of comparable scale, demonstrating that the hybrid models with $L=1$ – $7$ layers replaced lie on the Pareto front of similarly sized open-source models. Perplexity is evaluated on the Wikitext-v2 test set.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

SR Parameter	Configuration
Binary operators	+, *
Unary operators	inv(x)=1/x, sin, exp
Extra sympy mappings	"inv": lambda x: 1/x
Number of iterations	400
Complexity of operators	sin: 3, exp: 3

Appendix

Table 2: Default SymTorch SR configurations.

Simulation	Standard	Bottleneck	L1	KL	Pruning	Variance
Charge	18.20	19.12	18.06	39.80	19.40	56417.66
$r^{-1}$	0.26	0.25	0.32	15.40	0.28	87.69
$r^{-2}$	24.07	25.25	21.67	57.80	23.77	98641.90
Spring	0.24	0.16	0.20	7.29	0.23	55.84

Appendix

Table 3: Test set mean absolute error (MAE) for different GNN model variants across four force law simulations. Lower values indicate better performance. The pruning variant (introduced by SymTorch) achieves comparable performance to the bottleneck model while automatically discovering the optimal dimension. Variance values provide scale context for the prediction errors.

Simulation	Message	Standard	Bottleneck	L1	KL	Pruning
Charge	1	✗	✓ ‡	✗	✗	✓
	2	✗	✓ ‡	✓ §	✗	✓ §
$r^{-1}$	1	✗	✓	✓	✗	✓
	2	✓	✓	✓	✗	✓ ∗
$r^{-2}$	1	✓ §	✓	✓ §	✓ ‡	✓
	2	✗	✓	✗	✓ ‡	✓

Appendix

Table 4: Symbolic regression results for each message component. ✓= correct form of force law recovered; ✗= failure. ∗ Correct form, but with a small constant added to a term (e.g.,

1/(r+\text{const.})

). † Correct form, but only

\Delta y

apparent in both messages. ‡ Correct form with

\Delta x

in one message and

\Delta y

in the other. § Correct form with only

\Delta x

\Delta y

in at least one message.

Figure 6: The score of the equations found from PySR for the pruning model trained on the $r^{-2}$ as given in Table 5 . The score is calculated from Equation 3 . The best equation, highlighted in red, produces the largest drop in $\log(\text{loss})$ per unit of additional complexity.

Complexity	Equation	Loss
1	$\phi_{2}^{e}=0.08$	0.0888
5	$\phi_{2}^{e}=(\Delta x\cdot 0.00)+0.08$	0.0885
7	$\phi_{2}^{e}=((\Delta x+\Delta y)\cdot 0.00)+0.08$	0.0882
8	$\phi_{2}^{e}=((\Delta x\cdot\mathrm{inv}(r))\cdot 0.00)+0.08$	0.0880
9	$\phi_{2}^{e}=(m_{2}\cdot(\Delta y+\Delta x)\cdot 0.00)+0.08$	0.0877
10	$\phi_{2}^{e}=(\mathrm{inv}((r+\Delta x)\cdot r)\cdot-0.00)+0.08$	0.0843

Appendix

Table 5: Pareto front from PySR results on message 2 of pruning model on

r^{-2}

dataset. The highlighted equation shows the ’best equation’ chosen by PySR according to the metric given in Equation 3 . The constants have been truncated to two decimal points.

Model	Train MSE	Test MSE	Train variance	Test variance
NN	8.22	8.18	8646.09	8631.27
Symbolic	15.52	15.65
Finetune	12.51	12.66

Appendix

Table 6: MSE on train and test splits for each model variant on Lorenz trajectory data (

\dot{x}

). Variance is computed over all 10 output dimensions of

\dot{x}

	SINDy autoencoder	Ours (SymTorch)
SR method	Sparse regression over fixed library	PySR (genetic algorithm, library-free)
Function library	Must be specified in advance	Not required
Encoder architecture	Unconstrained MLP	Single linear layer
Latent dynamics	Linear combination of library terms	Arbitrary analytic expressions

Appendix

Table 7: Comparison of the SymTorch method and the SINDy autoencoder method of performing symbolic distillation on high-dimensional data transformed to a lower dimensional latent space.

System	True solution	Distilled expression	True constant	SR estimate
Heat equation	$e^{-\pi^{2}\alpha t}\sin(\pi x)$	$\sin(3.144\,x)\cdot e^{-1.981\,t}$	$\alpha=0.200$	$\begin{subarray}{c}\hat{\alpha}=1.981/\pi^{2}\\ \approx 0.201\end{subarray}$
Wave equation	$\sin(\pi x)\cos(c\pi t)$	$\sin(3.147\,x)\cdot\cos(2.105\,t)$	$c=0.670$	$\begin{subarray}{c}\hat{c}=2.105/\pi\\ \approx 0.670\end{subarray}$
Oscillator	$e^{-\delta t}\cdot 2A\cos(\phi+\omega t)$	$e^{-2.000\,t}\cdot\cos(-19.625\,t)$	$\delta=2,\;\omega\approx 19.90$	$\begin{subarray}{c}\hat{\delta}=2.000\\ \hat{\omega}\approx 19.625\end{subarray}$

Appendix

Table 8: Symbolic expressions recovered by SymTorch at complexity 13 for each physical system, alongside the true closed-form solutions and recovered physical constants.

Dataset	Task	$n_{\mathrm{train}}$	$n_{\mathrm{test}}$	$n_{\mathrm{features}}$	$J$	$n_{\mathrm{syn}}$
Iris	Classification	120	30	4	10	500
Wine	Classification	142	36	13	10	500
Breast Cancer	Classification	455	114	30	22	500
Pima Diabetes	Classification	614	154	8	30	500
German Credit	Classification	800	200	61	40	500
Adult	Classification	39,073	9,769	108	100	500

Appendix

Table 9: Summary of datasets used in the SLIME and LIME benchmarking.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
4116	0.73	0.61	0.01	1.00	0.23	0.61	0.00	1.00	$\operatorname{inv}(x_{24}+(x_{33}+x_{25})(x_{38}$ $+32.90(x_{48}+x_{47}-1.03(x_{24}+$ $x_{49})(x_{11}+x_{61})))+1.06)$
11317	0.95	1.17	0.89	0.05	0.00	1.69	0.08	0.95	$\operatorname{inv}(3.08((20.24(x_{58}+x_{24}+x_{41}$ $+x_{51}+x_{10}+x_{61})+x_{9})$ $(x_{53}+x_{23})+3.10))$
11038	0.31	0.48	0.29	1.00	0.00	0.10	0.02	1.00	$-0.79(((x_{24}+0.21+x_{25}+x_{47})$ $(\exp(x_{40})x_{33}+x_{67})$ $-1.25+x_{30}x_{41})$
184	-0.00	0.00	0.00	1.00	0.68	0.27	0.11	1.00	$\operatorname{inv}(x_{2}(x_{25}+x_{6}+x_{97}$ $+0.03x_{33}+x_{27})$ $+x_{11}+0.99)$
30797	0.38	1.17	0.70	0.32	0.04	15.94	0.98	0.32	$\operatorname{inv}(((x_{24}+x_{38}+0.34)x_{33}x_{63}$ $+1.71)(x_{21}x_{31}+0.41)$ $+x_{10}+x_{29}+x_{49}x_{30}-0.46)$

Appendix

Table 10: SLIME vs LIME results: Adult.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
47	0.66	0.77	0.56	0.09	0.00	0.04	0.00	1.00	$x_{10}\operatorname{inv}(x_{9})(x_{12}((x_{28}x_{1}$ $-3.04)(\epsilon x_{26}+\epsilon(x_{1}-$ $14.92)))+\epsilon)$
130	1.03	1.22	0.89	0.00	0.01	0.31	0.05	1.00	$(x_{27}-0.04\sin(x_{21}))((-\epsilon x_{23}-$ $0.11)(x_{3}-0.09x_{21}^{2})+$ $17.25)+0.19$
128	0.00	0.01	0.00	1.00	0.00	0.01	0.00	1.00	$\operatorname{inv}(x_{1})^{2}x_{3}^{2}(-\epsilon)+1.00$ $+x_{19}\operatorname{inv}(\operatorname{inv}(x_{7})(\sin(x_{20})$ $-0.15))$
2	0.04	0.07	0.00	1.00	0.00	0.04	0.00	1.00	$x_{6}(x_{16}-0.05x_{11})$ $+x_{21}(1.75(x_{11}-$ $1.37)x_{11}x_{14}^{2}+\epsilon)$
357	3.42	4.18	0.06	1.00	0.00	3.75	0.05	1.00	$x_{1}(-\epsilon x_{13}+\epsilon x_{1})$ $+6.91x_{28}(\epsilon x_{23}+x_{6}-0.03x_{2})$

Appendix

Table 11: SLIME vs LIME results: Breast cancer.

\epsilon

symbolizes a small constant.

Point index	WR 2	WMAE	WMSE	WR 2	WMAE	WMSE	Example equation
	LIME			SLIME
1739	-39.59	4.90	24.30	-0.17	0.67	0.70	$-0.36x_{2}+1.43x_{0}\operatorname{inv}(x_{5})$
4782	-74.08	4.21	18.47	0.21	0.35	0.19	$16.22x_{6}\operatorname{inv}(x_{7}+79.84)$ $+x_{0}\operatorname{inv}(x_{5})+13.07$
4664	-6.64	2.95	9.02	0.81	0.36	0.22	$3.00(\operatorname{inv}(x_{5})+\operatorname{inv}(x_{2}))$ $+((-\epsilon x_{7}-0.35)x_{6}+3.35)x_{0}-3.37$ $+\operatorname{inv}(x_{6}-2.27x_{0})$
77	-30.12	2.96	9.40	-0.07	0.41	0.32	$-0.35x_{2}+(x_{0}+\sin(x_{2}))$ $(\operatorname{inv}(x_{5})+0.15)$
13014	-31.57	3.28	11.10	-0.18	0.48	0.40	$x_{2}(0.08x_{0}x_{3}-0.59)$ $+\operatorname{inv}(x_{5}(\operatorname{inv}(x_{1})+0.24))$

Appendix

Table 12: SLIME vs LIME results: California Housing.

\epsilon

symbolizes a small constant.

Point index	WR 2	WMAE	WMSE	WR 2	WMAE	WMSE	Example equation
	LIME			SLIME
86	-22.94	1.28	1.68	0.21	0.19	0.06	$(-\epsilon x_{7}+0.04)x_{7}+0.01x_{0}$ $+x_{3}(-\epsilon x_{4}^{2}-0.02)$
237	-13.86	1.85	3.49	0.62	0.26	0.09	$-0.04x_{0}\operatorname{inv}(x_{3}\operatorname{inv}(-0.04x_{2}x_{7}-0.44x_{1})$ $-1.34)-2.53$
232	-54.81	1.38	2.00	-0.74	0.20	0.06	$(\operatorname{inv}(x_{3})x_{6}-2.14)(-\epsilon x_{6}+4.04)$ $((\operatorname{inv}(x_{3})x_{6}-3.58)(-0.01x_{6}+6.99)+1.56)$
3	0.69	0.18	0.05	0.81	0.13	0.03	$(0.07x_{1}+5.10)(\operatorname{inv}(x_{3})x_{7}-0.47$ $+\epsilon(x_{1}+x_{0})+\epsilon x_{2})$
648	-9.38	1.02	1.15	0.22	0.23	0.09	$\operatorname{inv}(x_{7}+55.98)(x_{2}\sin(x_{3})-166.70)$ $+(x_{0}+17.89x_{4})(\operatorname{inv}(x_{3})-\epsilon)$

Appendix

Table 13: SLIME vs LIME results: Concrete.

\epsilon

symbolizes a small constant.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
83	1.48	1.65	0.01	1.00	0.01	0.24	0.00	1.00	$-0.17x_{60}(\operatorname{inv}(x_{2})$ $+x_{57}(2.59x_{8}+(x_{18}$ $+x_{20})(x_{7}+x_{54}))+$ $x_{12}(x_{52}-0.60))+0.90$
230	0.01	0.39	0.01	1.00	0.00	0.76	0.01	1.00	$0.04(x_{32}+$ $x_{7}+x_{31}+x_{51}+$ $(x_{58}+x_{25})(x_{34}+$ $0.21x_{3}))+$ $0.07x_{10}+0.82$
225	0.06	0.85	0.15	0.05	0.00	0.74	0.05	0.05	$0.05(x_{56}+x_{41})-$ $0.10(x_{57}(x_{8}+x_{52}$ $+x_{11}+x_{38})+(x_{7}+$ $x_{11})\operatorname{inv}(x_{3}))+0.95$
3	3.42	5.14	0.07	1.00	0.00	0.48	0.00	1.00	$-0.07(x_{8}+(x_{53}+$ $x_{47})(x_{51}+x_{35}+$ $x_{33}))(x_{15}+$ $x_{28})+0.96-$ $0.20x_{15}x_{50}$
629	1.63	2.52	0.02	1.00	0.00	0.31	0.00	1.00	$2.67\operatorname{inv}((x_{45}+$ $0.60)((0.69(x_{8}+$ $0.22x_{54})+$ $x_{26})(x_{57}+$ $x_{20}+x_{33}))+$ $x_{15}+2.81)$

Appendix

Table 14: SLIME vs LIME results: German Credit.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
12	0.00	0.04	0.00	1.00	0.00	0.02	0.00	1.00	$(\exp(-x_{0}-x_{2}))(x_{3}+$ $2.60)(\operatorname{inv}(x_{2})-$ $0.07)\exp(x_{1})$
33	1.40	1.42	1.14	0.00	0.00	0.02	0.00	1.00	$x_{2}((x_{3}x_{2}^{2}+$ $0.17)x_{2}(\epsilon x_{0}$ $-\epsilon x_{2}^{2}\operatorname{inv}(x_{1}-$ $2.78)))+1.00$
119	0.66	0.69	0.48	0.48	0.00	0.06	0.00	1.00	$\epsilon\sin(x_{0})(-0.49x_{2}+$ $x_{1})(\operatorname{inv}(x_{3}+\sin(x_{2})-$ $0.98)-1.20)$
0	0.41	0.41	0.23	1.00	-0.00	0.00	0.00	1.00	$-\epsilon\operatorname{inv}(\operatorname{inv}(\operatorname{inv}(0.31x_{0}-$ $3.50x_{3}^{2})x_{2}^{2}+0.18)$ $+x_{1}-3.50)+1.00$
93	0.05	0.36	0.02	1.00	0.00	0.30	0.00	1.00	$\epsilon(x_{1}+x_{0})(x_{1}-5.92)x_{2}$ $+\epsilon$

Appendix

Table 15: SLIME vs LIME results: Iris.

\epsilon

symbolizes a small constant.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
64	0.17	0.70	0.15	0.34	0.01	0.55	0.00	0.99	$(0.02x_{7}^{2}+2.99x_{0}x_{6}+$ $x_{5})\operatorname{inv}(x_{2})-0.39$
176	0.51	0.77	0.43	0.04	0.00	0.30	0.00	1.00	$\epsilon x_{1}+(-0.01x_{5}+$ $0.04x_{0}x_{6})$ $(x_{0}+x_{2}+x_{3})\operatorname{inv}(x_{4})$
173	0.02	0.68	0.03	0.91	0.00	0.66	0.02	0.98	$((x_{6}((x_{6}x_{1}-85.14)x_{6})+$ $x_{0})\epsilon\sin(x_{0})$ $-0.01)x_{2}+0.68$
2	0.19	0.73	0.19	0.34	0.01	0.56	0.03	0.97	$\epsilon(\operatorname{inv}(\sin(x_{5})-$ $0.71)+x_{4}+x_{5}+x_{1}$ $+\operatorname{inv}(x_{6}-0.12))-0.45$
483	0.90	1.20	0.76	0.00	0.00	0.38	0.01	1.00	$x_{4}((-\epsilon(x_{2}+x_{7}+x_{2})+$ $\epsilon)x_{1}$ $-0.01)$

Appendix

Table 16: SLIME vs LIME results: Pima Diabetes.

\epsilon

symbolizes a small constant.

Point index	KL div	WCE	WP-MSE	Agree.	KL div	WCE	WP-MSE	Agree.	Example equation for prob. of class 0
	LIME				SLIME
14	0.38	0.79	0.25	1.00	0.00	0.34	0.00	1.00	$6.15(\operatorname{inv}(x_{2}x_{4}x_{3})(x_{8}+$ $0.03x_{12})$ $-\epsilon+\epsilon x_{5})$
39	1.04	1.52	0.06	1.00	0.00	0.25	0.00	1.00	$\epsilon x_{8}+\operatorname{inv}((5.23x_{7}+$ $x_{3})x_{2})$ $+\epsilon x_{12}-0.04$
40	0.28	0.52	0.13	1.00	0.00	0.23	0.00	1.00	$((x_{3}+\operatorname{inv}(x_{1})x_{5}x_{11})x_{10}+$ $x_{4})(-1.86\operatorname{inv}(x_{12}))+1.20$
0	0.24	0.45	0.03	1.00	0.00	0.24	0.00	1.00	$\operatorname{inv}((x_{1}+\sin(x_{3}))$ $(0.02\exp(x_{9}))$ $+x_{7}+1.99)+0.02$
111	2.65	3.54	0.07	1.00	0.00	0.57	0.00	1.00	$x_{5}^{4}((-0.01x_{4}+0.87)(-0.01x_{4}$ $+0.78))+0.02x_{8}x_{10}$

Appendix

Table 17: SLIME vs LIME results: Wine.

\epsilon

symbolizes a small constant.

Operation	Example Prompt
Addition of two 3-digit numbers	Return only the numeric answer in the format $boxed$ . What is $193+374=$ ?
Multiplication of two 3-digit numbers	Return only the numeric answer in the format $boxed$ . What is $484*726=$ ?
Counting 1s in a 6-digit string of 1s and 0s	Return only the numeric answer in the format $boxed$ . How many 1s are there in the string 000101?
Converting from Celsius to Fahrenheit	Return only the numeric answer in the format $boxed$ . What is $30$ degrees Celsius in Fahrenheit?

Appendix

Table 18: Prompts used to query Llama-3.2-1B-Instruct on four mathematical tasks: addition, multiplication, Celsius-to-Fahrenheit conversion, and counting 1s in binary strings. All prompts request answers in \boxed{} format to enable regex-based extraction of numeric results. For the temperature conversion task, we only inputted temperatures between

-20^{\circ}\mathrm{C}

and

200^{\circ}\mathrm{C}

Figure 7: Conceptual overview of the symbolic distillation pipeline applied to an LLM MLP layer. Input activations are projected to a reduced PCA subspace, symbolic regression learns a compact mapping between the compressed input and output representations, and the resulting equations replace the full MLP during inference.

Layer	PCA	PCA+SR
0	$89.5918$	$106.6565$
1	$4.8133$	$1981.6998$
2	$0.8865$	$0.8842$
3	$0.7172$	$0.7848$
4	$0.8810$	$0.8346$
5	$0.9624$	$0.9588$

Appendix

Table 19: Per-layer perplexity delta (

\Delta

PPL) for PCA and PCA+SR interventions. The validation set was used to produce these results.

Figure 8: Change in validation set perplexity under PCA compression and reconstruction of MLP activations, relative to the baseline perplexity of 10.62. Input activations are projected to the number of components shown on the $x$ -axis; output activations to the number shown on the $y$ -axis. The analysis was performed by adding PCA to the arbitrarily selected layers 7, 14, and 21.

	$\Delta$ PPL $\downarrow$			$\Delta$ Throughput (%) $\uparrow$		Memory Saved $\uparrow$
Layers	SR	PCA	Skip	SR	Skip	(MB)	(%)
1	$0.444$	$0.451$	$0.565$	$2.4$	$2.4$	$158$	$2.7$
2	$0.973$	$1.004$	$1.289$	$4.8$	$4.9$	$315$	$5.4$
3	$1.651$	$1.653$	$2.249$	$7.4$	$7.5$	$473$	$8.0$
4	$2.547$	$2.955$	$3.815$	$9.8$	$10.3$	$630$	$10.7$
5	$3.415$	$4.274$	$4.555$	$12.6$	$13.2$	$788$	$13.4$

Appendix

Table 20: Perplexity degradation (on the test set), throughput gain, and memory saved as a function of layers skipped, for Skip, SR, and PCA. Memory savings are identical across Skip and SR; PCA yields no memory saving.

Model	Perplexity on Validation Set	Avg. Latency (ms)	p95 Latency (ms)	Throughput (tokens/s)
Qwen2.5-1.5B (Baseline)	10.79	216.99	217.13	4719.07
Qwen2.5-1.5B SR (15)	11.24	211.87	212.07	4833.25
Qwen2.5-1.5B SR (15, 17)	11.77	207.14	207.58	4943.49
Qwen2.5-1.5B SR (15, 17, 10)	12.44	202.16	202.41	5065.23
Qwen2.5-1.5B SR (15, 17, 10, 13)	13.34	197.64	197.86	5181.18
Qwen2.5-1.5B SR (15, 17, 10, 13, 14)	14.21	193.12	193.32	5302.48

Appendix

Table 21: Perplexity on the test set and inference benchmarking for various open-source language models and SR-modified Qwen2.5-1.5B variants. SR rows list the greedily selected replaced layers.

Explore similar work

Sep 1, 2026

Neural Symbollic Regression Using Deep Learning and Sparse Modelling

Symbolic Regression (SR) seeks to find succinct mathematical expressions that represent the fundamental relationships within data, providing interpretability and scientific understanding that exceeds that of black-box models. Nevertheless, traditional methods like Genetic Programming face challenges with scalability and are highly sensitive to noise, while sparse regression techniques such as SINDy rely significantly on predetermined feature libraries. In this work, we present a Neural Symbolic Regression (NSR) framework that treats neural networks as functional preconditioners for symbolic discovery. Our approach uses a decoupled pipeline: a neural network first learns a smooth, noise-robust approximation of the target function in an interaction- aware nonlinear feature space. LASSO is then applied to extract sparse, interpretable closed-form expressions. To improve predictive accuracy and symbolic fidelity by integrating distributed hyperparameter optimization with Ray Tune and ASHA scheduling. Experiments on the Nguyen benchmark suite show that our approach consistently outperforms SINDy and non-tuned neural baselines in RMSE, noise robustness, and out-of-distribution generalization. Ablation studies confirm the significance of feature interactions, neural depth, and tuning strategies. In general, this study presents a scalable and understandable neural-symbolic framework, creating a solid link between neural approximation and the discovery of sparse equations for scientific machine learning.

Ravi Kumar U, Sumitra S

Department of Mathematics Indian Institute of Space Science and Technology Thiruvananthapuram, India

Scientific ML Neurosymbolic Learning Symbolic Regression

Ask PDF

Mar 25, 2026

Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning

Symbolic discovery of governing equations is a long-standing goal in scientific machine learning, yet a fundamental trade-off persists between interpretability and scalable learning. Classical symbolic regression methods yield explicit analytic expressions but rely on combinatorial search, whereas neural networks scale efficiently with data and dimensionality but produce opaque representations. In this work, we introduce Symbolic Kolmogorov-Arnold Networks (Symbolic-KANs), a neural architecture that bridges this gap by embedding discrete symbolic structure directly within a trainable deep network. Symbolic-KANs represent multivariate functions as compositions of learned univariate primitives applied to learned scalar projections, guided by a library of analytic primitives, hierarchical gating, and symbolic regularization that progressively sharpens continuous mixtures into one-hot selections. After gated training and discretization, each active unit selects a single primitive and projection direction, yielding compact closed-form expressions without post-hoc symbolic fitting. Symbolic-KANs further act as scalable primitive discovery mechanisms, identifying the most relevant analytic components that can subsequently inform candidate libraries for sparse equation-learning methods. We demonstrate that Symbolic-KAN reliably recovers correct primitive terms and governing structures in data-driven regression and inverse dynamical systems. Moreover, the framework extends to forward and inverse physics-informed learning of partial differential equations, producing accurate solutions directly from governing constraints while constructing compact symbolic representations whose selected primitives reflect the true analytical structure of the underlying equations. These results position Symbolic-KAN as a step toward scalable, interpretable, and mechanistically grounded learning of governing laws.

Salah A Faroughi, Farinaz Mostajeran, Amirhossein Arzani +1

Energy & Intelligence Lab, Department of Chemical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Department of Mechanical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah, 84112, USA +2

Interpretable ML Scientific ML Kolmogorov-Arnold Networks Symbolic Regression PDE Discovery Physics-Informed ML

Ask PDF

Sep 26, 2026

Recovery-Directed Symbolic Distillation of Neural Likelihoods

Amortized neural likelihoods enable computationally expensive inference for models with analytically intractable or unspecified likelihoods, but their black-box nature limits interpretability. We introduce a symbolic distillation pipeline that converts trained neural likelihoods into explicit, interpretable expressions optimized for efficient parameter estimation. Our approach uses a recovery-directed objective to guide symbolic regression toward expressions that preserve parameter-recovery accuracy rather than merely approximating the likelihood function. Candidate expressions are evaluated on held-out datasets and selected using a criterion that jointly accounts for expression complexity, parameter-recovery performance, and distributional distance from the learned likelihood. We evaluate the pipeline on the diffusion decision model, a classical cognitive model, whose analytically tractable likelihood provides ground truth for controlled evaluation. The proposed recovery-directed objective improves parameter recovery over standard symbolic-regression objectives. The resulting symbolic likelihoods enable over 100 times faster parameter evaluation than both neural likelihoods and, when available, the exact likelihood, while maintaining a manageable loss in precision. We further demonstrate these computational benefits in Bayesian hierarchical inference on empirical data. Our pipeline provides a lightweight interface for integrating symbolic distillation with existing neural-likelihood estimation methods and can be adapted to a range of simulation-based inference settings.

Kianté Fernandez, Xinwei Li

Department of Psychology University of California, Los Angeles Los Angeles, CA 90095, USA · Department of Civil and Environmental Engineering National University of Singapore Singapore

Symbolic Regression Simulation-Based Inference

Ask PDF