Using Transformers for Teaching and Learning via Symbolic Distillation of Neural A-life Agents

Shuwei Wang; Vadim Bulitko; Matthew R. G. Brown; Michael Youngblood; Ramon Lawrence; Roscoe Smith; Shinichi Nakagawa; William Yeoh

cs.LGNov 5, 2026

Using Transformers for Teaching and Learning via Symbolic Distillation of Neural A-life Agents

Authors: Shuwei Wang, Vadim Bulitko, Matthew R. G. Brown, Michael Youngblood, Ramon Lawrence, Roscoe Smith, Shinichi Nakagawa, William Yeoh

Organizations: Department of Computing Science, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada · Filuta AI, 1606 Headway Circle, Austin, TX 78754, USA · Department of Computer Science, Data Science, University of British Columbia, 6200 University Blvd, Vancouver, BC Canada V6T 1Z4, Canada · Department of Biology, University of Alberta, 116 Street and 85 Avenue, Edmonton, AB, T6G 2R3, Canada ·

Abstract

Neural networks controlling behavior of A-life agents can evolve over multiple generations. While effective in making the agents survive, such networks are generally opaque to a human observer. Furthermore, the neural weights cannot be transferred between agents with different neural architectures. Distilling such neural networks into compact symbolic policies can help humans understand agent behavior in A-life. Additionally, it allows the agents to share their knowledge in a compact fashion, improving population-level learning. Distilling a neural policy into a symbolic policy is typically performed by searching the space of symbolic policies to find one that approximates the neural policy. However, even with a domain-specific language the space of policies can be large, necessitating incomplete stochastic search such as genetic algorithms. Yet the distillation remains computationally expensive. We propose to instead use a transformer pre-trained on readily available random percept-action data. Doing so speeds up distillation substantially. We derive the transformer architecture via a neural architecture search. Empirical results compare different distillation methods and the influence of the symbolic space on A-life population dynamics through self-reflection and teaching and learning.

Figures & tables

parameter	value
grass growth	$\nicefrac{{1}}{{33.75}}$
grass growth decay	$1-\nicefrac{{1}}{{666.67}}$
move energy	$\nicefrac{{1}}{{10}}$
live energy	$\nicefrac{{1}}{{8.36}}$
power coefficient	$\frac{1}{4.78\times 10^{4}}$
max grass bite	$\nicefrac{{1}}{{1.33}}$

Table 1: Environment specifications.

component	minimum	maximum
number of layers	$1$	$5$
number of heads	$1$	$4$
embedding dimension	$64$	$240$
Q-K dimension	$64$	$256$
V dimension	$64$	$256$
MLP dimension	$64$	$512$

Table 2: Transformer architecture space.

component	maximum size
encoder depth	$5$
encoder heads	$(4,\ 1,\ 4,\ 2,\ 1)$
encoder embedding dimension	$150$
encoder Q-K dimension	$(139,\ 207,\ 85,\ 177,\ 138)$
encoder V dimension	$(223,\ 99,\ 97,\ 256,\ 183)$
encoder MLP dimension	$(330,\ 125,\ 459,\ 277,\ 147)$

Table 3: The NAS-found transformer architecture.

Figure 2: Distillation of percept-action pairs of A-life agents.

Figure 3: Influence of self-reflection (SR) and teaching and learning (TL) on A-life population dynamics with a strong symbolic bias.

Figure 4: Different subspaces of symbolic policies. Both self-reflection and teaching and learning are performed.

Figure 5: Influence of self-reflection (SR) and teaching and learning (TL) on A-life population dynamics with the transformer and the full symbolic space.

Policy Distillation

Policy distillation transfers behavior from expert or larger policies into compact, deployable learners while preserving task performance.

See all 54 papers

SymTorch: Symbolic Distillation of Neural Networks
Feb 24, 2026Elizabeth S. Z. Tan, Adil Soubki, Miles CranmerNeural Network Interpretability Scientific Discovery
Department of Applied Mathematics and Theoretical Physics University of Cambridge
Harness-Aware Distillation for Small Language Model Agents
Oct 2, 2026Moonseok Choi, Taehong Moon, Giung Nam +1LLM Agent Harnesses On-Policy Distillation
KAIST AI · Independent researcher
ActionOPD: Distilling Decisions, Not Tokens
Oct 14, 2026Huining Yuan, Xiangmin Yi, Zelai Xu +5LLM Agent Training Knowledge Distillation
Tsinghua University

Feb 24, 2026J/K move · Enter open · S save

SymTorch: Symbolic Distillation of Neural Networks

Elizabeth S. Z. Tan, Adil Soubki, Miles Cranmer

Department of Applied Mathematics and Theoretical Physics University of Cambridge

Neural Network Interpretability Scientific Discovery Symbolic Regression

Open paper AskPDF arXiv

What mathematical functions do neural network components learn? Symbolic distillation addresses this question by expressing neural network components with interpretable, closed-form mathematical expressions that expose the functional structure learned during training. We develop symbolic distillation as a systematic, architecture-agnostic methodology, and release our approach as the open-source SymTorch package - a PySR-powered library built natively for the PyTorch ecosystem. Applying this methodology across diverse architectures, we find that SymTorch is successful in the automated discovery of physical laws. Specifically, our approach (1) recovers pairwise interaction forces from graph neural networks trained on empirical

n

-body observations, (2) distills the exact closed-form PDE/ODE solutions of multiple physical systems, including the value of constants, from physics-informed neural networks trained on sparse data, and (3) uncovers the chaotic dynamics of the Lorenz system from high-dimensional data, ultimately outperforming the base neural network on downstream prediction tasks. We further demonstrate the utility of our framework for model interpretability by providing an optimized implementation of SLIME - a symbolic extension to the LIME explainability method. SLIME consistently outperforms LIME across predictive metrics across eight popular classification and regression benchmarks, while still providing an interpretable local symbolic model. Lastly, we investigate replacing transformer MLP layers with symbolic surrogates: replacing 1-7 layers with symbolic approximations yields 2-19% throughput improvements and up to 18.7% VRAM reduction, with the resulting hybrid models lying on the Pareto front of throughput versus perplexity among open-source LLMs of comparable scale.

Abstract

Policy Distillation

Explore similar work

SymTorch: Symbolic Distillation of Neural Networks

Harness-Aware Distillation for Small Language Model Agents

ActionOPD: Distilling Decisions, Not Tokens