What drives the emergence of symbolic mechanisms in large language models?
Authors: Melody Zixuan Li, Taylor Webb
Organizations: Princeton Neuroscience Institute, Princeton University · Princeton Neuroscience Institute, Department of Psychology, Princeton University
Recent work has identified a set of emergent symbolic mechanisms that support abstract reasoning in large language models, but it remains unclear what factors drive the emergence of these mechanisms. To address this question, we trained neural networks from scratch on abstract sequence tasks, varying both architectural and data distributional factors, and investigated the effects of these factors on the emergence of symbolic mechanisms. Using representational, attentional, and causal mediation analyses, we first confirmed that transformer-based language models trained from scratch on our task developed symbolic mechanisms. We then found that, across multiple axes of data diversity, greater diversity produced stronger symbolic signatures. Surprisingly, the same signatures emerged in modified transformer architectures and even multilayer perceptrons, indicating that architectural inductive biases are not necessary for the development of symbolic mechanisms. We also observed consistent patterns in an entity-attribute binding task. Together, these findings identify data diversity, not architecture, as the primary driver of emergent symbolic computation in neural networks.
Figures & tables
Figure 1: Illustration of Symbolic Mechanisms Learned by Model. Problems consist of an in-context example governed by either an ABA or ABB (pictured) rule constructed with arbitrary tokens, followed by an incomplete query example that must be completed according to the same rule. Symbol abstraction heads process the in-context example, identifying the symbol (A or B) associated with the third token. Query tokens are copied into role-specific (A vs. B) subspaces of the residual stream at the final position, and a rule-dependent (ABA vs. ABB) rotation is performed to gate the appropriate token (corresponding to A vs. B) into the output subspace.
Figure 2: Evidence for Symbol Abstraction Heads and Rotation Mechanism Learned by Model. (a) Layer-wise causal mediation analysis localizes symbolic encoding to layer 1. (b) Head-wise causal mediation analysis identifies symbol abstraction heads. (c) Query–key attention of the symbol abstraction heads. (d) RSA confirms that identified heads encode abstract role identity rather than token surface form. (e) – (f) Residual-stream PCA reveals a layer-wise rotation from abstract role representation to output prediction.
Figure 3: Assessing the Impact of Data Diversity on Emergent Symbol Processing. (a) Concept intervention pipeline. Role embeddings e^A and e^B are estimated from residual activations and used to swap the role at a target position. The flip rate is the fraction of prompts whose prediction switches to the answer of the opposite rule. (b) Flip rate increases with the number of unique training examples, (c) decreases as role-token bias increases, and (d) increases with vocabulary size and (e) the number of rules. Error bars show the standard deviation across 3 seeds.
Figure 4: Out-of-distribution generalization
Figure 5: Architectural inductive biases do not determine the emergence of symbolic mechanisms. (a) Flip rate at V=10k for Transformers with specific inductive biases removed, and for MLP models. Error bars show the standard deviation across 3 seeds. (b) Flip rate decreases with role-token bias in the same way for the pure MLP and the Transformer. (c) MLP models show the same dependence of out-of-distribution (reversed-role) accuracy on vocabulary size and role-token bias as transformers: with high bias and a small vocabulary they fail the reversed test, and reversed accuracy recovers as vocabulary grows. (d) Test accuracy over training.
Figure 6: Data diversity drives the emergence of symbolic binding mechanisms. (a) Increasing role-token bias results in decreased binding-ID flip rates. (b) The binding-ID flip rate increases with vocabulary size. (c) Both MLPs and Transformers display high flip rates, indicating that both have developed symbolic binding mechanisms.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Vocabulary
Train prompts
Validation prompts
Vocabulary size (ABA/ABB)
50–10,000
2,000,000
2,000
Number of training samples
10,000
200–200,000
500
Number of rules
100
2,000,000
5,000
Role-token bias
100–20k
5,000,000
5,000
Architecture variants
10,000
5,000,000
10,000
Binding (role-token bias)
20,000
5,000,000
20,000
Appendix
Table 1: Datasets. “Prompts” counts distinct training prompts.
Figure 7: Cross-rule decoding by layer. Probes trained on one rule and tested on the other. Layers 0–4 decode symbol identity perfectly; their correct/incorrect score is 0% because the same direction points to the wrong token under the new rule. Layers 7–11 show the reverse.
Figure 8: Timecourses by number of training samples (vocabulary size 10,000). Flip rate is measured on in-distribution prompts.
Figure 9: Timecourses by role-token bias (vocabulary size 100). Solid black: in-distribution accuracy. Dashed gray: accuracy on reversed-role (out-of-distribution) prompts. Blue: flip rate on reversed-role prompts. In-distribution accuracy is near ceiling at every bias level, while reversed accuracy and reversed flip rate fall as bias increases.
Figure 10: Timecourses by vocabulary size . Shaded boxes and insets zoom on the first 30,000 steps for the two smallest vocabularies. At small vocabularies ( V≤100 ) flip rate rises well after accuracy has converged; from V=500 upward the two curves rise together.
Figure 11: Timecourses by number of rules (vocabulary size 100). The 2-rule condition is the plain ABA/ABB task. The 10-rule condition uses ten length-5 rules.
Figure 12: Ratio of flip-rate to accuracy convergence step across diversity levels. A ratio of 1 means the mechanism converges together with accuracy. Points are means over runs. The ratio is about 1 for vocabulary sizes V≥500 and for the number-of-rules axis, and larger for the smallest vocabularies.
Figure 13: Effect of model size on the emergence of the symbolic mechanism (vocabulary size 50). Left: flip rate at the step where accuracy first converges, mean ± standard deviation over seeds. Right: training dynamics of the two models: accuracy and flip rate over training.
Figure 14: Causal mediation analysis for binding. Top: Stage 1 (binding-ID retrieval). Bottom: Stage 2 (token retrieval). Left: layer × token position (patching attention and MLP output at one position). Right: layer × head (patching one head’s output at the query position). Layers are numbered from 0.
Figure 15: CMA-weighted representational similarity for binding. Average cosine-similarity matrices over heads with CMA z>0.5 , ordered by binding ID (ID0 prompts first, ID1 second). Left: Stage 1 (binding-ID retrieval). Right: Stage 2 (token retrieval).
Large language models often reason beyond surface tokens, but the internal stage at which token-level information becomes abstract relational structure remains unclear. We investigate this question by analyzing how attention heads and layers transform information during autoregressive reasoning. Across mathematical and symbolic reasoning tasks, we observe a consistent layer-wise division of labor: outer layers mainly preserve and route input-related features, whereas middle layers reorganize them into more transferable rule-level representations. This interpretation is supported by representation geometry: middle-layer states occupy lower-dimensional manifolds and show stronger alignment across disjoint vocabularies that instantiate the same symbolic rules. It is further supported by causal interventions: removing middle-layer components identified by our interaction-based criterion produces substantially larger downstream changes and accuracy drops than removing components from other regions or at random. Together, these results suggest that abstract reasoning is not uniformly distributed across transformer layers, but is preferentially formed in a middle-layer computation stage that converts token-level information into reusable relational structure.
Junjie Zhang, Zhen Shen, Xisong Dong +1
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
We study structured abstraction-based reasoning for the Abstraction and Reasoning Corpus (ARC) and compare its generalization to test-time approaches. Purely neural architectures lack reliable combinatorial generalization, while strictly symbolic systems struggle with perceptual grounding. We therefore propose a neuro-symbolic architecture that extracts object-level structure from grids, uses neural priors to propose candidate transformations from a fixed domain-specific language (DSL) of atomic patterns, and filters hypotheses using cross-example consistency checks. Instantiated as a compositional reasoning framework based on unit patterns inspired by human visual abstraction, the system augments large language models (LLMs) with object representations and transformation proposals. On ARC-AGI-2, it improves over the base solver LLM (Grok-4, 16.0%) to 24.4% on the public evaluation set, and to 30.8% when combined with ARC Lang Solver via a meta-classifier. These results support the claim that in the evaluated settings, allocating test-time compute to object abstraction, neural-guided transformation proposal, and consistency filtering can improve accuracy over unstructured sampling-based scaling without task-specific finetuning of the system. We open-source the code: (https://github.com/CoreThink-AI/arc-agi-2-reasoner).
Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.