cs.CLAug 30, 2026

The Emergent Symbolic Structure of Artificial Neural Networks

Authors: R. Thomas McCoy, Paul Soulos, Tal Linzen, Paul Smolensky

Organizations: Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.

Figures & tables

Figure 1.1: The apparent incompatibility of neural and symbolic representations. Left: A symbolic sentence representation, specifically a (simplified) syntax tree. Right: A neural-network sentence representation, specifically a continuous vector. These two representations encode the same sentence, but they appear starkly different from each other. Nonetheless, neural networks are the state of the art in many areas traditionally viewed as symbolic. In this work, we study how vector representations can support such strong performance in domains that appear to require symbolic structure.
Figure 2.1: Linearly-transformed Tensor Product Representations (TPRs). A. Structure of a linearly-transformed TPR. (1) . Each filler and each role is encoded with a vector that stands for it. (2) Each filler and its corresponding role are bound together using the tensor product, illustrated in B . (3) The tensor products are summed. (4) A linear transformation is applied to give the final encoding, EE . C. Illustrations of TPRs used to encode pairs of letters with the roles 1st1^{\text{st}} and 2nd2^{\text{nd}} . In the special case, the encodings have a human-interpretable structure both when construed as lists of numbers and when construed as points in space. In the general case, the list of numbers is not readily interpretable, but the geometric arrangement remains intact (in a stretched, rotated, and translated coordinate system), illustrating how vectors can have systematic compositional structure even if no such structure is apparent to human observers. The example TPRs shown here are only two-dimensional for ease of visualization, but most of the TPRs we deal with in this paper are much larger (with hundreds or thousands of dimensions).
Figure 3.1: Left: The DISCOVER process. (i) We train the target model, whose encodings EE we wish to understand. (ii) We then train a DISCOVER model, which is explicitly structured as a linearly-transformed TPR, so that, for each input, its encoding ETPRE^{\text{TPR}} is as close as possible to the target model’s encoding EE . (iii) We then evaluate the DISCOVER model by plugging its encoding ETPRE^{\text{TPR}} into the decoder of the original target model, to see if the target model still produces the right output when provided with ETPRE^{\text{TPR}} instead of EE . Right: A tapir ( Tapirus indicus ). Tapirs are the mascot of DISCOVER because the word tapir can be created by interleaving TPR with AI . Tapir image © 2026 by Victoria E. McCoy under CC BY-SA 4.0 ( https://creativecommons.org/licenses/by-sa/4.0/ ).
Figure 3.2: Left: The role schemes that we use for analyzing letter-sequence-processing neural networks, with an example of the roles assigned by each role scheme to the sequence Q M Z C . Right: Performance of DISCOVER with various role schemes at approximating the encodings of GRUs trained to reverse letter sequences. The bars show the mean across 10 reruns, with error bars showing two standard deviations above and below the mean.
Figure 4.1: DISCOVER performance for models trained on letter-sequence-manipulation tasks. The bars show the mean across 10 reruns, with error bars showing two standard deviations above and below the mean.
Figure 5.1: Analyzing LLM representations of periods. A. Given some input, an LLM generates one vector per input token per layer. We analyze the vector EE that the LLM uses to represent a period at a particular layer (here, the middle layer). We hypothesize that EE encodes the full sentence preceding it; we test this hypothesis by training a period-unpacking model to take in EE and reconstruct the sentence preceding EE . B. To understand how EE encodes the information that the period-unpacking model extracts, we train a DISCOVER model to approximate EE with a TPR ( ETPRE^{TPR} ). C. To evaluate how successful ETPRE^{TPR} has been at capturing the structure of EE , we evaluate whether feeding ETPRE^{TPR} into the period-unpacking model yields the correct output sequence. ( ETPRE^{TPR} and the period-unpacking model were never trained together—they are paired up solely for evaluation.)
Figure 5.2: DISCOVER performance at approximating the encodings of periods in LLMs for lists of length 3 to 5 (top), subject-verb-object sentences (middle), and sentences with varying syntax (bottom). The points show the mean across 5 reruns, with error bars showing two standard deviations above and below the mean (the error bars are often too small to be visible). The dashed red lines show the performance of the trained period decoders when they are fed LLM period encodings; the remaining lines show the performance of the same period decoders when they are fed DISCOVER approximations. See Section 5.4 for descriptions of the role schemes.
TaskExampleAnswer
Arithmetic7 + -6 * 3-11
SyllogismsWe are given two premises: No polite agents are painters. All gamblers are painters. Please return the first conclusion listed below that follows from these premises: All gamblers are polite agents. Some gamblers are polite agents. No gamblers are polite agents. Some gamblers are not polite agents.No gamblers are polite agents.
Code executiondef change(sequence): return [“Q”] + sequence def alter(sequence): return sequence[::-1] y = [“S”, “M”, “V”] x = [“U”, “C”] change(y)[“Q”, “S”, “M”, “V”]
PassivizationPassivize the following sentence: The swimmer who the polite politician advised avoided the accountant near the clever gardener.The accountant near the clever gardener was avoided by the swimmer who the polite politician advised.
Tense reinflectionConvert the following sentence to the present tense: The important detective who recommended the astronauts recognized the tall detective.The important detective who recommends the astronauts recognizes the tall detective.
Question formationConvert the following sentence to a yes/no question: The professor who the painter will encourage would visit the musician near the famous runner.Would the professor who the painter will encourage visit the musician near the famous runner?
Table 1: Symbolic tasks that we use with GPT-OSS, along with examples and their answers. These tasks span 4 domains: mathematics (arithmetic), logic (syllogisms), computer code (code execution), and language (passivization, tense reinflection, and question formation).
Figure 6.1: Analyzing the complete input representation of an LLM (GPT-OSS). A: Standard LLM processing. The LLM is given a prompt made of some instructions and an input. The LLM uses many layers of processing (we show 3 layers here, but GPT-OSS in fact has 25 layers); it generates one vector per layer for each token in the prompt. It then generates the output, one token at a time. As each output token is generated, that token is appended to the prompt, and the LLM generates vector representations for it (one vector per layer), until the output is finished. B: Approximating the LLM with DISCOVER. We leave the representations of the instructions unchanged because they are identical across all inputs for a given task, but we replace all vectors that encoded the input—across layers and input tokens—with a TPR approximation (which is produced by an interpretable, closed-form equation). The LLM then generates its output based on the TPR approximations rather than based on its own input representations. We assess whether the resulting LLM output is correct. The instructions and input shown here are simplified for the sake of space.
 Bag-of-wordsBidirectionalBidirectionalTask-specificTask-specific
  (self)(all)(self)(all)
The{present:The}{(0,0):The}{(0,0):The}{subj_det:The}{subj_det-subj_det:The}
spy{present:The, present:spy}{(1,0):spy}{(0,1):The, (1,0):spy}{subj_n:spy}{subj_det-subj_n:The, subj_n-subj_n:spy}
helped{present:The, present:spy, present:helped}{(2,0):helped}{(0,2):The, (1,1):spy, (2,0):helped}{verb:helped}{subj_det-verb:The, subj_n-verb:spy, verb-verb:helped}
the{present:The, present:spy, present:helped, present:the}{(3,0):the}{(0,3):The, (1,2):spy, (2,1):helped, (3,0):the}{obj_det:the}{subj_det-obj_det:The, subj_n-obj_det:spy, verb-obj_det:helped, obj_det-obj_det:the}
poet{present:The, present:spy, present:helped, present:the, present:poet}{(4,0):poet}{(0,4):The, (1,3):spy, (2,2):helped, (3,1):the, (4,0):poet}{obj_n:poet}{subj_det-obj_n:The, subj_n-obj_n:spy, verb-obj_n:helped, obj_det-obj_n:the, obj_n-obj_n:poet}
Table 2: The five role schemes used to analyze GPT-OSS performing passivization, illustrated through the input The spy helped the poet (an actual stimulus would also contain formatting tokens, such as a period). Each row corresponds to one input token. Each table entry shows the role-filler pairs that the relevant token’s representation is hypothesized to contain under the indicated role scheme. For instance, under the bidirectional (all) role scheme, the token helped is hypothesized to contain three role-filler pairs: {(0,2):The, (1,1):spy, (2,0):helped}.
Figure 6.2: DISCOVER results at approximating GPT-OSS performing various symbolic tasks. The dashed lines indicate GPT-OSS’s performance on each task. Each bar is the mean across 5 reruns, with error bars showing two standard deviations.
Figure 7.1: Left: The constituent surgery process used to test if we can edit a model’s behavior via targeted edits to its representations ( Soulos et al., 2020 ) . (1) We obtain the target model’s encoding EE for an input (here, the input W,X,C is being provided to a target model that performs reversal). (2) We edit EE by subtracting the predicted representation for one role-filler pair and adding the predicted representation for another role-filler pair, producing an edited encoding E′E^{\prime} . (3) We feed E′E^{\prime} into the target model’s decoder and see if the decoder shows the intended change in behavior (here, producing D,X,W instead of C,X,W ). Right: Results of applying constituent surgery on models trained to perform transformations on sequences of letters.
Figure 7.2: Accuracy of causal interventions carried out on LLM period encodings of lists and sentences.
Figure 7.3: Causal interventions on fillers in GPT-OSS. For each type of intervention shown here, we run 100 interventions on examples of that type, yielding accuracies of 0.99 when changing an element in a list, 1.00 when changing the number of repetitions, 0.95 when changing which function is called, and 0.96 when changing which variable is used as the argument of the function call.
Figure 7.4: Causal interventions on GPT-OSS that change the structure of the input. Such interventions are conducted by using a DISCOVER approximation to edit the role of an element in the input. A. Swapping the positions of two numbers in an arithmetic expression (average intervention accuracy: 0.978). B. Moving a letter from one list to another in Python code (average intervention accuracy: 0.950). C. Moving modifiers within a sentence in the tense reinflection task (in these examples, the modifier always moves from the object to the subject). Moving a prepositional phrase or relative clause requires editing many role-filler pairs; further, the movement of the relative clause requires one verb to change from singular to plural in the output (underlined). Average intervention accuracy: 0.980 for adjectives, 0.892 for prepositional phrases, 0.958 for relative clauses.
Figure 8.1: Testing how DISCOVER generalizes to novel role-filler pairs when it is trained to approximate the encodings of models trained to manipulate sequences of letters. The dots show the mean across 10 reruns, while the error bars show two standard deviations (truncated at 0.0 and 1.0).
Figure 8.2: Testing how DISCOVER generalizes to novel role-filler pairs when it is trained to approximate LLM period encodings after lists of length 5 (top) or sentences of the form “the NOUN VERBED the NOUN, and the NOUN VERBED the NOUN.” (bottom). This plot shows the results for the middle layer of each LLM (the one labeled “50%” elsewhere in this paper); see Figure K.1 in the Appendix for other layers. The dots show the mean across 10 reruns, while the error bars show two standard deviations (truncated at 0.0 and 1.0).
Figure 8.3: Testing how DISCOVER generalizes to novel role-filler pairs when it is trained to approximate the encodings of GPT-OSS performing symbolic tasks. The dots show the mean across 5 reruns, while the error bars show two standard deviations (truncated at 0.0 and 1.0).
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: Target model accuracies on tasks that involve manipulating sequences of letters. The bars show the mean across 10 reruns, with error bars showing two standard deviations above and below the mean.
Figure C.1: Left: Target model accuracies on the task of sorting a list into alphabetical order. Right: DISCOVER performance for models trained on sorting a list of letters into alphabetical order. The bars show the mean across 10 reruns, with error bars showing two standard deviations above and below the mean.
LLMLayers analyzedHidden state size
Gemma-3-27b1, 16, 31, 46, 625376
GPT-2-XL1, 12, 24, 36, 481600
GPT-OSS-20b1, 6, 12, 18, 242880
Pythia-12b1, 9, 18, 27, 365120
Qwen3-14b1, 10, 20, 30, 405120
OLMo-2-13B1, 10, 20, 30, 405120
Appendix
Table 3: Information about the LLMs whose period encodings we analyze. For all models, layer 0 is the embedding layer, so layer 1 (the earliest one we analyze) is the first non-embedding layer. The last layer that we analyze is always the latest layer in the model.
Figure F.1: Accuracy of period-unpacking models trained to take in an LLM’s representation of a period and reconstruct the sentence that preceded that period.
Figure F.2: An example of a dependency parse used for the syntactic role scheme when analyzing the LLM period encodings of complex sentences.
Figure G.1: DISCOVER performance at approximating the encodings of periods in LLMs for the “complex sentences” condition, when the period-unpacking model into which the DISCOVER approximations are fed is trained to reproduce the original sentence as a linear string (top), a linearized constituency parse (middle), or a linearized dependency parse (bottom). Each point shows the mean over 5 re-runs, with error bars showing two standard deviations above and below the mean. The dashed red lines show the performance of the trained period-unpacking models when they are fed LLM period encodings; the remaining lines show the performance of the same period-unpacking models when they are fed DISCOVER approximations.
Figure H.1: Performance of GPT-OSS on the six symbolic tasks that we consider.
 OriginalEdit to make
The{subj_det-subj_det:The}None
spy{subj_det-subj_n:The, subj_n-subj_n:spy}−- subj_n-subj_n:spy ++ subj_n-subj_n:astronaut
helped{subj_det-verb:The, subj_n-verb:spy, verb-verb:helped}−- subj_n-verb:spy ++ subj_n-verb:astronaut
the{subj_det-obj_det:The, subj_n-obj_det:spy, verb-obj_det:helped, obj_det-obj_det:the}−- subj_n-obj_det:spy ++ subj_n-obj_det:astronaut
poet{subj_det-obj_n:The, subj_n-obj_n:spy, verb-obj_n:helped, obj_det-obj_n:the, obj_n-obj_n:poet}−- subj_n-obj_n:spy ++ subj_n-obj_n:astronaut
.{subj_det-punct:The, subj_n-punct:spy, verb-punct:helped, obj_det-punct:the, obj_n-punct:poet, punct-punct:.}−- subj_n-punct:spy ++ subj_n-punct:astronaut
Appendix
Table 4: Representational edits that would be made with the goal of changing spy to astronaut .
Figure J.1: Analysis of whether causal interventions on GPT-OSS representations are sensitive to sentence structure or are purely driven by linear order. See the text for descriptions of the terms in the plots. All results (right) show at least some version of the diagonal pattern predicted if the interventions are structure-sensitive (top left) rather than the uniform pattern predicted if the interventions are not structure-sensitive (bottom left).
Figure K.1: Testing how DISCOVER generalizes to novel role-filler pairs when it is trained to approximate LLM period encodings after lists of length 5 (top) or sentences of the form “the NOUN VERBED the NOUN, and the NOUN VERBED the NOUN.” (bottom). The dots show the mean across 10 reruns, while the error bars show two standard deviations (truncated at 0.0 and 1.0). The dashed lines show a strong baseline.
ArchitectureTaskBest λ\lambdaOOD accuracy
   Unreg.Reg.
MLPCopying0.01.01.0
 Reversal0.01.01.0
 Interleaving0.01.01.0
GRUCopying0.0010.770.98
 Reversal0.000011.01.0
Appendix
Table 5: OOD validation set results of the hyperparameter search for the λ\lambda hyperparameter that determines how much to weight the L2,1L_{2,1} regularization loss term for letter sequence models. The accuracies are on examples with 2 withheld role-filler pairs. We show accuracies in the unregularized setting (‘unreg.’—i.e., λ=0\lambda=0 ) and in the regularized setting (‘reg.’—i.e., with the best λ\lambda ).
ModelTaskBest λ\lambdaOOD accuracy
   Unreg.Reg.
Gemma (31)Sentences0.0010.000.86
Llama-3.1 (16)Sentences0.00050.000.90
OLMo-2 (20)Sentences0.0010.000.86
Appendix
Table 6: OOD validation set results of the hyperparameter search for the λ\lambda hyperparameter that determines how much to weight the L2,1L_{2,1} regularization loss term for LLM period encodings (left) and for GPT-OSS performing symbolic tasks (right). We show accuracies in the unregularized setting (‘unreg.’—i.e., λ=0\lambda=0 ) and in the regularized setting (‘reg.’—i.e., with the best λ\lambda ). On the left, the accuracies are on examples with 5 withheld role-filler pairs, and on the right the accuracies are on examples with 2 withheld role-filler pairs. In the model column on the left, the numbers in parentheses indicate the layer we were analyzing.
Figure M.1: DISCOVER performance on two white-box models: one that is known to have systematic role-filler binding (“TPR”) and one that is known not to have systematic role-filler binding (“Atomic Pair”).
Role schemeNumber of roles
Left-to-right6
Right-to-left6
Bidirectional21
Wickelroles729
Bag-of-words1
Appendix
Table 7: Number of roles for the DISCOVER models trained in the letter sequence experiments (left) and for the bidirectional (all) and task-specific (all) role schemes used in DISCOVER approximations of GPT-OSS (right).
More in this topic

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 333 papers

Explore similar work

CardsList
Jul 6, 2026

The Changing Role of Symbolic Methods in Artificial Intelligence

Why do intelligent systems need to perform explicit symbolic reasoning? Computer science has traditionally regarded symbolic reasoning as a defining component of intelligence. Yet the remarkable success of modern foundation models raises a fundamental question: if increasingly capable AI systems can operate with little explicit symbolic reasoning, what role do symbolic methods actually play? This article argues that explicit symbolic reasoning is not a fundamental property of intelligence, but a computational consequence of operating on simplified models of reality. We propose the Compression Principle: every computational model is a simplified representation of reality, and explicit symbolic reasoning compensates for information omitted during model construction. From this principle, we derive the Modeling--Reasoning Trade-off: as computational models preserve richer representations of the world, the need for explicit symbolic reasoning correspondingly decreases. This perspective provides a unified explanation for both the historical success of symbolic methods and the remarkable effectiveness of modern foundation models. Paradoxically, the same development makes symbolic methods increasingly important for humans. As intelligent systems become more capable and more opaque, symbolic representations increasingly serve as interfaces through which humans specify requirements, verify behavior, regulate autonomous systems, and establish trust. We therefore argue that the future of symbolic methods lies not primarily as the computational engine of intelligent systems, but as the symbolic interface between increasingly capable AI systems and the humans who build, govern, and depend upon them.

Jun Sun

Singapore Management University, Singapore

Logical ReasoningSymbolic ReasoningHuman Oversight of AI
AskPDF
Apr 2, 2026

Compositional Neuro-Symbolic Reasoning

We study structured abstraction-based reasoning for the Abstraction and Reasoning Corpus (ARC) and compare its generalization to test-time approaches. Purely neural architectures lack reliable combinatorial generalization, while strictly symbolic systems struggle with perceptual grounding. We therefore propose a neuro-symbolic architecture that extracts object-level structure from grids, uses neural priors to propose candidate transformations from a fixed domain-specific language (DSL) of atomic patterns, and filters hypotheses using cross-example consistency. Instantiated as a compositional reasoning framework based on unit patterns inspired by human visual abstraction, the system augments large language models (LLMs) with object representations and transformation proposals. On ARC-AGI-2, it improves base LLM performance from 16% to 24.4% on the public evaluation set, and to 30.8% when combined with ARC Lang Solver via a meta-classifier. These results demonstrate that separating perception, neural-guided transformation proposal, and symbolic consistency filtering improves generalization without task-specific finetuning or reinforcement learning, while reducing reliance on brute-force search and sampling-based test-time scaling. We open-source the ARC-AGI-2 Reasoner code (https://github.com/CoreThink-AI/arc-agi-2-reasoner).

Anugyan Das, Omkar Ghugarkar, Vishvesh Bhat +1

CoreThink AI · Stanford University

Neuro-Symbolic ReasoningVisual ReasoningLLM ReasoningProgram Synthesis
AskPDF
Mar 3, 2026

Neuro-Symbolic Artificial Intelligence: A Task-Directed Survey in the Black-Box Models Era

The integration of symbolic computing with neural networks has intrigued researchers since the first theorizations of Artificial intelligence (AI). The ability of Neuro-Symbolic (NeSy) methods to infer or exploit behavioral schema has been widely considered as one of the possible proxies for human-level intelligence. However, the limited semantic generalizability and the challenges in declining complex domains with pre-defined patterns and rules hinder their practical implementation in real-world scenarios. The unprecedented results achieved by connectionist systems since the last AI breakthrough in 2017 have raised questions about the competitiveness of NeSy solutions, with particular emphasis on the Natural Language Processing and Computer Vision fields. This survey examines task-specific advancements in the NeSy domain to explore how incorporating symbolic systems can enhance explainability and reasoning capabilities. Our findings are meant to serve as a resource for researchers exploring explainable NeSy methodologies for real-life tasks and applications. Reproducibility details and in-depth comments on each surveyed research work are made available at https://github.com/disi-unibo-nlp/task-oriented-neuro-symbolic.git.

Giovanni Pio Delvecchio, Lorenzo Molfetta, Gianluca Moro

Department of Computer Science and Engineering, University of Bologna, Cesena Campus Via dell’Universit`a 50, I-47522 Cesena, Italy

Neuro-Symbolic AINeuro-Symbolic ReasoningNeurosymbolic LearningExplainable Artificial Intelligence
AskPDF