Neural architectures for resolving references in program code

Gergő Szalay; Gergely Zsolt Kovács; Sándor Teleki; Balázs Pintér; Tibor Gregorics

cs.LGApr 15, 2026

Neural architectures for resolving references in program code

Authors: Gergő Szalay, Gergely Zsolt Kovács, Sándor Teleki, Balázs Pintér, Tibor Gregorics

Organizations: Faculty of Informatics, ELTE Eötvös Loránd University, Budapest, H-1117

Abstract

Resolving and rewriting references is fundamental in programming languages. Motivated by a real-world decompilation task, we abstract reference rewriting into the problems of direct and indirect indexing by permutation. We create synthetic benchmarks for these tasks and show that well-known sequence-to-sequence machine learning architectures are struggling on these benchmarks. We introduce new sequence-to-sequence architectures for both problems. Our measurements show that our architectures outperform the baselines in both robustness and scalability: our models can handle examples that are ten times longer compared to the best baseline. We measure the impact of our architecture in the real-world task of decompiling switch statements, which has an indexing subtask. According to our measurements, the extended model decreases the error rate by 42%. Multiple ablation studies show that all components of our architectures are essential.

Figures & tables

Figure 1: An example of decompiling a switch statement as a reference rewriting task. The assembly snippet on the left is decompiled into the C snippet on the right. Although the branches (green) appear in the same order on both sides, the order of the case labels (red) differs: they need to be reordered during decompilation. The connecting link between a branch and its corresponding case label is the jump label in the assembly. We can regard the jump label for the branch as the query (reference), the jump label for the case label as the key, and the case label itself as the data (referent).

Figure 2: The architecture created for the direct permutation problem. The encoder and the decoder can be seen in the top left and the top right corner, respectively. The inner structure of the decoder is shown at the bottom. The input is processed by two different embedding layers. One of these maps the tokens to a potential indexing distribution, the other one provides an input to the bidirectional encoder GRU . The output of the encoder GRU is given to a linear layer to specify the beginning of the DATA sequence. The hidden representation created by the encoder GRU is used by the decoder. The new hidden vector given by the GRU of the decoder is used to generate the logits for a fixed vocabulary, through a linear layer. The new hidden vector is also combined with the encoder outputs to select the current indexing position. This distribution is then element-wise multiplied with the index embedding values and accumulated to produce a single index distribution. This is then element-wise multiplied further with the sequence-beginning distribution, resulting in the logits that select the appropriate locations from the input. Both logits are summed to get the final output of the decoder step.

Figure 3: The architecture created for the indirect permutation problem with the attention mechanism (detailed in Section 3.2.1 ). The encoder can be seen on the top and the left of the figure, while the structure of the decoder can be seen at the bottom. The encoder processes the input token-wise with an embedding and a bidirectional GRU layer. The outputs of this GRU are given to a linear layer to produce a distribution that determines whether the given token is a DATA, a KEY, a QUERY, or NOT_RELEVANT. Using these values, associations are created from the QUERY tokens to the KEY tokens and from the KEY tokens to the DATA tokens respectively. These connections are then matrix-multiplied to produce the association from QUERY to DATA. The hidden vector created by the GRU of the encoder is given to the decoder for further processing. In each step, the decoder creates a new hidden vector, which is forwarded to a linear layer to predict the logits from a fixed vocabulary. Simultaneously, the new hidden vector is multiplied by the QUERY weights created by the encoder and further multiplied by the association matrix created in the encoder to predict the output. The two outputs are then summed, resulting in the output of the decoder step.

	1-10		10		20
	TA	WEA	TA	WEA	TA	WEA
GRU	21.75%	0%	17.17%	0%	–	–
GRU + Attn.	99.99%	99.98%	24.92%	0%	–	–
CopyNet	99.97%	99.86%	100 %	100 %	94.01%	0% 1 1 1 The learning of the Copying mechanism [ 3 ] was unstable. The highest token accuracies for different training runs range from 25% to 94%, but none could solve the task. See more details in Section 5.1 .
Transformer	41.87%	0%	39.24%	0%	–	–
PermNetD	99.99%	99.99%	100%	100%	99.95%	99.62%

Table 1: The results of different architectures on the direct permutation problem. If a model could not handle the PD

{}_{\text{10}}

dataset, no further evaluation was made on the PD

{}_{\text{20}}

dataset. TA marks the token accuracy and WEA marks the whole example accuracy.

	40		100
	TA	WEA	TA	WEA
PermNetD	99.99%	99.98%	99.99%	99.89%

Table 2: The results of PermNetD on the direct permutation problem. TA marks the token accuracy and WEA marks the whole example accuracy.

	1-10		10		20
	TA	WEA	TA	WEA	TA	WEA
GRU	19.20%	0%	16.83%	0%	–	–
GRU + Attn.	56.83%	17.98%	16.35%	0%	–	–
CopyNet	99.90%	99.32%	99.99%	99.97%	15.93%	0%
Transformer	99.97%	19.77%	39.15%	0%	–	–
PermNetI	99.99%	99.99%	100%	100%	99.99%	99.99%

Table 3: The results of different architectures on the indirect permutation problem. If a model could not handle the PI

{}_{\text{10}}

dataset, no further evaluation was made on the PI

{}_{\text{20}}

dataset. TA marks the token accuracy and WEA marks the whole example accuracy.

	40		100
	TA	WEA	TA	WEA
PermNetI	100%	100%	99.96%	98.26%

Table 4: The results of PermNetI on the indirect permutation problem. TA marks the token accuracy and WEA marks the whole example accuracy.

	Dict
	TA	WEA
CopyNet	83.33%	0.0%
PermNetI	99.94%	99.72%

Table 5: The results of copying mechanism [ 3 ] and the PermNetI on the PI

{}_{\text{DICT}}

permutation problem (defined in Section 4.1 ). TA marks the token accuracy and WEA marks the whole example accuracy.

Figure 4: The weights of the index embedding of the model (Eq. 5 ) for the direct permutation problem after being trained on the PD ${}_{\text{40}}$ benchmark. The most signicicant values (the orange-yellow diagonal) correspond to the correct indexing distribution for the relevant tokens. The values for non-index tokens are mostly zeroes.

Figure 5: An example of the $M_{dp}$ matrix (defined in Eq. 22 ) calculated by a trained PermNetI model on a PI ${}_{\text{20}}$ sample. This square matrix has dimensions equal to the length of the input sequence. The beginning of the sequence contains the DATA-KEY pairs, while the end of the sequence contains the QUERY tokens. The matrix connects the QUERY tokens to the DATA tokens, which is visible from the high values in the top right segment. These significant values mark the DATA tokens and the related KEY tokens as 2-by-1 areas, with the latter having smaller values. This confirms our intuition that the model focuses on the correct DATA tokens.

w/o	PD ${}_{\text{10}}$		PD ${}_{\text{20}}$		PD ${}_{\text{40}}$
	TA	WEA	TA	WEA	TA	WEA
base	100%	100%	99.95%	99.62%	99.99%	99.98%
fixed-vocab.	100%	100%	99.92%	98.67%	86.55%	0.17%
$M_{e}$	99.98%	99.93%	100%	100%	-	-
$M_{ind}$	99.99%	99.90%	71.51%	0.0%	-	-
$v_{arr}$	99.89%	99.29%	99.99%	99.89%	-	-

Table 6: The results of PermNetD after removing parts from it.

Figure 6: An example of the model focusing on the DATA-KEY pairs when removing $M_{d}$ from PermNetI on a PI ${}_{\text{10}}$ example. The matrix is similar to Figure 5 , but there are 10 2-by-1 yellow areas in the picture instead of 10 1-by-1 cells. For most of them, the value of the upper part (DATA) is less than or equal to the value of the lower one (KEY).

w/o	PI ${}_{\text{10}}$		PI ${}_{\text{20}}$		PI ${}_{\text{40}}$
	TA	WEA	TA	WEA	TA	WEA
base	100%	100%	99.99%	99.99%	100%	100%
fixed-vocab.	48.94%	0%	-	-	-	-
$M_{d}$ , $M_{k}$ , $M_{p}$	33.34%	0%	-	-	-	-
$M_{d}$	99.99%	99.95%	-	-	-	-
attn.	100%	100%	100%	100%	99.99%	99.96%

Table 7: The results of PermNetI after removing components from it.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Dataset
	Direct		Indirect
	PD ${}_{\text{1-10}}$	PD ${}_{\text{10}}$	PI ${}_{\text{1-10}}$	PI ${}_{\text{10}}$
Training data count	103515	90000	90292	90000
Test data count	11502	10000	10033	10000
Embedding size	128	128	128	128
Hidden size	1024	1024	1024	1024

Appendix

Table 8: Hyperparameters of GRU [ 2 ]

Name	Dataset
	Direct			Indirect
	PD ${}_{\text{1-10}}$	PD ${}_{\text{10}}$	PD ${}_{\text{20}}$	PI ${}_{\text{1-10}}$	PI ${}_{\text{10}}$
Training data count	102960	90000	90000	39329	180000
Test data count	11440	10000	10000	4370	20000
Embedding size	256	128	256	256	256
Hidden size	1024	1024	1024	1024	1024

Appendix

Table 9: Hyperparameters of attention mechanism [ 1 ]

Name	Dataset
	Direct			Indirect
	PD ${}_{\text{1-10}}$	PD ${}_{\text{10}}$	PD ${}_{\text{20}}$	PI ${}_{\text{1-10}}$	PI ${}_{\text{10}}$	PI ${}_{\text{20}}$
Training data count	100000	100000	100000	100000	100000	100000
Test data count	200017	200000	200000	209600	200000	200000
Embedding size	64	64	64	64	128	128
Hidden size	512	512	512	512	512	512

Appendix

Table 10: Hyperparameters of copying mechanism [ 3 ] (CopyNet)

Name	Dataset
	Direct		Indirect
	PD ${}_{\text{1-10}}$	PD ${}_{\text{10}}$	PI ${}_{\text{1-10}}$	PI ${}_{\text{10}}$
Training data count	152640	90000	90292	90000
Test data count	16960	10000	10033	10000
Embedding size	256	256	256	256
Hidden size	1024	1024	1024	1024

Appendix

Table 11: Hyperparameters of transformer [ 11 ]

Name	Dataset
	Direct			Indirect
	PD ${}_{\text{1-10}}$	PD ${}_{\text{10}}$	PD ${}_{\text{20}}$	PI ${}_{\text{1-10}}$	PI ${}_{\text{10}}$	PI ${}_{\text{20}}$
Training data count	100000	100000	100000	100000	100000	100000
Test data count	100000	100000	100000	100000	100000	100000
Embedding size	64	64	64	128	128	128
Hidden size	32	32	32	256	32	128

Appendix

Table 12: Hyperparameters of PermNetD (described in Section 3.1 ) and PermNetI (described in Section 3.2 ).

Name	Dataset
	Direct		Indirect
	PD ${}_{\text{40}}$	PD ${}_{\text{100}}$	PI ${}_{\text{40}}$	PI ${}_{\text{100}}$	PI ${}_{\text{DICT}}$
Training data count	100000	100000	100000	100000	100000
Test data count	100000	100000	100000	100000	100000
Embedding size	128	256	128	128	128
Hidden size	64	128	256	512	64

Appendix

Table 13: Hyperparameters of PermNetD and PermNetI tested for scalability and PI

{}_{\text{DICT}}

benchmarks (described in Section 4.1 ).

Sequence-to-Sequence Models

See all 11 papers

Decaf: Improving Neural Decompilation with Automatic Feedback and Search
May 12, 2026Alexander Shypula, Osbert Bastani, Edward SchwartzSoftware Reverse Engineering Compiler-Guided Program Synthesis
University of Pennsylvania Philadelphia, Pennsylvania 19104 · Carnegie Mellon University Pittsburgh, Pennsylvania 15213
CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference
Apr 14, 2026Qiang Zhang, Zhongnian LiAdaptive Inference Software Reverse Engineering
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, Xuzhou, China · Mine Digitization Engineering Research Center of the Ministry of Education, China University of Mining and Technology, Xuzhou, China
Echo: Learning-based Matching Decompilation using Trusted Back Translation
Sep 16, 2026Jun Bi, Xiangxin Fang, Aarsh Chaube +3Software Reverse Engineering Program Synthesis
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Edinburgh

May 12, 2026J/K move · Enter open · S save

Decaf: Improving Neural Decompilation with Automatic Feedback and Search

Alexander Shypula, Osbert Bastani, Edward Schwartz

University of Pennsylvania Philadelphia, Pennsylvania 19104 · Carnegie Mellon University Pittsburgh, Pennsylvania 15213

Software Reverse Engineering Compiler-Guided Program Synthesis

Open paper AskPDF arXiv

Decompilers are useful tools used in reverse engineering to understand compiled source code. Reconstructing source code from compiled binaries is a challenging task, because high-level syntax, identifiers, and custom data types are generally lost as the compiler translates human-readable code to low-level machine code. Deterministic decompilers are useful tools for binary analysis, but can struggle to infer idiomatic syntax and identifier names. Generative AI models are a natural fit for reconstructing high-level syntax, identifiers, and types, but they can still suffer by hallucinating improper programming constructs and semantics. Instead of attempting to improve neural decompilers with more data and more training, we argue that compiler feedback can be used to dramatically improve the semantic correctness of neural decompiler outputs via search. Our system, Decaf (DECompilation with Automated Feedback), raises the neural decompilation rate from 26.0% on ExeBench to 83.9% on the Real -O2 split without sacrificing similarity to the original source code. We also find our automatic feedback methodology is highly effective for improving weaker neural decompilation models.

Abstract

Sequence-to-Sequence Models

Explore similar work

Decaf: Improving Neural Decompilation with Automatic Feedback and Search

CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference

Echo: Learning-based Matching Decompilation using Trusted Back Translation