More Context, Mixed Results: Evaluating LLMs for Identifier Recovery and Code Deobfuscation
Authors: Simon Queyrut, Valerio Schiavoni, Giuseppe Antonio Di Luna, Daniele Cono D'Elia, Leonardo Querzoni, Pascal Felber
Organizations: Computer Science Department, University of Neuchâtel, Neuchâtel, Switzerland. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy.
Source code is often obfuscated through identifier renaming, incentivizing reverse engineers to develop techniques for recovering meaningful names. To assess how LLMs compare with these techniques, we evaluate 4 traditional baselines and 3 LLMs (Qwen, OLMo and DeepSeek) on identifier detection and semantic recovery across 51 recent Java projects, 3 obfuscators, and 6 context/retrieval settings. LLMs generally outperform traditional techniques on both tasks, but benefit from different forms of context and settings: agentic retrieval yields the strongest detection results, reaching 91.56% F1, while enclosing-class context generally improves semantic recovery at a higher token cost. Despite these gains, exact recovery peaks at 16.42% overall (under one-hop context) and 32.93% for field names (under repository retrieval), which are the easiest to recover. Class identifiers are the hardest to detect, with a mean detection F1 of 48.48% across models, settings and obfuscators. More context is not always beneficial: larger methods hinder semantic recovery. Beyond identifier renaming, our experiments show that control-flow obfuscation degrades both tasks more than string encryption. Our findings provide useful ad hoc insights for reverse engineers choosing models and context strategies to deobfuscate code snippets.
Figures & tables
Notation
Meaning
S(m)
All visible identifier occurrences in m .
U(m)⊆S(m)
Identifier occurrences whose project-owned class, method, or field declaration was genuinely obfuscated.
T(m)
The different declarations referred to by U(m) ; private mapping IDs distinguish them.
Table 1: Identifier occurrences and distinct targets in a focal-method m .
Figure 2: Inputs and outputs of the identifier-detection and semantic-recovery tasks. The highlighted subset U(m) contains six occurrences of five distinct targets in T(m) . Semantic recovery receives U(m) .
Table 2: Scope and availability of traditional identifier recovery systems. F, M, and C denote field-, method-, and class-name recovery targets.
Figure 3: Static context levels for identifier recovery.
Obfuscator
Focal-methods
F
M
C
Total
Allatori
58,065
71,508
118,212
73,462
263,182
ProGuard
58,721
74,855
126,875
70,823
272,553
yGuard
59,078
79,233
137,764
62,543
279,540
Table 3: Target counts by obfuscator, with counts of focal-methods inside the retained evaluation cohort of all three evaluated LLMs. F, M, and C mean field, method, and class.
Figure 4: Distribution of field (F), method (M), and class (C) targets per focal-method. The panels use different linear y-scales. Within each bin, bars show Allatori, ProGuard, and yGuard from left to right, with F, M, and C stacked from bottom to top. Bins are computed separately for each identifier type: each segment counts focal-methods whose number of distinct targets of that type falls in the bin. Small-stack annotations report these method counts in F/M/C order. Example: the outlined Allatori stack illustrates 1331 methods with 6–10 field targets, 2900 with 6–10 method targets, and 1520 with 6–10 class targets.
Figure 5: Overall workflow of the LLM evaluation. ➍ depends on the task, ➎ varies with the setting.
Figure 6: Retrieval workflow for the R experiments. The focal project is the repository m originates from.
Figure 7: Agentic workflow for the A experiments.
Figure 8: Inferred obfuscation families by model and context setting. Each stacked bar sums Allatori, ProGuard, and yGuard samples; segments show short-generated (S), high-entropy (H), other opaque (O), no inferred family (N), mixed naming strategies (Mix), and failed responses (Fail).
Figure 9: Mean type-agnostic detection F1 by target count. Scores are focal-method averaged; counts refer to distinct targets in T(m) . Thick curves show seven-bin moving averages. Example: the circled point means DeepSeek scored 65.5% under L1 across 903 evaluated focal-methods containing exactly 16 targets (any type).
Figure 10: Method-macro exact match recovery by identifier type across context settings, averaged with equal weight over evaluated focal-methods containing that type. Panels use different y-scales.
Figure 11: Mean focal-method type-agnostic semantic-recovery BERTScore F1 by distinct target count ∣T(m)∣ or its type-specific subset. Thick curves show seven-bin moving averages. Example: the circled point means DeepSeek scored 53.0% under L1 across 20 evaluated focal-methods containing exactly 32 method-identifier targets.
Figure 12: Identifier-detection quality vs. total token consumption (a) and mean inference time (b) per focal-method for our six settings. For scores and costs, we average over the same pool of focal-methods from Allatori, ProGuard, and yGuard. The connections between setting markers are arbitrary. Example: the circled R point in (b) shows DeepSeek’s mean detection F1 of 66.13% and mean inference time of 6.01 s per focal-method, computed over the same 175,864 methods pooled across Allatori, ProGuard, and yGuard.
Figure 13: Semantic-recovery quality vs. token consumption (a–b) and mean inference time (c–d) per focal-method for our six settings. Lines connect the setting markers arbitrarily. Example: the circled L3 point in (c) shows OLMo’s mean BERTScore F1 of 51.43% and mean inference time of 6.53 s per focal-method. Quality values match Table 5 ; token counts and inference times summarize the saved run logs.
Task
Model
Conditions
Cohort
Recovery
Qwen + Codex
Optional/required retrieval
Stratified 5%
Recovery
Qwen, OLMo
End-to-end A
Stratified 5%
Both
Qwen, OLMo
Allatori I , IS , IC , ISC
100 paired methods each
Table 6: Cohorts used for experiment campaigns of our ablation studies. For the main experiment campaigns, see Table 3 . I =identifier-renaming; S =string encryption; C =control-flow obfuscation.
Harness
Policy
ExM
RL- F1
B F1
Tokens
Time (h)
Task-specific
A
3.13
9.93
48.96
21.90M
18.80
Codex
Optional retrieval
2.55
7.65
46.88
115.74M
27.45
Required retrieval
2.57
7.73
47.09
115.67M
25.79
Table 7: Harness comparison with Qwen on the same stratified 5% cohort (8,771 methods). Scores are method-averaged percentages; tokens and cumulative workflow time include all three obfuscators. Time is summed per-method latency, not campaign wall time.
Figure 14: Paired Allatori factorial comparison over 100 focal-methods per condition. Lines link the results of one model for I (renaming alone), IS (added string encryption), IC (added control-flow obfuscation), and ISC (both transformations). Detection F1 is type-agnostic; semantic-recovery B F1 uses oracle targets and method-macro aggregation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Case
Samples
Total samples
175,864
No verified caller/callee
66,856
At least one caller/callee
109,008
Only callers
45,650
Only callees
26,748
Both callers and callees
36,610
Appendix
Table 8: Availability and composition of caller/callee evidence in L3 . Counts cover the evaluated cohort of 175,864 methods pooled across Allatori, ProGuard, and yGuard.
Metric
Our sample
Niroshan et al.
Valid APKs
1,000
548,967
Classified as obfuscated
563 ( 56.3% )
308,782 ( 56.25% )
Predicted tool among obfuscated APKs
ProGuard
220 ( 39.1% )
40.92%
Allatori
198 ( 35.2% )
36.64%
DashO
34 ( 6.0% )
1.01%
Appendix
Table 9: Comparison of predicted obfuscation prevalence with the large-scale study of Niroshan et al. ( Niroshan et al., 2025 ) .
Figure 15: Semantic-recovery performance conditioned on whether or not the distinct target was detected (both tasks performed under the A setting). μdifference reports the mean difference in B F1 between detected and missed targets, and r is the point-biserial correlation between detection and semantic recovery (0 means no correlation).
Figure 16: Effect of incomplete oracle occurrence inventories drawn from U(m) on semantic recovery (setting A ) for Qwen and OLMo on 8,771 focal-methods. Omitted targets are excluded from the scores: each supplied distinct target is scored once. The horizontal axis reports achieved coverage of T(m) : 38.88%, 58.07%, 85.12%, and 100% (these values are obtained after rounding the nominal 25%, 50%, 75%, and 100% inventories upwards).
Metric
Isobe before
Qwen → Isobe
OLMo → Isobe
Detection
98.89%
99.67% ( ↑ 0.78)
99.78% ( ↑ 0.89)
B F1
50.55%
57.29% ( ↑ 6.74)
54.77% ( ↑ 4.22)
RL- F1
7.62%
19.91% ( ↑ 12.29)
18.33% ( ↑ 10.71)
SemSim
99.74%
99.74% (0.00)
99.74% (0.00)
Appendix
Table 11: We compare Isobe–Tamada alone with Isobe–Tamada preceded by Qwen/OLMo detection and semantic recovery; scores are averages pooled over methods and obfuscators. Parentheses show the change in percentage points relative to Isobe–Tamada alone, calculated from the displayed scores.
When an LLM deobfuscates JavaScript, can poisoned identifier names in the string table survive into the model's reconstructed code, even when the model demonstrably understands the correct semantics? Using Claude Opus 4.6 across 192 inference runs on two code archetypes (force-directed graph simulation, A* pathfinding; 50 conditions, N=3-6), we found three consistent patterns: (1) Poisoned names persisted in every baseline run on both artifacts (physics: 8/8; pathfinding: 5/5). Matched controls showed this extends to terms with zero semantic fit when the string table does not form a coherent alternative domain. (2) Persistence coexisted with correct semantic commentary: in 15/17 runs the model wrote wrong variable names while correctly describing the actual operation in comments. (3) Task framing changed persistence: explicit verification prompts had no effect (12/12 across 4 variants), but reframing from "deobfuscate this" to "write a fresh implementation" reduced propagation from 100% to 0-20% on physics and to 0% on pathfinding, while preserving the checked algorithmic structure. Matched-control experiments showed zero-fit terms persist at the same rate when the replacement table lacks a coherent alternative-domain signal. Per-term variation in earlier domain-gradient experiments is confounded with domain-level coherence and recoverability. These observations are from two archetypes on one model family (Opus 4.6 primary; Haiku 4.5 spot-check). Broader generalization is needed
Code deobfuscation is the task of recovering a readable version of a program while preserving its original behavior. In practice, this often requires days or even months of manual work with complex and expensive analysis tools. In this paper, we explore an alternative approach based on Chain-of-Thought (CoT) prompting, where a large language model is guided through explicit, step-by-step reasoning tailored for code analysis. We focus on control flow obfuscation, including Control Flow Flattening (CFF), Opaque Predicates, and their combination, and we measure both structural recovery of the control flow graph and preservation of program semantics. We evaluate five state-of-the-art large language models and show that CoT prompting significantly improves deobfuscation quality compared with simple prompting. We validate our approach on a diverse set of standard C benchmarks and report results using both structural metrics for control flow graphs and semantic metrics based on output similarity. Among the tested models and by applying CoT, GPT5 achieves the strongest overall performance, with an average gain of about 16% in control-flow graph reconstruction and about 20.5% in semantic preservation across our benchmarks compared to zero-shot prompting. Our results also show that model performance depends not only on the obfuscation level and the chosen obfuscator but also on the intrinsic complexity of the original control flow graph. Collectively, these findings suggest that CoT-guided large language models can serve as effective assistants for code deobfuscation, providing improved code explainability, more faithful control flow graph reconstruction, and better preservation of program behavior while potentially reducing the manual effort needed for reverse engineering.
Seyedreza Mohseni, Sarvesh Baskar, Edward Raff +1
University of Maryland Baltimore County, Baltimore, USA · CrowdStrike, New York, USA
Reverse engineering obfuscated binaries is a core task in malware analysis and software security auditing. While neural decompilers based on Large Language Models (LLMs) have improved the readability and functional fidelity of recovered code over heuristic tools such as Ghidra and IDA Pro, a substantial gap remains between best-sample performance and practical single-output accuracy, particularly under code obfuscation. We study this gap through the lens of candidate generation and candidate selection. Using the LLM4Decompile architecture, we show that ground-truth tokens lie within the model's top-five next-token predictions in 95.6% of positions on average and that multi-sample decoding reaches a pass@100 of 93% on HumanEval-Decompile, indicating that correct reconstructions are typically already present in the model's sampling space. We then evaluate a generate-then-select pipeline that ranks 100 candidates per input with text-, syntax-, embedding-, consensus-, and likelihood-based selectors. Negative perplexity emerges as the strongest static selector on HumanEval-Decompile (73.3% pass@1), while execution-guided selection nearly closes the oracle gap on ExeBench (86.3% pass@1). Combined with one epoch of fine-tuning on our newly released ExeBench-Obfuscated dataset containing 362,647 obfuscated OLLVM samples, the same pipeline increases the pass@1 rate for obfuscated binaries to 88.8%, directly contradicting prior claims that released LLM4Decompile checkpoints are unsuitable for obfuscated inputs.
Felix Mächtle, Gautham Sriram, Manjil Neupane +2
Institute for IT Security, University of Lübeck, Lübeck, Germany