More Context, Mixed Results: Evaluating LLMs for Identifier Recovery and Code Deobfuscation
Authors: Simon Queyrut, Valerio Schiavoni, Giuseppe Antonio Di Luna, Daniele Cono D'Elia, Leonardo Querzoni, Pascal Felber
Organizations: Computer Science Department, University of Neuchâtel, Neuchâtel, Switzerland. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy.
Source code is often obfuscated through identifier renaming, incentivizing reverse engineers to develop techniques for recovering meaningful names. To assess how LLMs compare with these techniques, we evaluate 4 traditional baselines and 3 LLMs (Qwen, OLMo and DeepSeek) on identifier detection and semantic recovery across 51 recent Java projects, 3 obfuscators, and 6 context/retrieval settings. LLMs generally outperform traditional techniques on both tasks, but benefit from different forms of context and settings: agentic retrieval yields the strongest detection results, reaching 91.56% F1, while enclosing-class context generally improves semantic recovery at a higher token cost. Despite these gains, exact recovery peaks at 16.42% overall (under one-hop context) and 32.93% for field names (under repository retrieval), which are the easiest to recover. Class identifiers are the hardest to detect, with a mean detection F1 of 48.48% across models, settings and obfuscators. More context is not always beneficial: larger methods hinder semantic recovery. Beyond identifier renaming, our experiments show that control-flow obfuscation degrades both tasks more than string encryption. Our findings provide useful ad hoc insights for reverse engineers choosing models and context strategies to deobfuscate code snippets.
Figures & tables
Notation
Meaning
S(m)
All visible identifier occurrences in m .
U(m)⊆S(m)
Identifier occurrences whose project-owned class, method, or field declaration was genuinely obfuscated.
T(m)
The different declarations referred to by U(m) ; private mapping IDs distinguish them.
Table 1: Identifier occurrences and distinct targets in a focal-method m .
Figure 2: Inputs and outputs of the identifier-detection and semantic-recovery tasks. The highlighted subset U(m) contains six occurrences of five distinct targets in T(m) . Semantic recovery receives U(m) .
Table 2: Scope and availability of traditional identifier recovery systems. F, M, and C denote field-, method-, and class-name recovery targets.
Figure 3: Static context levels for identifier recovery.
Obfuscator
Focal-methods
F
M
C
Total
Allatori
58,065
71,508
118,212
73,462
263,182
ProGuard
58,721
74,855
126,875
70,823
272,553
yGuard
59,078
79,233
137,764
62,543
279,540
Table 3: Target counts by obfuscator, with counts of focal-methods inside the retained evaluation cohort of all three evaluated LLMs. F, M, and C mean field, method, and class.
Figure 4: Distribution of field (F), method (M), and class (C) targets per focal-method. The panels use different linear y-scales. Within each bin, bars show Allatori, ProGuard, and yGuard from left to right, with F, M, and C stacked from bottom to top. Bins are computed separately for each identifier type: each segment counts focal-methods whose number of distinct targets of that type falls in the bin. Small-stack annotations report these method counts in F/M/C order. Example: the outlined Allatori stack illustrates 1331 methods with 6–10 field targets, 2900 with 6–10 method targets, and 1520 with 6–10 class targets.
Figure 5: Overall workflow of the LLM evaluation. ➍ depends on the task, ➎ varies with the setting.
Figure 6: Retrieval workflow for the R experiments. The focal project is the repository m originates from.
Figure 7: Agentic workflow for the A experiments.
Figure 8: Inferred obfuscation families by model and context setting. Each stacked bar sums Allatori, ProGuard, and yGuard samples; segments show short-generated (S), high-entropy (H), other opaque (O), no inferred family (N), mixed naming strategies (Mix), and failed responses (Fail).
Figure 9: Mean type-agnostic detection F1 by target count. Scores are focal-method averaged; counts refer to distinct targets in T(m) . Thick curves show seven-bin moving averages. Example: the circled point means DeepSeek scored 65.5% under L1 across 903 evaluated focal-methods containing exactly 16 targets (any type).
Figure 10: Method-macro exact match recovery by identifier type across context settings, averaged with equal weight over evaluated focal-methods containing that type. Panels use different y-scales.
Figure 11: Mean focal-method type-agnostic semantic-recovery BERTScore F1 by distinct target count ∣T(m)∣ or its type-specific subset. Thick curves show seven-bin moving averages. Example: the circled point means DeepSeek scored 53.0% under L1 across 20 evaluated focal-methods containing exactly 32 method-identifier targets.
Figure 12: Identifier-detection quality vs. total token consumption (a) and mean inference time (b) per focal-method for our six settings. For scores and costs, we average over the same pool of focal-methods from Allatori, ProGuard, and yGuard. The connections between setting markers are arbitrary. Example: the circled R point in (b) shows DeepSeek’s mean detection F1 of 66.13% and mean inference time of 6.01 s per focal-method, computed over the same 175,864 methods pooled across Allatori, ProGuard, and yGuard.
Figure 13: Semantic-recovery quality vs. token consumption (a–b) and mean inference time (c–d) per focal-method for our six settings. Lines connect the setting markers arbitrarily. Example: the circled L3 point in (c) shows OLMo’s mean BERTScore F1 of 51.43% and mean inference time of 6.53 s per focal-method. Quality values match Table 5 ; token counts and inference times summarize the saved run logs.
Task
Model
Conditions
Cohort
Recovery
Qwen + Codex
Optional/required retrieval
Stratified 5%
Recovery
Qwen, OLMo
End-to-end A
Stratified 5%
Both
Qwen, OLMo
Allatori I , IS , IC , ISC
100 paired methods each
Table 6: Cohorts used for experiment campaigns of our ablation studies. For the main experiment campaigns, see Table 3 . I =identifier-renaming; S =string encryption; C =control-flow obfuscation.
Harness
Policy
ExM
RL- F1
B F1
Tokens
Time (h)
Task-specific
A
3.13
9.93
48.96
21.90M
18.80
Codex
Optional retrieval
2.55
7.65
46.88
115.74M
27.45
Required retrieval
2.57
7.73
47.09
115.67M
25.79
Table 7: Harness comparison with Qwen on the same stratified 5% cohort (8,771 methods). Scores are method-averaged percentages; tokens and cumulative workflow time include all three obfuscators. Time is summed per-method latency, not campaign wall time.
Figure 14: Paired Allatori factorial comparison over 100 focal-methods per condition. Lines link the results of one model for I (renaming alone), IS (added string encryption), IC (added control-flow obfuscation), and ISC (both transformations). Detection F1 is type-agnostic; semantic-recovery B F1 uses oracle targets and method-macro aggregation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Case
Samples
Total samples
175,864
No verified caller/callee
66,856
At least one caller/callee
109,008
Only callers
45,650
Only callees
26,748
Both callers and callees
36,610
Appendix
Table 8: Availability and composition of caller/callee evidence in L3 . Counts cover the evaluated cohort of 175,864 methods pooled across Allatori, ProGuard, and yGuard.
Metric
Our sample
Niroshan et al.
Valid APKs
1,000
548,967
Classified as obfuscated
563 ( 56.3% )
308,782 ( 56.25% )
Predicted tool among obfuscated APKs
ProGuard
220 ( 39.1% )
40.92%
Allatori
198 ( 35.2% )
36.64%
DashO
34 ( 6.0% )
1.01%
Appendix
Table 9: Comparison of predicted obfuscation prevalence with the large-scale study of Niroshan et al. ( Niroshan et al., 2025 ) .
Figure 15: Semantic-recovery performance conditioned on whether or not the distinct target was detected (both tasks performed under the A setting). μdifference reports the mean difference in B F1 between detected and missed targets, and r is the point-biserial correlation between detection and semantic recovery (0 means no correlation).
Figure 16: Effect of incomplete oracle occurrence inventories drawn from U(m) on semantic recovery (setting A ) for Qwen and OLMo on 8,771 focal-methods. Omitted targets are excluded from the scores: each supplied distinct target is scored once. The horizontal axis reports achieved coverage of T(m) : 38.88%, 58.07%, 85.12%, and 100% (these values are obtained after rounding the nominal 25%, 50%, 75%, and 100% inventories upwards).
Metric
Isobe before
Qwen → Isobe
OLMo → Isobe
Detection
98.89%
99.67% ( ↑ 0.78)
99.78% ( ↑ 0.89)
B F1
50.55%
57.29% ( ↑ 6.74)
54.77% ( ↑ 4.22)
RL- F1
7.62%
19.91% ( ↑ 12.29)
18.33% ( ↑ 10.71)
SemSim
99.74%
99.74% (0.00)
99.74% (0.00)
Appendix
Table 11: We compare Isobe–Tamada alone with Isobe–Tamada preceded by Qwen/OLMo detection and semantic recovery; scores are averages pooled over methods and obfuscators. Parentheses show the change in percentage points relative to Isobe–Tamada alone, calculated from the displayed scores.