Implicit Patterns in LLM-Based Binary Analysis

Qiang Li; XiangRui Zhang; Haining Wang

cs.AIMar 19, 2026

Implicit Patterns in LLM-Based Binary Analysis

Authors: Qiang Li, XiangRui Zhang, Haining Wang

Organizations: Beijing Jiaotong University China · Virginia Tech USA

Abstract

Binary vulnerability analysis is increasingly performed by LLM-based agents in an iterative, multi-pass manner, with the model as the core decision-maker. However, how such systems organize exploration over hundreds of reasoning steps remains poorly understood, due to limited context windows and implicit token-level behaviors. We present the first large-scale, trace-level study showing that multi-pass LLM reasoning gives rise to structured, token-level implicit patterns. Analyzing 521 binaries with 99,563 reasoning steps, we identify four dominant patterns: early pruning, path-dependent lock-in, targeted backtracking, and knowledge-guided prioritization that emerge implicitly from reasoning traces. These token-level implicit patterns serve as an abstraction of LLM reasoning: instead of explicit control-flow or predefined heuristics, exploration is organized through implicit decisions regulating path selection, commitment, and revision. Our analysis shows these patterns form a stable, structured system with distinct temporal roles and measurable characteristics. Our results provide the first systematic characterization of LLM-driven binary analysis and a foundation for more reliable analysis systems.

Figures & tables

Figure 1. (Left) One-pass paradigm: static analysis constructs a global program representation and vulnerability reasoning operates over this fixed view. (Right) Iterative paradigm: reasoning interleaves with repeated static analysis operations, proceeding incrementally through multiple tool invocations.

Figure 2. An example of step-by-step LLM-driven binary analysis, illustrating how reasoning states evolve through repeated interactions with external analysis tools.

Figure 3. Dataset overview: the relationship between the binary corpus, analysis sessions, reasoning traces, and output metadata.

Item	Count
Total analysis binaries	521
Architectures	ARM: 328; MIPS: 193
OS	Linux:519; Android:2
Endianness	Little Endian: 477; Big Endian: 44
Format	ELF32: 521
Reasoning sessions	521

Table 1. Dataset and session overview.

Pattern	Core Behavior	Primary Benefit
P1: Early Pruning	Trace exhibits early pruning behavior: candidate paths discarded early, rarely revisited.	Efficient reduction of large search spaces.
P2: Path Lock-in	Trace exhibits path lock-in behavior: reasoning remains focused on a selected path.	Sustained contextual coherence during deep analysis.
P3: Targeted Backtracking	Trace exhibits targeted backtracking behavior: previously deferred paths revisited.	Recovery from incomplete or unproductive analysis paths.
P4: Knowledge-Guided Prioritization	Trace exhibits knowledge-guided prioritization behavior: paths ranked using prior knowledge.	Rapid prioritization of analysis directions.

Table 2. Summary of token-level reasoning patterns in binary analysis.

Figure 4. Pattern 1: pruning behavior in traces—candidate paths discarded early, rarely revisited.

Figure 5. Pattern 2: lock-in behavior in traces—sustained reasoning within the same context, limited exploration of alternatives.

Figure 6. Pattern 3: backtracking behavior in traces—when the active path stalls, reasoning returns to a previously deferred candidate.

Figure 7. Pattern 4: prioritization behavior in traces—paths ranked early using prior knowledge, leading to uneven allocation of effort.

	Sessions w/ Pattern	Coverage	Total	Avg / Session	Avg / Active Session	Max
P1	435 / 521	83.5%	3,339	6.41	7.68	44
P2	509 / 521	97.6%	9,654	18.53	18.99	77
P3	489 / 521	93.8%	995	1.91	2.03	5
P4	509 / 521	97.6%	14,083	27.03	27.71	278

Table 3. Pattern prevalence and density per session.

Total Pattern Instances per Session	# Sessions	Percentage
0	6	1.15%
1-10	41	7.87%
11-30	104	19.96%
31-50	135	25.91%
51-100	188	36.08%
101-200	40	7.68%

Table 4. Distribution of total pattern instances per session.

Figure 8. Pattern roles mapped along two dimensions: reasoning control (X-axis) and activation level measured by average instances per session (Y-axis).

Figure 9. The heatmap showing the distribution of pattern instances across 10 phase bins (P1–P10) of normalized session progress. Each row represents one pattern, and each cell’s color intensity encodes the fraction of pattern instances occurring in that phase. Rows are independently normalized to highlight phase bias rather than absolute frequency.

Rank	Pattern Sequence	Frequency
1	P2 → P1	1,947
2	P1 → P2	1,918
3	P2 → P1 → P2	1,878
4	P1 → P2 → P1	1,547
5	P2 → P1 → P2 → P1	1,508
6	P1 → P2 → P1 → P2	1,488

Table 5. Most frequent subsequences across all reasoning sessions.

Figure 12. Joint distribution of normalized pattern counts: P2 vs P4. Each point represents a session. The strong negative correlation (r = -0.845) indicates complementary resource allocation: high P2 corresponds to low P4, and vice versa.

Pattern	Metric	Mean	Median	Std
P1 (3,339)	Path Length ( $L$ )	19.9	9.0	31.5
	Branching Factor ( $B$ )	1.035	1.000	0.141
	Forward Step Ratio ( $F$ )	0.869	0.947	0.226
	Backtrack Count ( $R$ )	0.175	0.000	0.534
P2 (9,654)	Path Length ( $L$ )	8.9	7.0	6.3
	Branching Factor ( $B$ )	1.051	1.000	0.197

Table 6. Metrics statistics: mean, median, and standard deviation are reported for each metric across all pattern instances.

Pattern			Exploration	Path Selection
	$B$ : Mean	$P$ : Mean	Level	Level
P1	1.035	0.118	Low	High
P2	1.051	0.043	Medium	Low
P3	1.040	0.064	Medium	Medium
P4	1.063	0.078	High	Medium

Table 7. Exploration vs. Path Selection metrics. Branching Factor (

B

) measures exploration breadth, while Pruning Rate (

P

) measures path elimination aggressiveness.

	Diversity	Length	Depth	Fan-out	Cycles	Entropy
	Mean	Mean	Mean	Mean	(%)	Mean
P1	5.96	17.48	2.77	2.65	70.9	0.60
P2	4.49	8.17	2.47	2.10	92.5	0.51
P3	2.68	4.71	1.76	1.22	51.4	0.23
P4	3.29	4.99	2.07	1.59	82.6	0.35

Table 8. Tool usage topology metrics by pattern type.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Command	Immediate Purpose	Context Gained / Used
0	i	Initialize binary metadata	Architecture, bitness, entry points
1	afl	Recover function list	Function boundaries and symbols
2	is~main	Locate main-related symbols	Candidate entry function
3	pdf @ entry0	Inspect entry function	Control-flow of program start
4	pdf @ fcn.0000db2c	Expand to adjacent function	Callee function structure
5	pdf @ fcn.0000b208	Continue function-space exploration	Alternative execution path

Appendix

Table 9. Annotated radare2 command trace (26 steps). Each command corresponds to a localized analysis operation that reveals a specific program property.

Vulnerabilities per Session	Count	Percentage
1 vulnerability	125	63.1%
2 vulnerabilities	45	22.7%
3 vulnerabilities	8	4.04%
4 vulnerabilities	9	4.5%
5 vulnerabilities	5	2.5%
6 vulnerabilities	1	0.5%

Appendix

Table 10. Vulnerability findings distribution across analysis sessions.

Rank	CWE Type	Count	Percentage
1	CWE-78 (OS Command Injection)	115	37.6%
2	CWE-120 (Classic Buffer Overflow)	52	16.7%
3	CWE-121 (Stack-based Buffer Overflow)	37	12.2%
4	CWE-134 (Format String)	23	7.2%
5	CWE-20 (Improper Input Validation)	16	5.3%
6	CWE-22 (Path Traversal)	12	3.8%

Appendix

Table 11. CWE type distribution across all vulnerability findings (Top 10).

LLMs for Cybersecurity

Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.

See all 255 papers

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning
Oct 5, 2026Boyue Caroline Hu, Kaivalya Ahir, Ronghao Ni +1LLM Auditing Software Vulnerability Detection
Carnegie Mellon University, Pittsburgh, Pennsilvanya, USA
Agentic Interpretation: Lattice-Structured Evidence for LLM-Based Program Analysis
May 12, 2026Jacqueline L. Mitchell, Chao WangStatic Code Analysis
University of Southern California, Los Angeles, California, USA
Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection
May 28, 2026Syafiq Al Atiiq, Chun Zhou, Christian GehrmannLLM Interpretability Software Vulnerability Detection
Lund University Lund, Sweden

Oct 5, 2026J/K move · Enter open · S save

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning

Boyue Caroline Hu, Kaivalya Ahir, Ronghao Ni, Limin Jia

Carnegie Mellon University, Pittsburgh, Pennsilvanya, USA

LLM Auditing Software Vulnerability Detection Faithfulness of Language Model Reasoning Structured Reasoning LLMs for Cybersecurity

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed for automated software vulnerability analysis. Binary classification alone is insufficient; practitioners need explanations to triage bugs and engineer patches. Standard practice relies on Chain-of-Thought (CoT) prompting, but free-form reasoning allows models to obscure logical leaps, hallucinated execution steps, and internal inconsistencies behind plausible prose. Our manual audit reveals that approximately 60% of correct vulnerability verdicts are accompanied by fabricated or unverifiable claims, and free-form explanations allow reasoning errors to evade LLM-as-a-judge evaluation. We present Vulnerability Explanation Reasoning Auditor (VERA), an automated framework for auditing LLM vulnerability reasoning. Rather than accepting free-form text, VERA asks models to output a Structured Reasoning Record (SRR) encoding tracked pointers, memory operations, and state transitions in machine-readable fields. A multi-stage judge audits each SRR against eight reasoning failure modes using deterministic checks, with LLM calls reserved for semantic interpretation. The standardized SRR schema also enables automated mutation testing to benchmark judges at scale without human annotation. Our evaluation shows reasoning flaws occur in correct verdicts just as frequently as incorrect ones, and VERA exposes 87% of reasoning errors that free-form LLM-as-judge systematically miss.

Abstract

LLMs for Cybersecurity

Explore similar work

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning

Agentic Interpretation: Lattice-Structured Evidence for LLM-Based Program Analysis

Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection