Security-by-design emphasises integrating security into software design and development. Security pattern catalogues operationalise this principle by prescribing reusable architectural solutions to recurring security problems. However, recovering their implementations from source code remains challenging because security patterns are abstract, span multiple components, and admit many valid implementations. We present SecPatRec, a hybrid approach for automated security pattern recovery from source code repositories. We treat a security pattern as a reference architecture and frame its recovery as a deductive task: determining which prescribed roles are present, grounding them in code entities, and recovering their interaction sequences. SecPatRec pairs a ReAct-style LLM agent equipped with semantic and lexical code search for role detection and grounding with static code property graph analysis for interaction recovery. We curate a human-validated ground-truth dataset of 100 pattern instances covering five security patterns across 93 open-source Java and Python repositories, and introduce a three-level evaluation framework measuring recovery correctness against this ground truth. We evaluate SecPatRec using four LLMs, including proprietary and open-source models. Averaged over all models and pattern instances, SecPatRec achieves role-detection accuracy of 0.84, role-grounding similarity of 0.84, and an interaction-recovery score of 0.66. Compared to a one-shot end-to-end LLM baseline, SecPatRec achieves higher aggregate recovery scores and lower estimated LLM API costs per run, while processing repositories that exceed the baseline's configured input limit. The recovered role mappings and statically derived interaction sequences provide structural and behavioural evidence that can support inspection of security-pattern implementations.
Figures & tables
Fig. 1: The Password-based authentication pattern of the catalogue [ 11 ] , reduced to its roles ri (boxes) and the actions bj between them (arrows), in the order of the pattern’s walkthrough.
Fig. 2: A login handler (left) and the architecture recovered from it for the password-based authentication pattern (right): roles ri , role mapping mi (dotted), pattern actions (dashed) and interaction sequence sj (solid). Boxes in the code give δ(ri) , dashed for external . The handler is simplified from the FastAPI documentation [ 30 ] .
Symbol
Meaning
Pattern specification
P=(R,B)
pattern specification
R , B
prescribed roles; their actions, in order
Codebase
C , E(C)
codebase, and its named code symbols extracted from union of all ASTs
Recovered Architecture
TABLE I: Notation.
Fig. 3: Overview of SecPatRec . The specification P is inserted into the prompt of the LLM, which iteratively searches the codebase C and produces the role mapping M^ and the presence statuses D^ . The CPG of C is built offline with Joern, and the interaction recoverer traverses it from the mapped symbols to produce the interaction sequence S^ .
#Repos
KLOC
#Methods
#Classes
#Tokens
#Files
CPG (MB)
Java
44
3,628.2
179,975
37,478
32.12M
32,628
228.3
Python
49
2,555.4
92,162
17,307
20.89M
10,622
590.2
Total
93
6,183.6
272,137
54,785
53.01M
43,250
818.5
TABLE II: Size of the 93 codebases per language.
Python
Java
Pattern
Pres.
Ext.
Miss.
Int.
Pres.
Ext.
Miss.
Int.
P1
32
17
21
31
33
14
23
39
P2
31
9
0
24
28
12
0
27
P3
33
0
7
19
30
6
4
28
P4
42
0
18
19
31
11
18
25
P5
40
0
0
34
34
6
0
34
TABLE III: Groundtruth roles by presence status and interactions per pattern, for Python and Java (10 codebases each).
Fig. 4: Presence status reported by each LLM against the groundtruth, over the roles of the 100 instances (share of the row, count in brackets). Red frames: a missing role reported as present or external.
Fig. 5: Mean RDA, MLS and IRC against codebase size per LLM: SecPatRec (solid) against the one-shot baseline (dashed), in five size bins of about 20 instances. Instances over the baseline’s input limit score 0 for the baseline.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 6: The ADL code generator on Password-based authentication (P1). Top: the six stages, grey where deterministic and tinted where a model call. Bottom: the artefact each stage writes, as verbatim excerpts (prose re-wrapped, runs elided). Stage 5 reads the page prose as well as the main flow; in its output, a response survives only as a sentence appended to the request it answers (shaded).
Fig. 8: Folding P1’s main flow. Its 16 arrows become the nine interactions b1,…,b9 of Figure 1 : each response (dashed) is folded into the request it answers, and a band spans the interaction from one to the other. The bands nest like a call stack. The System’s reply opens no band, since the Subject never called the System, and stays as b9 .
Fig. 16: RDA and MLS on a small example. (a) RDA: 4 of the 6 roles have the same presence status in the groundtruth and in the recovery; the 4 roles in F are found by both. (b) MLS: for each role in F , the distance d in the containment tree between the groundtruth symbol er (green) and the mapped symbol e^r (red), scored 1−d/2H .
Fig. 17: IRC on a small example, with interactions named by the actions bj of Figure 1 . S^ misses b5 and has an unexpected interaction u . (a) k=4 of the 6 interactions are shared, so J=4/6 . (b) Turning S^ into S takes q=3 neighbour swaps. (c) Of the 6 pairs of shared interactions, c=3 keep their order, so τ=3/6 .
Edge
Joern step
Connects
Label
Control flow
cfgFirst , _cfgOut
a node to the node of the same method that runs next
—
Call
call.callee
a call to the method it runs
call
Reference
identifier._refOut
a name to the variable, parameter or field it refers to
data
Method reference
methodRef.referencedMethod
a function passed as a value (a callback, a lambda, a nested function) to its method
route-registration if passed to a call, else nested-definition
Appendix
TABLE IV: The edges of Joern’s graph that our traversal follows. Joern step is the name of the traversal in Joern’s query language; Label is what our traversal records on an interaction reached through the edge.
Fig. 19: CFG edges: the chain runs from the METHOD node, through each call in the order it runs, to the METHOD_RETURN node. The argument nodes, which come before each call, are left out.
Fig. 20: The three outcomes of linking a call. Dashed orange arrows are CALL edges; the dashed box is a placeholder method, which has no code.
Fig. 21: REF edges: the name name refers to the method’s parameter, and the name tenant to a local variable.
Fig. 22: A method reference: the function tenant.get_tenant , passed as an argument, is a METHOD_REF node with a REF edge to its METHOD .
Fig. 23: Choosing the entry role from the actions B of the Verifiable token-based authentication pattern. Boxes are the pattern’s roles r∈R and numbered arrows its actions b1,…,b6 , in order. The sender of b1 , the Subject, is missing (dashed), so b1 is skipped (grey) and the sender of b2 , the Enforcer, becomes the entry role.
Fig. 25: The worked example. In (b), each box is one method. Grey arrows are control-flow edges; dashed orange arrows are call edges from Joern; the thick teal arrow is an edge we add. The dashed box is Joern’s placeholder for a library function, which has no code. Black circles give the order in which the traversal visits the statements, and the three interactions it records are numbered on the edges that produce them.
Added edge
Leads to
Built from
Label
Import binding
the repository method that the calling file’s import names
Joern’s import nodes
call
Receiver class (Python)
the method on the class the code assigns to the object a call is made on
Joern’s assignment nodes
call
Field access
the field that a read or write refers to
Joern’s field-access node and the object’s class
data
Override dispatch (Java)
the repository’s implementations of a library interface method
Joern’s class hierarchy, method names and parameter counts
dispatch
Framework route (Java)
the repository code that Spring Security passes a login request to
a rule written for that framework
dispatch
Injection provider (Java)
the @Bean method that builds the object a call is made on
Joern’s annotation nodes and return types
dependency-injection
Appendix
TABLE V: Edges we add to Joern’s graph. Label is recorded on every interaction the edge produces, so interactions that rest on an added edge can be told apart from ordinary calls.
Fig. 26: Import binding: Joern’s graph (a) and the edge we add (b).
Fig. 27: Receiver class: Joern’s graph (a) and the edge we add (b).
Fig. 28: Field access: Joern’s graph (a) and the edge we add (b). Unlabelled grey arrows are ARGUMENT edges.
Fig. 29: Override dispatch: Joern’s graph (a) and the edge we add (b).
Fig. 30: Framework route: Joern’s graph (a) and the edge we add (b). The dashed teal box is a rule in the Java profile, not a node of Joern’s graph.
Fig. 31: Injection provider: Joern’s graph (a) and the edge we add (b).
Fig. 32: Parameter default: Joern’s graph (a) and the edge we add (b). The dashed teal box is read from the source file, not from Joern’s graph.
Fig. 33: Annotation interface for P2 Verifiable-token authentication in freqtrade . Top: the interaction order derived from the current mappings. Bottom: every call between the mapped roles, with its code.
Fig. 34: Size of the 93 codebases (49 Python, 44 Java) in o200k_base tokens and in KLOC.
#Roles
Pattern
#Repos
Present
External
Missing
#Interactions
P1 Password-based authentication
10
33
14
23
39
P2 Verifiable-token authentication
10
28
12
0
27
P3 Obscure-token access control
10
30
6
4
28
P4 Session-based access control
10
31
11
18
25
P5 Rate limiting
10
34
6
0
34
Appendix
TABLE VI: Groundtruth per pattern for Java (10 codebases each): roles by presence status, and interactions.
#Roles
Pattern
#Repos
Present
External
Missing
#Interactions
P1 Password-based authentication
10
32
17
21
31
P2 Verifiable-token authentication
10
31
9
0
24
P3 Obscure-token access control
10
33
0
7
19
P4 Session-based access control
10
42
0
18
19
P5 Rate limiting
10
40
0
0
34
Appendix
TABLE VII: Groundtruth per pattern for Python (10 codebases each): roles by presence status, and interactions.
Fig. 35: Mean RDA, MLS and IRC of our pipeline per LLM and language. Bold: mean over the 50 instances; whiskers: lowest and highest pattern mean.
LLM
Highest MLS
MLS =1
GPT-6.1
54
27
DeepSeek
44
20
GLM
37
17
Muse-Spark
43
16
Appendix
TABLE VIII: Instances with the highest MLS (ties count for each LLM) and with MLS =1 , out of 100.
Fig. 36: Tokens and cost per run of our pipeline, per LLM and language. Bold: mean over the 50 runs; line: lowest to highest run. New input excludes prompt-cache hits; cost uses public API prices, also for the self-hosted DeepSeek and GLM.
Fig. 37: Mean RDA, MLS and IRC per pattern for (a) Python and (b) Java: one-shot baseline (hollow) against our pipeline (filled), with the difference beside each pair. Instances over the baseline’s 900K-token input limit score 0 for the baseline; their number is under each pattern.
Fig. 38: Tokens and cost per run of our pipeline (top) and the one-shot baseline (bottom), both languages, drawn as in Figure 36 . The baseline covers the 84 instances under its 900K-token input limit.
Fig. 39: Cost per run against codebase size, log-log: our pipeline (filled, solid fit) against the one-shot baseline (hollow, dashed fit). A slope of 1 means the cost grows in proportion to the size. Dotted: the baseline’s 900K-token input limit.
Fig. 40: IRC of our pipeline by MLS band, over 400 instances (4 LLMs × 100). Left: share of instances with a perfect, partial or zero IRC (the number of perfect ones in the bar) and the band’s mean IRC. Right: missed and unexpected interactions per instance.
RDA
MLS
IRC
LLM
Base
Ours
Base
Ours
Base
Ours
All 100 instances
GPT-6.1
0.74
0.79
0.70
0.83
0.51
0.72
DeepSeek
0.67
0.86
0.65
0.86
0.51
0.61
GLM
0.70
0.86
0.69
0.84
0.51
0.66
Muse-Spark
0.71
0.84
0.69
0.83
0.51
0.64
Appendix
TABLE IX: Mean RDA, MLS and IRC of the one-shot baseline (Base) and our pipeline (Ours) per LLM, over all 100 instances (the 16 over the baseline’s input limit score 0 for the baseline) and over the 84 instances both can run. Bold: the higher of the two.
Accuracy
New input (%)
Output (%)
Steps
Memory (MB)
LLM
Δ RDA
Δ MLS
Δ IRC
Py
Java
Py
Java
Py
Java
Py
Java
No keyword search
GPT-6.1
-0.009
-0.015
-0.001
0
-2
-8
-6
-0.6
-0.5
0
0
DeepSeek
-0.002
-0.002
+0.088
+16
+11
-11
+4
-0.6
+0.8
0
0
GLM
+0.008
+0.001
-0.032
+40
+12
-6
-3
-0.7
-0.5
0
0
Muse-Spark
+0.023
+0.009
+0.032
-33
-30
-7
-5
-0.4
-1.0
0
0
Appendix
TABLE X: Change from our pipeline when one search tool is removed: RDA, MLS and IRC over both languages; new input and output tokens (%), steps and memory (MB) per run and language. Without semantic search, neither the vector DB nor the embedding model is loaded.
Fig. 41: The two enforcers of session-based access control in ncats/gsrs-spring-starter . The Authentication enforcer (left) ends by passing the request to Spring (line 257). Spring then applies the Authorisation enforcer (right) to UserController . No call in the repository joins the two, so the code property graph has no edge between them. Yellow marks each enforcer’s entity, and red marks the two ends of the missing link.
The 100 groundtruth architectures; the 93 codebases and their CPGs; the outputs of every experiment; the five specifications P ; the Claude Code session that drafted the candidate mappings
Sections IV-B and V ; Appendices A-A and D
SecPatRec
https://github.com/anonymousspottedtoad/secpatrec
The role detection and mapping agent, the one-shot baseline, the two ablations and the secpat command that runs them
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.
Yihan Yang, Chenyuan Yang, Zijie Zhao +5
University of Illinois Urbana-Champaign · Quantum AI & Cyber Security Institute, FPT Corporation; work done during an internship at UIUC.
Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. Existing benchmarks do not faithfully evaluate LLMs in real-world bug hunting scenarios because they rely on fuzzing harnesses, target-specific descriptions, or vulnerability-reproduction tasks. We present SEC-bench Pro, a benchmark for measuring agent bug hunting on critical, high-complexity software systems. This work discloses reports with concrete PoC inputs and links fixes into reproducible tasks through a three-phase pipeline for vulnerability collection, environment reconstruction, and oracle-based validation. We instantiate SEC-bench Pro with 183 validated vulnerabilities across V8 and SpiderMonkey, including a V8 subset with more than $1.5 million in cumulative Google Vulnerability Reward Program awards. These instances span memory-safety, sandbox, JIT, and race-condition bugs under browser-grade and runtime-grade execution conditions. Our evaluation shows that coding agents with frontier models remain below 40% success on both evaluated engines. The open-weight Kimi-K2.6 baseline reaches 11.7% on V8, while the strongest frontier configuration reaches 32.0% on V8 and 38.8% on SpiderMonkey. ClaudeCode and Codex solve complementary instance sets, and their two-agent union reaches 37.9% on V8 and 48.8% on SpiderMonkey. SEC-bench Pro provides robust environments for assessing LLM-based security agents and exposes limitations in long-horizon bug hunting tasks.
Large Language Models (LLMs) can generate functionally correct code while introducing exploitable security weaknesses, particularly when developers specify functionality without explicit security requirements. Recent state-of-the-art approaches mitigate this problem using external security knowledge, but largely rely on flat similarity-based retrieval over independent artifacts, making it difficult to infer the relevant vulnerability space and retrieve targeted knowledge when direct matches are unavailable. We propose SEEK, an agentic Graph RAG framework for secure code generation that constructs a security knowledge graph using the Common Weakness Enumeration (CWE) taxonomy as its structural backbone and injects it with curated real-world CVE vulnerability-fix instances. At inference time, a Planner Agent predicts potential CWE weaknesses associated with the developer query, which guide Graph Scoping to restrict retrieval to relevant regions of the CWE taxonomy before deterministic multi-faceted candidate ranking. A Selection Agent subsequently validates the predicted weaknesses and retrieved candidates prior to code generation. Across five open-source LLMs, we evaluate SEEK in an out-of-distribution setting on CodeGuard+ and CyberSecEval, spanning C, C++, and Python. SEEK achieves the highest SecurePass@1 of 77.45% on CodeGuard+ and the highest aggregate ICD pass rate of 79.45% on CyberSecEval, surpassing prior approaches. Moreover, SEEK achieves 85.2% SecurePass@1 across 13 unseen CWE types with no directly linked vulnerability-fix instances, demonstrating strong generalizability, while maintaining acceptable inference latency costs.
Varun Gadey, Markus Tohn, Lichao Wu +2
University of Duisburg-Essen, Germany · University of Bristol, UK · Technical University of Darmstadt, Germany