Large language models (LLMs) and agentic frameworks are increasingly used for development tasks such as automated program repair (APR). Prior work has identified security risks in LLM-generated code, but most studies focus on synthetic, simplified, or isolated tasks. We present a large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues from SWE-bench. We evaluate developer-written patches, two standalone LLMs (Llama 3.3 Instruct-70B and GPT-5.4 nano), and four agentic frameworks (OpenHands, AutoCodeRover, HoneyComb, and SWE-agent). We also analyze code, issue, and project-level factors associated with insecure patches. Under our evaluated SWE-bench setup, Llama 3.3 introduced 135 manually confirmed vulnerabilities across the train and test sets, compared with 12 in developer-written patches for the same issues. On the SWE-bench test set, GPT-5.4 nano introduced 3 vulnerabilities compared with 2 in developer patches. All four evaluated agentic frameworks also introduced vulnerabilities. OpenHands had the highest retained count, with 44 vulnerabilities across 11 issue instances. Vulnerabilities are associated with broader code modifications and with issues missing specific information. These results suggest that APR systems require proactive risk assessment methods that consider both generated code and issue context.
Figures & tables
Figure 1. Overview of Our Methodology for LLM-Based Code Generation and Security Evaluation.
Patch Source
Split
Majority Vote
Retained Vulnerabilities
Affected Issue Instances
Llama 3.3
Train + Test
185
135
131
Developers
Train + Test
17
12
12
Llama 3.3
Test
14
9
8
GPT-5.4 nano
Test
6
3
3
Developers
Test
3
2
2
OpenHands
Test
195
44
11
Table 1. Number of retained vulnerabilities and affected issue instances across evaluation splits.
Llama 3.3-Generated Code
CWE-78 (OS Command Injection)
79
CWE-95 (Eval Injection)
39
CWE-502 (Insecure Deserialization)
17
CWE-22 (Path Traversal)
14
Developer-Written Code
CWE-78 (OS Command Injection)
4
Table 2. Most recurring CWE types found in standalone LLM-generated and developer-written patches. The full CWE distribution for the retained RQ1 vulnerabilities is provided in Appendix A .
Source / Metric
Vuln Mean
All Mean
p-value
Cliff’s δ
Llama 3.3
Files Added or Modified (All)
1.72
1.25
< 0.001
0.200
Files Added or Modified (Python)
1.60
1.16
< 0.001
0.199
Unique File Types Modified
1.10
0.99
0.007
0.081
Sensitive Files Modified
0.00
0.007
0.369
-0.006
LOC Generated by LLM (Python)
1740.50
2218.12
< 0.001
-0.197
Table 3. Summary statistics comparing vulnerable vs. all standalone LLM-generated code for code-level metrics.
Metric
OpenHands
AutoCodeRover
HoneyComb
SWE-agent
V
A
V
A
V
A
V
A
Files Added or Modified (All)
344.27 *
11.38
1.33
1.07
1.50
1.46
5.75
3.02
Files Added or Modified (Python)
68.83 *
4.46
1.33
1.07
1.50
1.40
5.25
1.99
Unique File Types Modified
3.73 *
1.41
1.00
0.98
1.00
0.90
1.50
1.08
Sensitive Files Modified
0.0909 *
0.0136
0.00
0.00
0.00
0.004
0.00
0.004
LOC Generated by LLM (Python)
1218.6
1591.0
387.33
1294.6
1591
1764
2104.0
1615.98
Table 4. Mean values for vulnerable (V) and all (A) agentic-generated patches across code-level metrics. * marks OpenHands metrics with statistically significant vulnerable versus all differences. AutoCodeRover, HoneyComb, and SWE-agent are interpreted descriptively because they have 3, 2, and 4 retained vulnerable instances, respectively.
Figure 3. Number of changed Python files per instance across standalone LLMs and agentic APR frameworks, capped at 10.
Bug Type
Llama
OpenHands
AutoCodeRover
HoneyComb
GPT
SWE-agent
Program Anomaly Issue
59
5
1
2
2
2
Configuration Issue
18
1
–
–
–
–
Miscellaneous/Other Issue
8
–
–
–
–
–
Performance Issue
5
–
–
–
–
–
GUI-Related Issue
3
–
–
–
–
–
Test Code-Related Issue
3
–
–
–
–
–
Table 5. Bug type distribution across vulnerable instances.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Source
CWE
Short Name
Count
Llama 3.3, Train+Test
CWE-78
OS Command Injection
79
CWE-95
Eval Injection
39
CWE-502
Insecure Deserialization
17
CWE-22
Path Traversal
14
CWE-939
Improper Authorization in Handler for Custom URL Scheme
8
CWE-732
Permission Misconfiguration
6
Appendix
Table 6. Full CWE distribution for manually retained RQ1 vulnerabilities. Counts are over retained vulnerability rows. Each row can contribute multiple CWE IDs when tools report multiple CWEs.