Security in LLM-Assisted Coding: A Systematic Literature Review

Liang Yuan; Maryam Khanian Najafabadi; Mashud Rana; Varvara Vetrova

cs.CROct 11, 2026

Security in LLM-Assisted Coding: A Systematic Literature Review

Authors: Liang Yuan, Maryam Khanian Najafabadi, Mashud Rana, Varvara Vetrova

Organizations: Australian Catholic University, North Sydney, New South Wales, Australia

Abstract

Large Language Models accelerate software development but introduce severe security vulnerabilities. This systematic review investigates security risks in automated coding, benchmark limitations, and the role of human developers. Findings indicate that models frequently generate insecure code and faulty patches due to training data biases. Additionally, the models themselves are susceptible to direct attacks. Current evaluation methods suffer from data contamination and lack robust security metrics. Consequently, human oversight remains essential. Safer integration of automation will require more security-aware model architectures, stronger security-specific benchmarks, and more effective human-AI collaboration frameworks to mitigate these systemic risks.

Figures & tables

Figure 1. A layered framework for analyzing security in LLM-assisted coding.

Survey	Year	Primary scope	Sec	Eval.	Human Factor	Agentic	Gov.	Main gap
Huang et al. ( Huang et al., 2023 )	2023	APR broadly	×	×	×	×	×	Not LLM-security focused.
Anand et al. ( Anand et al., 2024 )	2024	APR+code gen.	$\sim$	$\sim$	×	×	×	Broad overview; limited security synthesis.
Zhou et al. ( Zhou et al., 2024a )	2024	Vulnerability detect./repair	✓	$\sim$	×	×	×	Task-specific rather than integrated.
Zhang et al. ( Zhang et al., 2024 )	2024	LLMs for APR	$\sim$	$\sim$	×	×	×	Repair-focused; narrow security scope.
Basic and Giaretta ( BASIC and GIARETTA, 2025 )	2025	LLMs in code security	✓	$\sim$	×	×	×	Limited on deployment and governance.
Sheng et al. ( Sheng et al., 2025b )	2025	Vulnerability detection	✓	✓	×	×	×	Detection-oriented only.

Table 1. Comparison of closely related surveys and the scope of the present review. ✓ indicates substantial coverage;

\sim

indicates partial coverage; × indicates that the topic is not a central focus.

Category	Specific Threat	Potential Impact	References
Inherent Security Challenges	Trained with risk datasets	Introduction of exploitable flaws into applications.	( Nazzal et al., 2024 ; Mohsin et al., 2024 ; Gong et al., 2024 ; Khoury et al., 2023 ; Pearce et al., 2022 ; Goetz and Schaad, 2024 ; Tihanyi et al., 2025 )
	Lack of security principle understanding	Subtle, hard-to-detect vulnerabilities; insecure design patterns.	( Mohsin et al., 2024 ; Asare and Nagappan, 2023 ; Gong et al., 2024 ; He et al., 2024 ; Hajipour et al., 2024a ; Khoury et al., 2023 ; Tóth et al., 2024 )
	Insecure/incomplete patches	Regression of security posture; false sense of security.	( Belozerov et al., 2025 ; Tihanyi et al., 2025 ; Gong et al., 2024 ; Asare and Nagappan, 2023 ; Liu et al., 2024b )
	Propagation of outdated/vulnerable dependencies	System compromise through known exploits in third-party components.	( Kharma et al., 2025 ; Pearce et al., 2022 ; Sandoval et al., 2023 ; Siddiq et al., 2024 ; Mohsin et al., 2024 )
	Hallucinated APIs / semantic misassumptions	Incorrect security assumptions; broken validation logic; misuse of nonexistent or misunderstood protections.	( Liu et al., 2024a ; Mohsin et al., 2024 ; Tie et al., 2024 )
Human-Related Security Issues	Insecure prompt engineering	Generation of vulnerable code; exposure of sensitive information.	( Goetz and Schaad, 2024 ; Nazzal et al., 2024 ; Elgedawy et al., 2024 ; Chen et al., 2025b ; Li et al., 2024 ; He et al., 2024 )

Table 2. A Taxonomy of LLM Security Issues

Panel of Figure 3 — Figure 3. Illustrative example showing how weak prompt specification can lead to insecure but plausible code, and how a more security-aware reformulation can elicit safer implementation patterns. The prompts were manually written, and the sample code was generated using Gemini 3.1 Pro and subsequently reviewed by the authors for illustrative accuracy.

Vul. Family	Representative CWEs	Representative Works	Recurring Pattern
Input validation and injection	CWE-20, 22, 78, 79, 89, 94	( FU et al., 2024 ; Gong et al., 2024 ; Khoury et al., 2023 ; Liu et al., 2024b ; Nazzal et al., 2024 ; Rabbi et al., 2024 ; Tihanyi et al., 2025 ; Tóth et al., 2024 )	Models often generate functionally plausible code that omits sanitization, boundary checks, or command/query validation.
Memory safety	CWE-119, 120, 125, 416, 476, 787	( Hajipour et al., 2024b ; Li et al., 2024 ; Liu et al., 2024b ; Tihanyi et al., 2024 ; Tihanyi et al., 2025 )	Memory-unsafe languages show recurring low-level errors, especially when prompts do not explicitly encode safety constraints.
Weak crypto and insecure dependencies	CWE-327, 328, 759	( He and Vechev, 2023 ; Kharma et al., 2026 ; Mohsin et al., 2024 ; Pearce et al., 2022 ; Sandoval et al., 2023 ; Siddiq et al., 2024 )	Models reuse outdated algorithms, deprecated APIs, and vulnerable dependencies inherited from training corpora.
Incomplete or insecure repair	CWE-dependent	( Alrashedy et al., 2023 ; Asare and Nagappan, 2023 ; Belozerov et al., 2025 ; Liu et al., 2024b ; Nong et al., 2024 ; Pearce et al., 2023 ; Tihanyi et al., 2025 )	Generated patches may satisfy local tests while leaving root causes unresolved or introducing new flaws.
Data leakage and exposure	CWE-200 and related patterns	( FU et al., 2024 ; Gong et al., 2024 ; Hajipour et al., 2024b ; Hamer et al., 2024 ; He and Vechev, 2023 ; Li et al., 2024 ; Mohsin et al., 2024 ; Nazzal et al., 2024 ; Zeng et al., 2025 ; Zhong and Wang, 2024 )	Insecure handling of secrets, credentials, and sensitive data remains a recurrent failure mode.
Hallucination-related semantic failure	Non-CWE-specific	( Liu et al., 2024a ; Mohsin et al., 2024 ; Tie et al., 2024 )	Fabricated APIs or false assumptions about security behavior can produce plausible yet unsafe code and may propagate across workflows.

Table 3. Recurring vulnerability families in LLM-assisted coding studies.

Framework Family	Representative Works	Primary Oracle	Key Use&limitation
Function-level correctness	( Chen et al., 2021 ; Austin et al., 2021 ; Hendrycks et al., 2021 )	Unit tests	Useful for baseline synthesis ability, but weak in security realism and contextual coverage.
Security prompt benchmarks	( Siddiq et al., 2024 ; Siddiq and Santos, 2022 ; Tony et al., 2023 ; Wang et al., 2024a )	Curated prompts + static rules	Effective for probing known vulnerability classes, but limited in runtime realism and sensitive to analyzer imprecision.
Dynamic/exploit-driven security	( Dilgren et al., 2025 ; Nunez et al., 2024 ; Peng et al., 2025 ; Vero et al., ; Yang et al., 2024b )	Runtime tests, fuzzing, exploit execution	Better reflects exploitability in practice, but depends heavily on test quality, environment fidelity, and setup cost.
Repository/context-aware evaluation	( Li et al., 2025b ; Yu et al., 2024 ; Zheng et al., 2025 )	Project tests + contextual reasoning	Captures integration and cross-file failures, but is harder to standardize and compare across models.
Human-centered/manual evaluation	( Fakhoury et al., 2024 ; Miah and Zhu, 2024 ; Perry et al., 2023 )	Expert review/user study	Reveals review burden, usability, and trust effects, but is costly, subjective, and hard to scale.

Table 4. Representative families of evaluation frameworks for LLM-assisted coding security.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 324 papers

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
Jul 25, 2026Lixun Ma, Ruolong Ma, Bei Wang +4LLM Safety Benchmarks LLM Security
Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1
Security of LLM-generated Code: A Comparative Analysis
May 21, 2026Srivathsan G Morkonda, Mahmoud Selim, Hala AssalSoftware Security LLM Security
Carleton University, Ottawa, ON, Canada
Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
May 28, 2026Alexander Sternfeld, Andrei Kucharavy, Ljiljana DolamicPrompt Sensitivity Adversarial Attacks on LLMs
IEM, HES-SO, Le Foyer, Techno-Pôle 1, Sierre, Switzerland · Cyber-Defence Campus, armasuisse Science and Technology, Thun, Switzerland

Jul 25, 2026J/K move · Enter open · S save

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Lixun Ma, Ruolong Ma, Bei Wang, Feng Wei+3

Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China+1

LLM Safety Benchmarks LLM Security Secure Code Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently ambiguous or incomplete. In this paper, we adopt a developer-centric perspective and identify three representative risk scenarios that commonly lead to security vulnerabilities in LLM-generated code: Ambiguous Requirements, Under-Specified Operational Context, and Security--Functionality Conflict. Based on these scenarios, we construct a large-scale benchmark comprising 2,700 test cases, enabling fine-grained evaluation of LLM security under realistic conditions. Extensive evaluation of eight state-of-the-art LLMs reveals that all models exhibit average vulnerability rates exceeding 56% across risk scenarios. We further demonstrate that security-aware prompting can substantially mitigate these risks, achieving up to 45% improvement.

Abstract

LLM Security

Explore similar work

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Security of LLM-generated Code: A Comparative Analysis

Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs