Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Models

Manit Kaushik; Ishir Bhardwaj; Pranav Gupta; Pankaj Jalote; Arun Balaji Buduru

cs.CRSep 12, 2026

Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Models

Authors: Manit Kaushik, Ishir Bhardwaj, Pranav Gupta, Pankaj Jalote, Arun Balaji Buduru

Organizations: Indraprastha Institute of Information Technology Delhi New Delhi, India

Abstract

JavaScript powers approximately 98.8% of all websites, making vulnerabilities in its code a significant security risk, yet existing detection approaches such as Static Application Security Testing (SAST) tools often fail to identify many real-world vulnerabilities when applied to isolated code snippets. This paper presents an empirical study of Large Language Model (LLM)-based vulnerability identification for JavaScript programs, evaluating three LLM families (Gemini 1.5 Flash, GPT-4o Mini, DeepSeek-R1-Distill-Llama-8B) across multiple prompting strategies (zero-shot, chain-of-thought, few-shot) and fine-tuning approaches on a dataset of 1,125 JavaScript code snippets spanning five Common Weakness Enumeration (CWE) categories: Injection (CWE-74), OS Command Injection (CWE-78), Cross-Site Scripting (CWE-79), SQL Injection (CWE-89), and Uncontrolled Resource Consumption (CWE-400). Our experiments show that LLMs substantially outperform traditional SAST tools on snippet-level vulnerability identification, with a fine-tuned Gemini 1.5 Flash model achieving 60% detection accuracy compared to near-zero performance from rule-based analyzers. We find that fine-tuning improves accuracy from 29% to 60%, Chain-of-Thought prompting benefits reasoning-capable models such as GPT-4o Mini, few-shot prompting is effective for polymorphic vulnerabilities such as Cross-Site Scripting, and performance varies across vulnerability categories, reaching up to 84% accuracy for structured vulnerabilities such as SQL Injection. These results indicate that LLMs provide a practical approach for automated vulnerability identification in JavaScript code, particularly when combined with task-aligned supervision, though they should complement rather than replace existing security analysis workflows due to limited recall and uneven performance across vulnerability types.

Figures & tables

Figure 1: Experiment pipeline for vulnerability identification and patch generation.

Method	Type	Acc.	Prec.	Rec.	F1
Bearer	SAST	0.000	0.000	0.000	0.000
NodeJSScan	SAST	0.008	0.200	0.008	0.020
DeepSeek-R1-8B	LLM	0.080	0.260	0.080	0.120
Gemini 1.5 Flash	LLM	0.290	0.400	0.290	0.260
ChatGPT 4o-mini	LLM	0.340	0.430	0.340	0.310

Table 1: Performance Comparison: SAST Tools vs. Vanilla LLMs

Model	Strategy	Acc.	Prec.	Rec.	F1
Gemini 1.5 Flash	Zero-Shot	0.29	0.40	0.29	0.26
Gemini 1.5 Flash	CoT	0.18	0.43	0.18	0.21
ChatGPT 4o-mini	Zero-Shot	0.34	0.43	0.34	0.31
ChatGPT 4o-mini	CoT	0.35	0.47	0.35	0.32

Table 2: Effect of Prompting Strategies on Vanilla LLMs

LLM	Vanilla		Standard FT		CoT FT
LLM	Acc.	F1	Acc.	F1	Acc.	F1
DeepSeek-R1-8B	0.08	0.12	N/A	N/A	0.27	0.27
Gemini 1.5 Flash	0.29	0.26	0.60	0.60	0.22	0.21
ChatGPT 4o-mini	0.34	0.31	0.49	0.48	0.30	0.31

Table 3: Vanilla vs. Fine-Tuned LLMs (Macro Accuracy and F1)

S. No.	LLM	Method	Accuracy		F1-Score
			Vanilla	Fine-Tuned	Vanilla	Fine-Tuned
CWE 74
1	DeepSeek-R1-Distill-Llama-8B	Zero Shot	0.42	0.53	0.48	0.59
2	DeepSeek-R1-Distill-Llama-8B	Few Shot	0.53	0.53	0.57	0.56
3	Gemini 1.5 Flash	Zero Shot	0.68	0.53	0.67	0.57
4	Gemini 1.5 Flash	Few Shot	0.71	0.58	0.68	0.62

Table 4: Vanilla vs. Fine-Tuned LLMs Across CWE Categories and Prompting Methods

Software Vulnerability Detection

Software vulnerability detection identifies insecure code, pinpoints weaknesses, and helps developers validate or prevent exploitable bugs.

See all 143 papers

Apr 22, 2026

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

The rapidly evolving Node

.

js ecosystem currently includes millions of packages and is a critical part of modern software supply chains, making vulnerability detection of Node

.

js packages increasingly important. However, traditional program analysis struggles in this setting because of dynamic JavaScript features and the large number of package dependencies. Recent advances in large language models (LLMs) and the emerging paradigm of LLM-based agents offer an alternative to handcrafted program models. This raises the question of whether an LLM-centric, tool-augmented approach can effectively detect and confirm taint-style vulnerabilities (e.g., arbitrary command injection) in Node

.

js packages. We implement LLMVD

.

js, a multi-stage agent pipeline to scan code, propose vulnerabilities, generate proof-of-concept exploits, and validate them through lightweight execution oracles; and systematically evaluate its effectiveness in taint-style vulnerability detection and confirmation in Node

.

js packages without dedicated static/dynamic analysis engines for path derivation. For packages from public benchmarks, LLMVD

.

js confirms 84% of the vulnerabilities, compared to less than 22% for prior program analysis tools. It also outperforms a prior LLM-program-analysis hybrid approach while requiring neither vulnerability annotations nor prior vulnerability reports. When evaluated on a set of 260 recently released packages (without vulnerability groundtruth information), traditional tools produce validated exploits for few (

\leq 2

) packages, while LLMVD

.

js generates validated exploits for 36 packages.

Ronghao Ni, Mihai Christodorescu, Limin Jia

Carnegie Mellon University · Google

Automated Exploit Generation Software Vulnerability Detection Software Security LLMs for Cybersecurity

Ask PDF

Jun 19, 2026

Evaluating LLMs for Real-World Web Vulnerability Detection

Large Language Models (LLMs) have emerged as a promising tool for automated vulnerability detection, yet their effectiveness on web-specific vulnerabilities remains to be explored. This work benchmarks six frontier (Claude Opus 4.6, Codex GPT-5.4, Gemini 3.1-pro-preview) and open-weight models (Qwen 3.5, Qwen 3 Coder Next, MiniMax M2.5) on their ability to detect real-world web vulnerabilities using static analysis in WordPress plugins, including SQL injection, stored cross-site scripting, path traversal, and remote code execution. Using five prompt designs of varying structure, scope, and complexity across three experiment iterations, we aim to answer how model and prompt choice affects vulnerability detection. Our results show that all models are capable of detecting valid security issues, but the detection rate varies depending on the model and prompt. For example, Claude Opus 4.6 achieved the highest web vulnerability detection rate (63%), while open-weight MiniMax M2.5 performs on par with other frontier models (48%), and self-hosted Qwen 3.5 only achieved 35%. We show that scoped prompts that narrow the vulnerability scope outperform open-ended ones, whereas the prompt complexity has little impact. Surprisingly, no model achieved full reporting consistency across three experiment iterations, with some as low as 50%. Our experiments demonstrate the opportunities and limits of LLM-based vulnerability detection, as no model correctly identified one baseline vulnerability in one of the plugins. Additionally, we derive practical lessons learned for security practitioners and publish all code and data to support future research.

Sebastian Neef, Luca Jungnickel, Antonio Benjamin Buchholz +2

Technische Universität Berlin, Einsteinufer 17, 10587 Berlin, Germany · Freie Universität Berlin, Kaiserswerther Str. 16-18, 14195 Berlin, Germany

Software Vulnerability Detection LLM Reliability LLMs for Cybersecurity Prompt Engineering

Ask PDF

Jun 14, 2026

Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?

We ran 300 repeated vulnerability-finding scans to measure how repeatable agentic large language model (LLM) security review is on the same JavaScript code, prompt, and benchmark harness. The headline result is that LLM security findings were unevenly repeatable: reference-matched findings were stable, but extra model reports varied heavily from run to run. Across 250 model runs, 80 of 161 unique unmatched findings appeared in only one of five identical repetitions, while only 22 appeared in all five. By contrast, when Claude matched a Snyk Code reference finding, the behavior was much more stable: 134 of 158 unique reference-matched findings appeared in all five repetitions. The benchmark also shows complementarity. Models consistently found familiar, high-signal exploit shapes, and in one case surfaced a likely Snyk Code product gap. Snyk Code static application security testing (SAST) was deterministic and better at systematically enumerating repeated data-flow sinks. The results support combining agentic LLM review with deterministic SAST rather than treating either technique as a replacement for the other.

Liran Tal, Johannes Kloos, Arsenii Rudich +2

Snyk

Software Vulnerability Detection Static Code Analysis AI Agent Security Benchmarks ML Reproducibility LLM Agent Reliability

Ask PDF

Abstract

Software Vulnerability Detection

Explore similar work

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Evaluating LLMs for Real-World Web Vulnerability Detection

Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?