Found, but Not Fixed : From Software Vulnerability Discovery to Fixes with Iterative LLM Pipelines
Authors: Youssef Gamal, Mohamad Fakih, Halima Bouzidi, Rahul Dharmaji, Gustavo Quiros Araya, Oluwatosin Ogundare, Mst Ayesha Siddika, Mohammad Abdullah Al Faruque
Organizations: University of California, Irvine, Dept. of Electrical Engineering and Computer Science Irvine, CA, USA · Siemens Technology, Princeton, NJ, USA · Jahangirnagar University, Dept. of Computer Science and Engineering Dhaka, Bangladesh
Software vulnerabilities persist as a critical threat despite the deployment of sophisticated detection tools and AI assistants. While detection has advanced, the subsequent code repair phase remains a primary bottleneck; manual intervention is inherently slow, costly, and prone to introducing secondary regressions and issues. In this paper, we investigate whether an iterative, LLM-based pipeline can provide a robust solution for the automated repair of vulnerable functions in real-world code. We propose a generalized framework that leverages the reasoning capabilities of Large Language Models (LLMs) through a multi-stage feedback loop. We benchmark this approach against state-of-the-art models, including GPT-4o, GPT-5.1, and Gemini 3 Flash, demonstrating that such a pipeline can achieve high-quality code repair. To support the broader security community, we provide our evaluation framework, fine-tuned weights, and comprehensive experimental dataset on our website: https://sites.google.com/view/llm4cve
Figures & tables
Figure 1: Bridging the gap: How an iterative repair pipeline prevents exploitation by shortening the time-to-repair for software code.
Figure 2: The software vulnerability detection-and-repair lifecycle. Our iterative approach augments the traditional cycle by shortening the time-to-repair through autonomous feedback loops.
Figure 3: An Illustration of the Low-Rank Adaptation (LoRA) Process.
Figure 4: A visualization of how the LLM4CVE pipeline can automatically fix common software vulnerabilities
Language
CWE
Description
MITRE
Count
Link
C
CWE-125
Out-of-bounds Read
Link
452
CWE-119
Improper Buffer Access
Link
363
CWE-20
Improper Input Validation
Link
289
CWE-787
Out-of-bounds Write
Link
179
CWE-476
NULL Pointer Dereference
Link
176
Table 1: An overview of selected CWEs for the LLM4CVE pipeline
Model
Context Length (Tokens)
GPT-3.5-Turbo
16,385
GPT-4o
128,000
GPT-5.1
400,000
Gemini 3 flash
1,000,000
Llama 3 8B
8,192
Llama 3 70B
8,192
Table 2: Context lengths and parameter counts of selected LLMs.
Model
Iterative Pipeline Configuration
Unguided
Guided
Guided + Feedback
GPT-3.5-Turbo
✓
✓
✓
GPT-4o
✓
✓
✓
GPT-5.1
✓
✓
✓
Gemini 3 flash
✓
✓
✓
Llama 3 8B
✓
✓
✓
Table 3: Models & pipeline configurations used in our study.
Figure 6: CodeBLEU scores for all iterative pipeline configurations and LLMs. The shaded parts highlight the improvement relative to the previous configuration
Vulnerability Patching Technique
Setup Time
Execution Time
Human Intervention
N/A
28 days
Open-Source LLMs+LoRAs
24 hours
10 minutes
GPT LLMs
1 hour
5 minutes
Table 4: Engineer-hours for various vulnerability patching methods.
Pipeline
Model
CodeBLEU score
VulRepair
CodeT5
0.68
Iterative
GPT-3.5-Turbo
0.74
GPT-4o
0.77
GPT-5.1
0.78
Gemini 3 flash
0.82
Table 5: Semantic similarity score with marked C code.
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.
Om Nepal, Sushant Aryal, Oluseyi Olukola +1
School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA
Automated vulnerability repair has emerged as a promising direction to mitigate the growing number of software vulnerabilities. Recent advances in Large Language Models (LLMs) have further accelerated research in automated repair. However, existing frameworks remain largely restricted to memory-related vulnerabilities and locally repairable vulnerability settings, leaving generalization to unseen vulnerability types underexplored. Their evaluations are often limited to a single programming language, and largely rely on proprietary models. In this paper, we propose RAVEN, a scalable, efficient and autonomous framework that integrates an agentic retrieval-augmented generation (RAG) pipeline with controlled iterative repair in a unified framework. The framework utilizes open-source LLMs in a fully locally deployable setting with limited GPU requirements, while building a multi-faceted retrieval pipeline to retrieve historically relevant vulnerability fixes and guide the patch generation. In addition, RAVEN introduces a dedicated Curator Agent that retrieves cross-file dependencies from the target repository, to fix complex vulnerabilities that cannot be addressed using local vulnerable code alone. We evaluate RAVEN on 160 real-world CVE vulnerabilities across diverse vulnerability types, two programming languages, unseen CWE categories, and out-of-distribution settings. RAVEN achieves an overall repair success rate of 83.13%, outperforming all existing state-of-the-art repair frameworks, while also demonstrating strong generalization capabilities and maintaining the repair cost negligible.
AI-assisted development tools generate vulnerable code at significant rates, yet few automated mechanisms exist to detect, enrich, fix, and verify security issues at development velocity, particularly ones that ground remediation in real-world threat context. This paper presents an automated security evaluation pipeline that generates Python code from LLMSecEval prompts, scans for vulnerabilities using CodeQL and Bandit in parallel with an independent Code Validator LLM, enriches the Code Validator findings with MITRE ATT&CK techniques, CWE Observed Examples, and Python best practice guidelines, generates fixes via the Code Generation LLM, and re-scans with CodeQL and Bandit to verify outcomes. Two pipeline configurations were evaluated: Pipeline 1 (P1), using enriched Code Validator findings only, and Pipeline 2 (P2), where it additionally receives the initial CodeQL and Bandit findings. Both configurations were run across four Claude models: Opus 4.8, Sonnet 4.6, Sonnet 5, and Haiku 4.5, producing 80 runs against 26 LLMSecEval prompts covering 9 CWE categories. P1 reduced static analyzer findings across all four models, ranging from -9% (Opus 4.8) to -54% (Sonnet 5). P2 deepened these reductions further, ranging from -29% (Opus 4.8) to -69% (Haiku 4.5), with P2 outperforming P1 for every model. Verdict consistency averaged approximately 81% modal agreement across all configurations, with P2 marginally more stable than P1. Remediation introduced new vulnerabilities in 15-22% of cases: roughly 70% involved a single new finding, and P2 reduced churn for three of four models, with Sonnet 5 as the sole exception. Notably, the best Code Generation LLM (Opus 4.8) was not the best pipeline performer, as Sonnet 4.6 produced the lowest residual findings and highest pass rate after P2 remediation, suggesting that pipeline effectiveness and first-draft security are distinct properties.
Mikhail Surikov
Georgia Institute of Technology Atlanta, Georgia, USA