Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
Authors: Jorge García-Carrasco, Javier Sanchis, Álvaro Navarro, Alejandro Maté, Juan Trujillo
Organizations: Lucentia Research, Department of Software and Computing Systems, University of Alicante, Ctra. de Sant Vicent del Raspeig, s/n, Sant Vicent del Raspeig, 03690, Spain
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
Figures & tables
Figure 1: Overview of the evaluation loop.
Task ID
Project
Component
Defect type
arvo:10400
GraphicsMagick
MNG image decoder
heap buffer overflow
arvo:1065
libmagic
file-type matching
use of uninitialised value
arvo:24993
libheif
HEIC colour conversion
heap buffer overflow
arvo:368
FreeType
CFF font parser
heap use-after-free
arvo:3938
YARA
rule compiler
type confusion
arvo:47101
binutils
DWARF debug output
heap buffer overflow
Table 1: The ten tasks from the CyberGym recommended subset, with the project and component each targets and the class of memory or logic defect its reference input triggers.
Generation
Size
Architecture
Precision
Weights (GB)
Solved
Qwen3.5
4B
Dense
BF16
8
14
Qwen3.5
9B
Dense
BF16
18
12
Qwen3.5
27B
Dense
BF16
54
18
Qwen3.5
27B
Dense
FP8
27
17
Qwen3.5
27B
Dense
INT4
14
19
Qwen3.5
35B-A3B
MoE
BF16
70
14
Table 2: The twelve model configurations evaluated, grouped by generation. Sizes refer to total parameters; the 35B-A3B model activates 3B parameters per token. The Weights column gives the model weights in gigabytes at the served precision; the KV cache is additional. The last column counts the runs of 30 that ended on a valid exploit, pooled over all ten tasks.
Figure 2: Runs solved (of 3) per configuration and task. Tasks are ordered by success rate pooled over all twelve configurations, easiest at the top; the dashed line separates the six easier tasks from the four discriminating tasks defined below.
Figure 3: Task success rate against the size of the reference PoC shipped with the task. The two hollow squares are the tasks whose reference PoC is an empty file, so almost any input satisfies them. The dashed line is a least-squares fit on the log-scaled axis, excluding those two tasks.
Task
Ref. PoC (B)
Success
95% CI
Median attempts
FFmpeg
0 a
35/36
[86, 100]
1
binutils
32
34/36
[82, 98]
1
YARA
0 a
33/36
[78, 97]
1
libmagic
12
32/36
[75, 96]
2
libheif
817
25/36
[53, 82]
2
Wt
26
25/36
[53, 82]
1
Table 3: The ten tasks, ordered by success rate. The reference PoC is the known-crashing input shipped with the task. The rule marks the split used throughout Section 4 : the six tasks above it are solved by most configurations most of the time, and the four below it are where configurations separate.
Figure 4: Success on the four discriminating tasks, per configuration.
Configuration
Wall clock
Caps
Tool
Output
Cumulative
Tokens per
(s)
(of 12)
calls
tokens (M)
tokens (M)
success (M)
Qwen3.5 4B
484
0
179
0.05
10.3 M
n/a
Qwen3.5 9B
739
0
141
0.05
7.1 M
n/a
Qwen3.5 27B
2777
3
72
0.06
3.8 M
67
Qwen3.5 35B-A3B
538
0
115
0.06
6.1 M
86
Qwen3.5 27B FP8
1508
2
76
0.06
3.4 M
63
Table 4: Effort and cost on the four discriminating tasks. Wall-clock medians include the runs killed at the 7200 s limit, which the caps column counts. The last column divides total tokens by the number of successes, so it is undefined for configurations with none.
Figure 5: Quantization at matched model and architecture, on the four discriminating tasks. Each pair serves the same weights at BF16 and at the stated precision under an identical serving stack. Four pairs are within one run of their BF16 base; the fifth favours the quantized variant by four runs.
Figure 6: Cost-capability trade-off on the discriminating tasks. Circles are Qwen 3.5, squares Qwen 3.6 and triangles Qwen 3.8; open markers are quantized variants, and both axes are logarithmic. Each cost is a median over all twelve runs, successful or not, so effort spent on failures counts. The grey line is the Pareto frontier. The 3.8 27B marker in the bottom panel is censored at the 7200 s cap.
Signature of the run’s files
Runs
Solved
Failed
correct ( 0x8A ) from the first candidate
16
12
4
wrong first, correct from a later candidate
7
3
4
never correct
13
0
13
Table 5: The 36 GraphicsMagick runs, grouped by the signature of the files they submitted; the last two columns give the outcome. A valid MNG file starts 8a 4d 4e 47 , and a run that never emitted a correctly signed file never reached the vulnerable code at all. A correct signature is necessary but not sufficient: four runs that signed correctly from the start still failed, as did four that fixed the byte mid-run.
Vulnerability localization is a fundamental step in software security, requiring models to reason over large codebases and iteratively identify vulnerable implementations. We present Antares, a family of compact language models (350M, 1B, and 3B parameters) for agentic vulnerability localization. Based on IBM Granite base models, Antares is trained through a two-stage pipeline that combines supervised fine-tuning on cybersecurity reasoning and repository exploration data with reinforcement learning from verifiable rewards over vulnerable repositories. Across extensive evaluations, Antares-3B approaches GPT-5.5 while outperforming open-weight models over 200x larger in size. The Antares family further enables fast, low-cost local inference, completing a full 500-task evaluation sweep in approximately 15 minutes on a single H100 GPU, corresponding to an amortized evaluation time of under 2 seconds and less than $0.002 per task.
Supriti Vijay, Aman Priyanshu, Didier Chapoteau +8
1Foundation AI–Cisco Systems Inc. · 2Yale University
Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness. We introduce the Vulnerability Localization Benchmark (VLoc Bench), comprising 500 real world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. We evaluate 27 language models and four static-analysis tools under a common agent interface. Repository-scale vulnerability localization remains difficult: the strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model. We further find that stronger localization does not imply reliable behavior after remediation: systems that identify vulnerable files effectively can still report unsupported locations on patched repositories. These results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.
Aman Priyanshu, Supriti Vijay, Kimia Majd +8
1Foundation AI–Cisco Systems Inc. · 3Carnegie Mellon University · †Work done while at Foundation AI +1
Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense.
Nancy Lau, Louis Sloot, Jyoutir Raj +6
UC Santa Cruz · Carnegie Mellon University · Independent +2