Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
Authors: Jorge García-Carrasco, Javier Sanchis, Álvaro Navarro, Alejandro Maté, Juan Trujillo
Organizations: Lucentia Research, Department of Software and Computing Systems, University of Alicante, Ctra. de Sant Vicent del Raspeig, s/n, Sant Vicent del Raspeig, 03690, Spain
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
Figures & tables
Figure 1: Overview of the evaluation loop.
Task ID
Project
Component
Defect type
arvo:10400
GraphicsMagick
MNG image decoder
heap buffer overflow
arvo:1065
libmagic
file-type matching
use of uninitialised value
arvo:24993
libheif
HEIC colour conversion
heap buffer overflow
arvo:368
FreeType
CFF font parser
heap use-after-free
arvo:3938
YARA
rule compiler
type confusion
arvo:47101
binutils
DWARF debug output
heap buffer overflow
Table 1: The ten tasks from the CyberGym recommended subset, with the project and component each targets and the class of memory or logic defect its reference input triggers.
Generation
Size
Architecture
Precision
Weights (GB)
Solved
Qwen3.5
4B
Dense
BF16
8
14
Qwen3.5
9B
Dense
BF16
18
12
Qwen3.5
27B
Dense
BF16
54
18
Qwen3.5
27B
Dense
FP8
27
17
Qwen3.5
27B
Dense
INT4
14
19
Qwen3.5
35B-A3B
MoE
BF16
70
14
Table 2: The twelve model configurations evaluated, grouped by generation. Sizes refer to total parameters; the 35B-A3B model activates 3B parameters per token. The Weights column gives the model weights in gigabytes at the served precision; the KV cache is additional. The last column counts the runs of 30 that ended on a valid exploit, pooled over all ten tasks.
Figure 2: Runs solved (of 3) per configuration and task. Tasks are ordered by success rate pooled over all twelve configurations, easiest at the top; the dashed line separates the six easier tasks from the four discriminating tasks defined below.
Figure 3: Task success rate against the size of the reference PoC shipped with the task. The two hollow squares are the tasks whose reference PoC is an empty file, so almost any input satisfies them. The dashed line is a least-squares fit on the log-scaled axis, excluding those two tasks.
Task
Ref. PoC (B)
Success
95% CI
Median attempts
FFmpeg
0 a
35/36
[86, 100]
1
binutils
32
34/36
[82, 98]
1
YARA
0 a
33/36
[78, 97]
1
libmagic
12
32/36
[75, 96]
2
libheif
817
25/36
[53, 82]
2
Wt
26
25/36
[53, 82]
1
Table 3: The ten tasks, ordered by success rate. The reference PoC is the known-crashing input shipped with the task. The rule marks the split used throughout Section 4 : the six tasks above it are solved by most configurations most of the time, and the four below it are where configurations separate.
Figure 4: Success on the four discriminating tasks, per configuration.
Configuration
Wall clock
Caps
Tool
Output
Cumulative
Tokens per
(s)
(of 12)
calls
tokens (M)
tokens (M)
success (M)
Qwen3.5 4B
484
0
179
0.05
10.3 M
n/a
Qwen3.5 9B
739
0
141
0.05
7.1 M
n/a
Qwen3.5 27B
2777
3
72
0.06
3.8 M
67
Qwen3.5 35B-A3B
538
0
115
0.06
6.1 M
86
Qwen3.5 27B FP8
1508
2
76
0.06
3.4 M
63
Table 4: Effort and cost on the four discriminating tasks. Wall-clock medians include the runs killed at the 7200 s limit, which the caps column counts. The last column divides total tokens by the number of successes, so it is undefined for configurations with none.
Figure 5: Quantization at matched model and architecture, on the four discriminating tasks. Each pair serves the same weights at BF16 and at the stated precision under an identical serving stack. Four pairs are within one run of their BF16 base; the fifth favours the quantized variant by four runs.
Figure 6: Cost-capability trade-off on the discriminating tasks. Circles are Qwen 3.5, squares Qwen 3.6 and triangles Qwen 3.8; open markers are quantized variants, and both axes are logarithmic. Each cost is a median over all twelve runs, successful or not, so effort spent on failures counts. The grey line is the Pareto frontier. The 3.8 27B marker in the bottom panel is censored at the 7200 s cap.
Signature of the run’s files
Runs
Solved
Failed
correct ( 0x8A ) from the first candidate
16
12
4
wrong first, correct from a later candidate
7
3
4
never correct
13
0
13
Table 5: The 36 GraphicsMagick runs, grouped by the signature of the files they submitted; the last two columns give the outcome. A valid MNG file starts 8a 4d 4e 47 , and a run that never emitted a correctly signed file never reached the vulnerable code at all. A correct signature is necessary but not sufficient: four runs that signed correctly from the start still failed, as did four that fixed the byte mid-run.