Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured how much of that difference evaluation protocol accounts for, with model outputs held fixed. In a paired test, a model must flag a vulnerable function and clear its version after a fixing commit. Three choices that published evaluations make differently were varied one at a time: metric, verdict extraction and output budget. Seven frontier and large open models were evaluated on five released pair benchmarks and a set pooled for this work under one protocol, and 61 open models of 1.5B to 36B parameters on the pooled set. Function-level F1 follows how often a model flags both functions of a pair (Spearman +0.86 over 42 combinations) and is nearly unrelated to pair-level correctness (+0.16). On the pair score, extraction changes a model's number by +0.001 at the median and budget by +0.02 with an interval through zero, whereas the model changes a benchmark's number by up to 0.18 and the benchmark a model's by up to 0.16; a function-level score therefore measures flag rate more than model. For 37 of 68 models the difference between correct and reversed pairs is within its 95% interval of zero, the value for a null model that flags each function at a fixed rate, while both-flagged and both-cleared rates exceed that null by 0.055 on median, and for 64 of 68 both functions of a pair receive one answer more often than independence predicts: verdicts are determined by the text common to both functions. On length-matched pairs a linear probe on activations separates 0.78 by within-pair ranking, against 0.64 for a tf-idf baseline and 0.5 for length; the generated verdict is near chance for three of six models and at 0.55 to 0.57 for the other three, and a prompted logit is at chance for all six.
Figures & tables
Choice
Comparison
Change in pair-correct
Basis
Metric
function-level F1 in place of pair-correct
+0.29 to +0.59 (median +0.44 )
42 combinations
Extraction
language-model judge in place of the parse
median +0.001 , range −0.11 to +0.17
68 models, three judges
Budget
32,768 tokens in place of 1,024, gpt-5.2
+0.022 ( −0.022 to +0.067 )
90 pairs
Serving stack
API in place of self-hosted, qwen3-8b
+0.008 ( −0.021 to +0.038 )
1,000 pairs
Model
range over seven models, one benchmark
0.07 to 0.18
six benchmarks
Model
range over 61 open models
0.00 to 0.27
pooled set
Table 1: Effect of each choice on pair-correct, with the effect of the model and of the benchmark for comparison. Intervals are 95% bootstrap intervals over pairs; ranges are over models or benchmarks.
Figure 1: F1 and pair-correct order the same outputs differently. Pair-correct and function-level F1 for the seven API models on the 1,000-pair draw of the pooled set, sorted by pair-correct.
Model
P-C
P-V
P-B
P-R
net score
unparsed or tied
function-level F1
opus-4.8
0.177
0.533
0.271
0.019
0.158
0.000
0.628
gpt-5.2
0.111
0.819
0.051
0.019
0.092
0.000
0.672
grok-4.3
0.254
0.530
0.179
0.037
0.217
0.000
0.667
deepseek-r1
0.173
0.670
0.100
0.047
0.126
0.010
0.661
deepseek-v3.2
0.082
0.674
0.188
0.056
0.026
0.000
0.608
qwen3-coder
0.102
0.411
0.453
0.034
0.068
0.000
0.524
Table 2: The four pair outcomes, the share of pairs unparsed or tied, and function-level F1 on identical outputs, pooled-set draw ( n =1,000 pairs; the two self-hosted rows cover all 5,705). P-C: pair-correct; P-V: both sides flagged; P-B: both sides cleared; P-R: reversed; net score: P-C minus P-R.
Benchmark
opus-4.8
gpt-5.2
grok-4.3
deepseek-r1
deepseek-v3.2
qwen3-coder
qwen3-8b
PrimeVul ( n =200)
0.09
0.08
0.18
0.17
0.07
0.07
0.17
VulnPatchPairs ( n =200)
0.13
0.08
0.15
0.12
0.09
0.12
0.13
PairVul ( n =200)
0.19
0.12
0.18
0.18
0.15
0.06
0.14
JitVul ( n =200)
0.12
0.09
0.17
0.15
0.11
0.10
0.23
SecLLMHolmes ( n =33)
0.15
0.06
0.09
0.12
0.12
0.09
0.24
Ours ( n =1000)
0.18
0.11
0.25
0.17
0.08
0.10
0.21
Table 3: Pair-correct per benchmark and model under the protocol (32,768-token budget, K =3, deterministic parse). n is the number of drawn pairs; bootstrap intervals are about ±0.05 at n =200 and ±0.025 at n =1,000.
Figure 2: The strongest open models match the API models, and most of the population flags or clears nearly every pair. The four pair outcomes on the 1,000-pair draw of the pooled set for the seven API models (blue names), the fifteen open models with a net score above 0.05, and six open models that flag or clear nearly every pair, sorted by net score; the hatched remainder is pairs unparsed or tied. All 68 models are shown in Figure 5 (Appendix F ).
Figure 3: The probe separates the pairs that the prompted logit and the activation oracle do not. Within-pair accuracy (vulnerable side scored above its patch) of the three methods on the same 1,000 pairs for the six models with an activation oracle.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Source of pairs
Pairs
Evaluated
PrimeVul paired test split
security-fix commit, before/after function
435
200
VulnPatchPairs
security-fix commit, before/after function
1,146
200
PairVul
kernel security-fix commit, test split
578
200
JitVul
CVE-linked commit pairs
808
200
SecLLMHolmes
hand-curated real and synthetic pairs
33
33
Pooled set
PrimeVul, ARVO, CVE-fixing commits; two tiers
5,705
1,000; 5,705
Appendix
Table 4: Pair construction and counts. Pairs: pairs in the benchmark after exact-duplicate removal. Evaluated: pairs scored per model; for the pooled set, 1,000 for the panel and all 5,705 for the two full-corpus runs.
Figure 4: Most models are within noise of the null model at every flag rate. Every model on the 1,000-pair draw of the pooled set by flag rate p and net score. Vertical bars are 95% intervals of the net score.
Model
flag rate p
net score
P-V
p2
P-B
(1−p)2
Cohen’s κ
Qwen3.6-27B-abliterated
0.57
+0.219
0.404
0.324
0.265
0.185
0.36
grok-4.3 (API)
0.68
+0.217
0.530
0.456
0.179
0.105
0.37
Qwen3.6-27B
0.66
+0.205
0.519
0.440
0.192
0.113
0.38
Qwen3.6-35B-A3B-abliterated-v4
0.36
+0.191
0.196
0.132
0.469
0.405
0.30
opus-4.8 (API)
0.63
+0.158
0.533
0.398
0.271
0.136
0.59
Qwen3.6-35B-A3B
0.61
+0.155
0.460
0.373
0.238
0.152
0.38
Appendix
Table 5: Every model against the null model that flags each side with the model’s own rate p regardless of the side: predicted both-flagged p2 and both-cleared (1−p)2 next to the observed rates, and the net score, P-C minus P-R; all rates are over parsed pairs. κ : agreement between the two verdicts of a pair beyond chance, with each side’s own flag rate; values in italics are not distinguishable from zero (Fisher’s exact test, p≥0.05 ). Sorted by P-C minus P-R.
Benchmark
opus-4.8
gpt-5.2
grok-4.3
deepseek-r1
deepseek-v3.2
qwen3-coder
qwen3-8b
PrimeVul
0.05 (0.04)
0.07 (0.01)
0.14 (0.04)
0.14 (0.04)
-0.03 (0.10)
0.05 (0.01)
0.06 (0.11)
VulnPatchPairs
0.10 (0.03)
0.02 (0.06)
0.09 (0.07)
0.02 (0.10)
0.01 (0.08)
0.05 (0.07)
0.04 (0.10)
PairVul
0.15 (0.04)
0.05 (0.07)
0.11 (0.07)
0.08 (0.10)
0.07 (0.09)
0.06 (0.01)
0.02 (0.12)
JitVul
0.08 (0.04)
0.07 (0.01)
0.13 (0.04)
0.10 (0.05)
0.07 (0.04)
0.04 (0.05)
0.17 (0.06)
SecLLMHolmes
0.15 (0.00)
0.03 (0.03)
0.00 (0.09)
0.09 (0.03)
0.09 (0.03)
0.03 (0.06)
0.15 (0.09)
Ours
0.16 (0.02)
0.09 (0.02)
0.22 (0.04)
0.13 (0.05)
0.03 (0.06)
0.07 (0.03)
0.09 (0.11)
Appendix
Table 6: Pair-correct minus reversed per benchmark and model, with the reversed rate in parentheses; same outputs as Table 3 .
Benchmark
opus-4.8
gpt-5.2
grok-4.3
deepseek-r1
deepseek-v3.2
qwen3-coder
qwen3-8b
PrimeVul
0.60
0.66
0.61
0.63
0.53
0.43
0.53
VulnPatchPairs
0.54
0.60
0.56
0.56
0.52
0.41
0.55
PairVul
0.56
0.56
0.55
0.56
0.55
0.36
0.48
JitVul
0.61
0.67
0.63
0.64
0.61
0.47
0.59
SecLLMHolmes
0.68
0.65
0.60
0.65
0.63
0.57
0.61
Ours
0.63
0.67
0.67
0.66
0.61
0.52
0.59
Appendix
Table 7: Function-level F1 on the same outputs as Table 3 .
Figure 5: The four outcomes of every model on the 1,000-pair draw of the pooled set, sorted by pair-correct minus reversed; API models in blue; the hatched remainder is pairs unparsed or tied.
Model
family
group
reasoning block
budget (tokens)
no verdict line
Qwen3-8B-Base
qwen3
base
0.00
28672
0.020
DeepSeek-Coder-V2-Lite-Instruct
deepseek
code
0.00
32768
0.000
Qwen2.5-Coder-1.5B-Instruct
qwen2.5
code
0.00
28672
0.000
Qwen2.5-Coder-3B-Instruct
qwen2.5
code
0.00
28672
0.000
Qwen2.5-Coder-7B-Instruct
qwen2.5
code
0.00
28672
0.002
Yi-Coder-9B-Chat
yi
code
0.00
32768
0.006
Appendix
Table 8: The self-hosted models. Reasoning block: share of samples containing a completed reasoning block; gpt-oss-20b returns its reasoning separately from the sample text. Budget: the effective output budget in tokens. No verdict line: share of samples without one.
Model
group
P-C
P-V
P-B
P-R
unparsed or tied
function-level F1
Qwen3-8B-Base
base
0.170
0.375
0.232
0.186
0.037
0.526
DeepSeek-Coder-V2-Lite-Instruct
code
0.121
0.537
0.225
0.117
0.000
0.569
Qwen2.5-Coder-1.5B-Instruct
code
0.004
0.989
0.000
0.007
0.000
0.664
Qwen2.5-Coder-3B-Instruct
code
0.202
0.325
0.262
0.210
0.001
0.512
Qwen2.5-Coder-7B-Instruct
code
0.142
0.172
0.568
0.115
0.003
0.393
Yi-Coder-9B-Chat
code
0.091
0.683
0.123
0.092
0.011
0.610
Appendix
Table 9: Population outcome rates and function-level F1 on the 1,000-pair draw, protocol parse; columns as in Table 2 .
Judge
agreement with the parse
vulnerable
not vulnerable
no answer
pair-correct, judge minus parse
models judged
Llama-3.3-70B (FP8)
0.999 (0.63 to 1.00)
0.44
0.55
0.003
+0.001 (-0.03 to +0.17)
68
gpt-oss-120b
0.968 (0.66 to 1.00)
0.57
0.43
0.002
+0.001 (-0.11 to +0.15)
68
Qwen3-32B
0.999 (0.63 to 1.00)
0.45
0.54
0.002
+0.001 (-0.00 to +0.17)
68
Appendix
Table 10: The open judges. Agreement: median over models of the share of samples with a verdict line on which the judge agrees with the parse (range in parentheses). Vulnerable, not vulnerable, no answer: shares of the judge’s answers on samples without a verdict line. Pair-correct, judge minus parse: median over models of pair-correct under judge extraction minus pair-correct under the parse (range).
Model
verdict, net
verdict, ranking
logit
oracle
probe
Qwen3-1.7B
+0.033
0.516
0.551
0.505
0.805
Qwen3-4B
+0.096
0.548
0.497
0.495
0.816
Qwen3-8B
+0.122
0.561
0.473
0.433
0.816
Qwen3-14B
+0.141
0.571
0.448
0.517
0.816
Llama-3.1-8B-Instruct
+0.003
0.501
0.448
0.452
0.815
gemma-2-9b-it
-0.006
0.497
0.514
0.483
0.808
Appendix
Table 11: Four ways of obtaining a verdict on the same 1,000 pairs. Verdict, net: the net score of the generated verdict under the protocol. Verdict, ranking: the same verdict as within-pair ranking with ties as coin flips, 0.5+ net /2 . Logit, oracle, probe: within-pair ranking (vulnerable side scored above its patch) of the prompted logit, the activation oracle and the linear probe; chance is 0.5, tf-idf on the same folds 0.728.
Model
base, text
adapter, text
adapter, activations
check question
Qwen3-1.7B
1.00
0.99
1.00
+0.023
Qwen3-4B
0.92
0.85
0.87
+0.047
Qwen3-8B
0.98
0.89
0.32
+0.051
Qwen3-14B
0.93
0.88
0.29
+0.013
Llama-3.1-8B-Instruct
0.87
0.80
0.40
+0.004
gemma-2-9b-it
0.90
0.25
0.05
+0.046
Appendix
Table 12: Yes-rate on the vulnerable side when the base model is given the function text, when the oracle adapter is given the text, and when the adapter is given the activations; the last column is the difference in p(yes) on the bounds-check question, patched side minus vulnerable side.
Large language models are increasingly used for vulnerability detection, yet their reliability under different prompt formulations remains uncharacterized. We present PromptAudit, a controlled evaluation framework that isolates prompt effects by fixing the dataset, decoding, and parsing while varying only the prompting strategy. Using five prompting strategies across five open-weight models on 1,000 CVEs (6,074 code samples spanning 16 programming languages), we evaluate accuracy, recall, abstention, coverage, and effective F1. We find that standard chain-of-thought prompting achieves the strongest overall operational performance, while few-shot prompting provides model-dependent benefits that are most pronounced for prompt-sensitive models. In contrast, adaptive chain-of-thought frequently suppresses recall and self-consistency induces excessive abstention, sharply reducing effective performance. These results show that vulnerability detection behavior is jointly determined by the model and the prompt, and that prompt sensitivity is a first-class system property that must be explicitly characterized in evaluation and deployment.
Steffen J. Camarato, Yahya Hmaiti, Mandana Ghadamian +1
University of Central Florida Orlando, Florida, USA
Existing benchmarks for LLM-based vulnerability detection compress model performance into a single metric, which fails to reflect the distinct priorities of different stakeholders. For example, a CISO may emphasize high recall of critical vulnerabilities, an engineering leader may prioritize minimizing false positives, and an AI officer may balance capability against cost. To address this limitation, we introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories. The framework defines five role-specific weighting profiles: CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor. Each profile selects 12 to 16 dimensions with weights summing to 80, yielding a composite Decision Score between 0 and 100. We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects, covering 10 programming languages and 8 OWASP-aligned vulnerability categories. Evaluations are conducted across two settings: Code-in-Prompt (CIP) and Tool-Use (TU). Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model. For instance, Qwen3-Coder achieves an A (76.3) under the Head of Engineering profile but a D (45.2) under the CISO profile, while GPT-5.4 shows a similar disparity. These findings demonstrate that vulnerability detection is inherently a multi-objective problem and that stakeholder-aware evaluation provides insights that single aggregated metrics obscure.
AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.