A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

Maciej Cichoń; Bartłomiej Dmitruk

cs.LGSep 26, 2026

A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

Authors: Maciej Cichoń, Bartłomiej Dmitruk

Organizations: Striga, University of Lodz · Striga

Abstract

Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured how much of that difference evaluation protocol accounts for, with model outputs held fixed. In a paired test, a model must flag a vulnerable function and clear its version after a fixing commit. Three choices that published evaluations make differently were varied one at a time: metric, verdict extraction and output budget. Seven frontier and large open models were evaluated on five released pair benchmarks and a set pooled for this work under one protocol, and 61 open models of 1.5B to 36B parameters on the pooled set. Function-level F1 follows how often a model flags both functions of a pair (Spearman $+0.86$ over 42 combinations) and is nearly unrelated to pair-level correctness ( $+0.16$ ). On the pair score, extraction changes a model's number by $+0.001$ at the median and budget by $+0.02$ with an interval through zero, whereas the model changes a benchmark's number by up to 0.18 and the benchmark a model's by up to 0.16; a function-level score therefore measures flag rate more than model. For 37 of 68 models the difference between correct and reversed pairs is within its 95% interval of zero, the value for a null model that flags each function at a fixed rate, while both-flagged and both-cleared rates exceed that null by 0.055 on median, and for 64 of 68 both functions of a pair receive one answer more often than independence predicts: verdicts are determined by the text common to both functions. On length-matched pairs a linear probe on activations separates 0.78 by within-pair ranking, against 0.64 for a tf-idf baseline and 0.5 for length; the generated verdict is near chance for three of six models and at 0.55 to 0.57 for the other three, and a prompted logit is at chance for all six.

Figures & tables

Choice	Comparison	Change in pair-correct	Basis
Metric	function-level F1 in place of pair-correct	$+0.29$ to $+0.59$ (median $+0.44$ )	42 combinations
Extraction	language-model judge in place of the parse	median $+0.001$ , range $-0.11$ to $+0.17$	68 models, three judges
Budget	32,768 tokens in place of 1,024, gpt-5.2	$+0.022$ ( $-0.022$ to $+0.067$ )	90 pairs
Serving stack	API in place of self-hosted, qwen3-8b	$+0.008$ ( $-0.021$ to $+0.038$ )	1,000 pairs
Model	range over seven models, one benchmark	0.07 to 0.18	six benchmarks
Model	range over 61 open models	0.00 to 0.27	pooled set

Table 1: Effect of each choice on pair-correct, with the effect of the model and of the benchmark for comparison. Intervals are 95% bootstrap intervals over pairs; ranges are over models or benchmarks.

Figure 1: F1 and pair-correct order the same outputs differently. Pair-correct and function-level F1 for the seven API models on the 1,000-pair draw of the pooled set, sorted by pair-correct.

Model	P-C	P-V	P-B	P-R	net score	unparsed or tied	function-level F1
opus-4.8	0.177	0.533	0.271	0.019	0.158	0.000	0.628
gpt-5.2	0.111	0.819	0.051	0.019	0.092	0.000	0.672
grok-4.3	0.254	0.530	0.179	0.037	0.217	0.000	0.667
deepseek-r1	0.173	0.670	0.100	0.047	0.126	0.010	0.661
deepseek-v3.2	0.082	0.674	0.188	0.056	0.026	0.000	0.608
qwen3-coder	0.102	0.411	0.453	0.034	0.068	0.000	0.524

Table 2: The four pair outcomes, the share of pairs unparsed or tied, and function-level F1 on identical outputs, pooled-set draw (

n

=1,000 pairs; the two self-hosted rows cover all 5,705). P-C: pair-correct; P-V: both sides flagged; P-B: both sides cleared; P-R: reversed; net score: P-C minus P-R.

Benchmark	opus-4.8	gpt-5.2	grok-4.3	deepseek-r1	deepseek-v3.2	qwen3-coder	qwen3-8b
PrimeVul ( $n$ =200)	0.09	0.08	0.18	0.17	0.07	0.07	0.17
VulnPatchPairs ( $n$ =200)	0.13	0.08	0.15	0.12	0.09	0.12	0.13
PairVul ( $n$ =200)	0.19	0.12	0.18	0.18	0.15	0.06	0.14
JitVul ( $n$ =200)	0.12	0.09	0.17	0.15	0.11	0.10	0.23
SecLLMHolmes ( $n$ =33)	0.15	0.06	0.09	0.12	0.12	0.09	0.24
Ours ( $n$ =1000)	0.18	0.11	0.25	0.17	0.08	0.10	0.21

Table 3: Pair-correct per benchmark and model under the protocol (32,768-token budget,

K

=3, deterministic parse).

n

is the number of drawn pairs; bootstrap intervals are about

\pm 0.05

n

=200 and

\pm 0.025

n

=1,000.

Figure 2: The strongest open models match the API models, and most of the population flags or clears nearly every pair. The four pair outcomes on the 1,000-pair draw of the pooled set for the seven API models (blue names), the fifteen open models with a net score above 0.05, and six open models that flag or clear nearly every pair, sorted by net score; the hatched remainder is pairs unparsed or tied. All 68 models are shown in Figure 5 (Appendix F ).

Figure 3: The probe separates the pairs that the prompted logit and the activation oracle do not. Within-pair accuracy (vulnerable side scored above its patch) of the three methods on the same 1,000 pairs for the six models with an activation oracle.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Source of pairs	Pairs	Evaluated
PrimeVul paired test split	security-fix commit, before/after function	435	200
VulnPatchPairs	security-fix commit, before/after function	1,146	200
PairVul	kernel security-fix commit, test split	578	200
JitVul	CVE-linked commit pairs	808	200
SecLLMHolmes	hand-curated real and synthetic pairs	33	33
Pooled set	PrimeVul, ARVO, CVE-fixing commits; two tiers	5,705	1,000; 5,705

Appendix

Table 4: Pair construction and counts. Pairs: pairs in the benchmark after exact-duplicate removal. Evaluated: pairs scored per model; for the pooled set, 1,000 for the panel and all 5,705 for the two full-corpus runs.

Figure 4: Most models are within noise of the null model at every flag rate. Every model on the 1,000-pair draw of the pooled set by flag rate

p

and net score. Vertical bars are 95% intervals of the net score.

Model	flag rate $p$	net score	P-V	$p^{2}$	P-B	$(1-p)^{2}$	Cohen’s $\kappa$
Qwen3.6-27B-abliterated	0.57	+0.219	0.404	0.324	0.265	0.185	0.36
grok-4.3 (API)	0.68	+0.217	0.530	0.456	0.179	0.105	0.37
Qwen3.6-27B	0.66	+0.205	0.519	0.440	0.192	0.113	0.38
Qwen3.6-35B-A3B-abliterated-v4	0.36	+0.191	0.196	0.132	0.469	0.405	0.30
opus-4.8 (API)	0.63	+0.158	0.533	0.398	0.271	0.136	0.59
Qwen3.6-35B-A3B	0.61	+0.155	0.460	0.373	0.238	0.152	0.38

Appendix

Table 5: Every model against the null model that flags each side with the model’s own rate

p

regardless of the side: predicted both-flagged

p^{2}

and both-cleared

(1-p)^{2}

next to the observed rates, and the net score, P-C minus P-R; all rates are over parsed pairs.

\kappa

: agreement between the two verdicts of a pair beyond chance, with each side’s own flag rate; values in italics are not distinguishable from zero (Fisher’s exact test,

p\geq 0.05

). Sorted by P-C minus P-R.

Benchmark	opus-4.8	gpt-5.2	grok-4.3	deepseek-r1	deepseek-v3.2	qwen3-coder	qwen3-8b
PrimeVul	0.05 (0.04)	0.07 (0.01)	0.14 (0.04)	0.14 (0.04)	-0.03 (0.10)	0.05 (0.01)	0.06 (0.11)
VulnPatchPairs	0.10 (0.03)	0.02 (0.06)	0.09 (0.07)	0.02 (0.10)	0.01 (0.08)	0.05 (0.07)	0.04 (0.10)
PairVul	0.15 (0.04)	0.05 (0.07)	0.11 (0.07)	0.08 (0.10)	0.07 (0.09)	0.06 (0.01)	0.02 (0.12)
JitVul	0.08 (0.04)	0.07 (0.01)	0.13 (0.04)	0.10 (0.05)	0.07 (0.04)	0.04 (0.05)	0.17 (0.06)
SecLLMHolmes	0.15 (0.00)	0.03 (0.03)	0.00 (0.09)	0.09 (0.03)	0.09 (0.03)	0.03 (0.06)	0.15 (0.09)
Ours	0.16 (0.02)	0.09 (0.02)	0.22 (0.04)	0.13 (0.05)	0.03 (0.06)	0.07 (0.03)	0.09 (0.11)

Appendix

Table 6: Pair-correct minus reversed per benchmark and model, with the reversed rate in parentheses; same outputs as Table 3 .

Benchmark	opus-4.8	gpt-5.2	grok-4.3	deepseek-r1	deepseek-v3.2	qwen3-coder	qwen3-8b
PrimeVul	0.60	0.66	0.61	0.63	0.53	0.43	0.53
VulnPatchPairs	0.54	0.60	0.56	0.56	0.52	0.41	0.55
PairVul	0.56	0.56	0.55	0.56	0.55	0.36	0.48
JitVul	0.61	0.67	0.63	0.64	0.61	0.47	0.59
SecLLMHolmes	0.68	0.65	0.60	0.65	0.63	0.57	0.61
Ours	0.63	0.67	0.67	0.66	0.61	0.52	0.59

Appendix

Table 7: Function-level F1 on the same outputs as Table 3 .

Figure 5: The four outcomes of every model on the 1,000-pair draw of the pooled set, sorted by pair-correct minus reversed; API models in blue; the hatched remainder is pairs unparsed or tied.

Model	family	group	budget (tokens)	no verdict line
Qwen3-8B-Base	qwen3	base	28672	0.020
DeepSeek-Coder-V2-Lite-Instruct	deepseek	code	32768	0.000
Qwen2.5-Coder-1.5B-Instruct	qwen2.5	code	28672	0.000
Qwen2.5-Coder-3B-Instruct	qwen2.5	code	28672	0.000
Qwen2.5-Coder-7B-Instruct	qwen2.5	code	28672	0.002
Yi-Coder-9B-Chat	yi	code	32768	0.006

Appendix

Table 8: The self-hosted models. Reasoning block: share of samples containing a completed reasoning block; gpt-oss-20b returns its reasoning separately from the sample text. Budget: the effective output budget in tokens. No verdict line: share of samples without one.

Model	group	P-C	P-V	P-B	P-R	unparsed or tied	function-level F1
Qwen3-8B-Base	base	0.170	0.375	0.232	0.186	0.037	0.526
DeepSeek-Coder-V2-Lite-Instruct	code	0.121	0.537	0.225	0.117	0.000	0.569
Qwen2.5-Coder-1.5B-Instruct	code	0.004	0.989	0.000	0.007	0.000	0.664
Qwen2.5-Coder-3B-Instruct	code	0.202	0.325	0.262	0.210	0.001	0.512
Qwen2.5-Coder-7B-Instruct	code	0.142	0.172	0.568	0.115	0.003	0.393
Yi-Coder-9B-Chat	code	0.091	0.683	0.123	0.092	0.011	0.610

Appendix

Table 9: Population outcome rates and function-level F1 on the 1,000-pair draw, protocol parse; columns as in Table 2 .

Judge	agreement with the parse	vulnerable	not vulnerable	no answer	pair-correct, judge minus parse	models judged
Llama-3.3-70B (FP8)	0.999 (0.63 to 1.00)	0.44	0.55	0.003	+0.001 (-0.03 to +0.17)	68
gpt-oss-120b	0.968 (0.66 to 1.00)	0.57	0.43	0.002	+0.001 (-0.11 to +0.15)	68
Qwen3-32B	0.999 (0.63 to 1.00)	0.45	0.54	0.002	+0.001 (-0.00 to +0.17)	68

Appendix

Table 10: The open judges. Agreement: median over models of the share of samples with a verdict line on which the judge agrees with the parse (range in parentheses). Vulnerable, not vulnerable, no answer: shares of the judge’s answers on samples without a verdict line. Pair-correct, judge minus parse: median over models of pair-correct under judge extraction minus pair-correct under the parse (range).

Model	verdict, net	verdict, ranking	logit	oracle	probe
Qwen3-1.7B	+0.033	0.516	0.551	0.505	0.805
Qwen3-4B	+0.096	0.548	0.497	0.495	0.816
Qwen3-8B	+0.122	0.561	0.473	0.433	0.816
Qwen3-14B	+0.141	0.571	0.448	0.517	0.816
Llama-3.1-8B-Instruct	+0.003	0.501	0.448	0.452	0.815
gemma-2-9b-it	-0.006	0.497	0.514	0.483	0.808

Appendix

Table 11: Four ways of obtaining a verdict on the same 1,000 pairs. Verdict, net: the net score of the generated verdict under the protocol. Verdict, ranking: the same verdict as within-pair ranking with ties as coin flips,

0.5+

net

/2

. Logit, oracle, probe: within-pair ranking (vulnerable side scored above its patch) of the prompted logit, the activation oracle and the linear probe; chance is 0.5, tf-idf on the same folds 0.728.

Model	base, text	adapter, text	adapter, activations	check question
Qwen3-1.7B	1.00	0.99	1.00	+0.023
Qwen3-4B	0.92	0.85	0.87	+0.047
Qwen3-8B	0.98	0.89	0.32	+0.051
Qwen3-14B	0.93	0.88	0.29	+0.013
Llama-3.1-8B-Instruct	0.87	0.80	0.40	+0.004
gemma-2-9b-it	0.90	0.25	0.05	+0.046

Appendix

Table 12: Yes-rate on the vulnerable side when the base model is given the function text, when the oracle adapter is given the text, and when the adapter is given the activations; the last column is the difference in p(yes) on the bounds-check question, patched side minus vulnerable side.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection
May 22, 2026Steffen J. Camarato, Yahya Hmaiti, Mandana Ghadamian +1Prompt Sensitivity LLM Auditing
University of Central Florida Orlando, Florida, USA
Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection
Apr 2, 2026Subho Halder, Siddharth Saxena, Kashinath Kadaba Shrish +1LLMs for Cybersecurity Software Vulnerability Detection
Mattersec Labs · Kalmantic Labs
Activation Probes Surface Code-Security Signals that the Model's Output Misses
Aug 10, 2026Ivan WiryadiLLM Auditing Software Vulnerability Detection
Independent.

May 22, 2026J/K move · Enter open · S save

PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection

Steffen J. Camarato, Yahya Hmaiti, Mandana Ghadamian, David Mohaisen

University of Central Florida Orlando, Florida, USA

Prompt Sensitivity LLM Auditing Software Vulnerability Detection Prompt Engineering

Open paper AskPDF arXiv

Large language models are increasingly used for vulnerability detection, yet their reliability under different prompt formulations remains uncharacterized. We present PromptAudit, a controlled evaluation framework that isolates prompt effects by fixing the dataset, decoding, and parsing while varying only the prompting strategy. Using five prompting strategies across five open-weight models on 1,000 CVEs (6,074 code samples spanning 16 programming languages), we evaluate accuracy, recall, abstention, coverage, and effective F1. We find that standard chain-of-thought prompting achieves the strongest overall operational performance, while few-shot prompting provides model-dependent benefits that are most pronounced for prompt-sensitive models. In contrast, adaptive chain-of-thought frequently suppresses recall and self-consistency induces excessive abstention, sharply reducing effective performance. These results show that vulnerability detection behavior is jointly determined by the model and the prompt, and that prompt sensitivity is a first-class system property that must be explicitly characterized in evaluation and deployment.

Abstract

LLM Evaluation

Explore similar work

PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection

Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection

Activation Probes Surface Code-Security Signals that the Model's Output Misses