cs.CROct 28, 2026
SaveHow Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
Organizations: Snyk
Abstract
We ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file. A finding is credited only if it locates the vulnerability's attacker-controlled source and its dangerous sink, each within two lines. We evaluate nine productized configurations under one blind protocol: the Snyk Code static analyzer, Claude Code's /security-review with three model and effort settings (one with the Snyk MCP server), Codex Security with GPT-5.6 Luna, Terra and Sol, and the DeepSec scanner with two model backends. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart. Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined, and the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report. What an AI engine misses depends on its harness as much as on its model: without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in the classes that policy excludes or restricts (supplementary run on 19 of 20 applications); Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 at line level; DeepSec's model-free file selection kept a file from only 56 of the 108 flows. Coverage and cost were only loosely related. The harness, fixtures, reference sets and all run data are public.
Figures & tables
| Language family | Applications | Vulnerabilities | Cross-file | Same-file |
|---|---|---|---|---|
| Go | 6 | 32 | 25 | 7 |
| JavaScript / TypeScript | 7 | 30 | 10 | 20 |
| Python | 5 | 34 | 21 | 13 |
| Java | 2 | 12 | 6 | 6 |
| Total | 20 | 108 | 62 | 46 |
Table 1: Corpus summary by language family. An application with several languages is counted under its first-listed language; JavaScript and TypeScript are combined.
| Harness | Model | Effort, budget | Prompt | Primary metric |
|---|---|---|---|---|
| Snyk Code CLI 1.1305.1 c | — | — | none | AR recall |
| Claude Code 2.1.251 b /security-review | Opus 5 | xhigh, 30 turns | benchmark system prompt + /security-review | AR recall |
| Claude Code 2.1.251 b /security-review | Sonnet 5 | xhigh, 30 turns | as above | AR recall |
| Claude Code 2.1.251 b /security-review + Snyk MCP ( snyk@latest , 1.1307.0 d ) | Opus 5 | medium, 30 turns | as above + one required snyk_code_scan | AR recall |
| Codex Security 0.1.24 a (plugin 0.1.79 a ) | GPT-5.6 Luna | xhigh, standard scan | built-in | AR recall |
| Codex Security 0.1.24 a | GPT-5.6 Terra | xhigh, standard scan | built-in | AR recall |
Table 2: The nine configurations. Versions marked a are recorded in every run record; b was observed at preflight and in the bundled binary; c was reported by the operator after the run; d is inferred from the unpinned package and the run date. See Appendix B .
| Configuration | Primary ( 2 lines) | Exact line |
|---|---|---|
| Snyk Code | 89 | 85 |
| Claude Code /security-review , Opus 5 xhigh | 85 | 79 |
| Claude Code /security-review , Opus 5 medium + Snyk MCP | 80 | 74 |
| Claude Code /security-review , Sonnet 5 xhigh | 73 | 55 |
| Codex Security, GPT-5.6 Luna | 45 | 15 |
| Codex Security, GPT-5.6 Terra | 35 | 6 |
Table 3: Vulnerabilities matched (of 108) under the primary two-line rule and under exact-line matching.
| Configuration | Macro recall | Matched | 95% interval | Precision |
|---|---|---|---|---|
| Attacker-Reachable Vulnerability Recall | ||||
| Snyk Code | 82.4% | 89 of 108 | 74.0–90.5 | 20.9% (89/425) |
| Claude Code, Opus 5 xhigh | 79.3% | 85 of 108 | 71.6–86.7 | 74.6% (85/114) |
| Claude Code, Opus 5 medium + Snyk MCP | 75.2% | 80 of 108 | 68.2–82.3 | 82.5% (80/97) |
| Claude Code, Sonnet 5 xhigh | 67.0% | 73 of 108 | 57.7–76.2 | 74.5% (73/98) |
| Codex Security, GPT-5.6 Luna | 41.2% | 45 of 108 | 26.6–56.5 | 20.8% (45/216) |
Table 4: Main recall results. All Claude Code configurations run /security-review . Macro recall averages the 20 per-application recalls; the matched count is pooled. The interval comes from resampling applications (Section 2.6 ). Precision is a lower bound because the reference set may omit real vulnerabilities. DeepSec rows use localized recall, a different evidence requirement, and are not ranked with the others.
Figure 1: Main recall results. (A) The seven endpoint-aware configurations: macro Attacker-Reachable Vulnerability Recall with the matched count. The horizontal bar is a 95% interval from resampling the 20 applications; it shows how much the score depends on which applications are in the corpus, not run-to-run variation (Snyk Code’s output was identical across three runs, Section 6 ). The tick on the Claude Code rows marks the score under many-to-one credit, a sensitivity check (Section 6 ); the Codex Security configurations gain at most 3 matches under it and are not marked. (B) DeepSec, scored with localized recall, on its own axis. The top two cannot be separated: they disagree on 28 vulnerabilities, 16 vs 12 (McNemar , descriptive).
Figure 2: The ceiling (endpoint-aware configurations only; DeepSec excluded). (A) Greedy cover: Snyk Code matches 89; Opus 5 xhigh adds the 12 vulnerabilities that it matched and Snyk Code did not, reaching 101, the union of all seven endpoint-aware configurations (102 after the reference-set correction); no further configuration adds any. Opus 5 xhigh also misses 16 vulnerabilities that Snyk Code matched, which the cumulative count does not show. (B) The two sets of disagreements by class: left, matched by Snyk Code and missed by Opus 5 xhigh (16); right, the reverse (12). Hatching marks cross-file flows.
Figure 3: Exact overlap of the matched sets (UpSet plot). Each bar counts the reference vulnerabilities matched by exactly the combination of configurations marked by the dots beneath it; the three Codex Security configurations are pooled. The light bar counts the 7 vulnerabilities no endpoint-aware configuration matched; the dark bar counts the 10 only Snyk Code matched (5 under many-to-one credit). DeepSec is excluded.
Figure 4: Where each engine wins, by flow shape. (A) Recall on the 46 same-file (hollow) and 62 cross-file (filled) vulnerabilities. (B) What each configuration’s same-file misses are: classes the Claude Code review policy excludes or restricts, cross-site scripting, and all other classes. (C) Cross-file cross-site scripting, the one cell where every configuration matched at most 4 of 7. The three panels share their rows: each row is the same configuration in A, B and C.
Figure 5: The triage cost. (A) Findings that matched no reference vulnerability, by category; they are outside the reference set and not proven false. (B) Precision against the reference set (hollow) and over reports in the 14 scored classes only (filled); both are lower bounds. DeepSec, scored differently, is omitted: its Opus 5 backend reported 487 unmatched findings, 449 of them with labels that map to no scored class.
Figure 6: Classes the built-in review policy excludes or restricts. The Claude Code /security-review prompt (version 2.1.251) excludes ReDoS and asks that prototype pollution and open redirect be reported only at extremely high confidence; these classes were chosen from the prompt text, not from the data. (A) Credited recall on the 20 vulnerabilities in those classes, the 7 cross-site scripting vulnerabilities in React and Angular applications (which do not test the corresponding precedent), and the 81 others. (B) Per class: bars show vulnerabilities credited under the two-line rule, circles show vulnerabilities for which a finding of that class was reported anywhere in the application; labels read credited/reported. The dashed outline in both panels is the supplementary prompt-swap control (Opus 5 xhigh with the benchmark’s default audit prompt; 19 applications, single run, 2026-10-01), shown beside the main results and not merged into them.
Figure 7: Codex Security with GPT-5.6 Sol: for each of the 57 missed curated sinks with a reported location in the same file, the offset of the nearest reported location from the sink line (negative means above). Lighter bars are reported locations that are themselves a curated source, for which “above” is expected; among the other 41, 39 lie above the sink, median six lines. The shaded band is the two-line tolerance.
Figure 8: Matches against line tolerance. (A) The seven endpoint-aware configurations under the endpoint rule, from exact line to file level; only the two-line setting is used for the results, the others are counterfactual rescorings of the same findings. Labels give the change from the two-line setting to file level for Codex Security. (B) DeepSec under its own localized rule, on a separate axis.
Figure 9: Where DeepSec 2.3.7 loses the reference set. (A) Of 108 reference vulnerabilities, 56 had a flow file among the scan stage’s candidates (model-free, identical for both backends) and 44 had their sink file there; localized matches follow for each backend. (B) Candidate coverage by language family. Localized recall is DeepSec’s primary metric and is not comparable with Attacker-Reachable Vulnerability Recall.
Figure 10: Coverage against cost and time (endpoint-aware configurations; DeepSec is scored differently and excluded). (A) Macro recall against the mean known cost per application, on a log scale; Snyk Code has no per-scan model cost and is shown in a separate column, not on the dollar axis. Ringed points form the recall-versus-cost frontier; arrows mark the two same-harness contrasts discussed in the text. (B) Macro recall against mean wall time per application, on a log scale. One run per configuration and application; every ratio is a single observation.
Figure 11: Resources per matched reference vulnerability: totals over the 20 applications divided by the number of vulnerabilities matched. Rows are in the same order in every panel. (A) Wall time, log scale. (B) Known dollar cost, log scale; Snyk Code has no per-scan model cost. (C) Tokens, log scale; provider accounting differs, so compare within a family. (D) Reported findings per matched vulnerability, a measure of review volume.
Figure 12: Rank of each endpoint-aware configuration by macro recall under the scoring rule used for all results (outlined row) and 20 alternative rules, grouped as line tolerance, endpoint roles, matching and credit, and location-only or type-only rules. The right column gives Kendall’s against the outlined row. All alternatives rescore the same saved outputs; DeepSec is excluded because it reports no endpoint roles.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Reference vulnerabilities by class (108 in total).
Figure 14: Raw source and sink annotations by language family. Alternative anchors count separately, so a vulnerability can contribute more than one of each.
| Fixture | Family | Frameworks | Vulns | Cross | Same | Svc | Src/snk | Locs | Per vuln |
|---|---|---|---|---|---|---|---|---|---|
| ardanservice | Go | chi, sqlx, goth | 5 | 5 | 0 | 0 | 5/5 | 23 | 4.6 |
| blanq | JS/TS | nuxt, vue, drizzle | 7 | 0 | 7 | 2 | 7/7 | 19 | 2.7 |
| boilwater | JS/TS | express | 4 | 3 | 1 | 0 | 3/4 | 13 | 3.2 |
| coffeeshop | Java | spring-boot, thymeleaf, spring-security | 5 | 2 | 3 | 0 | 6/5 | 15 | 3.0 |
| conduit | Python | flask | 6 | 4 | 2 | 0 | 10/6 | 26 | 4.3 |
| flask-bones | Python | flask | 10 | 4 | 6 | 0 | 10/10 | 31 | 3.1 |
Table 5: Fixture lookup. Language family as in Table 1 . Cross = cross-file flows, same = same-file flows, svc = recorded as crossing a service boundary, src/snk = raw source and sink objects, locs = raw locations, per vuln = raw locations per vulnerability.
| Configuration | Model | Parameters |
|---|---|---|
| Snyk Code | — | snyk code test <fixture> --json |
| Claude Code /security-review | claude-opus-5 | effort xhigh; 30 turns |
| Claude Code /security-review | claude-sonnet-5 | effort xhigh; 30 turns |
| Claude Code /security-review + Snyk MCP | claude-opus-5 | effort medium; 30 turns; one required snyk_code_scan |
| Codex Security | gpt-5.6-luna , gpt-5.6-terra , gpt-5.6-sol | effort xhigh; standard mode |
| DeepSec | claude-opus-5 , gpt-5.6-sol | reasoning xhigh; 150 turns per batch; batch size 5; concurrency 1 |
Table 6: Run parameters as recorded in the configuration snapshot of the main execution.
| Application | Class | Source sink | Boundary crossed | Reported by |
|---|---|---|---|---|
| sassyreg | XSS | registry response field authorName ( PackageProfile.js:134 ) innerHTML in PropsPoints.js:13 | server/stored data browser DOM | 7 (CC 3, CxS 3, DS 1) |
| halloween | Prototype pollution | JSON body of PUT /users/:id/settings/notifications ( userController.ts:78 ) two-level computed property write ( userService.ts:141 ) | through third-party code | 6 (CC 3, CxS 3) |
| saas-starter-kit | SQL injection | order query parameter ( handlers/user.go:69 ) go-sqlbuilder OrderBy ( internal/user/user.go:189 ) | through third-party code | 5 (CC 1, CxS 3, DS 1) |
| coffeeshop | Mass assignment | POST /profile form fields bound to the Person entity ( ProfileController.java:33 ) personRepository.save ( PersonService.java:44 ) | through third-party code | 5 (CC 3, CxS 2) |
| flask-bones | XSS | username at registration ( auth/views.py:73 ) Bootbox dialog message rendered as HTML ( application.js:10 ) | server/stored data browser DOM; through third-party code | 4 (CC 1, CxS 3) |
| iguana | XSS | project name set through the project create and edit forms ( project/forms.py:63,79 ; Project.name , project/models.py:61 ) jQuery .append in draw_activity_heatmap.js:129 (reproduced live); a d3 tooltip leg in draw_last_seven.js was also reported but not reproduced | server/stored data browser DOM; through third-party code | 4 (CxS 3, DS 1) |
Table 7: Adjudicated real and new vulnerabilities outside the reference set that Snyk Code did not report. Reporting configurations: CC = Claude Code /security-review , CxS = Codex Security, DS = DeepSec. Every fix belongs in first-party code.
Figure 15: How many of the seven adjudicated vulnerabilities of Table 7 each configuration reported. Only findings reported by several configurations were adjudicated, so these counts are not rates and are never added to recall.
Figure 16: One-to-one credit, used for all results, against the alternative many-to-one credit, under which one finding may cover several reference vulnerabilities. (A) Vulnerabilities matched, of 108. (B) Same-class groups within an application (18 groups of two or more vulnerabilities) in which every member was matched. Snyk Code’s counts do not change; the two Opus 5 configurations gain the most.
Figure 17: Claude Code /security-review with Opus 5, totals over the 20 applications: medium effort with one required Snyk Code scan through the MCP server (main run), medium effort without it (supplementary arm A, 2026-10-01, dashed outline) and xhigh effort (main run). At equal effort the two medium-effort configurations matched about the same (80 and 81 of 108; they disagree on 5, split 2 to 3). The cost, time and noise differences are between two single runs about a month apart and are not an effect of grounding.
Figure 18: Wall time per application against approximate application size (non-blank source lines), both on log scales, for the seven endpoint-aware configurations. Sizes are approximate and ; the association is descriptive.
Figure 19: For each endpoint-aware configuration, the best location evidence it reported for each of the 108 reference vulnerabilities among findings of the right class, within two lines. A vulnerability is counted once per configuration under its best category, and findings are not consumed one-to-one, so the first category can exceed the matched counts of the main results. The Codex Security configurations most often reported the source without the sink, or the right file at other lines.
Figure 20: Matched reference vulnerabilities per application and endpoint-aware configuration, under the scoring rule used for all results.
Figure 21: Matched reference vulnerabilities per class and endpoint-aware configuration, under the scoring rule used for all results. Several classes have fewer than five vulnerabilities; read those cells as counts, not rates.
More in this topic
Software Vulnerability Detection
Software vulnerability detection identifies insecure code, pinpoints weaknesses, and helps developers validate or prevent exploitable bugs.
See all 146 papers