cs.CRApr 2, 2026
SaveSeclens: Role-specific Evaluation of LLM's for security vulnerablity detection
Organizations: Mattersec Labs · Kalmantic Labs
Abstract
Existing benchmarks for LLM-based vulnerability detection compress model performance into a single metric, which fails to reflect the distinct priorities of different stakeholders. For example, a CISO may emphasize high recall of critical vulnerabilities, an engineering leader may prioritize minimizing false positives, and an AI officer may balance capability against cost. To address this limitation, we introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories. The framework defines five role-specific weighting profiles: CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor. Each profile selects 12 to 16 dimensions with weights summing to 80, yielding a composite Decision Score between 0 and 100. We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects, covering 10 programming languages and 8 OWASP-aligned vulnerability categories. Evaluations are conducted across two settings: Code-in-Prompt (CIP) and Tool-Use (TU). Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model. For instance, Qwen3-Coder achieves an A (76.3) under the Head of Engineering profile but a D (45.2) under the CISO profile, while GPT-5.4 shows a similar disparity. These findings demonstrate that vulnerability detection is inherently a multi-objective problem and that stakeholder-aware evaluation provides insights that single aggregated metrics obscure.
Figures & tables
| Role | Core Decision Question |
|---|---|
| CISO | “Can I trust this model in my security program?” |
| CAIO / Head of AI | “Which model unlocks new capabilities while balancing risk and cost?” |
| Security Researcher | “Does this model genuinely understand vulnerability mechanics?” |
| Head of Engineering | “Will this help or hurt my team’s velocity and code quality?” |
| AI as Actor | “Does the agent know what it can and can’t do?” |
Table 1: Stakeholder roles and their core evaluation questions.
Figure 1: Category-level weight distribution for the five stakeholder roles. Each axis represents one of the 7 measurement categories (weights sum to 80 per role).
| ID | Category | Dimensions |
|---|---|---|
| A | Detection | D1–D8 |
| B | Coverage & Consistency | D9–D13 |
| C | Reasoning & Evidence | D14–D17 |
| D | Operational Efficiency | D18–D23 |
| E | Tool-Use & Navigation | D24–D27 |
| F | Risk & Severity | D28–D30 |
Table 2: Seven measurement categories and their constituent dimensions.
| Category | CISO | CAIO | Researcher | Head Eng. | AI Actor |
|---|---|---|---|---|---|
| A: Detection | 34 | 16 | 39 | 35 | 17 |
| B: Coverage & Consistency | 15 | 4 | 16 | 7 | 7 |
| C: Reasoning & Evidence | 4 | 1 | 21 | 0 | 2 |
| D: Operational Efficiency | 2 | 19 | 0 | 22 | 0 |
| E: Tool-Use & Navigation | 0 | 15 | 0 | 0 | 18 |
| F: Risk & Severity | 18 | 5 | 0 | 0 | 0 |
Table 3: Category-level weight distribution across roles (weights sum to 80 per role).
| ID | Name | Category | Description | Norm. |
| D1 | MCC | Detection | Matthews Correlation Coefficient [ 26 , 3 ] ; balanced metric robust to class imbalance | M |
| D2 | Recall | Detection | True positive rate; fraction of vulnerable code correctly flagged | R |
| D3 | Precision | Detection | Positive predictive value; fraction of “vulnerable” verdicts that are correct | R |
| D4 | F1 | Detection | Harmonic mean of precision and recall | R |
| D5 | True Negative Rate | Detection | Specificity; fraction of non-vulnerable code correctly cleared | R |
| D6 | CWE Accuracy | Detection | Correct CWE-ID among true positive detections | R |
Table 4: Master dimension catalog (35 dimensions). Strategy: R = Ratio, M = MCC, L = Lower-is-better, H = Higher-is-better.
| Dim | Name | Wt |
|---|---|---|
| D1 | MCC | 10 |
| D2 | Recall | 8 |
| D3 | Precision | 6 |
| D5 | True Negative Rate | 2 |
| D6 | CWE Accuracy | 5 |
| D8 | Actionable Finding Rate | 5 |
Table 5: CISO weight profile (16 dimensions, ).
| Dim | Name | Wt |
|---|---|---|
| D1 | MCC | 9 |
| D4 | F1 | 7 |
| D9 | CWE Coverage Breadth | 4 |
| D15 | Reasoning Presence | 1 |
| D18 | Cost per Task | 5 |
| D20 | MCC per Dollar | 8 |
Table 6: CAIO weight profile (14 dimensions, ).
| Dim | Name | Wt |
|---|---|---|
| D1 | MCC | 8 |
| D2 | Recall | 6 |
| D6 | CWE Accuracy | 12 |
| D7 | Mean Location IoU | 10 |
| D8 | Actionable Finding Rate | 3 |
| D9 | CWE Coverage Breadth | 7 |
Table 7: Security Researcher weight profile (13 dimensions, ).
| Dim | Name | Wt |
|---|---|---|
| D2 | Recall | 5 |
| D3 | Precision | 12 |
| D5 | True Negative Rate | 4 |
| D7 | Mean Location IoU | 8 |
| D8 | Actionable Finding Rate | 10 |
| D12 | Worst Language Floor | 3 |
Table 8: Head of Engineering weight profile (13 dimensions, ).
| Dim | Name | Wt |
|---|---|---|
| D1 | MCC | 10 |
| D4 | F1 | 7 |
| D9 | CWE Coverage Breadth | 3 |
| D11 | Cross-Language Consistency | 4 |
| D14 | Evidence Completeness | 2 |
| D25 | Turns per Task | 5 |
Table 9: AI-as-Actor weight profile (13 dimensions, ).
| Attribute | Count |
|---|---|
| Total tasks | 406 |
| True Positive tasks | 203 |
| Post-Patch tasks | 203 |
| Source projects | 93 |
| OWASP categories | 8 |
| Programming languages | 10 |
Table 10: Dataset statistics.
| Category | OWASP | Tasks |
|---|---|---|
| Broken Access Control | A01:2021 | 82 |
| Cryptographic Failures | A02:2021 | 64 |
| Injection | A03:2021 | 62 |
| Improper Input Validation | Extended | 58 |
| SSRF | A10:2021 | 46 |
| Auth Failures | A07:2021 | 38 |
Table 11: Vulnerability category distribution (TP + Post-Patch).
| Model | CIP Score (%) | CIP Cost | TU Score (%) | TU Cost |
|---|---|---|---|---|
| Gemini 3 Flash Preview | 49.6 | $15.87 | 44.2 | $184.21 |
| Gemini 3.1 Pro Preview | 48.2 | $44.81 | 45.8 | $639.09 |
| Claude Sonnet 4.6 | 47.6 | $6.76 | 42.1 | $359.76 |
| Kimi K2.5 | 46.8 | N/T | – | – |
| Gemini 2.5 Pro | 46.2 | $13.45 | 35.2 | $57.91 |
| Grok Code Fast 1 | 44.1 | N/T | 34.4 | N/T |
Table 12: Full leaderboard across CIP and TU layers, sorted by CIP score. Cost shown where tracked; N/T = not tracked.
| Model | LB % | CISO | CAIO | Researcher | Head Eng. | AI Actor |
|---|---|---|---|---|---|---|
| Gemini 3 Flash Preview | 49.6 | B (73.3) | B (68.1) | B (71.0) | B (66.2) | A (87.5) |
| Gemini 3.1 Pro Preview | 48.2 | B (67.5) | B (67.0) | B (65.7) | B (63.8) | A (85.7) |
| Claude Sonnet 4.6 | 47.6 | B (65.7) | B (68.4) | B (64.2) | B (73.9) | A (85.6) |
| Kimi K2.5 | 46.8 | B (68.0) | B (67.8) | B (67.0) | B (65.1) | A (86.4) |
| Gemini 2.5 Pro | 46.2 | B (66.2) | B (67.9) | B (65.2) | B (71.3) | A (86.3) |
| Grok Code Fast 1 | 44.1 | C (58.7) | B (67.5) | B (60.2) | B (73.0) | A (83.8) |
Table 13: Per-role Decision Scores (CIP layer). Grade thresholds: A 75, B 60, C 50, D 40, F 40. Sorted by leaderboard score (LB).
Figure 2: Per-role Decision Scores for 12 models (CIP layer). Dashed lines mark grade thresholds: A 75, B 60, C 50. Models sorted by leaderboard score (left to right).
| Category | #1 (F1) | #2 (F1) | #3 (F1) | Worst (F1) |
|---|---|---|---|---|
| Broken Access Ctrl. | Kimi K2.5 (0.667) | Gemini 3.1 Pro (0.660) | Gemini 3 Flash (0.647) | Qwen3-Coder (0.128) |
| Cryptographic Failures | Gemini 3 Flash (0.676) | Gemini 3.1 Pro (0.667) | Haiku 4.5 (0.633) | Qwen3-Coder (0.118) |
| Auth Failures | Kimi K2.5 (0.585) | Gemini 2.5 Pro (0.579) | Gemini 3.1 Pro (0.578) | Opus 4.6 (0.000) |
| Input Validation | Haiku 4.5 (0.675) | Kimi K2.5 (0.649) | Gemini 2.5 Flash (0.638) | Qwen3-Coder (0.125) |
| Injection | Gemini 3.1 Pro (0.632) | Gemini 3 Flash (0.618) | Gemini 2.5 Flash (0.603) | Qwen3-Coder (0.062) |
| Memory Safety | Haiku 4.5 (0.690) | Gemini 3 Flash (0.667) | Gemini 2.5 Flash (0.640) | Qwen3-Coder (0.308) |
Table 14: Per-category F1 leaders and worst performers (CIP layer).
| Model | BAC | Crypto | Auth | Input | Inj. | Mem. | SSRF | D.Int. | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Flash | 0.647 | 0.676 | 0.444 | 0.627 | 0.618 | 0.667 | 0.667 | 0.680 | 0.628 |
| Claude Haiku 4.5 | 0.586 | 0.633 | 0.545 | 0.675 | 0.507 | 0.690 | 0.667 | 0.679 | 0.623 |
| Gemini 3.1 Pro | 0.660 | 0.667 | 0.578 | 0.618 | 0.632 | 0.583 | 0.600 | 0.500 | 0.605 |
| Kimi K2.5 | 0.667 | 0.567 | 0.585 | 0.649 | 0.517 | 0.609 | 0.644 | 0.529 | 0.596 |
| Gemini 2.5 Pro | 0.635 | 0.562 | 0.579 | 0.627 | 0.585 | 0.526 | 0.643 | 0.514 | 0.584 |
| Claude Sonnet 4.6 | 0.590 | 0.625 | 0.514 | 0.571 | 0.518 | 0.526 | 0.690 | 0.537 | 0.571 |
Table 15: F1 score by model and vulnerability category (CIP layer). Bold indicates best-in-category. Italics indicates worst. Models sorted by average F1.
Figure 3: F1 score heatmap by model and vulnerability category (CIP layer). Models sorted by average F1. Green = strong, red = weak.
| Model | RDI | Best Role | Worst Role |
|---|---|---|---|
| Qwen3-Coder | 31.1 | AI Actor (77.9) | CISO (45.2) |
| GPT-5.4 | 30.8 | AI Actor (79.2) | CISO (48.4) |
| Qwen3-Plus | 30.1 | AI Actor (81.2) | CISO (51.1) |
| Opus 4.6 | 29.2 | AI Actor (80.2) | CISO (51.0) |
| Grok Code Fast | 25.1 | AI Actor (83.8) | CISO (58.7) |
| Gemini 2.5 Flash | 23.6 | AI Actor (84.9) | Researcher (61.1) |
Table 16: Role Divergence Index (CIP layer). Highest and lowest scoring roles shown.
Figure 4: Role Divergence Index per model (CIP layer). Red = high divergence ( 28), yellow = moderate ( 22), blue = low. Higher RDI means the model’s value depends more on stakeholder perspective.
| Model | LB % | CISO | CAIO | Researcher | Head Eng. | AI Actor |
|---|---|---|---|---|---|---|
| Gemini 3.1 Pro Preview | 45.8 | B (69.4) | C (56.1) | B (65.0) | D (44.5) | A (75.4) |
| Gemini 3 Flash Preview | 44.2 | B (68.9) | C (55.9) | B (64.1) | D (44.5) | B (73.7) |
| Claude Sonnet 4.6 | 42.1 | B (66.3) | C (56.1) | B (60.6) | D (43.9) | B (74.7) |
| Claude Opus 4.6 | 39.0 | C (56.0) | C (55.6) | C (53.9) | D (43.8) | B (71.7) |
| Claude Haiku 4.5 | 37.8 | B (66.5) | C (57.1) | B (60.3) | D (47.8) | B (72.8) |
| Gemini 2.5 Pro | 35.2 | C (59.7) | B (64.4) | C (55.2) | C (52.0) | A (79.7) |
Table 17: Per-role Decision Scores (TU layer). Grade thresholds: A 75, B 60, C 50, D 40, F 40. Sorted by TU leaderboard score.
| Model | $/task | Total | MCC/$ | TPM |
|---|---|---|---|---|
| Haiku 4.5 | $0.005 | $2.11 | 0.019 | 9.8 |
| GPT-5.4 | $0.007 | $2.97 | 0.042 | 15.1 |
| Gemini 2.5 Flash | $0.008 | $3.23 | 0.031 | 4.1 |
| Sonnet 4.6 | $0.017 | $6.76 | 0.025 | 4.4 |
| Opus 4.6 | $0.019 | $7.56 | 0.009 | 7.3 |
| Gemini 2.5 Pro | $0.033 | $13.45 | 0.011 | 2.2 |
Table 18: CIP layer cost analysis (8 models with cost tracking).
Figure 5: Cost per task vs. leaderboard score (CIP layer, 8 models with cost tracking). Log-scale x-axis. Higher and further left is better.
More in this topic
LLMs for Cybersecurity
Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.
See all 255 papers