Code-Grounded Agentic Explanations for Static Analysis Vulnerability Findings
Organizations: Fraunhofer IEM, Paderborn, Germany. · Heinz Nixdorf Institute, Paderborn University, Paderborn, Germany. · Carl Von Ossietzky Universität Oldenburg, Oldenburg, Germany. · Amazon Web Services, New York, United States.
Abstract
Static application security testing (SAST) tools surface vulnerabilities early and their verdicts are rule-based and auditable. Yet developers routinely dismiss findings they cannot interpret. Large language models can supply the missing explanation, but designs that also hand the model the detection give up the auditability that makes static analysis defensible. We keep detection with the analyser and give the model only the explaining. A preliminary user study with 20 participants and a workshop with 19 practitioners yielded five requirements. Guided by these we redesigned SAFE, an open-source IntelliJ IDEA plugin that explains findings from any analyser reporting in the Static Analysis Results Interchange Format. It tiers explanations by demand, ties them to the reported data flow, and replaces the single-prompt backend with an agent that must consult the repository and the rule that fired before answering. A within-subjects crossover experiment with 24 participants compared the redesign against its predecessor. Perceived usability rose from 77.3 to 85.0 on the System Usability Scale. Actionability gained most among the explanation-quality items, the aspect both earlier studies flagged as weakest, and the odds of a correct remediation were 2.6 times higher. Triage accuracy and time per finding were indistinguishable, which we read as uninformative rather than as equivalence. Trust was the one measure that did not move, and the closing interviews indicate that grounding made findings easier to understand rather than easier to believe. Participants also drew a boundary no requirement anticipated, wanting the tool to find and explain while they judge and apply.
Figures & tables
| Section | Required content |
|---|---|
| title | A short headline naming the data flow behind the finding. |
| tldr | One plain sentence stating what the reported vulnerability is and what an attacker gains from it. |
| what | Two sentences defining the vulnerability in terms of the real code identifiers involved rather than of the rule identifier that fired. |
| why | The concrete impact in this specific application rather than in the weakness class in general. |
| where | The exact file, line, and code construct at which the problem manifests. |
| how to fix | An attack scenario, followed by a targeted fix and an account of why that fix works. |
| Tool | Purpose |
|---|---|
| lookup_sast_rule | Looks up the definition of the rule that fired and returns its identifier, message, patterns, severity, and CWE and OWASP metadata, so that the explanation reflects what the rule actually checks. |
| get_code_structure | Returns the functions, classes, and methods of a file with their exact line ranges, taken from a precomputed index. |
| read_code | Reads source lines from a file, optionally restricted to a line range. |
| search_codebase | Searches the codebase for a pattern, optionally restricted to a list of files, and returns the matching paths. |
| list_files | Lists the entries of a directory, with the output truncated to a bounded number of entries. |
| find_files | Finds files whose name or path contains a given substring. |
| Group | Task Block 1 | Task Block 2 |
|---|---|---|
| G1 | SAFE-Blue on Set 1 | SAFE-Red on Set 2 |
| G2 | SAFE-Red on Set 2 | SAFE-Blue on Set 1 |
| G3 | SAFE-Blue on Set 2 | SAFE-Red on Set 1 |
| G4 | SAFE-Red on Set 1 | SAFE-Blue on Set 2 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Response Options |
|---|---|
| What is your current primary job role? | Software Developer, Student, Research, Security Engineer/Analyst, Other |
| How many years of professional software development experience do you have? | 1 year, 1–3 years, 3–7 years, 8+ years |
| How many years of experience do you have programming in Java? | 1 year, 1–3 years, 3–7 years (unused), 8+ years |
| How frequently do you use JetBrains products (for example IntelliJ IDEA, PyCharm, WebStorm, PhpStorm)? | Never, Rarely, Weekly, Daily |
| How would you rate your familiarity with secure coding practices (for example input validation, authentication, data protection, database security)? | Five-point scale, 1 Novice to 5 Expert (5 unused) |
| How frequently do you use Static Application Security Testing (SAST) tools (for example Semgrep, SonarQube, Qodana)? | Never, Occasionally, Weekly, Daily |
| Item | Response | Serves |
|---|---|---|
| Which finding is this? | The rule identifier shown in the plugin | Item identity |
| What is your final verdict on this finding? | True positive, false positive, or unsure | H1a |
| How would you fix or improve the code? | Free text | H1b |
| # | Item |
|---|---|
| 1 | I think that I would like to use this tool frequently. |
| 2 | I found the tool unnecessarily complex. |
| 3 | I thought the tool was easy to use. |
| 4 | I think that I would need the support of a technical person to be able to use this tool. |
| 5 | I found the various functions in this tool were well integrated. |
| 6 | I thought there was too much inconsistency in this tool. |
| Item | Hypothesis | Requirement |
|---|---|---|
| Clarity: The vulnerability explanations provided by the tool were easy to understand. | H3a | R1 |
| Actionability: The tool provided clear next steps to fix the vulnerabilities. | H3a | R3 |
| Confidence-support: The explanations made me feel confident in my decision to classify the finding as a true or false positive. | H3a | R5 |
| The user interface (tool window, gutter icons, and so on) helped me locate the issues efficiently. | H3c | R4 |
| I trust the tool’s assessment of the vulnerabilities. | H3d | R5 |
| # | Question | Probes | Asked |
|---|---|---|---|
| 1 | What did you like, not like, or wish to be improved in the usability of the plugins? | Usability, open | 20 |
| 2 | If the information from the SAFE-Red plugin were presented in the SAFE-Blue audience-toggle interface, do you think your accuracy or speed would have changed, and vice versa? Why? | Tool-bundle confound | 13 |
| 3 | In what way did the evidence links or the code annotations contribute more to your trust in the tool? | Trust, R5 | 11 |
| 4 | Do you think it is necessary to explain vulnerabilities to developers? In the age of AI, should problems be triaged automatically, and how involved should developers be? | Automation boundary | 17 |
| 5 | Any additional feedback, suggestions, or things you would like to change. | Open | 18 |
| Family | Weakness | Expected mechanism |
|---|---|---|
| F1 | Path traversal (CWE-23) | Canonicalise the resolved path and check that it stays inside the allowed base directory |
| F2 | SQL injection (CWE-89) | Prepared statement, parameterised query, or ORM binding |
| F3 | Server-side request forgery (CWE-918) | Constrain the host through an allowlist, or a trusted base URL with only the path and query under user control |
| F4 | Cross-site scripting (CWE-79) | Render through a view or template with contextual output encoding, or escape before writing |
| Level | Meaning | Decision rules |
|---|---|---|
| correct | Names the right mechanism for the family | Includes restating the fix the tool suggested, and naming the safe counterpart method |
| partial | Right family, mechanism vague or incomplete | Sanitisation or validation language with no named mechanism |
| incorrect | Wrong family, or a mechanism that does not address the weakness | A fix belonging to a different family |
| none | No remediation proposed | Commentary about the tool rather than the code, proposing detection rather than a fix, and declining to fix because the finding was judged not real |
| Code | Part. | Seg. | Agreement |
|---|---|---|---|
| Automation boundary | 22 | 40 | 67 per cent |
| Explanation structure and content | 21 | 38 | 0 per cent |
| Expertise level | 17 | 23 | 67 per cent |
| Importance of explaining | 14 | 19 | 67 per cent |
| Fix suggestions | 13 | 17 | 50 per cent |
| Data-flow navigation | 12 | 19 | 67 per cent |
Static Code Analysis
Static code analysis examines source code without running it to find defects, security risks, and maintainability problems.
See all 59 papers