cs.LGMar 5, 2026
SaveEVMbench: Evaluating AI Agents on Smart Contract Security
Organizations: OpenAI · Paradigm · OtterSec
Abstract
Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench, an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 117 curated vulnerabilities from 40 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security.
Figures & tables
Figure 1: Overview of the three evaluation modes in our benchmark. In Detect , the agent audits a smart contract repository and is evaluated via recall of ground-truth vulnerabilities. In Patch , the agent edits a vulnerable codebase, and we verify that original tests (minus those relying on vulnerable logic) still pass while exploits fail on the patched contracts. In Exploit , the agent interacts with a Ethereum instance via an RPC endpoint; we replay its transactions in an isolated grader container and run vulnerability-specific checks over contracts and balances.
Figure 2: An example of a vulnerable smart contract.
| Scope (sLOC) | Contracts | |
|---|---|---|
| min | 106 | 1 |
| 25% | 436 | 3 |
| 50% | 1,296 | 7 |
| mean | 2,045 | 16 |
| 75% | 3,394 | 22 |
| max | 10,108 | 116 |
Table 1: Summary statistics for audit repository scope size (sLOC) and contract counts.
Figure 3: Agent scores on the full evaluation across all task modes. We display bootstrap point estimates and confidence intervals for score rate across 3 independent runs. All agents are run at the maximum configurable reasoning level. “OC” denotes the OpenCode scaffold; models with no prefix are run using the CLI agent of their developer, i.e. GPT-5.3-Codex is run with Codex CLI, Claude models are run via Claude Code, and Gemini 3 Pro is run via Gemini CLI. Table J contains full results.
| Agent | Avg. Detect award ($) |
|---|---|
| Claude Opus 4.6 | 37,824.52 |
| OC-GPT-5.2 | 31,623.98 |
| Gemini 3 Pro | 25,112.46 |
| GPT-5.3-Codex | 19,915.38 |
| Claude Opus 4.5 | 18,069.79 |
| GPT-5 | 16,078.97 |
Table 2: Detect awards by agent. The maximum audit award is $218,434.
Figure 4: Scores of the best-performing agents versus output tokens.
Figure 5: Detect score against mean vulnerability disclosure count. Gemini 3 Pro, GPT-5.2, and GPT-5.3-Codex report the closest number of vulnerabilities when compared with the ground truth in the dataset.
Figure 6: GPT-5.2 executes a fund drain. Tool calls and responses are truncated and edited for readability. In the 2024-04-noya H-08 task, the agent discovers and executes the fund drain.
Figure 7: Left: Performance of GPT-5.3-Codex across all reasoning effort levels and task modes. Right: GPT-5.2 results across all hint levels.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Audit | Description | sLOC | # Contracts |
|---|---|---|---|
| 2023-07-pooltogether | Prize linked savings protocol that distributes yield as prizes. | 3,324 | 14 |
| 2023-10-nextgen | On-chain generative NFT platform with phased minting. | 1,265 | 8 |
| 2023-12-ethereumcreditguild | Credit guild with governance token and credit tokens. | 3,721 | 20 |
| 2024-01-canto | Rewards accounting for Canto lending market integrations. | 106 | 1 |
| 2024-01-curves | Curves protocol extending friend.tech with ERC20 export and fees. | 553 | 5 |
| 2024-01-init-capital-invitational | Composable liquidity hook money market for DeFi strategies. | 2,334 | 22 |
Table 4: Repositories included in the dataset with short descriptions, codebase size, and contract counts.
Figure 9: Distribution of auditor counts who found EVMbench vulnerabilities during Code4rena audits.
Figure 10: Distribution of lines of code in the oracle patches.
Figure 11: Distribution of per-vulnerability transactions in oracle exploits.
Figure 14: Input to the Model-Based Judge.
Figure 15: Under-Credit Prompt.
Figure 16: Over-Credit Prompt.
Figure 17: Prompt Injection.
| Model | Under-Credit | Over-Credit | Prompt Injection |
|---|---|---|---|
| GPT-5 | 100.00% | 1.94% | 2.78% |
Table 5: Performance of the judge on robustness stress tests. Higher is better for Under-Credit (correctly rewarding findings that identify the vulnerability despite slight technical errors), while lower is better for Over-Credit and Prompt Injection (avoiding rewarding incorrect findings). We compute results over the full set of vulnerabilities in the Detect mode.
Figure 18: All Instructions.
Figure 19: GPT-5 fails to drain ETH. Tool calls and responses are truncated and edited for readability. The agent discovers a vulnerability but has difficulty compiling a contract and fails to execute the exploit.
Figure 20: Detect Submission. In the 2024-08-phi Detect task, GPT-5 finds the H-06 reentrancy vulnerability (listed in its report as C-03). Parts of the submission have been removed to improve viewability; the full submitted report details 4 additional vulnerabilities and includes a summary section.
Figure 21: Patch Submission. In the 2024-08-phi H-06 Patch task, GPT-5 detects the H-06 reentrancy vulnerability. It makes the above changes to the code, fixing the reentrancy by adding the nonReentrant modifier. The existing tests pass and the exploit test fails after applying this change, so the agent is awarded.
Figure 22: Detect Submission. In the 2024-07-basin Detect task, GPT-5.1 identifies both ground-truth vulnerabilities and includes them in its submission. Parts of the submission have been removed to improve viewability. The rest of the report contains a detailed exploit scenario, impact analysis, and recommended mitigation of H-01; similar sections for H-02; and additional notes and a conclusion.
Figure 23: GPT-5.1 fails to drain ETH. Tool calls and responses are truncated and edited for readability. The agent discovers the vulnerability, deploys a malicious contract, and calls the method to drain funds to itself. However, the agent does not check the ETH balance of the contract and instead thinks that it is done after draining two tokens which the proxy address never had a balance of to begin with.
| Model | Snapshot |
|---|---|
| OpenAI o3 | o3 |
| GPT-5 | gpt-5-2025-08-07 |
| GPT-5.2 | gpt-5.2-2025-12-11 |
| GPT-5.3-Codex | gpt-5.3-codex |
| Claude Opus 4.5 | claude-opus-4-5-20251101 |
| Claude Opus 4.6 | claude-opus-4-6 |
Table 6: Model aliases and the snapshots used.
| Agent | CLI Version |
|---|---|
| Codex | 0.98.0 |
| Claude | 2.1.32 |
| Gemini | 0.25.0 |
| OpenCode | 1.1.26 |
Table 7: Agent CLI versions used in our runs.
More in this topic
AI Agent Security Benchmarks
AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.
See all 301 papers