Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench, an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 117 curated vulnerabilities from 40 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security.
Figures & tables
Figure 1: Overview of the three evaluation modes in our benchmark. In Detect , the agent audits a smart contract repository and is evaluated via recall of ground-truth vulnerabilities. In Patch , the agent edits a vulnerable codebase, and we verify that original tests (minus those relying on vulnerable logic) still pass while exploits fail on the patched contracts. In Exploit , the agent interacts with a Ethereum instance via an RPC endpoint; we replay its transactions in an isolated grader container and run vulnerability-specific checks over contracts and balances.
Figure 2: An example of a vulnerable smart contract.
Scope (sLOC)
Contracts
min
106
1
25%
436
3
50%
1,296
7
mean
2,045
16
75%
3,394
22
max
10,108
116
Table 1: Summary statistics for audit repository scope size (sLOC) and contract counts.
Figure 3: Agent scores on the full evaluation across all task modes. We display bootstrap point estimates and confidence intervals for score rate across 3 independent runs. All agents are run at the maximum configurable reasoning level. “OC” denotes the OpenCode scaffold; models with no prefix are run using the CLI agent of their developer, i.e. GPT-5.3-Codex is run with Codex CLI, Claude models are run via Claude Code, and Gemini 3 Pro is run via Gemini CLI. Table J contains full results.
Agent
Avg. Detect award ($)
Claude Opus 4.6
37,824.52
OC-GPT-5.2
31,623.98
Gemini 3 Pro
25,112.46
GPT-5.3-Codex
19,915.38
Claude Opus 4.5
18,069.79
GPT-5
16,078.97
Table 2: Detect awards by agent. The maximum audit award is $218,434.
Figure 4: Scores of the best-performing agents versus output tokens.
Figure 5: Detect score against mean vulnerability disclosure count. Gemini 3 Pro, GPT-5.2, and GPT-5.3-Codex report the closest number of vulnerabilities when compared with the ground truth in the dataset.
Figure 6: GPT-5.2 executes a fund drain. Tool calls and responses are truncated and edited for readability. In the 2024-04-noya H-08 task, the agent discovers and executes the fund drain.
Figure 7: Left: Performance of GPT-5.3-Codex across all reasoning effort levels and task modes. Right: GPT-5.2 results across all hint levels.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Audit
Description
sLOC
# Contracts
2023-07-pooltogether
Prize linked savings protocol that distributes yield as prizes.
3,324
14
2023-10-nextgen
On-chain generative NFT platform with phased minting.
1,265
8
2023-12-ethereumcreditguild
Credit guild with governance token and credit tokens.
3,721
20
2024-01-canto
Rewards accounting for Canto lending market integrations.
106
1
2024-01-curves
Curves protocol extending friend.tech with ERC20 export and fees.
553
5
2024-01-init-capital-invitational
Composable liquidity hook money market for DeFi strategies.
2,334
22
Appendix
Table 4: Repositories included in the dataset with short descriptions, codebase size, and contract counts.
Figure 9: Distribution of auditor counts who found EVMbench vulnerabilities during Code4rena audits.
Figure 10: Distribution of lines of code in the oracle patches.
Figure 11: Distribution of per-vulnerability transactions in oracle exploits.
Figure 14: Input to the Model-Based Judge.
Figure 15: Under-Credit Prompt.
Figure 16: Over-Credit Prompt.
Figure 17: Prompt Injection.
Model
Under-Credit ↑
Over-Credit ↓
Prompt Injection ↓
GPT-5
100.00%
1.94%
2.78%
Appendix
Table 5: Performance of the judge on robustness stress tests. Higher is better for Under-Credit (correctly rewarding findings that identify the vulnerability despite slight technical errors), while lower is better for Over-Credit and Prompt Injection (avoiding rewarding incorrect findings). We compute results over the full set of vulnerabilities in the Detect mode.
Figure 18: All Instructions.
Figure 19: GPT-5 fails to drain ETH. Tool calls and responses are truncated and edited for readability. The agent discovers a vulnerability but has difficulty compiling a contract and fails to execute the exploit.
Figure 20: Detect Submission. In the 2024-08-phi Detect task, GPT-5 finds the H-06 reentrancy vulnerability (listed in its report as C-03). Parts of the submission have been removed to improve viewability; the full submitted report details 4 additional vulnerabilities and includes a summary section.
Figure 21: Patch Submission. In the 2024-08-phi H-06 Patch task, GPT-5 detects the H-06 reentrancy vulnerability. It makes the above changes to the code, fixing the reentrancy by adding the nonReentrant modifier. The existing tests pass and the exploit test fails after applying this change, so the agent is awarded.
Figure 22: Detect Submission. In the 2024-07-basin Detect task, GPT-5.1 identifies both ground-truth vulnerabilities and includes them in its submission. Parts of the submission have been removed to improve viewability. The rest of the report contains a detailed exploit scenario, impact analysis, and recommended mitigation of H-01; similar sections for H-02; and additional notes and a conclusion.
Figure 23: GPT-5.1 fails to drain ETH. Tool calls and responses are truncated and edited for readability. The agent discovers the vulnerability, deploys a malicious contract, and calls the method to drain funds to itself. However, the agent does not check the ETH balance of the contract and instead thinks that it is done after draining two tokens which the proxy address never had a balance of to begin with.
We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing $57.4M in total exploit profit across the 200-case exploit set at a cost of $2.39 per case.
Jintao Huang, Fengqing Jiang, Radha Poovendran +1
1The Ohio State University · University of Washington
AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties. A reported exploit is evaluated by replaying it against the application and running the probes: a triggered probe indicates both that the exploit succeeded and which security property it violated. As a probe encodes a security property rather than a known vulnerability, it can detect vulnerabilities that were not known when the probe was written. We instantiate the framework as MobileCybench, a benchmark for vulnerability discovery by AI agents in 13 Android applications, with 495 probes written and reviewed by the authors. We evaluate 5 coding agents (OpenCode with GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code with Opus 4.8 and Opus 5) under 4 settings: as a malicious app on the victim's device or as a remote attacker with a low-privilege account, each with either only an obfuscated APK or access to the application's source code. Given only the obfuscated APK, the top agent, OpenCode with GPT-5.6-Sol, triggers probes in 53.8% of applications in the malicious-app setting and 16.7% in the remote-attacker setting. With source code, the trigger rate across all agents and both attack settings increases from 28.8% to 32.8%. Building and running the benchmark surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.
AI agents can inspect code and run tests to support smart contract security. SlowMist reports about $38.6 billion in losses across the broader blockchain ecosystem. Existing evaluations can reward memorized or copied answers, including attack code recovered from public blockchain transactions. Reconstructing contracts and validating tasks also require costly manual effort. To address these issues, we propose CONSTRAINTDIAL, a framework for lifelong construction of smart contract security tasks via protocol mutation. It automates local contract reconstruction and uses attack execution to select changes to the required calls or inputs. Execution checks guide revisions that make the original attack fail while preserving exploitability, the security objective, and normal functionality. New incidents and audit findings enter the same procedure. With CONSTRAINTDIAL, we construct 3,045 task inputs from 203 public cases, covering detect, exploit, and patch. Compared with reconstruction alone, protocol mutation reduces success with supplied public answers by 10.9 percentage points on average across detect, exploit, and patch. The fraction of tasks solved by copying online answers falls by 9.3 points.
Zifan Peng, Peng Huang, Jingyi Zheng +5
Shanghai Artificial Intelligence Laboratory · The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China +1