cs.AIOct 27, 2026

Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis

Authors: Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya Shah

Organizations: Columbia University New York, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked. This gap matters most for smart contracts, the programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. We ask how to check whether an AI-written contract behaves as it should, and where the requested behavior gets lost. We design an evaluation built on execution rather than opinion. We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes, compare against an answer key written without the description, and audit our own scorer until known wrong answers cannot pass. Common methods, such as AI judges and standard analysis tools, miss obvious mistakes, and scoring choices matter too. Checking only the functions every system includes hid about half the difference between pipelines. We then propose an elicitation-first architecture. Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key, and returns disagreements to the client instead of silently rewriting code. Asking first matters far more than reviewing or patching afterwards, across three AI models, while automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits, and their value can be estimated without the original contract. To our knowledge, this is the first study in smart contract synthesis to gather missing requirements before generation and check the result against an independently written executable model. The lessons extend to any setting where AI agents turn requests into programs. We release our tools, 600 tasks and all results.

Figures & tables

Figure 1. The two pipelines compared in this paper. (a) The original pipeline parses requirements without consulting their author, rewrites code in response to LLM security findings and grades itself with an LLM rubric. (b) The elicitation-first pipeline asks the client checklist questions before generation, checks the contract against an independently written reference model, shows each disagreement to the client as a blind choice and reports security findings without rewriting code. Badges give measured effects; percentages are transaction-level fidelity to the original contract on 500 held-out contracts (§ 4.5 ).
InstrumentCaughtFalse alarms
Slither (static analysis)1.5% [0.5, 4.3]n/a
LLM judge reading the code65.0%80%
Black-box LLM tester17.5%40%
LLM judge with executable proof30.0%40%
Reference model, with description94.3%14.4%
Reference model, function names only87.5%18.7%
Table 1. Existing instruments on planted mistakes. Slither covers 200 guard removals, the LLM rows 40 mutants of 20 contracts and the reference-model rows 66 held-out contracts. A mistake is caught when the instrument flags the mutated function on the mutant but not on its twin, so an instrument that flags most contracts somewhere can still catch many. False alarms are the share of untouched contracts flagged, or for reference models the share of steps on untouched contracts where they disagree.
Figure 2. Strict catch of contract-specific mistakes by reference models, by what the writer is given, for three model families. Whiskers are 95% intervals.
Figure 3. Question budget (92 contracts, transaction-level fidelity). (a) Asking the checklist in its fixed order gains little until question 6. Ordering questions by their measured value, learned on the other half of the contracts, reaches 68% with three questions and 71% with five. Points show adaptive agents at their mean number of questions. (b) Gain from each question asked alone; a few questions carry most of the value. Bands and whiskers are 95% intervals.
Figure 4. What the security audit-and-refine loop does. (a) Transaction-level fidelity per contract before and after the loop, with complete requirements as input; points below the diagonal lost fidelity. (b) LLM classification of a capped selection of 341 outcome changes in the 72 rewritten contracts against the complete requirements.
Figure 5. Information versus architecture. Each architecture receives either the short description or the same description with the checklist answers the elicitation-first pipeline obtained.
 TxnPassBuildFunc.Obs.
One-shot generation53.92.695.275.575.6
Original pipeline39.31.279.258.767.1
Elicitation-first, before triage72.217.297.084.777.7
Elicitation-first, full72.919.697.084.777.7
Table 2. End to end on 500 held-out contracts (499 with scorable steps), all from the same short description and code generator. Never-built outputs score zero. Columns give transaction-level fidelity, contracts passing all sampled tests, contracts that build, required functions implemented and original value changes observable.
Figure 6. Beyond the averages. (a) Cumulative distribution of per-contract transaction-level fidelity on 499 contracts; a curve further to the right is better, and the vertical jump at 100% is the share of contracts with every step correct. (b) Mean fidelity by contract type with 95% intervals; colors as in (a).
Figure 7. The same 500 sets of outputs scored four ways. (a) The shared-interface metric drops any function some system lacks and excludes contracts that do not build; adapted replay calls each contract through its own signatures; exact replay calls the original’s. (b) Share of the original contract’s contract-specific functions each system implements.
ConclusionChecks that support it
Information is lost before codefresh transaction sequences; five transaction-generator variants; three rulebook writer families; generated contracts as well as rulebooks; descriptions without factual errors; three generator families
Stages after writing add at most about one pointno detectable gain from extractor, critic, reviewer, ensemble, voting, proof-gated judging or rulebook repair; triage detectable but small and provisional
Security rewriting breaks requirements94 contracts; LLM classification of selected changes; holds on contracts without any selected justified fix
Elicitation-first beats the original96 and 499 contracts; four contract types; equal-information factorial; shared, adapted and exact scoring; failures counted or excluded; near-duplicates excluded
Few questions recover much of the gaincross-fitted ordering; four adaptive agents; a label-free separation score; which questions matter depends on description errors
Instrument fixes do not drive resultsrunner fix moved no catch rate by more than 0.6 points; exact replay preserved every ranking
Table 3. Each main conclusion and the independent checks behind it.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8. Architecture of the study. Inputs (left) feed the systems under test; every system’s output runs in the same in-process EVM, and reference models run in a separate sandboxed process. Two measurement families consume the traces, fidelity of generated contracts and validity of oracles on planted mistakes.
ModuleRole
harness.pydeploy at the declared compiler version, run steps with callers, ether and time travel
runner_state.py , model_runner3.pysandboxed reference-model execution with revert rollback and per-sequence reset
idea8_specific.pytransaction generator, seed pool, contract-specific mutation operators
rbeval.pystrict side-picking, right-side rate, bootstrap over contracts
fidelity.py , rescore.pyfidelity with exact replay, saved compile repairs, real-defect analysis
exp*.py , tests/one script per experiment with its criteria; known-answer tests
Appendix
Table 4. Main modules of the released code.
ExperimentCriterionOutcomeValue
Slither on guard removal §detection below 25%met1.5%
Conventional over-permission §violation rate at least 5%not met0.2%
Tester versus judge §tester at least judge plus 15not met17.5 vs 65.0
Proof-gated judge §recall at least 50%not met30%
Implementation voting §majority above mean singlenot met80.5 vs 80.9
Disagreement-guided clarification §+5+5 over none, +3+3 over randomnot met+1.7+1.7
Appendix
Table 5. Predefined criteria and outcomes, in the order the experiments were run. Criteria marked \lx@sectionsign\lx@sectionsign were committed together with their results and are retrospective; the others were committed before any output existed. Reference-model criteria ( †\dagger ) were re-graded with the corrected runner and none changed outcome. Criteria for generated contracts were set on an earlier metric and are applied unchanged to the exact-replay metric.
SettingValue
Sequences per contract, steps60, 8
Callersowner (deployer), two other accounts
Numeric arguments0, 1, 100, 999, 1000, 1001, seeded values
Seed poolcontract literals and getter values ±1\pm 1 , at most 16
Time steps15% of steps, 1 hour to 366 days
Clockfixed start, one second per block
Appendix
Table 6. Transaction generator and statistics.
Figure 9. Every intervention in the study, by where it acts. Interventions that add information before writing (blue) help in every model family and on both measures; agents and stages acting after writing (red) add at most about one point or cause harm. The first row uses the fresh-sequence evaluation.
AgentQuestionsFidelityMost frequent first question
Picks three3.067.2%roles (56 of 93)
Picks and stops4.770.6%roles (46 of 93)
Picks, stops, own questions6.677.7%roles (44 of 93)
Judge only stops10.073.1%never stopped early
All ten questions1076.0% 
Appendix
Table 7. Behavior of the adaptive question agents (92 contracts).
More in this topic

Agentic Code Generation

Agentic code generation uses AI agents to plan, write, test, and revise executable programs with tools and feedback.

See all 130 papers

Explore similar work

CardsList
May 10, 2026

SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.

Abhinav Goel, Agostino Capponi, Alfio Gliozzo +1

Columbia University New York, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA

Code GenerationCode Generation EvaluationLLM Code Generation
AskPDF
Jun 24, 2026

CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing $57.4M in total exploit profit across the 200-case exploit set at a cost of $2.39 per case.

Jintao Huang, Fengqing Jiang, Radha Poovendran +1

1The Ohio State University · University of Washington

Automated Program RepairSoftware Vulnerability DetectionSoftware SecurityAI Agent Security BenchmarksDecentralized Finance
AskPDF
Mar 5, 2026

EVMbench: Evaluating AI Agents on Smart Contract Security

Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench, an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 117 curated vulnerabilities from 40 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security.

Justin Wang, Andreas Bigger, Xiaohai Xu +5

OpenAI · Paradigm · OtterSec

AI Agent Security BenchmarksSoftware SecuritySoftware Vulnerability Detection
AskPDF