SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

Giovanni Pio Delvecchio; Stefano Fantazzini; Giacomo Frisoni; Gianluca Moro; Lorenzo Bacchiani; Mario Bravetti; Thomas Troels Hildebrandt

cs.SEOct 13, 2026

SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

Authors: Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni, Gianluca Moro, Lorenzo Bacchiani, Mario Bravetti, Thomas Troels Hildebrandt

Organizations: University of Bologna, Bologna, Italy · University of Copenhagen, Copenhagen, Denmark

Abstract

Large language models increasingly generate Solidity smart contracts that, once deployed, can never be patched, making a single latent fault permanent and exploitable. Yet current evaluations mostly ask only whether generated code compiles. We introduce SolBehave, a benchmark for evaluating the correctness of model-generated contracts at scale, moving beyond ad hoc unit tests. Across 250 real, reproducible Ethereum contracts, we apply three complementary tests of behavioural fidelity: differential fuzzing, historical-transaction replay, and, as our core contribution, synthetic-transaction replay guided by a Dynamic Condition Response (DCR) graph that enumerates the space of legitimate function-call sequences. Differential fuzzing compares each generated function with the original on automatically generated inputs; historical replay compares the whole contract on the transactions it actually received on-chain; DCR-guided replay compares it on sequences of calls that follow its intended usage, including orderings that never occurred on-chain. We evaluate seven state-of-the-art open-weight models after one round of automated repair. Compilation success ranges from 26.4% to 92.7%, but under differential fuzzing only 31.4% of all generated function bodies that compile, pooled over models and samples, are equivalent to the original. Divergence at the contract level is the norm rather than the exception: for six of the seven models, between 13% and 31% of the generated contracts that compile diverge from the original under historical replay, and between 13% and 44% under DCR-guided replay; only Gemma-4-31B stays below 3% on both. Compilation is therefore a necessary condition for correctness and far from a sufficient one.

Figures & tables

Fig. 1: Compiling is not behaving. Given a Vault contract with its function bodies removed (top), a model regenerates them one at a time (bottom left), shown against the original implementation (bottom right). initialize and deposit are regenerated correctly (

=

); withdraw is not (

\neq

). Both versions compile, so compile-rate metrics cannot tell them apart; the behavioural oracles of our benchmark are built to catch exactly this divergence.

Panel of Figure 2 — Fig. 2: Distribution of non-trivial functions per contract at the two stages of benchmark construction. Large contracts dominate both the crawled set and the subset that passes the reproducibility check; the size filter retains the shaded band, 1 to 20 functions, which contains the 250 contracts of the final benchmark (37, 40, and 173 in its three buckets).

Panel of Figure 3 — Fig. 3: Part of the DCR graph extracted for Gifto . Dashed lanes group events by caller role; the dotted node is an isolated event; italic labels on relations are guards. The subgraph exhibits all four relation types.

	Compile@10
Model	Raw	Repaired	Gain
Gemma-4-31B	89.1%	92.7%	$+3.6$
Qwen3.6-27B	64.7%	71.3%	$+6.6$
Qwen3-Coder-30B	53.2%	61.8%	$+8.6$
Gemma-4-12B	48.0%	57.9%	$+9.8$
Ornith-35B	41.2%	52.1%	$+10.8$

TABLE II: Compile@10 before and after one repair pass, per model. Raw : share of the ten samples per contract that compile as generated; Repaired : after a single attempt in which the same model fixes its non-compiling samples from the compiler error; Gain : their difference in percentage points.

Relation	judgements			P	R	F1
	TP	FP	FN
condition $A\mathrel{\rightarrow\!\bullet}B$	65	3	12	0.96	0.84	0.90
response $A\mathrel{\bullet\!\rightarrow}B$	18	6	1	0.75	0.95	0.84
inclusion $A\mathrel{\rightarrow\!+}B$	21	23	0	0.48	1.00	0.65
exclusion $A\mathrel{\rightarrow\!\%}B$	39	25	6	0.61	0.87	0.72
all	143	57	19	0.72	0.88	0.79

TABLE III: Human validation of the DCR relations extracted by Gemma-4-31B on 30 contracts. An extracted relation is a true positive (TP) if the annotator confirms it in the contract’s source and a false positive (FP) otherwise; a false negative (FN) is a relation the annotator finds in the source that the extractor did not emit.

Model		O2			O3
					matched			fixed
	Compile@10	$N$	Status	$\hat{C}\neq C^{*}$	$N$	Status	$\hat{C}\neq C^{*}$	$N$	Status	$\hat{C}\neq C^{*}$
Gemma-4-31B	92.7%	234	99.6%	$+2$	174	98.4%	$+5$	174	98.4%	$+5$
Qwen3.6-27B	71.3%	214	96.9%	$+29$	157	96.7%	$+20$	162	95.8%	$+24$
Qwen3-Coder-30B	61.8%	183	95.5%	$+38$	125	96.1%	$+18$	136	95.5%	$+22$
Gemma-4-12B	57.9%	169	95.4%	$+25$	127	94.6%	$+16$	129	94.6%	$+16$

Fig. 4: Compilation, historical replay (O2) and DCR-guided replay (O3) per model. (a)

N

: contracts the oracle could compare; Status : mean share of replayed calls on which the regenerated contract

\hat{C}

agrees with the original

C^{*}

on success or revert;

\hat{C}\neq C^{*}

: contracts with at least one disagreement, shaded in proportion to the value. O3 is run with the DCR graphs extracted by the model itself ( matched ) and with those extracted by Gemma-4-31B for every model ( fixed ). (b) Each bar splits a model’s divergent contracts into those flagged by O2 only, by both oracles, and by O3 only, on matched and fixed graphs.

Extractor	O3 coverage	Contracts O3 $>$ O2	Functions covered by one oracle only
	O3 coverage	Contracts O3 $>$ O2	O3 only	O2 only
Gemma-4-31B	47.3%	84	311	511
Qwen3.6-27B	46.1%	81	305	520
Gemma-4-12B	44.6%	72	292	537
Devstral-24B	44.2%	77	301	535
Ornith-35B	41.6%	68	276	566

TABLE IV: Function coverage of O3 per extractor, against the fixed coverage of O2. Coverage : mean share of a contract’s state-changing functions that O3 calls at least once, with the graphs of that extractor (O2:

55.7\%

for every row). Contracts : contracts on which O3 covers more functions than O2. Last two columns: functions, out of

1{,}759

, that only one of the two oracles calls.

Model	O1: equivalent
	Functions	Contracts
Gemma-4-31B	33.4%	34.0%
Qwen3.6-27B	35.5%	34.4%
Qwen3-Coder-30B	26.6%	29.9%
Gemma-4-12B	18.8%	13.1%
Ornith-35B	16.9%	15.6%

TABLE V: Differential fuzzing (O1) per model, over the

N{=}244

contracts with a compiling sample and a fuzzable function. Functions : share of (function, sample) pairs where the regenerated function is equivalent. Contracts : share with a sample where every fuzzable function is equivalent.

Model	$N$	one oracle only			two oracles			all
		O1	O2	O3	O1,O2	O1,O3	O2,O3
Gemma-4-31B	170	93	0	0	0	3	0	2
Qwen3.6-27B	158	70	0	0	5	8	0	15
Qwen3-Coder-30B	133	50	0	0	15	7	0	14
Gemma-4-12B	126	87	0	0	10	5	0	11
Ornith-35B	143	99	0	0	7	6	0	21

TABLE VI: Divergent contracts by the combination of oracles that flag them, per model. Counted over the

N

contracts evaluated by all three oracles, with O3 on the graphs of the reference extractor. A contract is flagged by O1 if at least one fuzzable function is non-equivalent in at least one compiling sample, and by O2 or O3 if at least one sample disagrees with the original on the status of at least one transaction. The seven columns partition the contracts flagged by at least one oracle and therefore sum to at most

N

; the remainder agreed with the original under every oracle.

Functions per contract	Compile@10		s/sample
	Mean	Range
1–5	$+72.4\%$	54.2–88.9%	$+4$
6–10	$+49.0\%$	35.0–69.4%	$+13$
11–15	$+43.3\%$	24.2–64.2%	$+24$
16–20	$+40.8\%$	16.7–70.6%	$+41$
21–25	$+27.9\%$	3.9–46.4%	$+75$

TABLE VII: Compile rate and generation cost by contract size. Mean and Range : Compile@10 after repair, averaged over the seven models and between the worst and the best of them. s/sample : seconds of generation per compiling sample. The dashed line is the benchmark’s 20-function bound.

Code Generation Evaluation

Code generation evaluation measures whether AI-produced programs are correct, reliable, secure, efficient, and faithful to their specifications.

See all 214 papers

SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
May 10, 2026Abhinav Goel, Agostino Capponi, Alfio Gliozzo +1Code Generation Code Generation Evaluation
Columbia University New York, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA
SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
May 27, 2026Kaihua Qin, Dawn Song, Arthur GervaisLLM Evaluation Differential Testing
University of Warwick · UC Berkeley · University College London
Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts
Feb 17, 2026Eduardo Sardenberg, Antonio José Grandson Busson, Daniel de Sousa Moraes +2Software Vulnerability Detection LLMs for Cybersecurity
TeleMídia Lab - PUC-Rio, Rio de Janeiro, RJ - Brazil · BTG Pactual, São Paulo, SP - Brazil · Instituto Militar de Engenharia, Rio de Janeiro, RJ - Brazil

May 10, 2026J/K move · Enter open · S save

SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya Shah

Columbia University New York, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA

Code Generation Code Generation Evaluation LLM Code Generation

Open paper AskPDF arXiv

We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.

Abstract

Code Generation Evaluation

Explore similar work

SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts