Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts
Authors: Eduardo Sardenberg, Antonio José Grandson Busson, Daniel de Sousa Moraes, Julio Cesar Duarte, Sérgio Colcher
Organizations: TeleMídia Lab - PUC-Rio, Rio de Janeiro, RJ - Brazil · BTG Pactual, São Paulo, SP - Brazil · Instituto Militar de Engenharia, Rio de Janeiro, RJ - Brazil
Smart contracts play a central role in blockchain systems by encoding financial and operational logic. Still, their susceptibility to subtle security flaws poses significant risks of financial loss and erosion of trust. LLMs create new opportunities for automating vulnerability detection, yet the effectiveness of different prompting strategies and model choices in real-world contexts remains uncertain. This paper evaluates state-of-the-art LLMs on Solidity smart contract analysis using a balanced dataset of 400 contracts under two tasks: (i) Error Detection, where the model performs binary classification to decide whether a contract is vulnerable, and (ii) Error Classification, where the model must assign the predicted issue to a specific vulnerability category. Models are evaluated using zero-shot prompting strategies, including zero-shot, zero-shot Chain-of-Thought (CoT), and zero-shot Tree-of-Thought (ToT). In the Error Detection task, CoT and ToT substantially increase recall (often approaching ~ 95--99%), but typically reduce precision, indicating a more sensitive decision regime with more false positives. In the Error Classification task, Claude 3 Opus attains the best Weighted F1-score (90.8) under the ToT prompt, followed closely by its CoT.
Figures & tables
Dataset
Total Files
Error Files
Error Rate (%)
DAppSCAN
200
200
100.0
JiuZhou
102
0
0.0
UniswapV2
12
0
0.0
UniswapV3
62
0
0.0
openZeppelin
24
0
0.0
Table 1: Dataset composition and vulnerability distribution
SWC Category
Count
Percent (%)
SWC-101-Integer Overflow and Underflow
70
18.23
SWC-135-Code With No Effects
64
16.67
SWC-104-Unchecked Call Return Value
39
10.16
SWC-114-Transaction Order Dependence
25
6.51
SWC-102-Outdated Compiler Version
25
6.51
SWC-128-DoS With Block Gas Limit
23
5.99
Table 2: Distribution of SWC vulnerability categories in the DAppSCAN dataset.
LLM
Prompt
Precision
Recall
F1-Score
gpt-4.1
zero-shot
70.12
89.73
78.83
gpt-4.1
zero-shot-cot
65.32
98.07
78.79
gpt-4.1
zero-shot-tot
64.10
99.08
78.06
Claude 3 Sonnet
zero-shot-cot
69.90
87.13
77.47
gpt-5
zero-shot-tot
62.56
98.57
76.61
Claude 3 Sonnet
zero-shot-tot
69.74
83.62
76.10
Table 3: Error detection (binary vulnerability classification) results across LLMs and zero-shot prompting strategies sorted by F1-score
LLM
Prompt
Micro
Macro
Weighted
P
R
F1
P
R
F1
P
R
F1
Claude 3 Opus
zero-shot-tot
89.7
89.7
89.7
78.1
78.1
77.1
94.8
89.7
90.8
Claude 3 Opus
zero-shot-cot
90.0
90.0
90.0
84.4
87.4
84.6
91.9
90.0
89.9
Claude 3 Opus
zero-shot
85.5
85.5
85.5
81.3
84.2
81.7
91.4
85.5
87.1
Claude 4.6 Opus
zero-shot-tot
82.4
82.4
82.4
70.3
66.8
67.1
93.1
82.4
85.4
gpt-4o
zero-shot-tot
82.8
82.8
82.8
83.9
85.0
81.2
92.5
82.8
83.4
Table 4: Multi-class error classification performance across LLMs and zero-shot prompting strategies, ordered by Weighted F1-score
Class
Count
Class
Count
SWC-101
22
SWC-105
5
SWC-114
21
SWC-108
5
SWC-135
21
SWC-113
5
SWC-103
19
SWC-115
5
SWC-104
18
SWC-112
3
SWC-107
18
SWC-119
3
Table 5: Distribution of SWC vulnerability categories predicted by our best model (Claude 3 Opus with Tree-of-Thought) for the evaluated subset of 200 smart contracts
Figure 1: Confusion matrix for the best-performing configuration (Claude 3 Opus with Tree-of-Thought), showing per-class confusions across SWC categories.
Figure 2: Weighted F1-score for the Error Classification task across evaluated LLMs under zero-shot, zero-shot-CoT, and zero-shot-ToT prompting strategies.
LLM
CoT F1-Score Gain
ToT F1-Score Gain
gpt-5.2
2.73
4.25
gpt-5
-2.32
4.11
gpt-4.1
-0.04
-0.77
gpt-4.1-mini
-0.52
-1.25
gpt-4.1-nano
1.78
1.72
gpt-4o
-3.91
-4.39
Table 6: F1-score gain of zero-shot-CoT and zero-shot-ToT over the zero-shot baseline in the Error Detection task. Gains are computed as ΔF1=F1prompt−F1zero-shot .
LLM
CoT F1-Score Gain
ToT F1-Score Gain
gpt-5
-8.5
9.5
o1
6.5
4.8
Claude 4.6 Opus
-8.5
4.9
o4-mini
6.3
4.3
gpt-4.1-mini
-9.5
4.2
gpt-4o-mini
-9.5
4.2
Table 7: Weighted F1-score gain of zero-shot-CoT and zero-shot-ToT over the zero-shot baseline in the Error Classification task. Gains are computed as ΔF1w=F1w,prompt−F1w,zero-shot , where F1w denotes the Weighted F1-score.
NetX Foundation Zug, Switzerland · Department of Computer Science University of Oxford Oxford, UK · Department of Computer Science The University of Manchester Manchester, UK