Organizations: Shanghai Artificial Intelligence Laboratory · The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · Wuhan University
AI agents can inspect code and run tests to support smart contract security. SlowMist reports about $38.6 billion in losses across the broader blockchain ecosystem. Existing evaluations can reward memorized or copied answers, including attack code recovered from public blockchain transactions. Reconstructing contracts and validating tasks also require costly manual effort. To address these issues, we propose CONSTRAINTDIAL, a framework for lifelong construction of smart contract security tasks via protocol mutation. It automates local contract reconstruction and uses attack execution to select changes to the required calls or inputs. Execution checks guide revisions that make the original attack fail while preserving exploitability, the security objective, and normal functionality. New incidents and audit findings enter the same procedure. With CONSTRAINTDIAL, we construct 3,045 task inputs from 203 public cases, covering detect, exploit, and patch. Compared with reconstruction alone, protocol mutation reduces success with supplied public answers by 10.9 percentage points on average across detect, exploit, and patch. The fraction of tasks solved by copying online answers falls by 9.3 points.
Figures & tables
Work
D
E
P
Tasks >3 k
Protocol mutation
Lifelong
Full automation
SCONE-bench
✗
✓
✗
✗
✗
✗
✗
EVMbench
✓
✓
✓
✗
✗
✗
✗
CyberChainBench
✓
✓
✓
✗
✗
✗
✗
Ours
✓
✓
✓
✓
✓
✓
✓
Table 1: Comparison of benchmarks. D/E/P denote detect/exploit/patch. ✓ / ✗ mean reported/not reported.
Figure 1: Overview of ConstraintDial . Protocol reconstruction, mutation, and execution checks support lifelong construction of detect, exploit, and patch tasks.
Figure 2: Contract counts.
Task
Model
Base
Equivalent control
Change 1
Change 2
Combined changes
Overall
Detect
GLM-5.2
37.7
31.1
31.1
34.4
37.7
34.4
Qwen3.8-Max
39.3
39.3
36.1
19.7
42.6
35.4
DeepSeek-V4.1-Flash
49.2
49.2
41.0
44.3
49.2
46.6
GLM-5.3-Flash
31.1
32.8
32.8
31.1
29.5
31.5
Qwen3.8-27B
29.5
29.5
29.5
27.9
26.2
28.5
Gemma-4-31B-IT
26.2
29.5
26.2
24.6
26.2
26.6
Table 2: One-hour success rates (%) on the same 305 tasks per task type. Each version type contains 61 tasks. Cell shading uses one shared 0–100% scale.
Figure 3: Success by 30 and 60 minutes on 305 tasks per panel and model. The 30-minute successes are a subset of the 60-minute successes.
Figure 4: Mean cost per task versus 1-hour success rate (upper left is better).
Figure 5: Success with usual materials and with a historical answer supplied. Every bar uses 61 cases, with open network access in all conditions.
Figure 6: SMOOFSStaking code excerpts. Shading marks the NFT return before position deletion and the agent’s repeated withdrawal. Other checks and calls are omitted.
Method
Accepted
Rate (%)
Simple preparation
105
86.1
Direct rewriting
65
53.3
Full method
100
82.0
Without execution traces
73
59.8
Without paired checks
75
61.5
Without test feedback
61
50.0
Table 3: Versions accepted for all 3 tasks.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Version type
Changed condition
Original attack
Base
Original local calling conditions
Succeeds
Equivalent control
Pause on a transfer path outside the reference attack
Succeeds
Change 1
Registration before the vulnerable deposit
Fails
Change 2
Open the schedule before advancing it
Fails
Combined changes
Both registration and schedule opening
Fails
Appendix
Table 4: Required checks for the reward allocation family. The reference attack is mechanically ported before comparison, producing the original attack. Every accepted changed protocol version must also admit an adapted attack and preserve the required normal operations.
Model
Location
Report
Trigger
Submission coverage
False positives
k/n
GLM-5.2
40.3
34.4
28.9
95.4
9.6
28/291
Qwen3.8-Max
41.0
35.4
30.5
95.7
8.2
24/292
DeepSeek-V4.1-Flash
51.5
46.6
41.3
99.3
6.3
19/303
GLM-5.3-Flash
38.0
31.5
25.6
94.8
9.0
26/289
Qwen3.8-27B
36.4
28.5
22.0
92.8
11.3
32/283
Gemma-4-31B-IT
34.1
26.6
20.3
91.1
10.8
30/278
Appendix
Table 5: Detect diagnostic rates (%). False positives use submitted repaired-control reports as their denominator.
Model
Build
Attack blocking
Normal operations
Joint
GLM-5.2
93.8
60.0
60.0
47.9
Qwen3.8-Max
95.4
68.2
65.9
55.1
DeepSeek-V4.1-Flash
97.7
72.5
73.8
62.3
GLM-5.3-Flash
91.5
55.7
53.8
42.3
Qwen3.8-27B
87.9
44.9
43.3
32.5
Gemma-4-31B-IT
91.1
47.5
47.2
36.1
Appendix
Table 6: Patch component success rates (%) on 305 tasks per model.
Detect
Exploit
Patch
Model
Time
Cost
Time
Cost
Time
Cost
GLM-5.2
25.5
0.29
41.1
1.05
32.6
0.99
Qwen3.8-Max
21.8
0.20
36.5
0.84
33.0
0.91
DeepSeek-V4.1-Flash
10.3
0.03
12.1
0.21
14.3
0.22
GLM-5.3-Flash
26.7
0.03
43.6
0.08
35.4
0.09
Qwen3.8-27B
29.6
0.19
42.1
0.68
40.6
0.72
Appendix
Table 7: Mean session duration (min) and cost (USD) over 305 sessions per model and task type.
Model
Single
95% interval
Combined
95% interval
Improved
Worse
Detect
GLM-5.2
0.0
[−14.7,14.7]
0.0
[−6.6,6.6]
2
2
Qwen3.8-Max
-3.3
[−16.4,9.8]
3.3
[−8.2,14.8]
7
5
DeepSeek-V4.1-Flash
-8.2
[−18.8,2.4]
0.0
[−6.6,6.6]
2
2
GLM-5.3-Flash
0.0
[−10.2,10.2]
-1.6
[−11.5,8.2]
4
5
Qwen3.8-27B
0.0
[−11.2,11.2]
-3.3
[−13.1,6.6]
4
6
Appendix
Table 8: Paired success differences in percentage points. Single compares Change 1 with Equivalent control. Combined compares Combined changes with Base.
Model
Condition
Successes
Identifiers only
Extra calls only
Other calls or inputs
Qwen3.8-Max
Simple preparation
47
0
42
5
Qwen3.8-Max
Protocol mutation
40
0
11
29
DeepSeek-V4.1-Flash
Simple preparation
53
0
49
4
DeepSeek-V4.1-Flash
Protocol mutation
48
0
17
31
Appendix
Table 9: Changes to successful exploit submissions with a public answer supplied.
Model / task
Historical
Base
Simple preparation
Protocol mutation
Qwen3.8-Max / Detect
53/57
37/45
32/44
27/40
DeepSeek-V4.1-Flash / Detect
57/60
47/56
44/54
36/49
Qwen3.8-Max / Exploit
50/55
17/49
0/47
0/40
DeepSeek-V4.1-Flash / Exploit
55/59
23/55
0/53
0/48
Qwen3.8-Max / Patch
49/54
34/50
31/49
25/44
DeepSeek-V4.1-Flash / Patch
54/58
41/55
38/54
32/49
Appendix
Table 10: Successful submissions with unchanged answer logic, as a fraction of all successful submissions.
Task
Environment
Answer found
Answer copied
Successful copies
Final successes after copying
Detect
Historical
111
88
70 (57.4)
80
Detect
Base
89
66
28 (23.0)
34
Detect
Protocol mutation
80
54
23 (18.9)
28
Exploit
Historical
112
94
68 (55.7)
77
Exploit
Base
91
69
21 (17.2)
42
Exploit
Protocol mutation
81
55
6 (4.9)
31
Appendix
Table 11: Online answer retrieval and copying without supplied answers. Each row combines 122 sessions from two models. Successful copies are counts (percentages) and allow identifier changes only.
Figure 7: Four information conditions on Base and Combined changes. Answer denotes a supplied Base answer. Every bar uses ten cases.
Task and transfer
Historical
Base
Combined
Detect: Unchanged
10
5
4
Detect: Mechanical transfer
10
8
7
Exploit: Unchanged
10
1
0
Exploit: Mechanical transfer
10
5
1
Patch: Unchanged
10
3
2
Patch: Mechanical transfer
10
7
6
Appendix
Table 12: Direct transfer successes out of ten public answers per task type.
Batch
Inspected
Constructed
Versions
Detect
Exploit
Patch
Total
Rate (%)
1
24
20
100
98
100
96
294
83.3
2
21
20
100
99
100
97
296
95.2
Total
45
40
200
197
200
193
590
88.9
Appendix
Table 13: Task construction from additional historical cases.
Table 14: Cumulative complete additions across 61 cases as the budget increases.
Method
Protocol
Detect
Exploit
Patch
Common
Mean revisions
Total cost (USD)
Cost per accepted version (USD)
Simple preparation
117
114
116
109
105
0.7
47.58
0.45
Direct rewriting
83
76
81
70
65
1.5
112.24
1.73
Full method
110
106
109
103
100
1.4
150.06
1.50
Without execution traces
89
84
86
76
73
1.9
142.74
1.96
Without paired checks
94
87
91
80
75
1.1
122.00
1.63
Without test feedback
81
72
78
65
61
0.0
95.16
1.56
Appendix
Table 15: Construction outcomes and effort from 122 proposals per method on the same 61 public cases.