Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models
Authors: Tenghui Huang, Jiawen Kang, Dongning Liu, Changyan Yi, Chengjun Cai, Anjia Yang, Li Li, Dong In Kim
Organizations: School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China · Department of Computer Science, City University of Hong Kong (Dongguan), Dongguan, Guangdong 518057, China · College of Cyber Security, Jinan University, Guangzhou, Guangdong 510632, China · Guangdong Institute of Science and Technology Information, Guangzhou, Guangdong, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, South Korea
Smart contract vulnerability detection with Large Language Models (LLMs) faces three causally linked challenges. First, new vulnerability categories demand parameter-efficient adaptation, since full retraining is prohibitive for sequentially arriving tasks. Second, training per-task adapters on a shared backbone causes catastrophic forgetting of previously learned vulnerabilities. Third, the resulting multiplicity of adapters must be consolidated into a single model, since task identity is unknown at inference time. Each challenge arises directly from the solution to its predecessor, making an integrated framework essential. We propose a three-stage pipeline in which each stage addresses one challenge and feeds into the next. The adaptation stage uses Frequency-Aware Low-Rank Adaptation (FA-LoRA), which performs adaptation in the Fourier domain with per-frequency importance gates, requiring only 0.4% trainable parameters while outperforming standard LoRA and QLoRA. The continual learning stage applies Forget-Aware Replay (FAR), which uses these frequency gates to estimate per-sample forgetting risk via loss dynamics and prioritizes vulnerable knowledge for rehearsal, achieving an average Micro-F1 of 0.8022 across sequential tasks. The deployment stage employs Anchor-Protected Progressive Merging (APPM), which exploits the asymmetric generalization produced by FAR training to identify the strongest-generalizing adapter as an anchor and consolidates all adapters into a single model via anchor-protected weighted merging with frequency-domain gate competition. APPM achieves a Micro-F1 of 0.8085, within 2.7% of the independent per-task upper bound, at a merge cost of 156 ms and no additional runtime memory. Experiments on DIVE confirm the framework effectively addresses all three challenges for evolving blockchain ecosystems.
Figures & tables
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 2: Architecture of FA-LoRA.
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 4: The two-stage APPM workflow.
Split
task_A
task_B
task_C
task_D
Train
5,262
5,262
5,262
5,262
Validation
542
504
536
651
Test
530
542
513
648
Total
6,334
6,308
6,311
6,561
TABLE I: DIVE benchmark statistics.
Hyperparameter
Value
FA-LoRA rank r [ 8 ]
16
Frequency mode [ 28 ]
High-frequency retention
Retain fraction γ [ 28 ]
0.2
Optimizer [ 46 ]
AdamW
Learning rate (CL) [ 8 ]
5×10−5
Learning rate (PEFT) [ 8 ]
3×10−5
TABLE II: Hyperparameter Configuration
Component
Params (M)
Fraction
Total
1,241.0
100.0%
Frozen (LLaMA base)
1,235.8
99.6%
Trainable (FA-LoRA)
5.2
0.4%
— LoRA ( U,V )
5.0
0.40%
— Gates ( g )
0.2
0.02%
Per-task storage
∼ 10 MB
TABLE III: Parameter breakdown of FA-LoRA.
Method
Quant.
Trainable (M)
Storage (MB)
Micro-F1
Macro-F1
Subset Acc.
1B
3B
1B
3B
1B
3B
WaRA [ 43 ]
✓
35.77
136.5
0.8398
0.8515
0.6529
0.7133
0.5544
0.5840
QLoRA [ 9 ]
✓
1.72
6.6
0.8185
0.8365
0.6133
0.6407
0.5181
0.5298
SLoRA [ 22 ]
✓
6.85
26.2
0.8138
0.8305
0.5945
0.6263
0.4966
0.5262
LoRA [ 8 ]
bf16
3.42
13.0
0.8094
0.8370
0.5613
0.6356
0.4845
0.5428
FourierFT [ 27 ]
✓
0.16
0.6
0.7449
0.7888
0.4724
0.5585
0.3471
0.4331
TABLE IV: Comparison of PEFT methods on the full DIVE dataset without task splitting.
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
Method
Merge (ms)
CPU Δ (MB)
Speedup
Δ Ind
Simple-Mean
102
0.0
70 ×
+5.5%
TIES [ 37 ]
2,123
0.0
3.4 ×
+5.5%
DARE [ 38 ]
4,204
0.0
1.7 ×
+7.9%
HAM g=2
7,164
68.1
1.0 ×
+11.0%
SFA a=0.5
72
0.0
100 ×
+33.0%
APPM (ours)
156
0.0
46 ×
+2.7%
TABLE V: Resource metrics for adapter merging methods. All methods produce identical model architectures.
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
After training
task_A
task_B
task_C
task_D
task_A
0.7495
0.6121
0.6331
0.6028
task_B
0.7253
0.7503
0.7303
0.7065
task_C
0.7162
0.7226
0.8974
0.8864
task_D
0.7128
0.7171
0.8935
0.8854
Forgetting
− 0.0367
− 0.0332
− 0.0039
—
TABLE VI: Backward evaluation matrix for FAR. Bold entries denote in-task F1, entries below the diagonal denote backward transfer, and entries above the diagonal denote forward transfer.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with Δ F1 = 0.0041. (b) Frequency retention γ is similarly insensitive with Δ F1 = 0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07 % monotonic improvement. (d) FAR temperature τ has negligible impact with Δ F1 = 0.0021.
Smart contracts on blockchains are prone to diverse security vulnerabilities that can lead to significant financial losses due to their immutable nature. Existing detection approaches often lack flexibility across vulnerability types and rely heavily on manually crafted expert rules. In this paper, we present an LLM-based framework for practical smart contract vulnerability detection. We construct and release a large-scale dataset comprising 31,165 professionally annotated vulnerability instances collected from over 3,200 real-world projects across 15 major blockchain platforms. Our approach leverages precise AST-based context extraction and vulnerability-specific prompt design to instantiate customized detectors for 13 prevalent vulnerability categories. Experimental results demonstrate strong effectiveness, achieving an average positive recall of 0.92 and an average negative recall of 0.85, highlighting the potential of carefully engineered contextual prompting for scalable and high-precision smart contract security analysis.
Xing Zhang, Keyu Zhang, Taohong Zhu +1
NetX Foundation Zug, Switzerland · Department of Computer Science University of Oxford Oxford, UK · Department of Computer Science The University of Manchester Manchester, UK
Smart contracts face critical security challenges that require thorough auditing in decentralized web services. While Large Language Models (LLMs) have shown promise in automated vulnerability detection, existing approaches lack severity evaluations with actionable remediation and demand unnecessarily massive computational overhead. In this study, we introduce an efficient end-to-end smart contract security audit framework utilizing lightweight, highly optimized open-source LLMs (0.6B-4B parameters). Our framework decouples comprehensive audit tasks into four interconnected components: vulnerability detection, explanation, severity classification, and remediation recommendation. To maintain high accuracy without massive parameters, we implement Rank-Stabilized Low-Rank Adapters (rsLoRA), knowledge distillation, and a custom Chain-of-Verification (CoVe) aggregation strategy to systematically screen and consolidate multiple draft responses from the model into a highly accurate audit report. Experimental results demonstrate that our lightweight pipeline consistently outperforms state-of-the-art open-source coder dense LLMs (7B to 34B parameters), achieving 98.25% accuracy in vulnerability detection and an alignment score of 0.4375 in generative explanation tasks. Furthermore, our extensive ablation studies empirically validate the superiority of our decoupled audit processes over unified prompting and uncover a novel severity centrality bias, establishing a critical benchmark for future research in LLM-assisted auditing.
Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan +1
Universitas Gadjah Mada, Indonesia · Monash University, Indonesia
Smart contracts play a central role in blockchain systems by encoding financial and operational logic. Still, their susceptibility to subtle security flaws poses significant risks of financial loss and erosion of trust. LLMs create new opportunities for automating vulnerability detection, yet the effectiveness of different prompting strategies and model choices in real-world contexts remains uncertain. This paper evaluates state-of-the-art LLMs on Solidity smart contract analysis using a balanced dataset of 400 contracts under two tasks: (i) Error Detection, where the model performs binary classification to decide whether a contract is vulnerable, and (ii) Error Classification, where the model must assign the predicted issue to a specific vulnerability category. Models are evaluated using zero-shot prompting strategies, including zero-shot, zero-shot Chain-of-Thought (CoT), and zero-shot Tree-of-Thought (ToT). In the Error Detection task, CoT and ToT substantially increase recall (often approaching ~ 95--99%), but typically reduce precision, indicating a more sensitive decision regime with more false positives. In the Error Classification task, Claude 3 Opus attains the best Weighted F1-score (90.8) under the ToT prompt, followed closely by its CoT.
Eduardo Sardenberg, Antonio José Grandson Busson, Daniel de Sousa Moraes +2
TeleMídia Lab - PUC-Rio, Rio de Janeiro, RJ - Brazil · BTG Pactual, São Paulo, SP - Brazil · Instituto Militar de Engenharia, Rio de Janeiro, RJ - Brazil