Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models
Authors: Tenghui Huang, Jiawen Kang, Dongning Liu, Changyan Yi, Chengjun Cai, Anjia Yang, Li Li, Dong In Kim
Organizations: School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China · Department of Computer Science, City University of Hong Kong (Dongguan), Dongguan, Guangdong 518057, China · College of Cyber Security, Jinan University, Guangzhou, Guangdong 510632, China · Guangdong Institute of Science and Technology Information, Guangzhou, Guangdong, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, South Korea
Smart contract vulnerability detection with Large Language Models (LLMs) faces three causally linked challenges. First, new vulnerability categories demand parameter-efficient adaptation, since full retraining is prohibitive for sequentially arriving tasks. Second, training per-task adapters on a shared backbone causes catastrophic forgetting of previously learned vulnerabilities. Third, the resulting multiplicity of adapters must be consolidated into a single model, since task identity is unknown at inference time. Each challenge arises directly from the solution to its predecessor, making an integrated framework essential. We propose a three-stage pipeline in which each stage addresses one challenge and feeds into the next. The adaptation stage uses Frequency-Aware Low-Rank Adaptation (FA-LoRA), which performs adaptation in the Fourier domain with per-frequency importance gates, requiring only 0.4% trainable parameters while outperforming standard LoRA and QLoRA. The continual learning stage applies Forget-Aware Replay (FAR), which uses these frequency gates to estimate per-sample forgetting risk via loss dynamics and prioritizes vulnerable knowledge for rehearsal, achieving an average Micro-F1 of 0.8022 across sequential tasks. The deployment stage employs Anchor-Protected Progressive Merging (APPM), which exploits the asymmetric generalization produced by FAR training to identify the strongest-generalizing adapter as an anchor and consolidates all adapters into a single model via anchor-protected weighted merging with frequency-domain gate competition. APPM achieves a Micro-F1 of 0.8085, within 2.7% of the independent per-task upper bound, at a merge cost of 156 ms and no additional runtime memory. Experiments on DIVE confirm the framework effectively addresses all three challenges for evolving blockchain ecosystems.
Figures & tables
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 2: Architecture of FA-LoRA.
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 4: The two-stage APPM workflow.
Split
task_A
task_B
task_C
task_D
Train
5,262
5,262
5,262
5,262
Validation
542
504
536
651
Test
530
542
513
648
Total
6,334
6,308
6,311
6,561
TABLE I: DIVE benchmark statistics.
Hyperparameter
Value
FA-LoRA rank r [ 8 ]
16
Frequency mode [ 28 ]
High-frequency retention
Retain fraction γ [ 28 ]
0.2
Optimizer [ 46 ]
AdamW
Learning rate (CL) [ 8 ]
5×10−5
Learning rate (PEFT) [ 8 ]
3×10−5
TABLE II: Hyperparameter Configuration
Component
Params (M)
Fraction
Total
1,241.0
100.0%
Frozen (LLaMA base)
1,235.8
99.6%
Trainable (FA-LoRA)
5.2
0.4%
— LoRA ( U,V )
5.0
0.40%
— Gates ( g )
0.2
0.02%
Per-task storage
∼ 10 MB
TABLE III: Parameter breakdown of FA-LoRA.
Method
Quant.
Trainable (M)
Storage (MB)
Micro-F1
Macro-F1
Subset Acc.
1B
3B
1B
3B
1B
3B
WaRA [ 43 ]
✓
35.77
136.5
0.8398
0.8515
0.6529
0.7133
0.5544
0.5840
QLoRA [ 9 ]
✓
1.72
6.6
0.8185
0.8365
0.6133
0.6407
0.5181
0.5298
SLoRA [ 22 ]
✓
6.85
26.2
0.8138
0.8305
0.5945
0.6263
0.4966
0.5262
LoRA [ 8 ]
bf16
3.42
13.0
0.8094
0.8370
0.5613
0.6356
0.4845
0.5428
FourierFT [ 27 ]
✓
0.16
0.6
0.7449
0.7888
0.4724
0.5585
0.3471
0.4331
TABLE IV: Comparison of PEFT methods on the full DIVE dataset without task splitting.
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
Method
Merge (ms)
CPU Δ (MB)
Speedup
Δ Ind
Simple-Mean
102
0.0
70 ×
+5.5%
TIES [ 37 ]
2,123
0.0
3.4 ×
+5.5%
DARE [ 38 ]
4,204
0.0
1.7 ×
+7.9%
HAM g=2
7,164
68.1
1.0 ×
+11.0%
SFA a=0.5
72
0.0
100 ×
+33.0%
APPM (ours)
156
0.0
46 ×
+2.7%
TABLE V: Resource metrics for adapter merging methods. All methods produce identical model architectures.
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
After training
task_A
task_B
task_C
task_D
task_A
0.7495
0.6121
0.6331
0.6028
task_B
0.7253
0.7503
0.7303
0.7065
task_C
0.7162
0.7226
0.8974
0.8864
task_D
0.7128
0.7171
0.8935
0.8854
Forgetting
− 0.0367
− 0.0332
− 0.0039
—
TABLE VI: Backward evaluation matrix for FAR. Bold entries denote in-task F1, entries below the diagonal denote backward transfer, and entries above the diagonal denote forward transfer.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with Δ F1 = 0.0041. (b) Frequency retention γ is similarly insensitive with Δ F1 = 0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07 % monotonic improvement. (d) FAR temperature τ has negligible impact with Δ F1 = 0.0021.
NetX Foundation Zug, Switzerland · Department of Computer Science University of Oxford Oxford, UK · Department of Computer Science The University of Manchester Manchester, UK
TeleMídia Lab - PUC-Rio, Rio de Janeiro, RJ - Brazil · BTG Pactual, São Paulo, SP - Brazil · Instituto Militar de Engenharia, Rio de Janeiro, RJ - Brazil