Model repositories such as Hugging Face increasingly distribute machine learning artifacts serialized with Python's pickle format, exposing users to remote code execution (RCE) risks during model loading. Recent defenses, such as PickleBall, rely on per-library policy synthesis that requires complex system setups and verified benign models, which limits scalability and generalization. In this work, we propose a lightweight, machine-learning-based scanner that detects malicious Pickle-based files without policy generation or code instrumentation. Our approach statically extracts structural and semantic features from Pickle bytecode and applies supervised and unsupervised models to classify files as benign or malicious. We construct and release a labeled dataset of 727 Pickle-based files from Hugging Face and evaluate our models on four datasets: our own, PickleBall (out-of-distribution), Hide-and-Seek (9 advanced evasive malicious models), and synthetic joblib files. Our method achieves 90.01% F1-score compared with 7.23%-62.75% achieved by the SOTA scanners (Modelscan, Fickling, ClamAV, VirusTotal) on our dataset. Furthermore, on the PickleBall data (OOD), it achieves 81.22% F1-score compared with 76.09% achieved by the PickleBall method, while remaining fully library-agnostic. Finally, we show that our method is the only one to correctly parse and classify 9/9 evasive Hide-and-Seek malicious models specially crafted to evade scanners. This demonstrates that data-driven detection can effectively and generically mitigate Pickle-based model file attacks.
Figures & tables
Fig. 2: Visualization of Pickle model remote code execution. Step (1) illustrates how easy it is to create the attack, Step (2) is the hex view, Step (3) visualizes how Fickling prints the attack steps, and Step (4) is the execution of the ransomware.
Fig. 5: Overview of the proposed detection pipeline. In the first stage, the Pickle-based model file is parsed using Python’s Pickletools library, and the extracted opcodes are converted into normalized opcode-frequency vectors. In the second stage, these vectors are evaluated using supervised or unsupervised learning techniques to determine whether the file is benign or malicious, ultimately producing a final prediction
Fig. 6: t-SNE representation of our curated dataset
Fig. 7: t-SNE representation of the PickleBall dataset
Dataset
Benign
Malicious
Curated (Hugging Face)
648
79
PickleBall [ 23 ]
255
82
Hide and Seek [ 25 ]
–
9
Malware
–
5
TABLE I: Overview of Pickle-based datasets used in this study.
Model
Unsupervised
Supervised
SOTA Scanners
LOF
IF
OC-SVM
RF
AE
CB
PS
C-AV
FIC
MS
VT
TP (%)
87.50
89.50
87.50
87.50
81.25
87.50
97.47
3.80
57.14
100.00
7.69
TN (%)
90.77
87.69
86.15
96.92
94.62
93.08
0.00
99.85
85.71
0.00
99.77
F1 (%)
66.67
60.87
58.33
90.01
86.68
83.72
19.15
7.23
62.75
18.66
14.04
TABLE II: Comparison of SafePickle Models and SOTA Scanners on our Curated Dataset.
Model
Unsupervised
Supervised
External Scanners
LOF
IF
OC-SVM
RF
AE
CB
PS
C-AV
FIC
MS
VT
PB
TP (%)
42.86
71.43
71.43
85.90
85.71
92.86
97.18
26.76
88.24
100.00
7.69
100.00
TN (%)
80.77
78.85
73.08
96.15
96.15
94.23
92.55
99.15
57.14
0.00
92.77
79.80
F1 (%)
40.00
57.14
52.63
90.94
90.93
91.37
86.79
41.35
54.88
39.14
11.81
76.09
TABLE III: Comparison of SafePickle Models and SOTA Scanners on PickleBall Dataset.
Model
SafePickle Supervised
SOTA Scanners
RF
AE
CB
PS
C-AV
FIC
MS
VT
PB
TP (%)
93.0
57.9
93.0
97.18
26.76
88.24
100.00
7.69
100.00
TN (%)
60.8
87.84
60.4
92.55
99.15
57.14
0.00
92.77
79.80
F1 (%)
81.22
75.12
81.15
86.79
41.35
54.88
39.14
11.81
76.09
TABLE IV: Cross-Dataset Evaluation on the PickleBall Dataset
Path
RF
CatBoost
AE
PickleScan
ClamAV
Fickling
ModelScan
VirusTotal
pkl
●
●
●
●
●
●
●
●
zip → pkl
●
●
●
●
○
○
●
○
zip → zip → pkl
●
●
◐
○
○
○
○
○
tar → pkl
●
◐
◐
○
○
○
○
◐
bz2 → pkl
●
●
◐
○
○
○
○
○
gz → pkl
●
●
◐
○
○
○
○
○
TABLE V: Scope of Pickle-Based Malware Detection Across File Loading Paths (Notation: ●= detected, ◐= scanned-not detected, ○= failed scan)
Scanner / Model
Avg. Runtime (ms)
SOTA Scanners
PickleScan
167.8
ClamAV
11,875.6
Fickling
44.4
ModelScan
2,776.8
VirusTotal
3,081.9
TABLE VI: Average Runtime per File for Detection Methods
Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditional on making a definitive judgment, ModelScan achieved 100% precision, recall, and F1. Fickling identified no unique true- positive families beyond those found by the combination of ModelAudit and ModelScan. Furthermore, for the 48 malicious families where ModelScan failed to complete its analysis, both ModelAudit and Fickling generated detections consistent with ground truth. These findings underscore the need to separate judgment accuracy from judgment availability, as well as incremental detection coverage from tool-level redundancy.
The growing reliance on pre-trained Machine Learning (ML) models has introduced new attack surfaces. Recent vulnerabilities demonstrate that malicious behavior can be embedded within model artifacts, often bypassing existing defenses. Current model-scanning solutions primarily rely on static, format-specific rules or known attack signatures, which limit their ability to generalize across frameworks and to detect novel exploitation paths. In contrast, we propose a solution that focuses on the effects an attack has on the host system executing the model and builds on foundational intuitions about ML model execution. In particular, we observe that ML models operate within well-defined lifecycle phases and that, within each phase, interactions with the host system are highly structured and predictable. We translate these intuitions into Moat, a dynamic lifecycle-aware approach for securing ML model execution, and instantiate this design in Re-Moat, our reference implementation. We evaluate Re-Moat across multiple ML frameworks using 77,974 real-world model artifacts from the Hugging Face Hub, 31 Proofs-of-Concept (PoCs) from CVEs, and 334 models from a state-of-the-art dataset, and compare it against state-of-the-art model-scanning solutions. Our results show that our approach detects all evaluated attack classes while maintaining a close-to-zero false-positive rate, validating our intuitions and motivating dynamic analysis for securing ML model execution.
Gabriele Digregorio, Marco Di Gennaro, Francesco Pastore +3
Malware continues to be a predominant operational risk for organizations, especially when obfuscation techniques are used to evade detection. Despite the ongoing efforts in the development of Machine Learning (ML) detection approaches, there is still a lack of feature compatibility in public datasets. This limits generalization when facing distribution shifts, as well as transferability to different datasets. This study evaluates the suitability of different data preprocessing approaches for the detection of Portable Executable (PE) files with ML models. The preprocessing pipeline unifies EMBERv2 (2,381-dim) features datasets, trains paired models under two training setups: EMBER + BODMAS and EMBER + BODMAS + ERMDS. Regarding model evaluation, both EMBER + BODMAS and EMBER + BODMAS + ERMDS models are tested against TRITIUM, INFERNO and SOREL-20M. ERMDS is also used for testing for the EMBER + BODMAS setup.
César Vieira, João Vitorino, Eva Maia +1
GECAD, ISEP, Polytechnic of Porto, Rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal