The use of ML in cybersecurity has long been impaired by generalization issues: Models that work well in controlled scenarios fail to maintain performance in production. The root cause often lies in ML algorithms learning superficial patterns (shortcuts) rather than underlying cybersecurity concepts. We investigate contrastive multi-modal learning as a first step towards improving ML performance in cybersecurity tasks. We aim at transferring knowledge from data-rich modalities, such as text, to data-scarce modalities, such as payloads. We set up a case study on threat classification and propose a two-stage multi-modal contrastive learning framework that uses textual vulnerability descriptions to guide payload classification. First, we construct a semantically meaningful embedding space using contrastive learning on descriptions. Then, we align payloads to this space, transferring knowledge from text to payloads. We evaluate the approach on a large-scale private dataset and a synthetic benchmark built from public CVE descriptions and LLM-generated payloads. The methodology appears to reduce shortcut learning over baselines on both benchmarks. We release our synthetic benchmark and source code as open source.
Figures & tables
Fig. 1 : High apparent accuracy masks poor generalization in threat classification. While some models exceed 0.92 accuracy in the easy scenario (solid bars), performance drops by almost 30% in the challenging scenario (dashed bars). SALM achieves the highest score in the challenging scenario.
Fig. 2 : Two-stage training procedure of SALM. Top : Contrastive learning structures the text embedding space around vulnerability types. Bottom : Modal alignment transfers this structure to payloads via a frozen text encoder, producing a unified semantic space.
Method
Real
Synthetic
Acc.
Macro F1
Acc.
Macro F1
TF-IDF + RF
65.7
29.8
16.0
9.0
FT CodeBERT + MLP
62.3
26.0
20.6
13.2
Embedding Similarity
48.9
20.9
8.1
4.3
SALM
68.1
30.1
24.4
14.3
Relative Improvement
+4%
+1%
+19%
+8%
TABLE I : Performance (%) on real and synthetic sets. Relative improvements computed against best baseline per metric.
Fig. 3 : t-SNE projections of vulnerability-description embeddings. Dots represent individual descriptions; Stars mark the generic anchor for each vulnerability type. Stage 1 contrastive learning structures the space around these anchors.
Fig. 4 : t-SNE projections of payload embeddings across training stages (dots are descriptions, stars are payloads). Semantic structure emerges only after Stage 2.
Class
FT CB
Embed.
TF-IDF
SALM
Code-execution
51.5
43.9
60.2
50.3
Dir-traversal
63.0
46.6
60.1
78.6
DoS
–
–
–
–
Info-Disclosure
57.3
51.3
63.0
70.8
Injection
73.4
55.7
69.8
69.2
RFI
–
–
–
–
TABLE II : Per-class F1-scores (%) on the challenging temporal split. Only classes present in this partition are shown. A dash (–) indicates zero F1. Best results per class in bold.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Vulnerability Type
Prototype Description
Vulnerability Type
Prototype Description
Backdoor
A hidden entry point allowing unauthorized system access. Data breaches and loss of control.
Injection
A vulnerability that lets attackers inject malicious code. Unauthorized database manipulation and system compromise.
Botnet
A network of compromised computers controlled remotely. Large-scale cyber-attacks and unauthorized system exploitation.
Overflow
A memory-related flaw causing crashes or unauthorized access. Unexpected execution paths and system instability.
CGI
A flaw in web applications enabling unauthorized actions. Data theft, code execution, and service disruption.
Remote-File-Inclusion
A vulnerability permitting attackers to include external malicious files. Unauthorized remote file execution and system compromise.
Code-Execution
A security flaw allowing arbitrary code execution. Unauthorized system control and potential data breaches.
Scanner
Automated tools probing for system vulnerabilities. Potential exposure of security gaps exploitable by attackers.
Dir-Traversal
A security issue enabling attackers to retrieve sensitive files. Unauthorized file access and system compromise.
Trojan
Malware disguised as legitimate software for system compromise. Unauthorized access and data theft.
DoS
An attack aimed at rendering a system unusable. Network disruption and resource exhaustion.
Webshell
A hidden backdoor allowing remote system control. Data manipulation and complete system takeover.
Appendix
TABLE III : Textual vulnerability-type prototypes used as anchors during semantic retrieval. Each description provides a concise characterization of the vulnerability type for the contrastive learning stage.
Class
Description
Example CWE
Backdoor
Traffic associated with hidden methods to gain unauthorized remote access
CWE-912
Botnet
Traffic from compromised systems controlled by attackers for coordinated attacks
N/A
CGI
Payload targeting CGI script vulnerabilities due to improper input validation
CWE-77, CWE-20
Code-execution
Payload carrying attempts to execute arbitrary commands or code on the target system
CWE-94
Dir-traversal
Path traversal payload attempting to access files outside the web root directory
CWE-22
DoS
Denial of Service traffic attempting to exhaust resources and make them unavailable
CWE-400
Appendix
TABLE IV : The 15 vulnerability classes used for threat classification. CWE mappings are approximate as the taxonomy reflects operational requirements rather than strict CWE alignment.
Vulnerability Type
Payloads (Real)
Payloads (Synthetic)
Descriptions
Backdoor
586
1,000
946
Botnet
978
0
676
CGI
192
1,000
9
Code-execution
166,359
0
7,805
Dir-traversal
90,636
1,000
811
DoS
610
1,000
1,164
Appendix
TABLE V : Dataset statistics per class. Payloads in the operational dataset reflect labels provided by the operator’s deployment; synthetic payloads are balanced for controlled evaluation. Descriptions drive Stage 1 contrastive learning.
Fig. 5 : Traditional TF-IDF + Random Forest pipeline.