MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports
Organizations: Zhejiang University Hangzhou, China · University College London London, United Kingdom · Jiangsu Police Institute Nanjing, China · Alibaba Group Beijing, China
Abstract
Mapping behavioral intelligence in malware analysis reports to ATT&CK techniques supports threat analysis and defense, but reliable identification remains challenging. Supporting evidence is often scattered across long, heterogeneous reports, while closely related techniques share behavioral cues and differ only in subtle but decisive details. While large language models (LLMs) can use external knowledge to interpret reports, they may assign techniques based on plausible associations rather than sufficient evidence. We present MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification. Its core is a shared behavioral representation connecting report observations to technique definitions. This representation is built through agentic mining of technique documentation and supports two online stages: 1) primitive-guided extraction uses behavioral primitives to structure LLM-extracted observations, link them to source passages, and group dispersed evidence; 2) evidence-aligned retrieval matches this evidence to technique profiles through the same concepts, retaining likely candidates and confusable alternatives for LLM comparison of distinguishing behaviors. Controls at both stages validate evidence references and check technique-specific requirements. The representation also supports adaptation without parameter updates, where offline agentic audits diagnose concept weaknesses and refine extraction or retrieval policies. We evaluate MEvince against 11 baselines spanning four method families using a dataset of 610 public malware reports referenced by MITRE ATT&CK. MEvince improves exact-ID F1 over the strongest baseline by 17.4%, and its concept-guided retrieval increases gold-label candidate coverage from 31.6% to 47.9%. With a small adaptation set, offline refinement yields further gains and outperforms baselines that update model parameters.
Figures & tables
| Method | Representative Correct Mappings | Representative Errors (FNs / FPs) | Observation |
|---|---|---|---|
| Direct | Replication Through Removable Media ( T1091 ); PowerShell ( T1059.001 ); Web Protocols ( T1071.001 ); Registry Run Keys / Startup Folder ( T1547.001 ) | 16 FNs , including Query Registry ( T1012 ), Process Discovery ( T1057 ), Modify Registry ( T1112 ), and Non-Standard Port ( T1571 ); 3 FPs | Captures five salient local behaviors, but misses when the mapping depends on composing weaker evidence distributed across the report. |
| CTIKG | [Sub-technique Level] Adds: File Deletion ( T1070.004 ) | 15 FNs (1 fewer) ; 6 FPs (3 more) , including Spearphishing Attachment ( T1566.001 ), Archive via Utility ( T1560.001 ), and Local Data Staging ( T1074.001 ) | Preserves structured intermediates with local relations, while higher pipeline complexity does not clearly improve final mapping quality. |
| RAG | [Parent-technique Level] Adds: Ingress Tool Transfer ( T1105 ); Dynamic Resolution ( T1568 ) | 18 FNs (2 more) ; 12 FPs (9 more) , including AutoHotKey & AutoIT ( T1059.010 ), Remote Access Tools ( T1219 ), Active Setup ( T1547.014 ), and Hide Infrastructure ( T1665 ) | Recovers additional true parent-level positives, but also raises substantially more FPs, suggesting that the model tends to overcommit to retrieved candidates even when evidence is weak. |
| Method | Exact ATT&CK IDs | Parent IDs | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Type | Name | F1 | Recall | Prec. | Jacc. | F1 | Recall | Prec. | Jacc. |
| Classifier | RoBERTa | 0.250 | 0.234 | 0.294 | 0.158 | 0.351 | 0.369 | 0.389 | 0.233 |
| CySecBERT | 0.140 | 0.214 | 0.111 | 0.079 | 0.295 | 0.503 | 0.225 | 0.183 | |
| Extraction | NER-base | 0.105 | 0.166 | 0.091 | 0.059 | 0.151 | 0.212 | 0.139 | 0.089 |
| NER-full | 0.144 | 0.268 | 0.108 | 0.082 | 0.214 | 0.347 | 0.172 | 0.127 | |
| AttacKG | 0.038 | 0.069 | 0.028 | 0.020 | 0.099 | 0.180 | 0.072 | 0.055 | |
| Exact ATT&CK IDs | ||||
| Decision stage | F1 | Recall | Prec. | Jacc. |
| Mapping accepts only | 0.306 | 0.308 | 0.366 | 0.194 |
| document verifier | 0.310 | 0.302 | 0.384 | 0.200 |
| Parent IDs | ||||
| Mapping accepts only | 0.383 | 0.384 | 0.458 | 0.256 |
| document verifier | 0.386 | 0.383 | 0.463 | 0.260 |
| Method | Exact ATT&CK IDs | Parent IDs | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Type | Name | F1 | Recall | Prec. | Jacc. | F1 | Recall | Prec. | Jacc. |
| Classifier | RoBERTa | 0.285 | 0.271 | 0.328 | 0.183 | 0.358 | 0.365 | 0.391 | 0.237 |
| CySecBERT | 0.199 | 0.187 | 0.242 | 0.119 | 0.306 | 0.313 | 0.343 | 0.195 | |
| SFT | Llama-3.1-8B | 0.260 | 0.294 | 0.263 | 0.161 | 0.360 | 0.400 | 0.366 | 0.238 |
| Llama-3.1-70B | 0.292 | 0.337 | 0.296 | 0.187 | 0.385 | 0.444 | 0.389 | 0.260 | |
| Qwen2.5-14B | 0.291 | 0.331 | 0.293 | 0.184 | 0.368 | 0.429 | 0.366 | 0.241 | |
| Shortest | Median | Longest | ||||
|---|---|---|---|---|---|---|
| Method | Calls | Tokens (K) | Calls | Tokens (K) | Calls | Tokens (K) |
| Direct Prompt | 1 | 0.97 | 1 | 3.55 | 1 | 10.11 |
| RAG | 4 | 4.25 | 24 | 25.26 | 72 | 79.01 |
| CTIKG | 39 | 35.01 | 249 | 254.30 | 794 | 791.34 |
| MEvince | 6 | 13.28 | 12 | 41.29 | 45 | 144.11 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Count | Role | Representative entries |
|---|---|---|---|
| Action | 36 | What the threat does; preserves the behavioral predicate. | execute; modify; query; inject; connect |
| Object | 94 | What the action affects; identifies the artifact or resource. | process; file; registry key; credential; network configuration |
| Goal | 18 | Why the behavior is performed; separates similar operations with different purposes. | persistence; discovery; credential access; command and control |
| Mechanism | 302 | How the behavior is carried out; captures technique-specific means. | Run-key startup; PowerShell; process hollowing; non-standard port |
| Role template | 34 | Combines concepts that describe the same behavior. | registry persistence; fileless execution; discovery; C2; exfiltration |
| Coupling motif | 17 | Connects related behaviors described in nearby evidence units. | delivery–execution; Run-key–fileless execution; C2–exfiltration |
| Shortest | Median | Longest | ||||
|---|---|---|---|---|---|---|
| Method | Input | Output | Input | Output | Input | Output |
| Direct Prompt | 0.65 | 0.32 | 2.91 | 0.64 | 9.18 | 0.93 |
| RAG | 3.81 | 0.44 | 22.92 | 2.34 | 70.90 | 8.11 |
| CTIKG | 28.54 | 6.47 | 206.02 | 48.28 | 648.18 | 143.16 |
| MEvince | 9.62 | 3.66 | 28.49 | 12.81 | 99.25 | 44.86 |
Malware Analysis
Malware analysis studies how to detect, classify, attribute, and understand malicious software and its behavior.
See all 64 papers