Small Language Models (SLMs) are increasingly attractive for resource-constrained environments due to their low computational footprint. These characteristics also make them candidates for future autonomous malware, where connectivity, latency, privacy, or operational constraints may preclude cloud-based inference. Yet the ability of ultra-compact models to support malware during post-exploitation remains poorly understood. We systematically evaluate seven open-source SLMs (20M-1.5B parameters) on two post-exploitation code-generation tasks: Directory Enumeration and Backdoor Generation. We compare Python (high-resource) and Lua (low-resource) across prompting strategies, task formulations, iterative reasoning, grammar-constrained decoding, and execution-feedback loops. Results show that offensive code-generation capabilities do not increase linearly with model size. Successful execution depends on the interaction of model capacity, task formulation, and prompting strategy, with task formulation exerting a greater influence than prompt engineering alone. Algorithmic formulation combined with few-shot prompting enables Qwen2.5-Coder 1.5B to reach Pass@5 of 100% in both Python and Lua for Directory Enumeration, 100% in Python for Backdoor Generation, and 99.2% in Lua. Although the evaluated models remain too computationally demanding for widespread deployment in contemporary malware, rapid progress in compact models may reduce this barrier. These findings help anticipate AI-assisted malware and inform future countermeasures.
Figures & tables
Figure 1: Research questions organizing the experimental evaluation of Small Language Models in post-exploitation scenarios. The study is structured around four complementary research dimensions: (i) architectural and domain limits, (ii) interaction strategies, (iii) autonomy and reflection mechanisms, and (iv) efficiency and failure analysis. Each dimension is addressed in the Experimental Evaluation and Discussion section through dedicated experiments.
Figure 2: Operational pipeline of the embedded Small Language Model (SLM)-powered malware. The schematic illustrates the autonomous, closed-loop execution flow designed for air-gapped environments. The central malicious core sequentially drives the process: (1) reading task-specific instructions; (2) invoking the embedded SLM to dynamically generate operational functional code; (3) validating the output through static rules and unit tests within an internal sandbox; and (4) delivering the verified payload to the target system.
Variable
Evaluated Values
Task
Directory Enumeration, Backdoor
Task Formulation
Basic, Constrained, Algorithmic
Prompt Strategy
Zero-Shot, Few-Shot, CoT, FS + CoT
Language
Python, Lua
Output Structure
None, EBNF Grammar
Table 1: Summary of experimental variables
Architecture
Params
Tuning
Target Domain
EleutherAI/pythia-70m
70M
Base
General
EleutherAI/pythia-160m
160M
Base
General
HF/SmolLM2-135M
135M
Instruct
General
HF/SmolLM2-360M
360M
Instruct
General
Qwen2.5-Coder-0.5B
0.5B
Instruct
Coder
Qwen2.5-Coder-1.5B
1.5B
Instruct
Coder
Table 2: List of SLMs evaluated.
Figure 3: Experimental methodology adopted to evaluate the offensive capabilities of compact language models. The upper panel summarizes the controlled experimental variables, including tasks, programming languages, model families, task formulations, prompting strategies, and structured decoding. The lower panel illustrates the three interaction paradigms investigated in this work: standard one-shot generation, execution-feedback interaction, and iterative Least-to-Most decomposition. For each paradigm, the evaluated variables and fixed experimental settings are explicitly reported to ensure a controlled comparison across all experiments.
Figure 4: Validation pipeline adopted to assess generated programs. Candidate scripts are first subjected to static and heuristic verification, including syntax validation, structural integrity checks, and task-specific constraint verification. Successfully validated programs are subsequently executed inside an isolated sandbox and evaluated through dynamic execution and unit tests.
Figure 5: High-level architecture of the experimental framework. The framework orchestrates the complete evaluation pipeline, from experiment configuration and task specification to prompt construction, SLM inference, code extraction, automated validation, and metric collection. Modular components enable language-specific prompt adaptation, output enforcement, and reproducible large-scale experimentation across different interaction paradigms.
Parameter
Value
Rationale / Application
Global Inference Parameters
Decoding Temperature
0.2
Low variance to preserve structural formatting while enabling narrow exploration for Pass@k.
Max Generation Tokens
1024
Sufficient context window to accommodate both reasoning (CoT) and final payloads.
Sampling Budget ( n )
10
Number of independent completions generated per prompt to compute the Pass@k estimator.
Macro-Runs
10
Number of full pipeline executions to calculate statistical mean and standard deviation ( ± ).
Table 4: Pass@1 and Pass@5 scores for the Directory Enumeration and Backdoor tasks.
Figure 6: Effect of the constrained task formulation on directory enumeration performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.
Figure 7: Effect of the constrained task formulation on backdoor generation performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.
Figure 8: Effect of task formulation on directory enumeration performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.
Figure 9: Effect of task formulation on backdoor generation performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.
Model
Size
Language
Prompting Strategy (Pass@5 %)
Core
Few Shot
Thinking
FS + Thinking
Task Formulation: BASIC
Qwen2.5-Coder
1.5B
Python
72.2 ± 11.7
100.0 ± 0.1
50.6 ± 21.2
32.8 ± 29.5
Lua
30.0 ± 25.8
0.0 ± 0.0
17.8 ± 29.6
5.0 ± 15.8
Qwen2.5-Coder
0.5B
Python
81.2 ± 17.6
85.5 ± 15.3
69.8 ± 31.8
76.4 ± 19.0
SmolLM2
360M
Python
53.9 ± 33.1
0.0 ± 0.0
10.0 ± 21.1
10.0 ± 21.1
Table 5: Cross-Analysis: Directory Enumeration. Impact of Prompting Strategies on Task Complexity (Pass@5 with Variance).
Model
Size
Language
Prompting Strategy (Pass@5 %)
Core
Few Shot
Thinking
FS + Thinking
Task Formulation: BASIC
Qwen2.5-Coder
1.5B
Python
5.0 ± 15.8
0.0 ± 0.0
43.3 ± 32.3
5.0 ± 15.8
Lua
0.0 ± 0.0
0.0 ± 0.0
10.0 ± 21.1
0.0 ± 0.0
Qwen2.5-Coder
0.5B
Python
19.2 ± 32.9
5.0 ± 15.8
25.6 ± 34.3
5.0 ± 15.8
Task Formulation: CONSTRAINED
Table 6: Cross-Analysis: Backdoor Generation. Impact of Prompting Strategies on Task Complexity (Pass@5 with Variance).
Figure 10: Impact of strict syntactic forcing via EBNF Guided Decoding on the Directory Enumeration task in Python (a) and Lua (b) and Backdoor task in Python(c) and Lua (d) under the basic task formulation
Figure 11: Performance comparison on the Direnum task across prompting strategies. The chart tracks the evolution of Pass@1 and Pass@5 metrics from standard One-Shot, to Algorithmic One-Shot, to Iterative (L2M) prompting. The Qwen 1.5B model (blue) demonstrates maximum efficacy at the Algorithmic stage, peaking at 99.8% (Python) and 91.1% (Lua) in Pass@5. However, enforcing the Iterative (L2M) strategy causes a noticeable regression, dropping performance back to 75.0%. Smaller models (¡ 1B parameters) show only marginal, transient gains from algorithmic hints before completely collapsing under multi-step constraints.
Figure 12: Performance trajectories on the Backdoor generation task. The inclusion of algorithmic instructions produces a dramatic but brittle spike in performance for the Qwen 1.5B model (blue), notably jumping from 0.0% to 90.0% Pass@5 in Lua. Moving to the Iterative (L2M) framework, however, causes a severe performance collapse, completely zeroing out Lua success rates and reducing Python gains. This volatility highlights the profound difficulty of maintaining malicious code constraints across multiple iterative reasoning steps, especially for smaller language models which fail entirely (0.0%) regardless of the prompting technique.
Figure 13: Iterative drop-off curve for the directory enumeration task. The Y-axis represents the survival rate, defined as the percentage of trajectories successfully maintaining a functional state up to a given iterative step.
Figure 14: Iterative drop-off curve for the backdoor generation task. The Y-axis represents the survival rate, defined as the percentage of trajectories successfully maintaining a functional state up to a given iterative step.
Figure 15: Sequential taxonomy of failure modes observed during iterative (Least-To-Most) malicious code generation. Failure categories follow the sequential evaluation pipeline adopted in our framework: Syntax Errors prevent successful parsing or code generation; Constraint Errors correspond to executable programs that violate the functional requirements of the task; Runtime Errors occur only after syntactically valid and functionally compliant code reaches execution (for the iterative it happens only when the last step is reached). Each cell reports the percentage of failures observed for Directory Enumeration (left value) and Backdoor Generation (right value), while the color intensity represents the maximum percentage between the two tasks, facilitating cross-task comparison across model sizes and programming languages.
Figure 16: Cumulative Pass Rate trend across the 10 steps of the iterative Reflection cycle (Language: PYTHON). Only the experimental configurations (Strategy × Task Formulation) that exhibited a positive responsiveness to error-message feedback (Pass Rate variation >0 ) are reported.
Figure 17: Cumulative Pass Rate trend across the 10 steps of the iterative Reflection cycle (Language: LUA). Only the experimental configurations (Strategy × Task Formulation) that exhibited a positive responsiveness to error-message feedback (Pass Rate variation >0 ) are reported.
Figure 18: Comprehensive evaluation of the computational overhead vs. code quality for both the directory enumeration task and backdoor task type. The x-axis displays the evaluated Large Language Models, while the y-axis represents the hierarchical nesting of Task Formulations and Prompting Strategies. The size of each marker is inversely proportional to the average number of generated tokens (larger markers indicate higher computational efficiency). The color gradient maps the success rate achieved via the Pass@5 metric. Circles ( ∘ ) and diamonds ( ⋄ ) distinguish between Lua and Python implementations, respectively.
Figure 19: Distribution of the earliest validation stage at which unsuccessful generations fail across model sizes and programming languages. Each generated program is evaluated through a sequential fail-first validation pipeline consisting of (i) syntax verification, (ii) task-specific constraint validation, and (iii) runtime execution. Evaluation terminates immediately after the first failed stage; consequently, each unsuccessful generation contributes to exactly one failure category. Cell values report the percentage of failed generations whose earliest failure occurred at each validation stage, expressed as Directory Enumeration — Backdoor Generation. This representation isolates the nature of failures among unsuccessful generations, independently of the overall success rate, allowing comparison of how model size, task complexity, and programming language influence where the generation process first breaks down.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Attribute
Specification
Task ID
DIR_ENUM
Target Domain
System Interaction / Post-Exploitation
Supported Languages
Lua ( DIR_ENUM ), Python ( DIR_ENUM_PY )
Objective
Implement a function enumerate_dir(target_path) that returns a collection (Table/List) of all files and subdirectories contained within the target path.
Evaluation Criteria
Isolated sandbox execution; Verification of return type; Positive assertion on the presence of a dynamically generated test file ( target_file.txt ) and subdirectory.
Implement a function start_tcp_server() that creates a TCP socket, binds it to a fixed port (7456), listens for connections, executes received commands on the host system, and sends the textual output back to the client.
Evaluation Criteria
Isolated sandbox execution with a mocked or threaded networking environment; Verification of socket initialization and binding; Positive assertion on the successful receipt of command output (e.g., executing echo ) by a simulated client.
Table 9: Computational Overhead and Inference Dynamics (Lua). Comparison of Pass@5, generated tokens, inference time, and throughput (TPS) across prompting strategies for the directory enumeration task. The data highlights two notable patterns: (1) Contextual Collapse in Qwen 0.5B under ”Few Shot + Thinking” (generating only ∼ 120 tokens before aborting), and (2) Over-generation in SmolLM2 360M during ”Thinking”, where it exhausts token limits without achieving a single success.
Model
Size
Lang
Strategy
Pass@5 (%)
Avg Tokens
Time (s)
TPS
Task Formulation: BASIC
Qwen2.5-Coder
1.5B
Lua
Core
0.0 ± 0.0
2790.7 ± 128.9
6.8 ± 1.0
416.8 ± 48.9
Few Shot
0.0 ± 0.0
2374.3 ± 252.6
7.6 ± 1.8
324.6 ± 61.4
Thinking
10.0 ± 21.1
4419.3 ± 458.9
12.2 ± 2.3
370.0 ± 56.4
FS + Thinking
0.0 ± 0.0
3235.3 ± 414.4
10.7 ± 3.3
322.5 ± 75.1
Qwen2.5-Coder
0.5B
Lua
Core
0.0 ± 0.0
3383.3 ± 337.9
7.5 ± 1.1
454.4 ± 52.7
Appendix
Table 10: Computational Overhead and Inference Dynamics (Lua). Comparison of Pass@5, generated tokens, inference time, and throughput (TPS) across prompting strategies for the backdoor generation task. The data highlights the recurrence of Contextual Collapse in Qwen 0.5B under ”Few Shot + Thinking” (drastically reducing generated tokens across all formulations) and the high computational cost of ”Thinking” strategies in SmolLM2 360M without corresponding performance gains.
Model
Size
Pass@1 (%)
Pass@5 (%)
Latent Retention
Python
Lua
Python
Lua
Task Formulation: BASIC
Qwen2.5-Coder
1.5B
18.0 ± 4.2
6.0 ± 5.2
72.2 ± 11.7
30.0 ± 25.8
41.5%
Qwen2.5-Coder
0.5B
25.0 ± 9.7
0.0 ± 0.0
81.2 ± 17.6
0.0 ± 0.0
0.0%
SmolLM2
360M
14.0 ± 10.7
0.0 ± 0.0
53.9 ± 33.1
0.0 ± 0.0
0.0%
Models < 300M*
–
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
N/A
Appendix
Table 11: Impact of Task Formulation on Directory Enumeration task across Python (High-Resource) and Lua (Low-Resource). The table illustrates how cognitive friction (Constrained) degrades baseline capabilities, while Process-Driven Scaffolding (Algorithmic) substantially reduces the observed Python–Lua performance gap (Retention ≥ 91.3%). Models under 300M parameters are aggregated as they consistently exhibit zero capability.
Model
Size
Pass@1 (%)
Pass@5 (%)
Latent Retention
Python
Lua
Python
Lua
Task Formulation: BASIC
Qwen2.5-Coder
1.5B
1.0 ± 3.2
0.0 ± 0.0
5.0 ± 15.8
0.0 ± 0.0
0.0%
Qwen2.5-Coder
0.5B
5.0 ± 9.7
0.0 ± 0.0
19.2 ± 32.9
0.0 ± 0.0
0.0%
SmolLM2
360M
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
N/A
Models < 300M*
–
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
0.0 ± 0.0
N/A
Appendix
Table 12: Impact of Task Formulation on Backdoor Generation across Python (High-Resource) and Lua (Low-Resource). The table highlights a severe vulnerability: while Constrained and Basic formulations yield minimal malicious code, the Process-Driven (Algorithmic) formulation inadvertently acts as a jailbreak, causing Qwen 1.5B’s backdoor success rate to skyrocket to 90.0% in Lua. Models under 300M parameters are aggregated as they consistently exhibit zero capability.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.
Malware analysis starts with the raw bytes of an executable program, and tools to "lift" these to higher-level representations, such as assembly, are expensive and subject to error. Large Language Models (LLMs) cannot process raw byte representations and answer questions about them. To this end, we present the first byte-native LLM. Based on a vocabulary expansion technique using a bespoke byte tokenizer, such a model is capable of responding to complex questions about malware binaries, with accuracies ranging from 69% for malware family classification to 98% for architecture classification. Our findings indicate that providing domain knowledge during training is essential for this application -- off-the-shelf models lack both accuracy and insight. We've deployed this emerging solution to a limited number of analysts to gather feedback for further improvements.
Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4
CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1
Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target. However, existing LLM security benchmarks treat a crash as exploitation success. That single binary outcome collapses the hard parts of exploitation: the transition from triggering a bug to constructing reusable primitives and control. We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags, from coverage and crash through sandbox primitives, arbitrary read/write, control-flow hijack, and arbitrary code execution. Each capability is verified by a deterministic oracle that uses a per-run randomized challenge-response for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution. We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened. We report three arms: <model,env> as the primary measurement of model-environment capability, <model,env, adaptive coaching> as a secondary arm that adds adaptive coaching to test whether targeted feedback shifts outcomes, and <model,env,harness> as an ablation that swaps in the model's native CLI to check whether vendor-side optimizations increase exploitation capabilities. Our results show a sharp capability split between publicly deployed frontier models and the private frontier. Across the 8 publicly deployed models tested, reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not. The private model shows arbitrary code execution on approximately half. Overall, results suggest that exploit construction against hardened targets is an emerging frontier capability.
Seunghyun Lee, David Brumley
Carnegie Mellon University · Pittsburgh, PA 15213 · Carnegie Mellon University and Bugcrowd +1