From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation

Gabriel Hortea; Simon Pietro Romano; Juan Tapiador; Mario Varlese

cs.CROct 16, 2026

From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation

Authors: Gabriel Hortea, Simon Pietro Romano, Juan Tapiador, Mario Varlese

Organizations: Università degli Studi di Napoli Federico II, Naples, Italy · Universidad Carlos III de Madrid, Madrid, Spain

Abstract

Small Language Models (SLMs) are increasingly attractive for resource-constrained environments due to their low computational footprint. These characteristics also make them candidates for future autonomous malware, where connectivity, latency, privacy, or operational constraints may preclude cloud-based inference. Yet the ability of ultra-compact models to support malware during post-exploitation remains poorly understood. We systematically evaluate seven open-source SLMs (20M-1.5B parameters) on two post-exploitation code-generation tasks: Directory Enumeration and Backdoor Generation. We compare Python (high-resource) and Lua (low-resource) across prompting strategies, task formulations, iterative reasoning, grammar-constrained decoding, and execution-feedback loops. Results show that offensive code-generation capabilities do not increase linearly with model size. Successful execution depends on the interaction of model capacity, task formulation, and prompting strategy, with task formulation exerting a greater influence than prompt engineering alone. Algorithmic formulation combined with few-shot prompting enables Qwen2.5-Coder 1.5B to reach Pass@5 of 100% in both Python and Lua for Directory Enumeration, 100% in Python for Backdoor Generation, and 99.2% in Lua. Although the evaluated models remain too computationally demanding for widespread deployment in contemporary malware, rapid progress in compact models may reduce this barrier. These findings help anticipate AI-assisted malware and inform future countermeasures.

Figures & tables

Figure 1: Research questions organizing the experimental evaluation of Small Language Models in post-exploitation scenarios. The study is structured around four complementary research dimensions: (i) architectural and domain limits, (ii) interaction strategies, (iii) autonomy and reflection mechanisms, and (iv) efficiency and failure analysis. Each dimension is addressed in the Experimental Evaluation and Discussion section through dedicated experiments.

Figure 2: Operational pipeline of the embedded Small Language Model (SLM)-powered malware. The schematic illustrates the autonomous, closed-loop execution flow designed for air-gapped environments. The central malicious core sequentially drives the process: (1) reading task-specific instructions; (2) invoking the embedded SLM to dynamically generate operational functional code; (3) validating the output through static rules and unit tests within an internal sandbox; and (4) delivering the verified payload to the target system.

Variable	Evaluated Values
Task	Directory Enumeration, Backdoor
Task Formulation	Basic, Constrained, Algorithmic
Prompt Strategy	Zero-Shot, Few-Shot, CoT, FS + CoT
Language	Python, Lua
Output Structure	None, EBNF Grammar

Table 1: Summary of experimental variables

Architecture	Params	Tuning	Target Domain
EleutherAI/pythia-70m	70M	Base	General
EleutherAI/pythia-160m	160M	Base	General
HF/SmolLM2-135M	135M	Instruct	General
HF/SmolLM2-360M	360M	Instruct	General
Qwen2.5-Coder-0.5B	0.5B	Instruct	Coder
Qwen2.5-Coder-1.5B	1.5B	Instruct	Coder

Table 2: List of SLMs evaluated.

Figure 3: Experimental methodology adopted to evaluate the offensive capabilities of compact language models. The upper panel summarizes the controlled experimental variables, including tasks, programming languages, model families, task formulations, prompting strategies, and structured decoding. The lower panel illustrates the three interaction paradigms investigated in this work: standard one-shot generation, execution-feedback interaction, and iterative Least-to-Most decomposition. For each paradigm, the evaluated variables and fixed experimental settings are explicitly reported to ensure a controlled comparison across all experiments.

Figure 4: Validation pipeline adopted to assess generated programs. Candidate scripts are first subjected to static and heuristic verification, including syntax validation, structural integrity checks, and task-specific constraint verification. Successfully validated programs are subsequently executed inside an isolated sandbox and evaluated through dynamic execution and unit tests.

Figure 5: High-level architecture of the experimental framework. The framework orchestrates the complete evaluation pipeline, from experiment configuration and task specification to prompt construction, SLM inference, code extraction, automated validation, and metric collection. Modular components enable language-specific prompt adaptation, output enforcement, and reproducible large-scale experimentation across different interaction paradigms.

Parameter	Value	Rationale / Application
Global Inference Parameters
Decoding Temperature	$0.2$	Low variance to preserve structural formatting while enabling narrow exploration for Pass@k.
Max Generation Tokens	$1024$	Sufficient context window to accommodate both reasoning (CoT) and final payloads.
Sampling Budget ( $n$ )	$10$	Number of independent completions generated per prompt to compute the $Pass@k$ estimator.
Macro-Runs	$10$	Number of full pipeline executions to calculate statistical mean and standard deviation ( $\pm$ ).
Deterministic Seeds	42, 123, 456, 789, 1,024, 2,048, 4,096, 8,192, 16,384, 32,768	Fixed global seeds across all 10 macro-runs to guarantee exact experimental replicability.

Table 3: Hyperparameters and reproducibility configuration across experimental phases.

Model	Size	Pass@1 (%)		Pass@5 (%)
Model	Size	Python	Lua	Python	Lua
Task: Directory Enumeration (Task Formulation: BASIC)
Qwen2.5-Coder	1.5B	18.0 $\pm$ 4.2	6.0 $\pm$ 5.2	72.2 $\pm$ 11.7	30.0 $\pm$ 25.8
Qwen2.5-Coder	0.5B	25.0 $\pm$ 9.7	0.0 $\pm$ 0.0	81.2 $\pm$ 17.6	0.0 $\pm$ 0.0
SmolLM2	360M	14.0 $\pm$ 10.7	0.0 $\pm$ 0.0	53.9 $\pm$ 33.1	0.0 $\pm$ 0.0
Models $<300$ M ∗	–	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0

Table 4: Pass@1 and Pass@5 scores for the Directory Enumeration and Backdoor tasks.

Figure 6: Effect of the constrained task formulation on directory enumeration performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.

Figure 7: Effect of the constrained task formulation on backdoor generation performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.

Figure 8: Effect of task formulation on directory enumeration performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.

Figure 9: Effect of task formulation on backdoor generation performance. Pass@1 (a) and Pass@5 (b) scores are reported for Python and Lua across the evaluated models.

Model	Size	Language	Prompting Strategy (Pass@5 %)
Model	Size	Language	Core	Few Shot	Thinking	FS + Thinking
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	Python	72.2 $\pm$ 11.7	100.0 $\pm$ 0.1	50.6 $\pm$ 21.2	32.8 $\pm$ 29.5
Qwen2.5-Coder	1.5B	Lua	30.0 $\pm$ 25.8	0.0 $\pm$ 0.0	17.8 $\pm$ 29.6	5.0 $\pm$ 15.8
Qwen2.5-Coder	0.5B	Python	81.2 $\pm$ 17.6	85.5 $\pm$ 15.3	69.8 $\pm$ 31.8	76.4 $\pm$ 19.0
SmolLM2	360M	Python	53.9 $\pm$ 33.1	0.0 $\pm$ 0.0	10.0 $\pm$ 21.1	10.0 $\pm$ 21.1

Table 5: Cross-Analysis: Directory Enumeration. Impact of Prompting Strategies on Task Complexity (Pass@5 with Variance).

Model	Size	Language	Prompting Strategy (Pass@5 %)
Model	Size	Language	Core	Few Shot	Thinking	FS + Thinking
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	Python	5.0 $\pm$ 15.8	0.0 $\pm$ 0.0	43.3 $\pm$ 32.3	5.0 $\pm$ 15.8
Qwen2.5-Coder	1.5B	Lua	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	10.0 $\pm$ 21.1	0.0 $\pm$ 0.0
Qwen2.5-Coder	0.5B	Python	19.2 $\pm$ 32.9	5.0 $\pm$ 15.8	25.6 $\pm$ 34.3	5.0 $\pm$ 15.8
Task Formulation: CONSTRAINED

Table 6: Cross-Analysis: Backdoor Generation. Impact of Prompting Strategies on Task Complexity (Pass@5 with Variance).

Figure 10: Impact of strict syntactic forcing via EBNF Guided Decoding on the Directory Enumeration task in Python (a) and Lua (b) and Backdoor task in Python(c) and Lua (d) under the basic task formulation

Figure 11: Performance comparison on the Direnum task across prompting strategies. The chart tracks the evolution of Pass@1 and Pass@5 metrics from standard One-Shot, to Algorithmic One-Shot, to Iterative (L2M) prompting. The Qwen 1.5B model (blue) demonstrates maximum efficacy at the Algorithmic stage, peaking at 99.8% (Python) and 91.1% (Lua) in Pass@5. However, enforcing the Iterative (L2M) strategy causes a noticeable regression, dropping performance back to 75.0%. Smaller models (¡ 1B parameters) show only marginal, transient gains from algorithmic hints before completely collapsing under multi-step constraints.

Figure 12: Performance trajectories on the Backdoor generation task. The inclusion of algorithmic instructions produces a dramatic but brittle spike in performance for the Qwen 1.5B model (blue), notably jumping from 0.0% to 90.0% Pass@5 in Lua. Moving to the Iterative (L2M) framework, however, causes a severe performance collapse, completely zeroing out Lua success rates and reducing Python gains. This volatility highlights the profound difficulty of maintaining malicious code constraints across multiple iterative reasoning steps, especially for smaller language models which fail entirely (0.0%) regardless of the prompting technique.

Figure 13: Iterative drop-off curve for the directory enumeration task. The Y-axis represents the survival rate, defined as the percentage of trajectories successfully maintaining a functional state up to a given iterative step.

Figure 14: Iterative drop-off curve for the backdoor generation task. The Y-axis represents the survival rate, defined as the percentage of trajectories successfully maintaining a functional state up to a given iterative step.

Figure 15: Sequential taxonomy of failure modes observed during iterative (Least-To-Most) malicious code generation. Failure categories follow the sequential evaluation pipeline adopted in our framework: Syntax Errors prevent successful parsing or code generation; Constraint Errors correspond to executable programs that violate the functional requirements of the task; Runtime Errors occur only after syntactically valid and functionally compliant code reaches execution (for the iterative it happens only when the last step is reached). Each cell reports the percentage of failures observed for Directory Enumeration (left value) and Backdoor Generation (right value), while the color intensity represents the maximum percentage between the two tasks, facilitating cross-task comparison across model sizes and programming languages.

Figure 16: Cumulative Pass Rate trend across the 10 steps of the iterative Reflection cycle (Language: PYTHON). Only the experimental configurations (Strategy

\times

Task Formulation) that exhibited a positive responsiveness to error-message feedback (Pass Rate variation

>0

) are reported.

Figure 17: Cumulative Pass Rate trend across the 10 steps of the iterative Reflection cycle (Language: LUA). Only the experimental configurations (Strategy

\times

Task Formulation) that exhibited a positive responsiveness to error-message feedback (Pass Rate variation

>0

) are reported.

Figure 18: Comprehensive evaluation of the computational overhead vs. code quality for both the directory enumeration task and backdoor task type. The x-axis displays the evaluated Large Language Models, while the y-axis represents the hierarchical nesting of Task Formulations and Prompting Strategies. The size of each marker is inversely proportional to the average number of generated tokens (larger markers indicate higher computational efficiency). The color gradient maps the success rate achieved via the $Pass@5$ metric. Circles ( $\circ$ ) and diamonds ( $\diamond$ ) distinguish between Lua and Python implementations, respectively.

Figure 19: Distribution of the earliest validation stage at which unsuccessful generations fail across model sizes and programming languages. Each generated program is evaluated through a sequential fail-first validation pipeline consisting of (i) syntax verification, (ii) task-specific constraint validation, and (iii) runtime execution. Evaluation terminates immediately after the first failed stage; consequently, each unsuccessful generation contributes to exactly one failure category. Cell values report the percentage of failed generations whose earliest failure occurred at each validation stage, expressed as Directory Enumeration — Backdoor Generation. This representation isolates the nature of failures among unsuccessful generations, independently of the overall success rate, allowing comparison of how model size, task complexity, and programming language influence where the generation process first breaks down.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	Specification
Task ID	DIR_ENUM
Target Domain	System Interaction / Post-Exploitation
Supported Languages	Lua ( DIR_ENUM ), Python ( DIR_ENUM_PY )
Objective	Implement a function enumerate_dir(target_path) that returns a collection (Table/List) of all files and subdirectories contained within the target path.
Evaluation Criteria	Isolated sandbox execution; Verification of return type; Positive assertion on the presence of a dynamically generated test file ( target_file.txt ) and subdirectory.
Prompt Complexity Variations (Cognitive Load Modulation)

Appendix

Table 7: Task Definition Card: Directory Enumeration ( DIR_ENUM )

Attribute	Specification
Task ID	BACKDOOR
Target Domain	System Interaction / Remote Access
Supported Languages	Lua ( BACKDOOR_BASIC ), Python ( BACKDOOR_BASIC_PY )
Objective	Implement a function start_tcp_server() that creates a TCP socket, binds it to a fixed port (7456), listens for connections, executes received commands on the host system, and sends the textual output back to the client.
Evaluation Criteria	Isolated sandbox execution with a mocked or threaded networking environment; Verification of socket initialization and binding; Positive assertion on the successful receipt of command output (e.g., executing echo ) by a simulated client.
Prompt Complexity Variations (Cognitive Load Modulation)

Appendix

Table 8: Task Definition Card: TCP Command Execution Server ( BACKDOOR )

Model	Size	Lang	Strategy	Pass@5 (%)	Avg Tokens	Time (s)	TPS
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	Lua	Core	30.0 $\pm$ 25.8	2507.8 $\pm$ 151.3	6.2 $\pm$ 0.5	403.7 $\pm$ 21.8
			Few Shot	0.0 $\pm$ 0.0	1720.4 $\pm$ 95.9	4.8 $\pm$ 0.5	358.6 $\pm$ 25.4
			Thinking	17.8 $\pm$ 29.6	3155.7 $\pm$ 223.0	8.7 $\pm$ 0.7	365.1 $\pm$ 23.5
			FS + Thinking	5.0 $\pm$ 15.8	3044.9 $\pm$ 223.3	8.2 $\pm$ 1.4	378.5 $\pm$ 41.6
Qwen2.5-Coder	0.5B	Lua	Core	0.0 $\pm$ 0.0	2581.7 $\pm$ 121.3	5.4 $\pm$ 0.3	476.2 $\pm$ 18.8

Appendix

Table 9: Computational Overhead and Inference Dynamics (Lua). Comparison of Pass@5, generated tokens, inference time, and throughput (TPS) across prompting strategies for the directory enumeration task. The data highlights two notable patterns: (1) Contextual Collapse in Qwen 0.5B under ”Few Shot + Thinking” (generating only

\sim

120 tokens before aborting), and (2) Over-generation in SmolLM2 360M during ”Thinking”, where it exhausts token limits without achieving a single success.

Model	Size	Lang	Strategy	Pass@5 (%)	Avg Tokens	Time (s)	TPS
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	Lua	Core	0.0 $\pm$ 0.0	2790.7 $\pm$ 128.9	6.8 $\pm$ 1.0	416.8 $\pm$ 48.9
			Few Shot	0.0 $\pm$ 0.0	2374.3 $\pm$ 252.6	7.6 $\pm$ 1.8	324.6 $\pm$ 61.4
			Thinking	10.0 $\pm$ 21.1	4419.3 $\pm$ 458.9	12.2 $\pm$ 2.3	370.0 $\pm$ 56.4
			FS + Thinking	0.0 $\pm$ 0.0	3235.3 $\pm$ 414.4	10.7 $\pm$ 3.3	322.5 $\pm$ 75.1
Qwen2.5-Coder	0.5B	Lua	Core	0.0 $\pm$ 0.0	3383.3 $\pm$ 337.9	7.5 $\pm$ 1.1	454.4 $\pm$ 52.7

Appendix

Table 10: Computational Overhead and Inference Dynamics (Lua). Comparison of Pass@5, generated tokens, inference time, and throughput (TPS) across prompting strategies for the backdoor generation task. The data highlights the recurrence of Contextual Collapse in Qwen 0.5B under ”Few Shot + Thinking” (drastically reducing generated tokens across all formulations) and the high computational cost of ”Thinking” strategies in SmolLM2 360M without corresponding performance gains.

Model	Size	Pass@1 (%)		Pass@5 (%)		Latent Retention
Model	Size	Python	Lua	Python	Lua	Latent Retention
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	18.0 $\pm$ 4.2	6.0 $\pm$ 5.2	72.2 $\pm$ 11.7	30.0 $\pm$ 25.8	41.5%
Qwen2.5-Coder	0.5B	25.0 $\pm$ 9.7	0.0 $\pm$ 0.0	81.2 $\pm$ 17.6	0.0 $\pm$ 0.0	0.0%
SmolLM2	360M	14.0 $\pm$ 10.7	0.0 $\pm$ 0.0	53.9 $\pm$ 33.1	0.0 $\pm$ 0.0	0.0%
Models $<$ 300M*	–	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	N/A

Appendix

Table 11: Impact of Task Formulation on Directory Enumeration task across Python (High-Resource) and Lua (Low-Resource). The table illustrates how cognitive friction (Constrained) degrades baseline capabilities, while Process-Driven Scaffolding (Algorithmic) substantially reduces the observed Python–Lua performance gap (Retention

\geq

91.3%). Models under 300M parameters are aggregated as they consistently exhibit zero capability.

Model	Size	Pass@1 (%)		Pass@5 (%)		Latent Retention
Model	Size	Python	Lua	Python	Lua	Latent Retention
Task Formulation: BASIC
Qwen2.5-Coder	1.5B	1.0 $\pm$ 3.2	0.0 $\pm$ 0.0	5.0 $\pm$ 15.8	0.0 $\pm$ 0.0	0.0%
Qwen2.5-Coder	0.5B	5.0 $\pm$ 9.7	0.0 $\pm$ 0.0	19.2 $\pm$ 32.9	0.0 $\pm$ 0.0	0.0%
SmolLM2	360M	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	N/A
Models $<$ 300M*	–	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	N/A

Appendix

Table 12: Impact of Task Formulation on Backdoor Generation across Python (High-Resource) and Lua (Low-Resource). The table highlights a severe vulnerability: while Constrained and Basic formulations yield minimal malicious code, the Process-Driven (Algorithmic) formulation inadvertently acts as a jailbreak, causing Qwen 1.5B’s backdoor success rate to skyrocket to 90.0% in Lua. Models under 300M parameters are aggregated as they consistently exhibit zero capability.

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 276 papers

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis
Jul 22, 2026Adel ElZemity, Shujun Li, Budi AriefMulti-Agent LLM Systems Small Language Models
University of Kent, Canterbury, United Kingdom
Large Byte Model: Teaching Language Models About Compiled Code
Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware Classification Byte-Level Language Model
CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1
ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
May 13, 2026Seunghyun Lee, David BrumleyAI Agent Security Benchmarks LLMs for Cybersecurity
Carnegie Mellon University · Pittsburgh, PA 15213 · Carnegie Mellon University and Bugcrowd +1

Jul 22, 2026J/K move · Enter open · S save

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

Adel ElZemity, Shujun Li, Budi Arief

University of Kent, Canterbury, United Kingdom

Multi-Agent LLM Systems Small Language Models Language Model Ensembles Malware Analysis

Open paper AskPDF arXiv

Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.

Abstract

Small Language Models

Explore similar work

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

Large Byte Model: Teaching Language Models About Compiled Code

ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents