cs.CRDec 7, 2025

Effective and Efficient Threat Hunting with Small Language Models

Authors: Saleha Muzammil, Rahul Reddy, Vishal Kamalakrishnan, Hadi Ahmadi, Wajih Ul Hassan

Abstract

Analysts in Security Operations Centers query massive telemetry streams using Kusto Query Language (KQL), but writing correct KQL demands specialized expertise that bottlenecks scaling security teams. We investigate how Small Language Models (SLMs) can enable accurate, cost-effective translation from natural language queries (NLQs) to KQL. We propose a three-knob framework spanning prompting, fine-tuning, and architecture. First, we adapt NL2KQL for SLMs with lightweight retrieval and introduce error-aware prompting that targets common parser failures with a handful of mined tips, at a fraction of the tokens KQL's full rule set would require. Second, we apply LoRA fine-tuning with rationale distillation augmenting each NLQ-KQL pair with a brief chain-of-thought to transfer teacher reasoning. This yields an informative negative result, as neither variant surpasses targeted prompting. Third, we propose a two-stage architecture pairing an SLM drafter with a low-cost LLM judge for schema-aware refinement. We evaluate nine models (five SLMs, four LLMs) on syntax correctness, semantic accuracy, table selection, filter precision, latency, and token cost. On Microsoft's NL2KQL Defender Evaluation dataset, our two-stage approach reaches 0.987 syntax and 0.906 schema-valid ("semantic") accuracy, exceeding every baseline we run under equivalent infrastructure, and it generalizes to independently authored queries over the same schema (0.964 syntax, 0.831 schema-valid). The only baselines within 0.05 schema-valid are NL2KQL+GPT-4o (0.878) and NL2KQL+GPT-5 (0.861), which cost USD 2.998 and USD 2.018 for 230 queries against USD 0.213 for ours, a 9.5-14x reduction at matched accuracy. These results establish SLMs as a practical foundation for natural-language querying in security operations.

Explore similar work

CardsList
Oct 15, 2025cs.CL

Toward Cybersecurity-Expert Small Language Models

Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training datasets. To address this gap, we present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters. To train CyberPal 2.0, we generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0, which integrates expert-in-the-loop steering of reasoning formats alongside LLM-driven multi-step grounding, yielding higher-fidelity, task-grounded reasoning traces for security tasks. Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and matches or surpasses various open and closed-source frontier models, while remaining a fraction of their size. On core cyber threat intelligence knowledge tasks, our models outperform almost all tested frontier models, ranking second only to Sec-Gemini v1. On core threat-investigation tasks, such as correlating vulnerabilities and bug tickets with weaknesses, our best 20B-parameter model outperforms GPT-4o, o1, o3-mini, and Sec-Gemini v1, ranking first, while our smallest 4B-parameter model ranks second.
Matan Levi, Daniel Ohayon, Ariel Blobstein +3
CybersecurityLarge Language Model SafetySmall Language Models
AskPDF
Apr 21, 2026cs.CR

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.
Alankrit Chona, Igor Kozlov, Ambuj Kumar
Security AnalysisCyber Threat IntelligenceThreat ModelsText CorporaStochastic Optimal Control
AskPDF
Apr 30, 2026cs.CR

Toward Autonomous SOC Operations: End-to-End LLM Framework for Threat Detection, Query Generation, and Resolution in Security Operations

Security Operations Centers (SOCs) face mounting operational challenges. These challenges come from increasing threat volumes, heterogeneous SIEM platforms, and time-consuming manual triage workflows. We present an end-to-end threat management framework that integrates ensemble-based detection, syntax-constrained query generation, and retrieval-augmented resolution support to automate critical security workflows. Our detection module evaluates both traditional machine learning classifiers and large language models (LLMs), then combines the three best-performing LLMs to create an ensemble model, achieving 82.8% accuracy while maintaining 0.120 false positive rate on SIEM logs. We introduce the SQM (Syntax Query Metadata) architecture for automated evidence collection. It uses platform-specific syntax constraints, metadata-based retrieval, and documentation-grounded prompting to generate executable queries for IBM QRadar and Google SecOps. SQM achieves a BLEU score of 0.384 and a ROUGE-L score of 0.731. These results are more than twice as good as the baseline LLM performance. For incident resolution and recommendation generation, we demonstrate that integrating SQM-derived evidence improves resolution code prediction accuracy from 78.3% to 90.0%, with an overall recommendation quality score of 8.70. In production SOC environments, our framework reduces average incident triage time from hours to under 10 minutes. This work demonstrates that domain-constrained LLM architectures with retrieval augmentation can meet the strict reliability and efficiency requirements of operational security environments at scale.
Md Hasan Saju, Akramul Azim
SecurityTriageDetection FrameworkData Generation
AskPDF