cs.CROct 7, 2026

Ask the Expert: LLM-Guided Reinforcement Learning for Autonomous Cyber Defense

Authors: Fernando Martinez, Abhishek Satyam, Tao Li, Junaid Farooq, Ying Wang, Juntao Chen

Organizations: Department of Computer and Information Sciences, Fordham University, New York, USA · Department of Systems Engineering, Stevens Institute of Technology, New Jersey, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Policy-based reinforcement learning (RL) approaches have produced promising results for autonomous cyber defense; however, they are sample-inefficient in settings where defenders must respond under delayed, partial observations with actions from large action spaces. While large language models (LLMs) may reason semantically about security state space, high latency and trust assumptions prevent attractive in-line deployment models. We introduce Ask the Expert, a training-time guidance framework which first summarizes hard cyber-defense states, then intermittently queries an LLM for host-level defensive recommendations via a constrained action interface, and finally transforms those recommendations into tiered reward shaping for use with PPO. Because the LLM is discarded after training, deployment is a pure RL policy. Across TTCP CAGE CC1 and CC2 and both attacker types, this asymmetric design improves sample efficiency over PPO and outperforms the evaluated potential-based reward shaping (PBRS) baselines, while retaining the strongest terminal mean and requiring no LLM dependency at deployment time.

Figures & tables

Fig. 1: The Ask the Expert architecture shared across both B_lineAgent and RedMeanderAgent experiments. Reward-breakdown telemetry is encoded into compromised, spreading, and clean host sets; spreading hard states are stored in the candidate bag; a recent-frontier sampler builds the LLM prompt; and the returned host-level advice is merged into a tiered reward-shaping policy applied at rollout end before PPO advantage computation. Deployment remains pure RL with no LLM calls.
Fig. 2: Mean checkpoint evaluation reward for PPO, Ask the Expert , and PBRS variants in CC1. Top: B_lineAgent . Bottom: RedMeanderAgent . Shaded bands denote ±1\pm 1 standard error across independent training seeds.
Fig. 3: Mean checkpoint evaluation reward for PPO, Ask the Expert , and PBRS variants in CC2. Top: B_lineAgent . Bottom: RedMeanderAgent . Shaded bands denote ±1\pm 1 standard error across independent training seeds.
Fig. 4: Frontier-sampling mechanics from one representative B_lineAgent run. Panel (a) compares the normalized Shannon entropy [ 27 ] of host frequencies in the raw rollout stream and in the sampled prompt over training. Panel (b) shows the host-wise reweighting induced by the sampler, measured as prompt share minus rollout share at each query. Positive values indicate that a host is emphasized more in the prompt than in the rollout stream.

Explore similar work

CardsList

  1. Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

    Jul 30, 2026Konur Tholl, François Rivest, Mariam El Mezouar +2Cybersecurity

    Department of Electrical and Computer Engineering, Royal Military College of Canada, Kingston, Canada · Department of Mathematics and Computer Science, Royal Military College of Canada, Kingston, Canada · Defence Research and Development Canada, Ottawa, Canada +1

  2. Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution

    Sep 30, 2026Harshith Doppalapudi, Nathaniel D. Bastian, Ankit ShahHierarchical Reinforcement LearningRetraining

    Indiana University Bloomington, IN, USA · Johns Hopkins University Baltimore, MD, USA

  3. Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

    Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Red-TeamingLarge Language Model Safety

    University of California, Irvine · Northeastern University · Johns Hopkins University