ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises
Organizations: DEVCOM Army Research Laboratory · The Univeristy of Texas at El Paso
Abstract
Cyber-range and capture-the-flag scenario construction remains labor-intensive because authors must coordinate infrastructure, vulnerable services, attack paths, artifacts, and success conditions. LLMs can propose detailed exercises, but plausible descriptions do not guarantee deployable software or verifiable objectives. We present ScenarioForge, which turns high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, then instantiates and validates them in the Common Open Research Emulator (CORE). The ScenarioForge Execution Dataset contains 226 materialized scenarios covering 294 Vulhub images and all 85 custom-built challenge-resource generators developed with our AI-assisted scaffolding. All 226 passed construction, preview, sequencing, deployment, strict runtime validation, and live artifact checks; their 867 steps, each an individually verifiable challenge objective, comprise 457 vulnerability-backed and 410 generated-resource-backed instances. In a separate 50-prompt matched benchmark, ScenarioForge+Qwen3.8 and ScenarioForge+GPT5.6 each achieved 50/50 (100%) topology/build success and 160/160 intent fields; direct frontier generation reached 41/50 builds and 147/160 fields only with the full cumulative scaffold: structure, vocabulary, policy, and container-image guidance. The test still omitted ScenarioForge's downstream challenge-resource generation, dependency/flag sequencing, deployment, and live validation. The artifacts support operationally rich, reproducible exercises for training and evaluation.
Figures & tables
| Component | Grounding and preflight | Execution evidence |
|---|---|---|
| Vulnerable service | Resolve to a validated catalog entry with a deployment definition, exposed ports, startup procedure, and inject plan. | Container is placed on the intended node; required processes and ports are active; exploit prerequisites and injected files are present. |
| Generated challenge | Bind to an installed flag-generator or flag-node-generator that produces an artifact, expected flag, and solution procedure. | Output and inject artifacts exist at the designated target; the prescribed solution action retrieves the expected flag. |
| Topology and segmentation | Compile nodes, links, addresses, routes, and policy rows into CORE-supported operations; reject invalid references or dependency cycles. | Expected nodes and paths exist; allowed communication succeeds and prohibited communication is denied. |
| Traffic workload | Resolve protocol, endpoints, payload type, rate, and timing to supported traffic-agent configurations. | Sender and receiver agents are active and the requested flows traverse the instantiated topology. |
| Participant attachment | Bind the participant requirement to a configured hardware-in-the-loop interface and attachment target. | The participant host reaches permitted entry points through the selected CORE bridge or interface. |
| Challenge dependency | Resolve each prerequisite and produced artifact to a concrete prior or current step, including any required credential, host access, or pivot. | The Solutions Script traverses the dependencies in order, automatically checks supported steps, and records pass, fail, inconclusive, or skip. |
| Category | Measure | Scale / incidence | Composition and coverage |
|---|---|---|---|
| Dataset scope | Scenarios | 226 specifications | Fixed and materialized; one challenge chain per scenario. |
| Challenge objectives (steps) | 867 total | 457 vulnerability-backed (52.7%); 410 generated-resource-backed (47.3%). | |
| Chain structure | 212 multi-objective (93.8%) | Mean 3.836 objectives per scenario. | |
| Chain-length distribution | 1–10 objectives per chain | Scenarios: 14 have 1 objective; 24 have 2; 36 have 3; 113 have 4; 12 have 5; 11 each have 6 or 7; 4 have 8; none have 9; 1 has 10. | |
| Resource catalog | Vulnerability resources | 294 enabled images | 457 selections; 181/226 scenarios (80.1%); 136 component types; 8 semantic families. |
| Vulnerability severity | 242/294 map to CVEs (82.3%) | Published CVSS: 183/294 (62.2%), mean ; 107 critical (58.5%), 56 high (30.6%), 20 medium (10.9%). CISA Known Exploited Vulnerabilities (KEV): 97/242 (40.1%). |
| Condition | Available scaffolding | Topology/build success | Intent-field accuracy | Mean total tokens | Median AI time |
|---|---|---|---|---|---|
| ScenarioForge+Qwen3.8 | Schema, compiler, catalogs, validation, repair | 50/50 (100%) | 100% (160/160) | 89,680 | 94.4 s |
| ScenarioForge+GPT5.6 | Same framework-assisted path | 50/50 (100%) | 100% (160/160) | 41,437 | 37.4 s |
| Frontier R1 | CORE XML request | 4/49 (8.2%) | 55.0% (88/160) | 5,812 | 76.5 s |
| Frontier R2 | R1 + structural example | 28/50 (56%) | 85.0% (136/160) | 2,808 | 27.1 s |
| Frontier R3 | R2 + node/service vocabulary | 38/50 (76%) | 86.9% (139/160) | 2,870 | 23.4 s |
| Frontier R4 | R3 + segmentation vocabulary | 38/50 (76%) | 91.3% (146/160) | 3,109 | 26.6 s |
| Dimension | Representative run | Challenge structure | Representative artifacts |
|---|---|---|---|
| Access dependencies | dataset-segmented-enterprise-pivots_run01 | 6 total: 2 vulnerability, 4 generated; 2 access requirements | ChatOps transcript; wallet note; parser endpoint; SSH key; HTTP credential page |
| CI supply chain | dataset-mixed-ci-supply-chain_run01 | 3 total: 1 vulnerability, 2 generated | Traversal result; deploy-values.yaml ; SQL lookup |
| Segmented scale | dataset-scale-segmented-combined_run05 | 10 total: 5 vulnerability, 5 generated | CSV; packet capture; encoded blobs; MD5 hash; risk register; audit dump |
| Multi-artifact access | dataset-scale-multi-artifact_run03 | 5 generated; 1 access requirement | SSH keys; lab notebook; runbook; backup/finance records; NFS evidence |
| Messaging chain | dataset-mixed-collaboration-messaging_run03 | 4 total: 1 vulnerability, 3 generated | SMTP bounce; IMAP message; DMARC XML; SHA-256 secret |
Adversarial Scenario Generation
Adversarial scenario generation creates challenging, targeted situations to expose weaknesses and improve the robustness of AI systems.
See all 40 papers