We introduce ShIOEnv, a Gymnasium-compatible Bash environment for synthesizing execution-annotated shell input traces. Datasets containing such traces are useful for training models to perform execution-free shell simulation, which in turn is useful for deploying honeypots for cyber deception. Existing datasets, such as NL2CMD and other Bash command corpora, do not include complete execution artifacts (stdout, stderr, exit status, and changes to the system's state) associated with a command's execution. ShIOEnv synthesizes shell inputs, executes them in a controlled system context, and records stdout, stderr, exit status, and structured filesystem and environment patches. Unconstrained (random) generation of shell inputs (command and arguments) results in multiple instances of arguments that are mostly redundant or result in execution errors. We introduce grammar-constrained synthesis to generate syntactically valid arguments with high input irreducibility, which we estimate through self-ablative contrasts of execution artifacts. We prepare 1.6M samples using grammar-constrained and unconstrained synthesis and collect their execution artifacts for 86 Linux/Bash utilities. Using this dataset, we train small language models (CodeT5) and evaluate their accuracy in predicting the actual execution behavior of attacker-sourced Bash commands. These models outperform commonly used rule-based simulation (Cowrie) and prompted large language models by up to 14.2 pp in output edit similarity (O-ES). We enhance the NL2CMD dataset by constructing 500K samples executable in a grounded system context along with their complete execution artifacts. We empirically demonstrate that models trained on grammar-constrained samples outperform those trained on NL2CMD samples by 22.5 pp in O-ES. ShIOEnv, command grammars, and 2.1M samples used in our primary evaluations are released for use by other researchers.
Figures & tables
Figure 1: Synthesis of shell input containing a df command and argument actions. Execution artifacts are collected upon completion.
Figure 2: Expansion of --output=source argument by df grammar productions.
Table 3: CodeT5 performance by training dataset and volume.
Dataset
EM
O-ES
Δ -JS
NL2CMD
0.245 ± 0.00
0.465 ± 0.04
0.447 ± 0.04
UCS
0.372 ± 0.04
0.608 ± 0.03
0.347 ± 0.04
GCS
0.462 ± 0.01
0.719 ± 0.01
0.591 ± 0.06
Table 4: CodeT5 performance distribution across execution environments (OS + filesystem).
Figure 3: Approximation error of the budgeted irreducibility estimator. The left plot reports the command-level error distribution for each contrast budget. The right plot reports mean absolute error between R⋆ and RM by argument length. Shaded regions indicate 95% confidence intervals.
Figure 4: Global distribution of irreducibility by synthesis method (left) and number of arguments (right) using RM=64 . UCS denotes unconstrained synthesis. GCS denotes grammar-constrained synthesis. Shaded regions indicate standard deviation.
Dataset
Filter
EM
O-ES
Δ -JS
UCS
R64<0.5
0.269
0.476
0.141
UCS
R64≥0.0
0.333
0.605
0.383
UCS
R64≥0.5
0.375
0.632
0.455
GCS
R64<0.5
0.359
0.616
0.392
GCS
R64≥0.0
0.457
0.735
0.578
GCS
R64≥0.5
0.482
0.735
0.631
Table 5: CodeT5 execution-free modeling performance by dataset irreducibility. The threshold R64≥0.0 denotes the unfiltered training set.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Type
Description
session_id
Integer
Unique sample identifier
input
String
Constructed input sequence
input_args
List
Input separated into argument units
exit_code
Integer
Exit code returned by the executed input
output
String
Observed stdout and stderr from executing the input
context_patch
String
Latent changes to the abstract system environment
Appendix
Table 6: ShIOEnv dataset fields
Figure 6: System contexts used in evaluation. The Ubuntu 22.04 and Debian 12 file systems are only used for system context ablations reported in Table 4 .