ShIOEnv: A Command Evaluation Environment for Grammar-Constrained Synthesis and Execution Behavior Modeling

Jarrod Ragsdale; Rajendra Boppana

cs.CLMay 23, 2025

ShIOEnv: A Command Evaluation Environment for Grammar-Constrained Synthesis and Execution Behavior Modeling

Authors: Jarrod Ragsdale, Rajendra Boppana

Organizations: Department of Computer Science, University of Texas at San Antonio, San Antonio, USA.

Abstract

We introduce ShIOEnv, a Gymnasium-compatible Bash environment for synthesizing execution-annotated shell input traces. Datasets containing such traces are useful for training models to perform execution-free shell simulation, which in turn is useful for deploying honeypots for cyber deception. Existing datasets, such as NL2CMD and other Bash command corpora, do not include complete execution artifacts (stdout, stderr, exit status, and changes to the system's state) associated with a command's execution. ShIOEnv synthesizes shell inputs, executes them in a controlled system context, and records stdout, stderr, exit status, and structured filesystem and environment patches. Unconstrained (random) generation of shell inputs (command and arguments) results in multiple instances of arguments that are mostly redundant or result in execution errors. We introduce grammar-constrained synthesis to generate syntactically valid arguments with high input irreducibility, which we estimate through self-ablative contrasts of execution artifacts. We prepare 1.6M samples using grammar-constrained and unconstrained synthesis and collect their execution artifacts for 86 Linux/Bash utilities. Using this dataset, we train small language models (CodeT5) and evaluate their accuracy in predicting the actual execution behavior of attacker-sourced Bash commands. These models outperform commonly used rule-based simulation (Cowrie) and prompted large language models by up to 14.2 pp in output edit similarity (O-ES). We enhance the NL2CMD dataset by constructing 500K samples executable in a grounded system context along with their complete execution artifacts. We empirically demonstrate that models trained on grammar-constrained samples outperform those trained on NL2CMD samples by 22.5 pp in O-ES. ShIOEnv, command grammars, and 2.1M samples used in our primary evaluations are released for use by other researchers.

Figures & tables

Figure 1: Synthesis of shell input containing a df command and argument actions. Execution artifacts are collected upon completion.

Panel of Figure 2 — Figure 2: Expansion of --output=source argument by df grammar productions.

arch	basename	cat	cd 1	chgrp	chmod	chown	comm 2
cp	cpio 1,2	curl 1	cut 2	date	dd	df	dir
dirname	dmesg 1	du	echo	env	ethtool 1	export 1	false
file 1	find 1	fold 2	free 1	grep 1,2	groups	gzip 2	head 2
hostid	id	ip-addr 1	ip-link 1	ip-route 1	last 1	link	ln
logname	ls	lsblk 1	lscpu 1	mkdir	mkfifo	mktemp	mv
netstat 1	nl 2	nproc	od 2	paste 2	ping 1	printenv	ps 1

Table 1: Evaluated commands

Method	EM	O-ES	$\Delta$ -JS
Cowrie	0.165	0.345	0.248
GPT-4o-mini	0.171	0.582	0.164
GPT-4.1-mini	0.247	0.593	0.570
CodeT5 + NL2CMD	0.241	0.510	0.488
CodeT5 + UCS	0.333	0.605	0.383
CodeT5 + GCS	0.457	0.735	0.578

Table 2: Bash command execution-free simulation performance (UCS=Unconstrained sampling, GCS=Grammar-constrained sampling).

Size	Dataset	EM	O-ES	$\Delta$ -JS
100K	NL2CMD	0.171	0.401	0.402
	UCS	0.207	0.446	0.191
	GCS	0.353	0.661	0.582
300K	NL2CMD	0.227	0.465	0.354
	UCS	0.286	0.529	0.268
	GCS	0.426	0.715	0.523

Table 3: CodeT5 performance by training dataset and volume.

Dataset	EM	O-ES	$\Delta$ -JS
NL2CMD	0.245 $\pm$ 0.00	0.465 $\pm$ 0.04	0.447 $\pm$ 0.04
UCS	0.372 $\pm$ 0.04	0.608 $\pm$ 0.03	0.347 $\pm$ 0.04
GCS	0.462 $\pm$ 0.01	0.719 $\pm$ 0.01	0.591 $\pm$ 0.06

Table 4: CodeT5 performance distribution across execution environments (OS + filesystem).

Panel of Figure 3 — Figure 3: Approximation error of the budgeted irreducibility estimator. The left plot reports the command-level error distribution for each contrast budget. The right plot reports mean absolute error between $R^{\star}$ and $\widehat{R}_{M}$ by argument length. Shaded regions indicate 95% confidence intervals.

Panel of Figure 4 — Figure 4: Global distribution of irreducibility by synthesis method (left) and number of arguments (right) using $\widehat{R}_{M=64}$ . UCS denotes unconstrained synthesis. GCS denotes grammar-constrained synthesis. Shaded regions indicate standard deviation.

Dataset	Filter	EM	O-ES	$\Delta$ -JS
UCS	$\widehat{R}_{64}<0.5$	0.269	0.476	0.141
UCS	$\widehat{R}_{64}\geq 0.0$	0.333	0.605	0.383
UCS	$\widehat{R}_{64}\geq 0.5$	0.375	0.632	0.455
GCS	$\widehat{R}_{64}<0.5$	0.359	0.616	0.392
GCS	$\widehat{R}_{64}\geq 0.0$	0.457	0.735	0.578
GCS	$\widehat{R}_{64}\geq 0.5$	0.482	0.735	0.631

Table 5: CodeT5 execution-free modeling performance by dataset irreducibility. The threshold

\widehat{R}_{64}\geq 0.0

denotes the unfiltered training set.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Type	Description
session_id	Integer	Unique sample identifier
input	String	Constructed input sequence
input_args	List	Input separated into argument units
exit_code	Integer	Exit code returned by the executed input
output	String	Observed stdout and stderr from executing the input
context_patch	String	Latent changes to the abstract system environment

Appendix

Table 6: ShIOEnv dataset fields

Panel of Figure 6 — Figure 6: System contexts used in evaluation. The Ubuntu 22.04 and Debian 12 file systems are only used for system context ablations reported in Table 4 .

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

Aug 13, 2026

QuoteBench: How Matched Scores Can Hide Command-Path Failures

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

Shangao Li, Yao Zhang, Volker Tresp +1

1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning

LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks Code Generation Evaluation LLM Agent Reliability

Ask PDF

May 28, 2026

LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents

Mastering terminal environments requires language agents capable of multi-step planning, feedback-grounded execution, and dynamic state adaptation. However, training such agents is currently bottlenecked by a reliance on scraped external repositories, which limits domain diversity, environment controllability, and the targeting of specific capability deficits. We introduce LiteCoder-Terminal-Gen, a zero-dependency synthesis pipeline that autonomously generates executable and verifiable terminal training environments directly from domain specifications. Using this framework, we construct two large-scale resources: LiteCoder-Terminal-SFT, comprising 11,255 expert trajectories across 10 domains, and LiteCoder-Terminal-RL, featuring 602 verifiable environments for trajectory-level preference optimization. Supervised fine-tuning of Qwen-family models on our SFT dataset yields agents that significantly outperform their base counterparts. Notably, our 32B variant achieves 29.06%, 18.54%, and 34.00% pass@1 on Terminal Bench 1.0, 2.0, and Pro, respectively. Furthermore, applying Direct Multi-turn Preference Optimization (DMPO) on our RL environments yields additional performance gains. These results systematically demonstrate that fully synthetic, executable environments offer a scalable and verifiable supervision signal for mastering complex, real-world command-line workflows.

Xiaoxuan Peng, Kaiqi Zhang, Xinyu Lu +5

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences

RL for Language Models Terminal Agents LLM Agent Training Synthetic Data Generation for Language Models Long-Horizon LLM Agents

Ask PDF

Sep 11, 2026

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data

Large language models trained recursively on their own or other models' outputs undergo model collapse, in which distributional tails and factual accuracy deteriorate while fluency survives. Prior work diagnoses collapse after training; the actionable problem is screening a corpus of unknown provenance before training. We introduce SynthSentry, a corpus-level, model-agnostic contamination signal requiring no access to the generating model, no generation history, and no synthetic labels. The score is a distributional divergence over three statistics: lexical diversity collapse, n-gram tail truncation, and perplexity variance across reference models. We evaluate on corpora contaminated by small open-weight generators and an instruction-tuned open-weight model under a leave-one-generator-out protocol. A domain-stratified study measures false positives on naturally repetitive human text (legal, clinical, source code). The score ranks corpora by severity with little loss when whole generator families are held out. Per-domain calibration holds near its nominal false-positive budget once covariance shrinkage and a bootstrap threshold replace a naive quantile, which runs four times over budget. A downstream fine-tuning check showed no contamination-driven accuracy deficit at our scale, so whether pruning recovers one remains open; the same run shows over-pruning risk once pruning exceeds the true contamination fraction. We frame screening as a data-curation defense rather than a post-hoc diagnosis and release the scoring toolkit. All results are small-scale; scope is English-language, batch-mode corpus screening. Contamination sources are single-generation or hand-authored rather than recursively generated, so results speak to synthetic contamination generally and not to recursion depth.

Praveen Kumar Myakala, Ravichandra Namburi, Sowmya Keragodu Jayaramu +1

Independent AI Researcher Melissa, USA · Independent Researcher Frisco, USA · Independent Researcher Austin, USA

AI-Generated Text Detection Training Data Curation Data Contamination in Language Models

Ask PDF

Abstract

Synthetic Data Generation for Language Models

Explore similar work

QuoteBench: How Matched Scores Can Hide Command-Path Failures

LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data