Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

Bhanu Prakash Vangala; Tanu Malik

cs.SESep 30, 2026

Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

Authors: Bhanu Prakash Vangala, Tanu Malik

Organizations: Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA

Abstract

Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, defined as the accurate identification of the dependencies required to execute generated code, is equally critical. We develop an agent protocol for environment specification and introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies to systematically assess coding agents for environment specification. Using this protocol, we evaluate the extent to which coding agents systematically misspecify software environment dependencies and how this misspecification varies across three agents, four languages, and fifty programming tasks. Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared and runtime dependency layers, implicating environment priors learned from the models' training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.

Figures & tables

Panel of Figure 1 — Figure 1: Relationships among the three dependency layers. $D^{(1)}$ is extracted from the initial manifest, $D^{(2)}$ from the final successfully installed environment, and $D^{(3)}$ from independent runtime tracing via strace . No fixed set-inclusion is assumed between $D^{(1)}$ and $D^{(2)}$ . Pairwise gaps define phantom dependencies (declared but not needed), hidden dependencies (needed but not declared or installed), and bloat (installed but unused).

Figure 2: Environment Evaluation Pipeline. Left: the agent $A$ exercises its Gen skill to produce an initial project $(C^{(0)},M^{(0)})$ , which is executed in an isolated environment $I$ . On failure, the Repair skill is invoked with the runtime error $e^{(i)}$ for up to $k$ rounds, yielding the final artifact $(C^{(k)},M^{(k)})$ . Right: three extraction primitives operate on the pipeline outputs to produce the three dependency layers: $D^{(1)}$ via manifest parsing of the initial declaration $M^{(0)}$ ; $D^{(2)}$ via package-manager resolution of the final manifest $M^{(k)}$ ; and $D^{(3)}$ via strace -based runtime tracing of $C^{(k)}$ in $I$ , serving as ground truth.

Domain	$n$	Representative tasks
Data Processing	10	CSV statistical analyzer, JSON schema validator, XML-to-JSON converter, YAML config merger
Cryptography	7	AES-256 file encryption, RSA digital signatures, TOTP generator, bcrypt hasher
Image Processing	6	Format converter, watermarking, histogram analyzer, EXIF metadata extractor
Networking	6	HTTP downloader with retry, port scanner, DNS resolver, static file server
Text Processing	6	Markdown-to-HTML, spell checker, TF-IDF search engine, template engine
Compression	5	ZIP archive manager, GZIP tool, file deduplicator, compression benchmark

Table 1: The 50 benchmark programming tasks span eight application domains.

Dimension	Count
Benchmark programming tasks	50
Target language ecosystems	4
LLM coding agents	3
Primary configurations ( $50{\times}4{\times}3$ )	600
Stochastic trials (Claude Code, 2 extra ${\times}$ 200)	400
Maximum agent interactions per instance	11

Table 2: Study scale.

Ecosystem	Layer 1: $D^{(1)}=\mathrm{ParseManifest}_{\ell}(M^{(0)})$	Layer 2: $D^{(2)}=\mathrm{Resolve}_{\ell}(M^{(k)})$	Layer 3: $D^{(3)}=\mathrm{Trace}_{\ell}(C^{(k)},I)$
Python	Parse requirements.txt line by line; strip version constraints	Diff of pip list before vs. after pip install	Syscall READ s under dist-packages/ or site-packages/
Java	<artifactId> values in pom.xml <dependencies>	mvn dependency:list artifact IDs (full transitive closure)	Syscall READ s of target/dependency/*.jar via classpath execution
JavaScript	Keys of "dependencies" in package.json	npm list --all --json ; full node_modules/ tree	Syscall READ s under node_modules/
C++	FetchContent_Declare and find_package names in CMakeLists.txt	CMake configure log ( $D^{(2)}\approx D^{(1)}$ ; transitive expansion typically absent)	Syscall READ s of .so files; seven base system libraries excluded

Table 3: Per-ecosystem realization of

\mathrm{ParseManifest}_{\ell}

\mathrm{Resolve}_{\ell}

, and

\mathrm{Trace}_{\ell}

for the three dependency layers.

Figure 4: Gen skill: first-attempt success across all 600 instances (3 agents

\times

4 languages

\times

50 tasks). Each cell is one (agent, language, task) triple; green indicates first-attempt success and red indicates failure.

Figure 5: Repair skill: distribution of repair cycles needed to converge for each (agent, language) pair, bucketed into first-attempt success, 1–3, 4–7, and 8–10 cycles, and failure within the budget.

Figure 6: Protocol convergence by language. Solid bars show the first-attempt success rate

\mathrm{SR}^{(0)}

and hatched extensions show the self-correction gain contributed by Repair .

Figure 7: Phantom, hidden, and installed-but-unused bloat rates by ecosystem, averaged over successful projects with zero-denominator cases included as zero.

Figure 8: Initial-to-final environment growth by ecosystem. Each box summarizes the distribution of the inflation ratio

\rho

of Section 3 over all successful projects in that ecosystem (log scale).

Agent	Language	Precision	Recall	F1
Claude	Python	0.990	0.642	0.733
Claude	Java	0.935	0.739	0.758
Claude	JavaScript	0.929	0.406	0.493
Claude	C++	0.037	0.781	0.031
Codex	Python	0.938	0.827	0.837
Codex	Java	0.939	0.684	0.735

Table 4: Initial-manifest accuracy: treating runtime dependencies

D^{(3)}

as ground truth.

Figure 9: Cross-agent agreement on declared dependencies. Each bar reports the mean Jaccard similarity between the declared dependency sets extracted from two agents’ initial manifests for the same task in the same ecosystem; the dotted line marks the level at which half of the declared packages would be shared.

Language	Mean $\bar{J}_{\mathrm{intra}}$	Median	UCR	$\bar{V}_{\cup}$	$\bar{V}_{\cap}$
Python	0.113	0.083	0/50 (0.0%)	5.5	0.2
Java	0.138	0.111	0/48 (0.0%)	5.6	0.1
JavaScript	0.068	0.000	0/50 (0.0%)	5.9	0.0
C++	0.093	0.067	0/35 (0.0%)	5.3	0.1

Table 5: Stochastic variability: Claude’s three independent trials produce near-zero agreement.

Agent	Fail	SysLib	SLAR	Rec.	EGAR
Claude	32	28	87.5%	28	100.0%
Codex	28	5	17.9%	5	100.0%
Gemini	10	0	0.0%	0	–

Table 6: Environment Gap: C++ system library assumption failures and recovery. “Fail” counts first-attempt failures; “SysLib” counts those whose terminal error is a missing system library; “Rec.” counts SysLib failures eventually repaired.

Figure 10: Protocol convergence by task domain, with solid and hatched bars as in Figure 6 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Ecosystem	$\rho_{S}$	$p$ -value
Python	$-0.276$	0.0006
Java	$-0.260$	0.0013
JavaScript	$-0.093$	0.2564
C++	$-0.410$	0.0000

Appendix

Table 7: Spearman correlation between manifest size and first-attempt success.

ID	Metric	Key quantity	Results
M1	Success rate under self-correction	$\mathrm{SR}^{(0)}$ , $\mathrm{SR}^{(\leq k)}$	§ 6.1
M2	Environment growth	$\rho$	§ 6.2
M3	Phantom / hidden / bloat rates	$\phi$ , $\eta$ , $\beta$	§ 6.2
M4	Manifest accuracy	P, R, F1	§ 6.2
M5	Cross-agent consistency	$\bar{J}_{\ell}$ , CDR	§ 6.3
M6	Stochastic variability	$\bar{J}_{\mathrm{intra}}$ , UCR	§ 6.3

Appendix

Table 8: Metric summary: definitions, key quantities, and result section.

ID	Problem	Domain	Cx.	ID	Problem	Domain	Cx.
p_01	CSV Statistical Analyzer	Data Processing	M	p_26	DNS Resolver	Networking	M
p_02	JSON Schema Validator	Data Processing	M	p_27	HTTP REST Client w/ Retry	Networking	H
p_03	XML to JSON Converter	Data Processing	M	p_28	Static HTTP File Server	Networking	H
p_04	YAML Config Merger	Data Processing	M	p_29	Network Latency Monitor	Networking	M
p_05	Log File Pattern Analyzer	Data Processing	M	p_30	Markdown to HTML Converter	Text Processing	M
p_06	SQLite Database Manager	Data Processing	H	p_31	Spell Checker	Text Processing	M

Appendix

Table 9: All 50 programming problems.

Ecosystem	Most frequent phantom packages
Python	pycparser (6), pandas (5), tabulate (2), tzdata (1), python-dateutil (1)
Java	gson (2), zxing-javase (2), zxing-core (2), slf4j-api (1), javalin (1)
JavaScript	yargs (46), sharp (8), fast-csv (4), lodash (4), js-yaml (3)
C++	nlohmann_json (65), openssl (16), opencv (11), yaml-cpp (5), mbedtls (5)

Appendix

Table 10: Most frequent phantom packages

D^{(1)}\setminus D^{(3)}

by ecosystem.

Ecosystem	Most frequent hidden packages
Python	six (31), python-dateutil (23), pytz (20), packaging (14), kiwisolver (11)
Java	jackson-annotations (21), jackson-core (20), slf4j-api (14), commons-logging (12), error_prone_annotations (12)
JavaScript	ansi-styles (15), supports-color (13), has-flag (13), emoji-regex (9), is-fullwidth-code-point (9)
C++	crypto (16), z (14), zstd (6), nettle (6), krb5 (5)

Appendix

Table 11: Most frequent hidden packages

D^{(3)}\setminus D^{(1)}

by ecosystem.

Explore similar work

Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
May 7, 2026Francesco Dente, Dario Satriani, Paolo PapottiCode Generation Large Language Model Agents
EURECOM, France · University of Basilicata, Italy
WiseSpec: Requirements-Driven Agents for Code Generation
Sep 1, 2026Zhao TianCode Generation Software Engineering
School of Computer Software, Tianjin University Tianjin, China
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Sep 29, 2026Hantian Ding, Chloe Bi, Jiacheng Zhu +5Codebases Model Auditing
Meta Superintelligence Labs

May 7, 2026, cs.SEJ/K move · Enter open · S save

Constraint Decay: The Fragility of LLM Agents in Backend Code Generation

Francesco Dente, Dario Satriani, Paolo Papotti

EURECOM, France · University of Basilicata, Italy

Code Generation Large Language Model Agents Structural Constraints Fragility

Open paper AskPDF arXiv

Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications. However, production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings. Existing benchmarks often overlook these non-functional requirements, rewarding functionally correct but structurally arbitrary solutions. We present a systematic study evaluating how well agents handle structural constraints in multi-file backend generation. By fixing a unified API contract across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks, we isolate the effect of structural complexity using a dual evaluation with end-to-end behavioral tests and static verifiers. Our findings reveal a phenomenon of constraint decay: as structural requirements accumulate, agent performance exhibits a substantial decline. Capable configurations lose 30 points on average in assertion pass rates from baseline to fully specified tasks, while some weaker configurations approach zero. Framework sensitivity analysis exposes significant performance disparities: agents succeed in minimal, explicit frameworks (e.g., Flask) but perform substantially worse on average in convention-heavy environments (e.g., FastAPI, Django). Finally, error analysis identifies data-layer defects (e.g., incorrect query composition and ORM runtime violations) as the leading root causes. This work highlights that jointly satisfying functional and structural requirements remains a key open challenge for coding agents.

Abstract

Explore similar work

Constraint Decay: The Fragility of LLM Agents in Backend Code Generation

WiseSpec: Requirements-Driven Agents for Code Generation

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch