LLM agents are increasingly used for code generation, but physics-based simulation poses a deeper challenge: natural-language descriptions of simulation models are inherently underspecified, and different admissible resolutions of implicit choices produce physically valid but scientifically distinct configurations. Without explicit detection and resolution of these ambiguities, neither the correctness of the result nor its reproducibility from the original description can be assured. This paper investigates agentic scientific simulation, where model construction is organized as an execution-grounded interpret-act-validate loop and the simulator serves as the authoritative arbiter of physical validity rather than merely a runtime. We present JutulGPT, a reference implementation built on the fully differentiable Julia-based reservoir simulator JutulDarcy. The agent combines structured retrieval of documentation and examples with code synthesis, static analysis, execution, and systematic interpretation of solver diagnostics. Underspecified modelling choices are detected explicitly and resolved either autonomously (with logged assumptions) or through targeted user queries. The results demonstrate that agent-mediated model construction can be grounded in simulator validation, while also revealing a structural limitation: choices resolved tacitly through simulator defaults are invisible to the assumption log and to any downstream representation. A secondary experiment with autonomous reconstruction of a reference model from progressively abstract textual descriptions shows that reconstruction variability exposes latent degrees of freedom in simulation descriptions and provides a practical methodology for auditing reproducibility. All code, prompts, and agent logs are publicly available.
Figures & tables
Figure 1: Typical iterative interpret–act–validate loop used by JutulGPT . The agent incrementally interprets user intent, detects ambiguities, and resolves them either autonomously (with explicit assumption logging) or via targeted user queries. Code generation is grounded in retrieved documentation and validated through static analysis, execution and simulator diagnostics, with failures triggering revision cycles. The loop terminates when the simulator runs to completion; an event that constitutes a validity certificate by virtue of JutulDarcy’s internal enforcement of conservation tolerances, closure consistency, and solver convergence.
Documentation Retrieval
Keyword search
Retrieves documentation using keyword matching.
Docstring lookup
Displays function, class, or module docstrings.
Example search
Finds examples related to a query.
Code Analysis
Linter
Analyzes code for errors, style issues, and common pitfalls.
Code runner
Executes code snippets in a controlled environment.
Table 1: Overview of tools available to JutulGPT.
Figure 2: Taxonomy of well models produced by JutulGPT in response to a documentation query.
Figure 3: Saturation profiles extracted at fixed fractions of injected pore volume (PVI), providing a normalized basis for comparison independent of injection rate or grid resolution.
Figure 4: Explanation generated by the agent when asked to provide a formal description of the simulated problem, including assumptions, governing equations, constitutive laws, introduction of fractional form and mobility ratio, physical explanation of favorable versus unfavorable flow, and brief discussion of the importance of the quarter-five-spot problem.
Figure 5: 3D reservoir model generated by JutulGPT . The front half uses semi-transparent colors to distinguish the three stratigraphic layers (each with ten grid layers of different average permeability and porosity), while the back half shows log 10 of permeability to reveal the stochastic heterogeneity correctly produced within each layer. Four corner injectors (red, I1–I4) and three interior producers (blue, P1–P3) are placed for a peripheral waterflood. In this classic setup, water is injected to displace more viscous oil toward the producers; the anticlinal trap and layered heterogeneity together control sweep efficiency.
Figure 6: Excerpts from the reproduction prompt for the 3D waterflooding case generated by the agent.
Figure 7: Water saturation for the compressible reference case (left) and the incompressible reproduction from the technical report (right). Densities are specified at surface conditions (1 bar), but the simulation runs at reservoir pressures (88–90 bar). In the reference case, higher oil compressibility substantially reduces the effective density contrast and weakens buoyancy; the diffusive pressure response slows vertical redistribution, allowing viscous fingering to produce a mixed, voxelated pattern. In contrast, the incompressible case maintains the full surface density contrast, enabling rapid buoyancy segregation: denser water quickly underrides lighter oil and channels along the high-permeability bottom layer, forming sharp, bottom-conformant fronts.
Figure 8: Journal-style description of the 3D waterflooding case generated by the agent.
Figure 9: Comparison of vertical cross-sections through x=400 for the reference model and the model reconstructed from the journal-style description.
Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.
Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.
Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2
Agent-based models (ABMs) are difficult to reproduce: their behavior is spread across prose narratives, platform-specific code, and implicit assumptions, so that two readers routinely reconstruct different models from the same documentation. We present VISA, a structured, symbol-based description protocol that specifies a model in eight interconnected tables---four at the agent level (Agent, Variable, Sensing, Internal Function) and four at the model level (Associated Data, Input/Output, Schedule, Validation)---under the principle of minimality with completeness. VISA makes a model machine-parseable and unambiguous via two artifacts: nineteen executable consistency rules that turn model validity into a checkable property, and three reusable LLM-executable skills (authoring, checking, and code generation) that operationalize the full author--check--code--reproduce loop. We validate the protocol on three external, independently authored ABMs spanning three platforms: we reproduce two cross-language (NetLogo to Python) directly from their VISA specifications, and we capture a third, an industrial AnyLogic model, in eight tables (passing all nineteen rules) while honestly demarcating where reproduction is blocked by a proprietary movement library and unavailable data---itself a transparency contribution. VISA moves the reproduction barrier from the model, where it is invisible, to a named, localized dependency, where it is actionable.
Zhou He
University of Chinese Academy of Sciences, Beijing, China