cs.AIOct 30, 2026

From 1D Benchmarks to Complex 3D Geometries: Can Agents Reliably Specify Problems for Automated PDE Simulation?

Authors: Akshat Jani, Sakhinana Sagar Srinivas, Venkataramana Runkana

Organizations: Tata Research Development and Design Center, Pune, India 411057

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Turning an engineering request into a simulation requires consistent geometry, physics, boundary conditions, and numerical settings. Recent agent-assisted systems automate simulation setup, but successful execution alone does not establish whether the specification preserves the intended problem or correctly handles missing information. We study whether agents can reliably construct complete executable JSON specifications for automated geometry creation, meshing, PDE solving, and result export. Our workflow combines model extraction with deterministic scientific and capability checks, explicit clarification, and validated local patches that preserve unrelated accepted fields. We evaluate specification correctness, execution completion, and numerical verification separately. On 120 template-related requests, the full workflow achieves 98.3% correct first-turn specification and decision outcomes, compared with 92.5% for schema-assisted single shot and 82.5% for model-only: gains of 5.8 and 15.8 percentage points, respectively. It makes no false-ready decisions among the 42 requests requiring clarification or rejection. Component ablations identify contributions from scientific checks and conditional repair. Separate downstream studies cover 1D, 2D, and 3D steady and unsteady, linear and nonlinear problems, including flow and elasticity with complex three-dimensional wing geometries. A 30-case gallery demonstrates automated execution from retained JSON specifications, while 9 of 12 selected live user-to-results tests achieve joint specification, execution, export, and numerical-reference success (75.0%). These results support traceable, engineer-reviewed simulation assistance within the tested scope, while fresh-response regressions identify consistency as a priority for further improvement.

Figures & tables

Figure 1: Overall workflow, from left to right. (a) User/frontend: receives PDE requests and clarification replies. (b) Agentic and validation system: constructs and checks the specification while retaining session history. (c) Validated JSON: records the problem and numerical settings. (d) Execution backend: Gmsh creates geometry and meshes, FEniCSx and PETSc solve the PDEs, and ParaView supports visualization. (e) Outputs: returns fields, plots, diagnostics, and downloadable artifacts. The feedback loop supports validated user revisions.
JSON fieldsContents
geometry, meshDomain parameters, mesh backend, and resolution.
equationEquation family, coefficients, sources, and nonlinear law.
boundary_conditions, initial_conditionBoundary selectors, condition types and values, and initial fields.
finite_element, timeElement family and degree; steady/transient regime, time step, final time, and integration scheme.
solver, outputLinear/nonlinear solver settings, tolerances, and requested exports.
Table 1: Main fields of the executable JSON specification. The original request, schema version, and descriptive labels are also retained.
Figure 2: Specification construction and validation. The model proposes a draft or local repair, while deterministic checks assess structure, request evidence, and solver compatibility. Missing information returns to the user for clarification. Session storage retains the specification, revisions, and audit records. Execution requires a complete specification with no blocking findings.
IDExperimentEvaluationRequests
A1Specification comparisonCompare model-only, schema-assisted single shot, and the full workflow120
A2Component ablationRemove one component at a time; input-changing controls use a fixed 40-request subset120
A3Clarification completionApply fixed authorized replies and check whether a correct complete specification is reached30
Table 2: Agentic studies: user requests to JSON. Studies reuse requests, so their counts are not added together.
IDExperimentVerification setupSolves
V12D elasticityFour mesh levels; displacement and energy checks4
V32D steady flowThree mesh levels and one pressure patch test4
V42D unsteady heatFive step sizes for each BE/CN scheme; one equilibrium control11
V62D annular heatBE/CN refinement and spatial-isolation check8
V72D rectangular heatBE/CN refinement and spatial-isolation check8
V23D elasticityFour mesh levels; displacement and energy checks4
Table 3: Numerical verification studies. Counts include 56 completed study solves, including studies with failed checks. Development smoke runs are excluded.
IDExperiment groupScopeCases
G1Rod and polyline diffusion1D; steady/unsteady; linear3
G2Linear scalar equations2D; steady; linear6
G3Nonlinear diffusion2D; steady; nonlinear2
G4Heat and reaction-diffusion2D; unsteady; linear/nonlinear4
G7Airfoil and cylinder flow2D; steady; nonlinear2
G9Cantilever and perforated plate2D; steady; linear elasticity2
Table 4: Solution gallery: JSON to results. Each case belongs to one group.
SystemCorrect first-turn outcomesCorrect complete specificationsFalse-ready decisions
Model-only99/120 (82.5%)70/78 (89.7%)6/42
Schema-assisted single shot111/120 (92.5%)75/78 (96.2%)1/42
Full workflow118/120 (98.3%)76/78 (97.4%)0/42
Table 5: Specification and decision correctness for A1 (v1.8). Denominators are 120 total requests, 78 complete requests, and 42 requests requiring clarification or rejection. Failed cases remain in the denominators.
ConditionCorrect / evaluatedComponent exercised
120-request comparisons; reference: 118/120 (98.3%)
Full workflow118/120 (98.3%)—
Without model repair115/120 (95.8%)5/120 (4.2%)
Without scientific grounding117/120 (97.5%)117/120 (97.5%)
Without semantic/capability reconciliation114/120 (95.0%)12/120 (10.0%)
Without readiness gate117/120 (97.5%)4/120 (3.3%)
Table 6: Component ablation results for A2 (v1.8). Entries show counts and percentages. Component exercised counts requests that used the removed component in the reference workflow. Each block states its full-workflow reference; all evaluated requests remain in the denominators.
Figure 3: Component ablation results. Values are full-workflow accuracy minus ablated accuracy, in percentage points, on matched requests. Positive values favour the full workflow. The 120-request comparisons use a reference score of 118/120118/120 ; the 40-request input controls use 40/4040/40 . All groups share one horizontal scale. Output-format removal causes parsing failures on all 120 requests. The hollow marker identifies unexercised mapping hints. Schema and routing controls use new model draws, so their differences also include variation between responses.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

FieldContents and requirements
“schema_version”Identifies the problem contract; version 2.0 is used.
“title”Descriptive label; not scientific evidence.
“original_query”Original request, retained unchanged. Later user replies are stored separately.
“geometry”Geometry type, dimension, and required parameters, such as extents, radii, or wing dimensions.
“mesh”Mesh backend, resolution, and cell settings, consistent with the geometry and dimension.
“finite_element”Element family and degree. Lagrange degrees one and two are supported, subject to solver requirements.
Appendix
Table 7: Fields of the solver-facing JSON specification. Requirements depend on the selected geometry and equation.
ItemSetting
Model snapshotgpt-5.4-mini-2026-03-17
Reasoning effort“none”
Maximum output6,400 tokens per model call
Provider storageDisabled
Assisted extractionStrict JSON-schema output; the schema is also supplied in the instructions
Model-only baselineJSON-object output with the generic problem schema supplied as text; separate prompt and model response
Appendix
Table 8: Generation settings for the principal agentic studies. Requests and responses are retained locally.
MemberMeaning and requirement
“contract_version”Must identify “phase2-context-patch-1.0”.
“base_revision”Must match the current context revision.
“source_turn”Identifies the authorizing user turn or repair event.
“operations”Up to 32 explicit “add”, “replace”, or “remove” operations. An empty proposal is recorded as declined.
“path”A permitted, non-root JSON Pointer. Duplicate or overlapping paths are rejected.
“value_json”JSON-encoded value for “add” and “replace”; null for “remove”.
Appendix
Table 9: Patch contract used for clarification, amendment, and repair.
IDProblem and domainMesh and time sequenceSolves
V12D elasticity; unit square; plane stressP1 triangles; n=8,16,32,64n=8,16,32,644
V32D steady flow; unit square with a circular holeP2/P1 velocity/pressure; h=0.06,0.03,0.015h=0.06,0.03,0.015 ; pressure patch at h=0.06h=0.064
V42D unsteady heat; unit-radius diskP1; h=0.3h=0.3 ; five step sizes per scheme and one equilibrium control11
V62D annular heat; radii 0.350.35 and 11P1; h=0.00175h=0.00175 ; three BE and five CN step sizes8
V72D rectangular heat; unit squareP1 triangles; 768×768768\times 768 cells; three BE and five CN step sizes8
V23D elasticity; unit cubeP1 tetrahedra; n=4,8,16,24n=4,8,16,244
Appendix
Table 10: Setup for the original verification campaign. For structured meshes, nn denotes cells per coordinate direction. For Gmsh, hh is the requested characteristic mesh length.
ItemSetting
SoftwarePython 3.12.3; DOLFINx 0.11.0.post0; UFL 2026.1.0; Gmsh 4.12.1; PETSc 3.19.6; NumPy 1.26.4
Linear solverPETSc preonly with LU; relative tolerance 10−810^{-8} , absolute tolerance 10−1010^{-10} ; maximum 10,000 iterations
Nonlinear solverNewton with backtracking; relative tolerance 10−810^{-8} , absolute tolerance 10−1010^{-10} ; maximum 50 iterations; nested linear settings specify GMRES with ILU
Tighter-tolerance controlsRelative tolerance 10−1010^{-10} and absolute tolerance 10−1210^{-12} in the linear, nonlinear, and nested linear settings
Error integrationQuadrature degree 12 for V1–V5; degree 8 for V6–V7 and cantilever quantities
Execution checksFinite solution coefficients, converged solver steps, and no invalid cells
Appendix
Table 11: Software and requested solver settings for the original campaign. Complete case settings are retained in the input JSON files.
Figure 4: Spatial convergence for V1–V3. Orders are fitted using the finest three levels. Elasticity uses P1 displacement elements; flow uses P2 velocity and P1 pressure. The L2L^{2} and H1H^{1} seminorm errors have different units.
IDFieldL2L^{2} order and errorH1H^{1} order and errorOutcome
V12D displacement2.000 4.46×10−74.46\times 10^{-7}1.000 9.02×10−59.02\times 10^{-5}Pass
V23D displacement2.000 3.17×10−63.17\times 10^{-6}1.000 2.41×10−42.41\times 10^{-4}Pass
V32D velocity2.976 2.11×10−82.11\times 10^{-8}2.004 1.09×10−51.09\times 10^{-5}Pass
Appendix
Table 12: Spatial verification results. Errors are absolute integrated norms on the finest mesh. The H1H^{1} column reports seminorm errors.
Figure 5: Temporal convergence for V4 and V5 at T=0.4T=0.4 . All five time steps are shown; fitted orders use the finest three. Stationary controls are checked separately.
IDStudyBE orderCN orderSpatial ratioOutcome
V4Affine heat0.9912.000Control passedPass
V5Affine reaction–diffusion1.0052.000Control passedPass
V6Annular heat1.0111.94110.84%10.84\%Fail
V7Rectangular heat1.0081.9616.83%6.83\%Pass
Appendix
Table 13: Temporal verification results. Accepted order bands are [0.8,1.2][0.8,1.2] for BE and [1.65,2.35][1.65,2.35] for CN. The spatial ratio is the larger of the two scheme ratios; CN determines it for both V6 and V7.
Figure 6: Temporal refinement with nonzero spatial error. Upper panels show reference errors at the three primary time steps. Two additional CN runs estimate the spatial error. The lower panel shows the CN ratio from Equation 3 . V6 exceeds the 10%10\% limit; V7 passes.
Figure 7: MPI agreement on the same cantilever mesh. Displacement-coefficient differences are normalized by the serial coefficient-vector norm. The second curve gives the largest relative difference in tip deflection, strain energy, or integrated stress, normalized by the corresponding absolute serial value. The test measures consistency, not parallel speedup.
IDComparisonNative-field differenceReported-quantity differenceLimit
V84 MPI processes2.08×10−112.08\times 10^{-11}3.70×10−113.70\times 10^{-11}10−710^{-7}
 8 MPI processes9.79×10−129.79\times 10^{-12}1.59×10−111.59\times 10^{-11}10−710^{-7}
 16 MPI processes2.40×10−112.40\times 10^{-11}4.44×10−114.44\times 10^{-11}10−710^{-7}
 32 MPI processes2.11×10−112.11\times 10^{-11}3.84×10−113.84\times 10^{-11}10−710^{-7}
V92D flow, tighter tolerances8.12×10−108.12\times 10^{-10}3.01×10−73.01\times 10^{-7}10−610^{-6}
 3D reaction–diffusion, tighter tolerances000010−610^{-6}
Appendix
Table 14: Parallel and tighter-tolerance agreement. The reported-quantity column gives the largest relative difference among cantilever quantities for V8 or recorded error metrics for V9. Zero indicates no difference at saved numerical precision.
StudyRepeated setupSolvesResultOutcome
V12D elasticity; n=8,16,32n=8,16,323Orders: 2.000 / 1.000Pass
V32D flow; three mesh levels and pressure patch4Orders: 2.917 / 1.938; pressure patch passedPass
V42D heat; three steps per scheme and equilibrium control7Orders: 0.975 / 2.003; control passedPass
V23D elasticity; n=4,8,16n=4,8,163Orders: 2.000 / 1.000Pass
V53D reaction–diffusion; three steps per scheme and equilibrium control7Orders: 1.012 / 2.000; control passedPass
V9Flow and reaction–diffusion with tighter tolerances2Field differences: 8.12×10−108.12\times 10^{-10} and 00Pass
Appendix
Table 15: Additional verification repetitions. Spatial results give L2/H1L^{2}/H^{1} seminorm orders; temporal results give BE/CN orders. Orders use the two finest levels, whereas the original campaign uses three-level fits.
IDRequested problemOutcome
E11D Poisson, interval ( L=1.340L=1.340 )Verified
E21D Poisson, interval ( L=2.153L=2.153 )Verified
E31D Poisson, interval ( L=2.257L=2.257 )Blocked
E42D Poisson, rectangleVerified
E52D Poisson, annulusVerified
E62D unsteady heat, rectangleBlocked
Appendix
Table 16: All selected live requests, ordered by dimension. Verified means joint success: correct specification, converged solve, complete exports, and a passed reference check. Blocked means no solver invocation.
Figure 8: Outcomes and timings for all 12 requests. Bars show elapsed time from request receipt to the saved result or blocked outcome. Diamonds show worker time for executed cases. The clocks are measured separately and are not additive. Blocked cases have no numerical-execution time.
StudyModel-onlySSFull workflow
Development—38/60 (63.3%)46/60 (76.7%)
Initial holdout—22/40 (55.0%)20/40 (50.0%)
Confirmation 1—40/40 (100.0%)26/40 (65.0%)
Confirmation 2—37/40 (92.5%)38/40 (95.0%)
Confirmation 3—38/40 (95.0%)37/40 (92.5%)
Confirmation 432/40 (80.0%)37/40 (92.5%)39/40 (97.5%)
Appendix
Table 17: Originally recorded first-turn specification and decision outcomes. Entries are correct/evaluated requests (percentage). SS denotes schema-assisted single shot. A dash means that no model-only result was recorded for that study. Model-only uses a separate prompt and model draw.
Request groupMSF
Task
Extraction42/48 (87.5%)47/48 (97.9%)47/48 (97.9%)
Complete specification28/30 (93.3%)28/30 (93.3%)29/30 (96.7%)
Clarification17/30 (56.7%)30/30 (100.0%)30/30 (100.0%)
Rejection/conflict12/12 (100.0%)6/12 (50.0%)12/12 (100.0%)
Wording
Appendix
Table 18: Main-benchmark first-turn outcomes by task, wording, and difficulty. Entries are correct/evaluated requests (percentage). M is model-only, S is schema-assisted single shot, and F is the full workflow. Each block separately accounts for all 120 requests.
Figure 9: First-turn outcomes for all 120 main-benchmark requests. Each column is one request, with aligned cells for all three systems. Requests are grouped by task and then by their original record ID; labels give task counts. Correct clarification and rejection decisions count as correct outcomes. The cells do not indicate solver execution.
ConditionConfirmation 4 (v1.7)Main study (v1.8)
Full-cohort comparisons
Full-workflow reference39/40 (97.5%)118/120 (98.3%)
Without repair39/40 (97.5%)115/120 (95.8%)
Without scientific grounding39/40 (97.5%)117/120 (97.5%)
Without semantic/capability checks37/40 (92.5%)114/120 (95.0%)
Without readiness gate38/40 (95.0%)117/120 (97.5%)
Appendix
Table 19: Component-removal outcomes in two studies. Entries are correct/evaluated requests (percentage). The first block uses the full cohorts; the second uses the input-control requests and its own reference. No combined accuracy is formed across the two studies.
StudyCorrect / evaluatedPercentage
Confirmation 4 (v1.7)10/10100.0%
Main study: A3 (v1.8)28/3093.3%
Earlier-cohort rerun (v1.8)32/5064.0%
Main-study subset (v1.9)14/1877.8%
Appendix
Table 20: Correct complete specifications after fixed authorized clarification replies. Each row uses its own retained responses and selected clarification requests; these are not first-turn accuracy scores. Later rows reuse known requests rather than adding independent cases.
Test or systemEarlier v1.8Updated v1.9
A6: saved-final-JSON replay, full workflow
Main-benchmark contexts118/120 (98.3%)119/120 (99.2%)
Broader-input contexts86/130 (66.2%)111/130 (85.4%)
A7: fresh responses, 150 matched requests
Model-only118/150 (78.7%)127/150 (84.7%)
Schema-assisted single shot112/150 (74.7%)89/150 (59.3%)
Appendix
Table 21: Updated-workflow tests. Entries are correct/evaluated requests (percentage). A6 scores retained final JSON contexts; A7 scores first-turn specification and decision outcomes from new responses. Earlier A7 scores use exactly the same 150 selected requests. Replay and fresh-response results must not be pooled.
Figure 10: Fresh-response regression A7, with earlier and updated scores calculated on the same requests in each panel: (a) 72 main-benchmark requests, (b) 78 broader-input requests, and (c) all 150 selected requests. The first two panels partition the third and are not additional tests. Values are percentages of correct first-turn specification and decision outcomes. Single shot and the full workflow share their initial response; model-only uses a separate prompt and draw. New responses accompany the workflow update, so changes cannot be attributed to the version alone. All panels use the same scale.
More in this topic

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 868 papers

Explore similar work

CardsList
May 28, 2026

SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents

Software engineering (SWE) agents are transitioning from code generation to full software development lifecycle automation. A critical phase in this lifecycle is specification design: transforming initial proposals into carefully considered requirements through expert review. Existing benchmarks such as SWE-Bench are implementation-focused by measuring the agent's ability to generate code given fixed, precise design requirements. This formulation assumes specifications are correct and complete. In real-world complex and critical software systems, initial specifications are often incomplete and flawed, requiring extensive expert reviews and revisions before being accepted for implementation. To fill this gap, we introduce SpecBench to evaluate specification-level reasoning: the ability to generate complete, unambiguous, consistent, and correct system specifications. SpecBench tasks are derived from the Request for Comments (RFC) process used by mature open-source projects. For each task, an agent is given an initial design proposal, the project codebase, and all past project RFC discussions. The agent is tasked with identifying specification deficiencies: omissions, ambiguities, inconsistencies, or incorrect assumptions in the initial proposal. We evaluate predictions against critiques raised by expert maintainers during historical RFC reviews. SpecBench contains tasks from 5 diverse repositories: Kubernetes, React, Rust, TVM, and vLLM. We evaluate state-of-the-art SWE agents on SpecBench, analyzing their capacity to reason about system design without execution feedback. The best performing agent, GPT-5.4, achieves 44.4% accuracy.

Grant Hamblin, Kevin Song, Zhanda Zhu +4

University of Toronto · Toronto, Ontario, Canada · University of Waterloo +3

Requirements EngineeringSoftware Engineering AgentsSoftware Engineering BenchmarksLLM Agent Evaluation
AskPDF
Sep 24, 2026

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Haiqing Li, Xin Ma, Yinhao Wu +7

Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA

LLM Agent VerificationRuntime Enforcement for AI AgentsLLM Agent Harnesses
AskPDF
Mar 26, 2026

A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation

Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.

Chengshuai Yang

NextGen PlatformAI C Corp, USA

Scientific Code GenerationLLM ReliabilityCode Generation Benchmarks
AskPDF