Scientific Coding with Large Language Models: Evidence from Finite Element Analysis
Organizations: Department of Mechanical Engineering, Boston University, Boston, MA 02215 · AI Bio Design, Allen Institute & Move37 Labs · Department of Mechanical Engineering, Division of Systems Engineering, Boston University, Boston, MA 02215
Abstract
Large language models (LLMs) offer a potential route to automating scientific programming, but their usefulness depends on consistently producing numerically correct code. We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models on implementation and test generation without execution feedback. The best model succeeds in 87.1% of single-pass implementation trials, but rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%. For six models, we compare single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur. Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3, the probability of at least one success in three attempts, but 77.1% Pass, the probability that all three succeed. No model-harness combination exceeds 78.8% Pass. To understand why agents submit incorrect code after passing their own checks, we analyze 249 failed agentic trajectories and identify six categories of self-verification failure. The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%), and in 28.5% of cases the tests share an error with the implementation. Motivated by these findings, an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass. These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.
Figures & tables
| Domain | Task Name | Description | H | T |
|---|---|---|---|---|
| FEM 1D | isotropic hardening | Elastoplastic stress update with linear isotropic hardening | 0 | 0 |
| FEM 1D | kinematic hardening | Elastoplastic stress and back-stress update with linear kinematic hardening | 0 | 0 |
| FEM 1D | linear elastic dynamics | Elastic bar response using quadratic elements and explicit Newmark integration | 1 | 1 |
| FEM 2D | circular mesh | Triangular mesh on a circular domain using prescribed connectivity | 0 | 0 |
| FEM 2D | circular mesh (Delaunay) | Triangular mesh on a circular domain using Delaunay triangulation | 0 | 0 |
| FEM 2D | Specht shape functions and derivatives | Shape functions and first and second derivatives for a Specht triangular element | 0 | 0 |
| Model | Developer | Access | Provider | Reasoning | Output cap | Price input/output |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | Anthropic | Proprietary | AWS Bedrock | Adaptive (high) | 128,000 | 10.00 / 50.00 |
| Claude Opus 5 | Anthropic | Proprietary | AWS Bedrock | Adaptive (high) | 128,000 | 5.00 / 25.00 |
| Claude Sonnet 5 | Anthropic | Proprietary | AWS Bedrock | Adaptive (high) | 128,000 | 2.00 / 10.00 |
| DeepSeek V4 Flash 0731 | DeepSeek | Open weights | Together AI, OpenRouter a | High | 393,216 | 0.14 / 0.28 |
| DeepSeek V4 Pro 0813 | DeepSeek | Open weights | Together AI | High | 393,216 | 1.32 / 3.96 |
| Gemini 3.1 Pro Preview | Proprietary | High | 65,536 | 2.00 / 12.00 |
| Configuration | Single-pass | SimpleAgent | OpenHands | coder-tester |
| Models | 15 models | 6 models | 6 models | Fable/Astra pair |
| Tracks | Implementation, Test generation | Implementation | Implementation | Implementation |
| Tools and actions | No tools | Python execution, submit | Terminal, file editor, browser, planning, finish | Python execution, feedback, submit |
| Execution state | No execution loop | Fresh sandbox for each execution | Persistent workspace files | Fresh sandbox, tests retained |
| Internet access | — | Disabled | Enabled | Disabled |
| Out-token cap per call | Model-specific | 30,000 | 30,000 | 32,000 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Function implementation (%) | Joint test success rate (%) |
|---|---|---|
| Claude Fable 5.1 | 87.1 | 49.9 |
| Claude Opus 5 | 78.8 | 64.6 |
| Claude Sonnet 5 | 68.2 | 42.8 |
| GPT-6 Astra | 82.4 | 82.6 |
| GPT-5.6 Sol | 68.2 | 76.5 |
| GPT-5.6 Luna | 54.1 | 74.3 |
| Model | Method | Newly solved | Improved | Lost | Degraded | Unchanged |
|---|---|---|---|---|---|---|
| Claude Opus 5 | SimpleAgent | 1 | 4 | 0 | 1 | 11 |
| OpenHands | 0 | 5 | 0 | 2 | 10 | |
| GPT-5.6 Sol | SimpleAgent | 0 | 5 | 0 | 1 | 11 |
| OpenHands | 0 | 6 | 0 | 0 | 11 | |
| GPT-5.6 Luna | SimpleAgent | 2 | 3 | 0 | 2 | 10 |
| OpenHands | 2 | 2 | 1 | 1 | 11 |
| Model | Single-pass | SimpleAgent | OpenHands |
|---|---|---|---|
| Claude Fable 5.1 | 93.5 | — | — |
| Claude Opus 5 | 90.6 | 96.5 | 91.2 |
| Claude Sonnet 5 | 79.4 | — | — |
| GPT-6 Astra | 93.5 | — | — |
| GPT-5.6 Sol | 84.1 | 85.3 | 88.2 |
| GPT-5.6 Luna | 70.6 | 80.6 | 76.5 |
| Model | Single-pass | SimpleAgent | OpenHands |
|---|---|---|---|
| Claude Fable 5.1 | 79.4 | — | — |
| Claude Opus 5 | 65.9 | 77.1 | 78.8 |
| Claude Sonnet 5 | 54.7 | — | — |
| GPT-6 Astra | 68.8 | — | — |
| GPT-5.6 Sol | 52.4 | 64.1 | 72.4 |
| GPT-5.6 Luna | 38.8 | 38.2 | 37.6 |
| Model | Cases | VF-01 | VF-02 | VF-03 | VF-04 | VF-05 | VF-06 | NV | Other |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 19 | 10 | 13 | 0 | 7 | 1 | 0 | 0 | 0 |
| GPT-5.6 Sol | 26 | 16 | 17 | 0 | 7 | 0 | 1 | 0 | 0 |
| GPT-5.6 Luna | 68 | 59 | 55 | 2 | 11 | 4 | 10 | 0 | 0 |
| Kimi K3 | 37 | 25 | 21 | 4 | 16 | 10 | 2 | 0 | 0 |
| GLM-5.2 | 40 | 29 | 23 | 4 | 13 | 6 | 1 | 0 | 0 |
| DeepSeek V4 Flash 0731 | 59 | 43 | 46 | 1 | 17 | 9 | 1 | 0 | 0 |
| Code | Category | SimpleAgent | OpenHands | All | All (%) |
|---|---|---|---|---|---|
| VF-01 | Inadequate Test Configuration and Coverage | 105 | 77 | 182 | 73.1 |
| VF-02 | Generic Consistency Checks | 103 | 72 | 175 | 70.3 |
| VF-04 | Non-Independent Verification | 40 | 31 | 71 | 28.5 |
| VF-05 | Test Accommodation | 17 | 13 | 30 | 12.0 |
| VF-06 | Unverified Post-Test Modification | 14 | 1 | 15 | 6.0 |
| VF-03 | Ineffective Test Assertions | 9 | 2 | 11 | 4.4 |
| Setting | SimpleAgent | OpenHands | coder-tester |
|---|---|---|---|
| Output tokens per call | 30,000 | 30,000 | 32,000 (either role) |
| Iterations / steps | 200 | 200 | 200 per agent |
| Python / scratch timeout | 120 s | – | 60 s |
| Full-suite timeout | – | – | 300 s |
| Execution-output limit | 10,000 characters | – | 10,000 characters |
| Coder scratch executions | – | – | 3 per candidate |
| Model | Role | Provider | Reasoning | Output token cap |
|---|---|---|---|---|
| GPT-6.1 Sol | Initial judge | AWS | High | 131,072 |
| Claude Opus 5.5 | Initial judge | AWS | Adaptive (high) | 128,000 |
| Grok 4.7 | Initial judge | AWS | High | 128,000 |
| GPT-6 Astra | Adjudicator | AWS | High | 128,000 |
| Code | Category | Prevalence (%) | |
|---|---|---|---|
| VF-01 | Inadequate Test Configuration and Coverage | 68.1 | 0.612 |
| VF-02 | Generic Consistency Checks | 66.1 | 0.665 |
| VF-03 | Ineffective Test Assertions | 3.1 | 0.551 |
| VF-04 | Non-Independent Verification | 23.4 | 0.731 |
| VF-05 | Test Accommodation | 10.0 | 0.718 |
| VF-06 | Unverified Post-Test Modification | 4.7 | 0.880 |
Scientific Code Generation
Scientific code generation develops systems that turn scientific questions, papers, and workflows into executable, scientifically valid programs.
See all 54 papers