Scientific Coding with Large Language Models: Evidence from Finite Element Analysis
Authors: Erfan Hamdi, Joel Shor, Emma Lejeune
Organizations: Department of Mechanical Engineering, Boston University, Boston, MA 02215 · AI Bio Design, Allen Institute & Move37 Labs · Department of Mechanical Engineering, Division of Systems Engineering, Boston University, Boston, MA 02215
Large language models (LLMs) offer a potential route to automating scientific programming, but their usefulness depends on consistently producing numerically correct code. We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models on implementation and test generation without execution feedback. The best model succeeds in 87.1% of single-pass implementation trials, but rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%. For six models, we compare single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur. Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3, the probability of at least one success in three attempts, but 77.1% Pass3, the probability that all three succeed. No model-harness combination exceeds 78.8% Pass3. To understand why agents submit incorrect code after passing their own checks, we analyze 249 failed agentic trajectories and identify six categories of self-verification failure. The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%), and in 28.5% of cases the tests share an error with the implementation. Motivated by these findings, an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass3. These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.
Figures & tables
Domain
Task Name
Description
H
T
FEM 1D
isotropic hardening
Elastoplastic stress update with linear isotropic hardening
0
0
FEM 1D
kinematic hardening
Elastoplastic stress and back-stress update with linear kinematic hardening
0
0
FEM 1D
linear elastic dynamics
Elastic bar response using quadratic elements and explicit Newmark integration
1
1
FEM 2D
circular mesh
Triangular mesh on a circular domain using prescribed connectivity
0
0
FEM 2D
circular mesh (Delaunay)
Triangular mesh on a circular domain using Delaunay triangulation
0
0
FEM 2D
Specht shape functions and derivatives
Shape functions and first and second derivatives for a Specht triangular element
0
0
Table 1: Summary of the 17 tasks evaluated in this study. Tasks are grouped by domain. H denotes the reference-helper count, and T the helper-availability tier. Section 3.1 defines these annotations.
Model
Developer
Access
Provider
Reasoning
Output cap
Price input/output
Claude Fable 5.1
Anthropic
Proprietary
AWS Bedrock
Adaptive (high)
128,000
10.00 / 50.00
Claude Opus 5
Anthropic
Proprietary
AWS Bedrock
Adaptive (high)
128,000
5.00 / 25.00
Claude Sonnet 5
Anthropic
Proprietary
AWS Bedrock
Adaptive (high)
128,000
2.00 / 10.00
DeepSeek V4 Flash 0731
DeepSeek
Open weights
Together AI, OpenRouter a
High
393,216
0.14 / 0.28
DeepSeek V4 Pro 0813
DeepSeek
Open weights
Together AI
High
393,216
1.32 / 3.96
Gemini 3.1 Pro Preview
Google
Proprietary
Google
High
65,536
2.00 / 12.00
Table 2: Models and single-pass generation settings. All 15 models were evaluated in the implementation and test-generation tracks. Provider identifies the API service used in the retained runs. Developer identifies the model developer. Output caps are the recorded token limits per call in the single-pass setting. Reasoning denotes the configured setting. Prices are uncached input/output rates in USD per million tokens. Agentic output caps are given separately in Table 3 .
Configuration
Single-pass
SimpleAgent
OpenHands
coder-tester
Models
15 models
6 models
6 models
Fable/Astra pair
Tracks
Implementation, Test generation
Implementation
Implementation
Implementation
Tools and actions
No tools
Python execution, submit
Terminal, file editor, browser, planning, finish
Python execution, feedback, submit
Execution state
No execution loop
Fresh sandbox for each execution
Persistent workspace files
Fresh sandbox, tests retained
Internet access
—
Disabled
Enabled
Disabled
Out-token cap per call
Model-specific
30,000
30,000
32,000
Table 3: Experimental configurations. Each setting uses the 17 tasks in Table 1 , with five trials per task and model or coder-tester pairing. Single-pass implementation and test generation are separate tracks. The six shared models and Fable/Astra pairings are identified in Table 8 .
Figure 1: Overview of agentic harnesses studied. (a) Overview of SimpleAgent. SimpleAgent follows a ReAct-style loop with two actions: Python execution (test) and submission (submit). The Python execution action runs code in a fresh sandbox preloaded with the task’s required imports and helper functions. Execution is stateless: variables and files do not persist between executions, while the LLM retains the conversation history and receives execution output and exit codes as observations. Submission is permitted only after at least one execution and only if the most recent execution returned exit code 0. (b) Overview of the OpenHands coding agent. OpenHands is an open-source framework for coding agents. In this study, the agent had access to tools for web browsing, terminal execution, file editing, task tracking, planning, delegation to subagents, and a finish action to end the session. The agent ran inside a Singularity container configured to isolate it from the local reference implementations while retaining internet access.
Figure 2: Overview of the LLM-as-judge workflow for categorizing self-verification failures. Three initial judges independently assess each input package using the codebook, assigning all applicable labels with a short rationale for each. Their label sets are then compared case by case. When all three judges assign identical label sets, the shared classification is retained as final. Cases with any label disagreement are passed to a final adjudicator. The final adjudicator uses the original input package, all three judges’ verdicts and rationales, and the codebook to determine the final classification.
Figure 3: Single-pass performance on implementation and test-generation tasks. Each point represents one of the 15 evaluated models listed in Table 2 . Results are computed separately for the implementation and test-generation tracks. Each track uses five trials per task (5 × 17 = 85 runs per model). The horizontal axis shows function correctness: the percentage of generated implementations that pass the benchmark evaluation. The vertical axis shows the average joint test success rate: the fraction of generated tests that both pass the reference implementation and reject the designated buggy implementations. Both of these metrics are averaged across tasks and trials. Higher values indicate better performance on each metric. The differing model rankings across the two axes show that stronger implementation performance does not necessarily mean stronger test-generation performance.
Figure 4: Changes in task reliability relative to the single-pass baseline. Each row represents one model–harness combination across 17 tasks, with reliability measured by the number of successful runs out of five trials. Tasks are classified as newly solved when the success count increases from zero to a positive value, and as improved reliability when it increases from an already positive value. Degraded reliability indicates a decrease that remains above zero, whereas lost tasks have a positive baseline success count that falls to zero. Improvements extend to the right and regressions to the left. Unchanged task counts are shown separately. Most gains reflect improved reliability on previously solved tasks.
Figure 5: Pass@3 performance versus recorded cost. Pass@3 is estimated from five trials per task and averaged across 17 tasks. The horizontal axis shows the total recorded cost across all 85 runs on a logarithmic scale. Dashed lines connect the Pareto-optimal configurations within the single-pass, SimpleAgent, and OpenHands settings. The star denotes a multi-agent configuration built on SimpleAgent, with GPT-6 Astra as the coder and Claude Fable 5.1 as the tester. SimpleAgent offers competitive performance at lower recorded cost than OpenHands for the evaluated models, while the coder-tester configuration achieves the highest Pass@3 among the configurations shown.
Figure 6: Pass 3 performance versus recorded cost. Pass 3 estimates the probability that all three attempts succeed, using five trials per task and averaging across 17 tasks. The horizontal axis shows the total recorded cost across all 85 runs on a logarithmic scale. Dashed lines connect the Pareto-optimal configurations within the single-pass, SimpleAgent, and OpenHands settings. The star denotes a multi-agent configuration built on SimpleAgent, with GPT-6 Astra as the coder and Claude Fable 5.1 as the tester. SimpleAgent achieves higher Pass 3 than OpenHands for three of the six shared models at lower recorded cost, while the coder-tester configuration achieves the highest Pass 3 among the configurations shown.
Figure 7: Self-verification failure categories among completed failed submissions. (a) Categories grouped by model. Each bar shows the number of analyzed failed cases assigned to a particular category, pooled across SimpleAgent and OpenHands. Models are ordered by increasing number of analyzed failed cases. Inadequate Test Configuration and Coverage and Generic Consistency Checks are the most frequently assigned categories. A case may receive multiple labels, so category counts can sum to more than the number of cases for a model. (b) Distribution of self-verification failure categories. Bars show the percentage of the 249 analyzed failed cases assigned to each category, pooled across models and agentic harnesses. Categories are ordered by decreasing frequency, and numbers beside the bars indicate case counts. A case may receive multiple labels, so the percentages do not sum to 100%.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Function implementation (%)
Joint test success rate (%)
Claude Fable 5.1
87.1
49.9
Claude Opus 5
78.8
64.6
Claude Sonnet 5
68.2
42.8
GPT-6 Astra
82.4
82.6
GPT-5.6 Sol
68.2
76.5
GPT-5.6 Luna
54.1
74.3
Appendix
Table 4: Single-pass implementation and test-generation performance. Function implementation measures the fraction of submissions that pass the benchmark evaluation. Joint test success rate measures the fraction of generated tests that accept the reference implementation and reject all designated faulty implementations. Both metrics follow Section 3.3 and are percentages averaged over five trials per task, with equal weight across 17 tasks. Bold marks the highest value in each column.
Model
Method
Newly solved
Improved
Lost
Degraded
Unchanged
Claude Opus 5
SimpleAgent
1
4
0
1
11
OpenHands
0
5
0
2
10
GPT-5.6 Sol
SimpleAgent
0
5
0
1
11
OpenHands
0
6
0
0
11
GPT-5.6 Luna
SimpleAgent
2
3
0
2
10
OpenHands
2
2
1
1
11
Appendix
Table 5: Changes in task reliability relative to single-pass generation. Each task has five implementation trials. Newly solved tasks increase from zero to at least one successful trial. Improved tasks increase from a positive success count. Lost tasks fall to zero successes, while degraded tasks decrease but retain at least one success. Unchanged tasks have the same success count. Entries count tasks, and each row sums to 17.
Model
Single-pass
SimpleAgent
OpenHands
Claude Fable 5.1
93.5
—
—
Claude Opus 5
90.6
96.5
91.2
Claude Sonnet 5
79.4
—
—
GPT-6 Astra
93.5
—
—
GPT-5.6 Sol
84.1
85.3
88.2
GPT-5.6 Luna
70.6
80.6
76.5
Appendix
Table 6: Pass@3 (implementation) by model and method. Pass@3 estimates the probability of at least one successful implementation in three attempts (Equation 4 ). Values are percentages estimated from five trials per task and averaged equally across 17 tasks. Dashes indicate unevaluated combinations. Bold marks the highest score in each method column.
Model
Single-pass
SimpleAgent
OpenHands
Claude Fable 5.1
79.4
—
—
Claude Opus 5
65.9
77.1
78.8
Claude Sonnet 5
54.7
—
—
GPT-6 Astra
68.8
—
—
GPT-5.6 Sol
52.4
64.1
72.4
GPT-5.6 Luna
38.8
38.2
37.6
Appendix
Table 7: Pass 3 (implementation) by model and method. Pass 3 estimates the probability that all three implementations succeed in three attempts (Equation 5 ). Values are percentages estimated from five trials per task and averaged equally across 17 tasks. Dashes indicate unevaluated combinations. Bold marks the highest score in each method column.
Model
Cases
VF-01
VF-02
VF-03
VF-04
VF-05
VF-06
NV
Other
Claude Opus 5
19
10
13
0
7
1
0
0
0
GPT-5.6 Sol
26
16
17
0
7
0
1
0
0
GPT-5.6 Luna
68
59
55
2
11
4
10
0
0
Kimi K3
37
25
21
4
16
10
2
0
0
GLM-5.2
40
29
23
4
13
6
1
0
0
DeepSeek V4 Flash 0731
59
43
46
1
17
9
1
0
0
Appendix
Table 10: Self-verification failure categories by model. Counts pool failed SimpleAgent and OpenHands submissions. Cases gives the number of analyzed failed submissions for each model. Final labels combine 104 unanimous three-judge decisions and 145 adjudicated decisions. A case may receive multiple categories, so category counts can exceed the number of cases. Codes are expanded in Table 11 .
Code
Category
SimpleAgent
OpenHands
All
All (%)
VF-01
Inadequate Test Configuration and Coverage
105
77
182
73.1
VF-02
Generic Consistency Checks
103
72
175
70.3
VF-04
Non-Independent Verification
40
31
71
28.5
VF-05
Test Accommodation
17
13
30
12.0
VF-06
Unverified Post-Test Modification
14
1
15
6.0
VF-03
Ineffective Test Assertions
9
2
11
4.4
Appendix
Table 11: Distribution of self-verification failure categories. Counts cover 146 SimpleAgent and 103 OpenHands failed submissions. Percentages use all 249 analyzed failed cases as the denominator. A case may receive multiple labels, so category percentages need not sum to 100%. The last row reports cohort sizes rather than a sum of category counts.
Setting
SimpleAgent
OpenHands
coder-tester
Output tokens per call
30,000
30,000
32,000 (either role)
Iterations / steps
200
200
200 per agent
Python / scratch timeout
120 s
–
60 s
Full-suite timeout
–
–
300 s
Execution-output limit
10,000 characters
–
10,000 characters
Coder scratch executions
–
–
3 per candidate
Appendix
Table 12: Agent harness operational limits. Timeouts apply separately to each action or request. Other limits apply per trial unless stated. A dash indicates that no explicit setting is reported here.
Model
Role
Provider
Reasoning
Output token cap
GPT-6.1 Sol
Initial judge
AWS
High
131,072
Claude Opus 5.5
Initial judge
AWS
Adaptive (high)
128,000
Grok 4.7
Initial judge
AWS
High
128,000
GPT-6 Astra
Adjudicator
AWS
High
128,000
Appendix
Table 13: LLM judge configuration. Output token caps are measured in tokens per call. High denotes the configured reasoning effort. Claude Opus 5.5 uses adaptive reasoning with high effort.
Code
Category
Prevalence (%)
κ
VF-01
Inadequate Test Configuration and Coverage
68.1
0.612
VF-02
Generic Consistency Checks
66.1
0.665
VF-03
Ineffective Test Assertions
3.1
0.551
VF-04
Non-Independent Verification
23.4
0.731
VF-05
Test Accommodation
10.0
0.718
VF-06
Unverified Post-Test Modification
4.7
0.880
Appendix
Table 14: Initial-judge agreement and category prevalence. Fleiss’ κ measures agreement among three judges across 249 failed submissions before adjudication. Prevalence is the percentage of the 747 individual judgments assigning each category.
The State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences, Beijing 100190, China · School of Engineering Science, University of Chinese Academy of Sciences, Beijing 100049, China