cs.SENov 16, 2026

Scientific Coding with Large Language Models: Evidence from Finite Element Analysis

Authors: Erfan Hamdi, Joel Shor, Emma Lejeune

Organizations: Department of Mechanical Engineering, Boston University, Boston, MA 02215 · AI Bio Design, Allen Institute & Move37 Labs · Department of Mechanical Engineering, Division of Systems Engineering, Boston University, Boston, MA 02215

Abstract

Large language models (LLMs) offer a potential route to automating scientific programming, but their usefulness depends on consistently producing numerically correct code. We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models on implementation and test generation without execution feedback. The best model succeeds in 87.1% of single-pass implementation trials, but rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%. For six models, we compare single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur. Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3, the probability of at least one success in three attempts, but 77.1% Pass3^3, the probability that all three succeed. No model-harness combination exceeds 78.8% Pass3^3. To understand why agents submit incorrect code after passing their own checks, we analyze 249 failed agentic trajectories and identify six categories of self-verification failure. The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%), and in 28.5% of cases the tests share an error with the implementation. Motivated by these findings, an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass3^3. These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6Scientific Code GenerationCode Generation Evaluation

  2. Re4: Scientific Computing Agent with Rewriting, Resolution, Review and Revision

    Aug 28, 2025Ao Cheng, Lei Zhang, Guowei HeScientific ComputingScientific Code Generation