Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science
Authors: Maohao Ran, Chendong Ma, Yanting Zhang, Dailing Jiang, Yusen Huang, Meng Gao, Jun Song
Organizations: Department of Geography, Hong Kong Baptist University, Hong Kong, China. · Hong Kong University of Science and Technology, Hong Kong, China. · The Chinese University of Hong Kong (Shenzhen), Shenzhen, China.
Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.
Figures & tables
Category
Problems
Follow-up
Numeric
Paraphrase
queries
variants
variants
Atmospheric dynamics
116
47
520
580
Atmospheric thermodynamics
89
59
395
445
Atmospheric chemistry
51
76
160
255
Air quality
37
8
130
185
Boundary layer
32
16
145
160
Table 1: Composition of AtmosCoder-Bench by category. 436 base problems from 13 established textbooks spanning introductory to advanced level (undergraduate to graduate), each labelled with one of ten atmospheric–environmental categories. Follow-up queries counts the additional sub-questions a problem poses beyond its first (a problem asking for parts (1)–(3) contributes two); every sub-question is graded independently. Each perturbable problem carries five parameter-perturbed numeric variants (346 problems; contamination resistance) and every problem five paraphrase variants (phrasing robustness; answer unchanged).
Figure 1: The AtmosCoder-Bench admission pipeline. Textbook items pass a sequence of gates—extraction for self-contained numeric answerability, an executability gate, cross-vendor independent-derivation agreement (textbook-anchored or consensus-frozen), an AST + perturbation faithfulness audit, a well-posedness audit and triple-gated de-scaffolding—before certified, contamination-resistant variants are generated. Items failing a gate are removed (right), so the admitted set carries verifiable, execution-grounded ground truth.
Model
Option mode
Code mode (grading tolerance)
Δ
raw
chance-corr.
±5%
±10%
±20%
Gemini-3.1-Pro (r)
94.5%
92.7%
60.0%
61.0%
61.9%
+30.8
DeepSeek-R1 (r)
88.5% ∗
84.7%
53.0%
54.6%
55.7%
+29.0
GPT-5.5
80.1%
73.5%
49.6%
50.6%
53.0%
+20.5
DeepSeek-V4-flash
88.1%
84.1%
43.3%
44.0%
45.8%
+38.3
DeepSeek-V3
63.3% ∗
51.1%
27.2%
28.1%
29.6%
+21.5
Table 2: Multiple-choice inflation: option mode versus computed answers on identical problems. Δ is chance-corrected option accuracy minus computed accuracy at the most permissive ±20% tolerance; “(r)” denotes a reasoning configuration. ∗ Option accuracy from the source benchmark [ 33 ] .
Figure 2: Overall model performance under execution-grounded evaluation. a , Code-mode accuracy (mean ± s.d. over three runs) for the 16 leaderboard configurations, coloured by reasoning setting (labels ending in (R) denote reasoning enabled), together with the two domain-adapted ClimateGPT models shown separately as an out-of-distribution reference. b , Distribution of output tokens per problem under the two protocols, for the six configurations run under both, with the median direct-to-code ratio at the right. c , Accuracy against token cost; arrows join the non-reasoning and reasoning settings of the same backbone, dashed contours mark equal tokens per correct answer, and † marks the two configurations whose token counts are lower bounds because the endpoint returns only a summary of the reasoning.
Figure 3: Performance variation across atmospheric domains and failure modes. a , Accuracy across atmospheric subfields and difficulty levels. b , Per-trap outcomes for each configuration across the six trap mechanism families. Red denotes failures returning the exact shortcut value predicted during trap construction, whereas grey denotes other failures; pale grey marks a solved trap.
Figure 4: Failure modes on the Dogpatch Skunk Works Gaussian-plume task. No configuration solves it, and the failures differ in kind: strong models misreport the parameterisation used, intermediate models misclassify regimes or alter formulas, and weaker models show unstable sign, unit, and arithmetic errors.
Hefei National Laboratory College of Mechanical and Electrical Engineering University of Science and Technology of China · Hefei National Laboratory Institute of Theoretical Physics University of Science and Technology of China Chinese Academy of Sciences Hefei 230026, China · Chinese Academy of Sciences Beijing 100190, China
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom