cs.PFOct 13, 2026
SaveHow Reliable Are Large Language Models for Soil Organic Carbon Assessment? An Australian Benchmark
Organizations: School of Mathematics and Physics, The University of Queensland, QLD 4072, Australia · School of Agriculture and Food Sustainability, The University of Queensland, QLD 4072, Australia · College of Jiyang, Zhejiang A&F University, Zhuji 311800, China · Department of Soil, Water, and Ecosystem Sciences, University of Florida-IFAS, FL 32611, USA
Abstract
Soil organic carbon (SOC) is an indicator of agricultural soil condition and carbon storage, but numerical outputs from large language models (LLMs) require validation before use in monitoring or management. We evaluated seven general-purpose LLM configurations on three direct-response tasks: estimating 0--10 cm SOC concentrations for 308 Australian paddocks under two information conditions, calculating SOC stock in 120 matched scenarios, and comparing 300 paddock pairs. References were measured SOC records or deterministic calculations; response validity, repeated-call dispersion and estimated API cost were assessed separately. Additional measured soil attributes reduced concentration mean absolute error by 13--62% across configurations, although high-SOC underprediction and paddock-specific errors persisted. Six configurations were highly accurate in stock calculations using equation-native units without coarse fragments, but accuracy declined with physically equivalent alternative units. Pairwise accuracy was 90.5--94.6% overall and 59.1--70.9% for 52 closely separated pairs; some configurations were sensitive to presentation order. Nearly all outputs were parseable, without necessarily being correct. The benchmark identifies three distinct requirements for AI-mediated SOC assessment: agreement with measured concentrations, unit-consistent stock calculations and reliable discrimination between sites. These findings support task-specific validation and auditable calculations, rather than unverified use of LLM outputs as ecological evidence.
Figures & tables
Figure 1: Geographical distribution of the study paddocks in Australia. Points show the dataset-reported sampling locations of the 308 paddocks with 0–10 cm reference SOC values across six Australian states; they do not mark paddock centroids or boundaries, and colour does not encode SOC concentration. No sampling point was moved or omitted. Coordinates were projected to Australian Albers (EPSG:3577), and coastline and state polygons were simplified for display. Data sources: SOC-M v1 ( Karunaratne and others, 2026 ) , CSIRO Data Access Portal, DOI: 10.25919/z378-4286, © CSIRO 2025; and Australian Bureau of Statistics (ABS) Australian Statistical Geography Standard (ASGS) Edition 3 States and Territories 2021, © Commonwealth of Australia. Both datasets are used under CC BY 4.0 ( https://creativecommons.org/licenses/by/4.0/ ).
| State | Paddocks | Records | Records/site | Mean | SD | Median | Q1–Q3 | Min–max |
|---|---|---|---|---|---|---|---|---|
| NSW | 103 | 363 | 3.5 | 20.1 | 10.2 | 18.3 | 14.5–24.0 | 3.0–69.1 |
| WA | 79 | 326 | 4.1 | 23.0 | 20.0 | 14.1 | 5.9–35.7 | 3.0–72.4 |
| VIC | 64 | 210 | 3.3 | 30.4 | 23.1 | 22.1 | 13.5–44.7 | 1.1–93.0 |
| TAS | 25 | 160 | 6.4 | 42.1 | 16.9 | 39.7 | 29.0–51.8 | 15.7–82.4 |
| QLD | 21 | 93 | 4.4 | 11.6 | 4.3 | 11.8 | 9.7–14.1 | 4.8–18.7 |
| SA | 16 | 61 | 3.8 | 15.2 | 5.2 | 13.6 | 12.4–16.6 | 10.0–30.2 |
Table 1: Paddock-level reference SOC concentrations and retained analytical records for the 0–10 cm layer by Australian state
| Configuration | Provider | Model ID | Reasoning | Token limit |
|---|---|---|---|---|
| GPT-5.6 Sol | OpenAI | gpt-5.6-sol | none | 1,024 |
| GPT-5.6 Terra | OpenAI | gpt-5.6-terra | none | 1,024 |
| Claude Sonnet 5 | Anthropic | claude-sonnet-5 | disabled | 1,024 |
| Gemini 3.8 Flash | gemini-3.8-flash | low | 1,024 | |
| Grok 4.6 | xAI | grok-4.6 | low | 1,024 |
| DeepSeek V4 Pro | DeepSeek | deepseek-v4-pro | disabled | 1,024 |
Table 2: Evaluated model configurations
Figure 2: Benchmark design and evaluation framework. (A) T1 evaluated SOC concentration estimation for 308 paddocks under two nested information conditions: T1a added soil properties and historical land use to the location and depth information in T1b; B0, B1, B2 and B2 ′ are prespecified supervised reference baselines. T2 crossed a single-dimension unit-representation condition (U0/U1) with a coarse-fragment dry-mass condition (F0/F1) across 30 base scenarios, yielding 120 items; each response gave SOC stock in the requested unit. T3 sampled 300 pairs without replacement from 47,278 eligible unordered pairs and presented each pair ten times in each presentation order; GAP, the absolute difference in reference SOC between the two paddocks, was analysed as a continuous variable. (B) Seven configurations were evaluated through stateless, closed-book calls without tools or retrieval. Responses were scored before task-specific aggregation. Performance, repeated-call variability, response validity, token use and estimated API cost were reported separately, without a cross-task composite score. The total of 145,040 refers to planned responses.
| Task | Benchmark items | Primary metric | Main contrasts | Uncertainty |
|---|---|---|---|---|
| T1 | 308 paddocks 2 conditions (616 items); 20 responses per configuration–item; 86,240 planned responses | MAE (g C kg -1 ): absolute errors averaged within paddock and condition, then equally across paddocks | Paired T1b T1a error difference; supervised baselines B0–B2 ′ | Point estimates only |
| T2 | 30 base scenarios 4 matched versions (120 items); 16,800 planned responses | Accuracy (absolute relative error ): mean binary score across 20 responses per item, then equally across items | Matched U1 U0; F1 F0; U F interaction | Pointwise repeated-call 95% intervals |
| T3 | 300 paddock pairs 2 presentation orders; 42,000 planned responses | Direct accuracy: mean binary score within each order, then equal weighting of orders and pairs | 21 pairwise configuration differences; GAP and position associations | Pointwise repeated-call 95% intervals; pointwise Wald intervals for secondary estimates |
Table 3: Benchmark tasks, performance metrics and statistical contrasts
| Configuration | T1a MAE | T1b MAE | T1b - T1a | T2 accuracy (95% interval) | T3 accuracy (95% interval) |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 4.24 | 10.11 | +5.87 | 84.0% (83.2–84.7%) | 94.0% (93.9–94.2%) |
| GPT-5.6 Terra | 5.11 | 9.97 | +4.86 | 78.2% (77.3–79.1%) | 94.2% (94.0–94.3%) |
| Claude Sonnet 5 | 6.65 | 9.87 | +3.22 | 76.7% (75.6–77.7%) | 94.3% (94.2–94.4%) |
| Gemini 3.8 Flash | 3.62 | 8.78 | +5.17 | 89.0% (88.4–89.7%) | 94.6% (94.5–94.7%) |
| Grok 4.6 | 3.82 | 10.02 | +6.20 | 83.2% (82.6–83.8%) | 93.7% (93.4–93.9%) |
| DeepSeek V4 Pro | 6.34 | 9.87 | +3.53 | 76.1% (74.8–77.4%) | 93.5% (93.2–93.8%) |
Table 4: Summary performance across the three quantitative SOC tasks
Figure 3: Paddock-level SOC concentration error distributions. (A) For each of the seven configurations, each T1a or T1b box contains 308 paddock-level mean absolute errors, calculated by averaging call-level absolute error over the valid finite responses for that paddock and condition. Open diamonds show the mean of these 308 values, which equals the reported MAE; box centre lines show medians. (B) Each box contains 308 within-paddock differences in mean absolute error (T1b T1a). Positive differences indicate lower error under T1a; open diamonds show the mean paired difference. Boxes span Q1–Q3, whiskers reach the most extreme values within , and outliers are retained. Box statistics were computed in the original g C kg -1 units; Panel A uses a log1p display and Panel B a symmetric-log display. No T1 confidence interval or hypothesis test is implied. Supervised baselines are reported in Section 3.1 and Table S6.
Figure 4: Measured versus predicted SOC concentration. Each of the seven facets shows the measured 0–10 cm paddock reference against the mean valid finite prediction for that paddock under T1a (green circles) and T1b (pink squares), with 308 paddocks per condition. The dashed line is 1:1 equality; solid condition-specific lines are descriptive ordinary-least-squares fits with freely estimated intercepts, drawn only over the observed reference range. Axes are linear, share the same scale across facets, and express SOC concentration in g C kg -1 . A plotted point averages valid repeated predictions; it is not an additional paddock or an MAE observation. The fitted lines are descriptive; they do not calibrate predictions and are not used for tests or intervals. MAE was scored from individual responses and generally differs from the absolute error of a mean prediction.
Figure 5: SOC stock calculation under matched conditions. (A) Accuracy in the four matched unit-representation (U) and coarse-fragment (F) conditions for each configuration. Open versus filled symbols indicate U0 versus U1; circles versus squares indicate F0 versus F1. U1 changes one unit-representation dimension (SOC concentration, bulk density, layer thickness, or requested output unit) relative to U0. F1 changes the physical coarse-fragment condition and its reference stock. (B) Matched U1 U0 accuracy difference, averaged equally over F and the 30 base scenarios. (C) Matched F1 F0 accuracy difference, averaged equally over U and base scenarios. Points are the matched contrasts and horizontal bars their pointwise repeated-call 95% intervals, in percentage points; dashed lines indicate zero. Panels B and C share a to percentage-point display range. Accuracy uses the prespecified absolute-relative-error tolerance of , with invalid and terminally unavailable responses scored as incorrect. U F interaction contrasts are reported in the Results section.
Figure 6: Pairwise SOC comparison accuracy and association with SOC separation. (A) Direct accuracy gives each of the 300 paddock pairs equal weight and balances the two presentation orders within pair. Points and horizontal bars are the direct accuracy estimates and pointwise repeated-call 95% intervals; the values at right show one-decimal percentages. The horizontal axis in panel A is restricted to 80–100% for display. (B) Conditional predicted accuracy from the secondary model against the continuous absolute measured SOC difference (GAP, g C kg -1 ; log1p display), separately for the higher-SOC paddock shown as A or B. Shading gives pointwise, not simultaneous, 95% Wald intervals conditional on estimated penalty parameters. Shared-paddock, pair and configuration-by-pair random effects are set to zero for these curves; they are not an average of the direct scores in A. Rug marks show the 300 observed pair GAP values. The fitted model has no GAP position interaction on the logit scale.
Figure 7: Repeated-call dispersion, response validity and estimated API cost. (A) Post-collection descriptive summary of within-cell numerical-answer dispersion for T1 and T2. T1 shows the interquartile range (IQR) of valid finite SOC predictions within each configuration–paddock–condition cell (g C kg -1 ), separately for T1a and T1b. T2 shows 100 IQR of valid finite stock predictions divided by the absolute reference stock in the same requested output unit, within each configuration–item cell. Each estimable cell contributes once; at least two valid responses are required. Zeros and the complete observed upper tail are retained, and log1p affects only display, not box statistics. (B) Invalid returned outputs plus responses that remained unavailable after operational failures, as percentages of planned responses; both are counted separately in Table S5. This is response validity, not accuracy. (C) Estimated API cost (US1.55 reservation for attempts without interpretable usage is excluded. T3 within-order choice disagreement is reported in Supplementary Table S12.
| Indicator operation | Benchmark evidence | Interpretive limitation | Minimum check before applied use |
|---|---|---|---|
| SOC concentration (T1) | Added measurements improved average MAE; high-SOC underprediction and local errors remained. | Estimated SOC may mischaracterise carbon-rich sites; no soil-health classes or management effects were tested. | Verify against independent 0–10 cm measurements, suitable local references and concentration-dependent bias. |
| SOC stock (T2) | Equivalent-unit re-expression reduced correctness under a supplied stock equation. | A unit-induced stock difference is an arithmetic artefact, not a change in soil carbon storage. | Document concentration mass basis, depth, bulk density and fragment convention; recalculate deterministically. |
| Paddock comparison (T3) | Accuracy fell for close SOC pairs; some A/B position effects were observed. | Relative SOC ranking does not establish the potential for carbon gain or a causal management response. | Compare like soil contexts; quantify measured uncertainty; verify both presentation orders. |
| Operational quality | Almost all responses were parseable, although repeated-call dispersion and estimated costs varied. | Formatting, reproducibility and price are not substitutes for ecological or numerical validity. | Audit strict validity, input provenance, correctness, repeat variation and costs separately. |
Table 5: Practical interpretation of the benchmark for soil organic carbon indicator workflows.
More in this topic
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 802 papers