MineralExp: Benchmarking LLM Agents for Experimental Design in Real-World Mineral Exploration
Authors: Bing Wang, Mingyang Wang, Yuanhang Mi, Yue Xu, Kaibo Su, Li'ang Xiao, Chunguo Wu, Changchun Li, Ximing Li, Kun-Feng Qiu, Bo Yang
Organizations: College of Computer Science and Technology, and Key Laboratory of Symbolic Computation and Knowledge Engineering of the MoE, Jilin University, Changchun, China · State Key Laboratory of Geological Processes and Mineral Resources, Frontiers Science Center for Deep-time Digital Earth, School of Earth Science and Resources, China University of Geosciences (Beijing), Beijing, China
In mineral exploration, verifying geological hypotheses relies on designing a variety of geochemical experiments. While large language model (LLM) agents increasingly automate downstream geoscientific data analysis, a crucial upstream capability remains largely unexplored: automated geochemical experimental design, i.e., deciding which experiments to conduct before committing to costly laboratory work. Evaluating this capability of LLM agents is difficult for two reasons. First, geochemical experiments exhibit high methodological diversity and subjectivity: to test the same hypothesis, different geologists may choose different minerals and instruments, so no single protocol can serve as the reference answer. Second, open-ended experimental plans resist representation as structured labels, which makes objective scoring difficult. To address these challenges, we introduce MineralExp, a benchmark that reformulates geochemical experimental design as a hypothesis-based experiment selection problem, i.e., a multi-label classification problem. We abstract concrete instrument names and procedural details into a shared taxonomy of 19 experimental types across five mineral-system processes: Source, Transport, Deposition, Modification, and Preservation. This taxonomy yields deterministic multi-label targets while accommodating diverse experimental protocols. Constructed from published ore-deposit literature and verified by domain experts, MineralExp comprises 277 cases and 1,572 reference experimental plans. Given a geological hypothesis and initial context, an LLM agent selects the relevant tasks and generates concrete experimental plans, which are scored by multi-label classification metrics and an LLM-as-a-Judge metric. The experimental results show that current agents generally achieve high recall but low precision, tending to propose more experiments than are necessary to test the hypothesis.
Figures & tables
Figure 1. A representative example from MineralExp . Given a geological hypothesis and its initial context, an LLM agent selects the relevant experiments and proposes the corresponding experimental plans.
Figure 2. Overall construction pipeline of MineralExp . From 353 candidate studies, we extract (O,E,F) records to separately construct decontaminated inputs Xi=(Hi,Ci,Ai) and mapped reference targets (Oi∗,Yi∗) , followed by expert review and validation to yield 277 cases and 1,572 reference plans.
Process
Task IDs
Tasks
Positive Cases
Source
S1–S4
4
160
Transport
T1–T5
5
170
Deposition
D1–D4
4
146
Modification
M1–M3
3
143
Preservation
P1–P3
3
46
Total
-
19
277
Table 1. Process-level composition of MineralExp .
ID
Experimental Types
Cases
S1
Magma sources and evolution
50
S2
Magmatic metallogenic potential
19
S3
Sources of metals and sulfur
128
S4
Metamorphic fluids and metal release
1
T1
Ore-fluid sources
95
T2
Fluid evolution and episodes
95
Table 2. The 19 experimental types in MineralExp and their corresponding number of positive cases.
Agent
LLM
Source
Transport
Deposition
Modification
Preservation
Overall
R.
F1
R.
F1
R.
F1
R.
F1
R.
F1
R.
F1
OpenCode
GPT-6.1 Sol
72.08
68.77
76.19
67.16
63.86
56.09
62.80
57.87
81.25
72.90
70.76
63.84
GPT-6 Luna
79.19
53.70
84.03
54.01
79.21
46.51
76.83
46.24
79.17
46.91
80.58
50.53
Claude Sonnet 5.5
78.68
68.28
82.91
65.92
71.29
50.70
71.95
51.75
85.42
68.33
77.89
60.42
Claude Haiku 5.5
84.26
54.16
82.91
59.86
67.82
50.37
76.83
46.75
83.33
51.61
79.03
53.87
DeepSeek-V4.1-Flash
86.29
48.57
88.80
50.28
82.18
41.60
88.41
45.74
79.17
41.76
86.36
46.77
Table 3. Experimental results of 4 open-source agent harnesses and 7 foundation LLMs on our benchmark MineralExp .
Figure 3. Detailed selection metrics of MineralExp on OpenCode.
Agent
DeepSeek-V4.1-Flash
GPT-6.1 Sol
#Strict
#Part
#Weig
#Strict
#Part
#Weig
OpenCode
32.44
87.53
59.99
24.05
83.59
53.82
OpenHands
36.83
93.51
65.17
27.48
85.81
56.65
Biomni
35.56
92.49
64.03
22.65
84.99
53.82
ChemToolAgent
32.63
91.28
61.96
24.49
83.78
54.13
Table 4. LLM-as-a-Judge scoring of the experimental plans.
Figure 4. Token consumption of MineralExp .
Figure 5. Statistics of used tools of MineralExp .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10Figure 11Figure 12
Figure 6. Detailed selection metrics of MineralExp on OpenHands, Biomni, and ChemToolAgent.
Component
Case A (ID: G19-0224): Incomplete Plan Coverage
Case B (ID: G19-0175): Full Plan Coverage
Input X(H,C,A)
∙ Hypothesis ( H ): Test whether sulfur-gas and cover indicators developed in arid regions remain informative for detecting concealed gold mineralization beneath semi-humid grassland cover at the Zhaishang deposit. ∙ Context ( C ): Sediment-hosted gold deposit under semi-humid cover; applicability of soil-gas indicators is unresolved. ∙ Conditions ( A ): Exploration lines 143 and 175, a cover profile, and known orebody positions for validation.
∙ Hypothesis ( H ): Determine the formation temperatures, oxygen/sulfur fugacities ( fO2,fS2 ), and pH conditions of arsenopyrite and chlorite across successive mineralization stages at the Jinshan gold deposit. ∙ Context ( C ): Prior work links mineralization to magmatism, deformation, and mixed fluids, but stage-wise physicochemical evolution is unresolved. ∙ Conditions ( A ): Existing ore, arsenopyrite, chlorite, and host-strata samples.
Ground Truth (Y∗,P∗)
∙ Reference Tasks ( Y∗ ): P2, P3 (2 tasks) ∙ Reference Operations ( P∗ , 7 ops): - O03-O05 : ICP-MS assay of Au, AES assay of Ag, and HG-AFS assay of As/Hg across two soil grain-size fractions . - O07-O08 : ICP-MS assay of operationally extractable Au and four operational Au phase pools (water, clay, organic, Fe/Mn-oxide). - O09 : Field soil-gas measurements ( H2S,SO2,CO2 ). - O10 : Total organic carbon (TOC) in two soil fractions.
∙ Reference Tasks ( Y∗ ): M2, T2, T3 (3 tasks) ∙ Reference Operations ( P∗ , 3 ops): - O02 : Transmitted/reflected light microscopy on ore sections to establish mineral assemblages and relationships across three paragenetic stages . - O04 : Quantitative EPMA compositional analysis of stage-resolved arsenopyrite generations (Apy-II1, II2, III). - O05 : Quantitative EPMA compositional analysis of stage-resolved chlorite generations (Chl-I, II, III).
∙ Predicted Tasks ( Y ): M2 ✓ , T2 ✓ , T3 ✓ , D1 × , D2 ×∙ Generated Plans ( P ): - Design ( dt ): First establish a paragenetic scheme from polished thin sections via reflected/transmitted light microscopy and SEM-BSE; then sample coexisting arsenopyrite and chlorite from each recognized stage for quantitative EPMA and fluid-inclusion microthermometry. - Rationale ( rt ) & Evidence ( et ): Prevent mixing records from different stages; obtain stage-specific compositions to constrain temperature, fO2/fS2 , and pH shifts.
Evaluation & Diagnosis
∙ Task Metrics: Recall =100.0% , Precision =33.3% , F1 =50.0%∙ Evidence Coverage ( Cov ): 21.4% ( 1.5/7 operations) ∙ Judge Diagnosis: Full credit ( 1.0 ) on O09 (soil gas) and partial credit ( 0.5 ) on O10 (organic matter without grain-size fractions). Scored 0 on O03-O05 and O07-O08 : generic “pathfinder elements” and sulfur leaching fail to specify grain-size fractionated Au/Ag/As/Hg assays and four-phase extractable-Au extraction.
∙ Task Metrics: Recall =100.0% , Precision =60.0% , F1 =75.0%∙ Evidence Coverage ( Cov ): 100.0% ( 3.0/3 operations) ∙ Judge Diagnosis: Full credit ( 1.0 ) on all three operations ( O02, O04, O05 ). The generated plans explicitly align scientific purpose, analytical method, and target material—linking stage-resolved optical petrography directly to generation-specific EPMA of arsenopyrite and chlorite.
Appendix
Table 5. Case study comparing two instances evaluated with OpenHands and DeepSeek-V4.1-Flash. Both cases achieve 100% task recall on Y , but exhibit a sharp contrast in plan-level evidence coverage ( Cov : 21.4% vs. 100.0% ). Correct task predictions are marked with ✓ , and over-selected false positives with × .