InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

Oliver Bentham; Vivek Srikumar

cs.CLApr 14, 2026

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

Authors: Oliver Bentham, Vivek Srikumar

Organizations: Kahlert School of Computing University of Utah

Abstract

Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.

Figures & tables

Figure 1: Data repositories are generated in a top-down manner. 1) Scientific Context : A field, domain, and subdomain are randomly sampled from a hierarchical taxonomy. 2) Project Specification : A project title, description (containing a plan, hypotheses, variables, and confounders), and abstract are generated with the LLM, conditioned on the scientific context. 3) Directory Structure : The project specification is used to create a plausible directory structure in which folder and file names describe the variables. 4) File Structure : Conditioned on all prior information, independent and dependent variables are generated using named distributions and functions of other variables, respectively. The right side shows one such instantiation, corresponding to seed $118$ .

		Paraphrase Model
Evaluated Model	Templated	GPT-OSS 20B	Gemma 3 27B it	Qwen3 4B Inst.
GPT-5.4	$49.2\pm 2$	$48.0\pm 2$	$39.1\pm 2$	$43.0\pm 2$
Claude Opus 4.6	$43.7\pm 2$	$41.9\pm 2$	$27.8\pm 2$	$28.7\pm 2$
GPT-OSS 20B	$29.7\pm 2$	$30.0\pm 2$	$26.9\pm 2$	$29.7\pm 2$
Qwen3 4B Inst.	$26.5\pm 2$	$24.2\pm 2$	$22.3\pm 2$	$25.5\pm 2$
Gemma 3 27B it	$24.2\pm 2$	$23.2\pm 2$	$22.7\pm 2$	$22.2\pm 2$

Table 1: Accuracy by evaluated model and template/paraphrase variant. Each score also shows the corresponding 95% confidence interval. We do not report results averaged across paraphrases; our analysis in Appendix L shows that paraphrase quality varies significantly across the paraphrasing models, and averaging can be misleading.

Figure 2: Model accuracy, separated by question category.

Figure 3: Distribution of error types by model, based on samples of 100 incorrect QA pairs per model (paraphrased using GPT-OSS 20B). Distributions are normalized within each model. Note that the error categories have a sequential structure. See the text for details.

Figure 4: Each evaluated model’s precision and recall for detecting unanswerable questions.

Figure 5: Comparisons between accuracy, token use, and tool use, per model. Although increased tool use correlates with accuracy, token use does not because high token use models typically attempt to load files into the LLM’s context window, a suboptimal strategy.

	Paraphrase Model
Evaluated Model	GPT-OSS 20B	Qwen3 4B Inst.	Gemma 3 27B it
GPT-5.4	0.74	0.66	0.58
Claude Opus 4.6	0.59	0.49	0.48
GPT-OSS 20B	0.72	0.69	0.67
Qwen3 4B Inst.	0.76	0.80	0.83
Gemma 3 27B it	0.81	0.83	0.85

Table 2: Agreement between per-question correctness on templated and paraphrased variants, measured using Krippendorff’s alpha. Rows denote evaluated models and columns denote paraphrasing models. Red cells indicate cases where the evaluated model and paraphrasing model are the same.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Field	# Domains	# Subdomains
Computer Science	11	41
Engineering - Bioengineering	9	32
Engineering - Chemical and Materials	7	31
Engineering - Civil and Environmental	9	25
Engineering - Electrical and Computer	17	59
Engineering - Industrial and Systems	8	24

Appendix

Table 3: The numbers of domains and subdomains per field in the scientific taxonomy

Name (short name)	Values	Description
growth phase (gphase)	early, mid, late	The stage of microbial growth in the culture, indicating whether cells are in early (exponential), mid (mid-exponential), or late (stationary) phase, with no physical units.
Genotype Modification Type (gtype)	knockout, overexpression, promoter_swap	The type of genetic modification applied to the strain (e.g., gene knockout, overexpression, or promoter swap), with no physical units.
Date (date)	01/05/2025 to 11/11/2025	Date. Includes day, month and optional year.
Time point of ethanol measurement during fermentation (tpt)	0, 6, 12, 24, 48	The time at which ethanol yield and other phenotypic measurements are taken during fermentation, in hours.
Sequential Number (seq_number)	1, 2, 3, 4, 5	A sequential trial or run number.
fermentation pH (pH)	4.0, 5.0, 6.0, 7.0	The acidity or alkalinity of the fermentation medium, measured on the pH scale (0–14), with values ranging from 4.0 to 7.0.

Appendix

Table 4: The variables used in the directory/file paths for repository #118.

Name (short)	Description	Type	Values (Probs.)
Fermentation temperature (temp)	The temperature at which fermentation occurs, in degrees Celsius	ind/cat	15°C (25.0%), 25°C (25.0%), 30°C (25.0%), 35°C (25.0%)
Glucose concentration (glucose_conc)	The initial concentration of glucose in the fermentation medium, in grams per liter	ind/cat	50 g/L (33.3%), 100 g/L (33.3%), 200 g/L (33.3%)
Oxygen level (oxygen_level)	The level of oxygen present during fermentation: anaerobic, microaerobic, or aerated	ind/cat	anaerobic (50.0%), microaerobic (30.0%), aerated (20.0%)
Ethanol yield (ethanol_yield)	Amount of ethanol produced per liter of fermentation medium, in grams per liter	dep/con
Glucose consumption rate	Rate at which glucose is consumed, in grams per liter per hour	dep/con
Biomass concentration (biomass)	Total cell mass concentration, in grams per liter	dep/con

Appendix

Table 5: The variables used as file columns for repository #118. The first type value is either independent (ind) or dependent (dep), and the second value is either categorical (cat) or continuous (con). The last column shows the values and probabilities for independent, categorical variables. The short name for “Glucose consumption rate” (omitted for formatting) is “glucose_consumption_rate”.

Question	Values
The folder and file organization looks typical for projects in my field.	Strongly Disagree, Somewhat Disagree, Neither, Somewhat Agree, Strongly Agree
The file naming conventions look realistic for my field.	Strongly Disagree, Somewhat Disagree, Neither, Somewhat Agree, Strongly Agree
The level of messiness/incompleteness looks realistic (e.g. missing values, inconsistent file/directory structure, etc.).	Strongly Disagree, Somewhat Disagree, Neither, Somewhat Agree, Strongly Agree
The column names look plausible for this kind of project.	Strongly Disagree, Somewhat Disagree, Neither, Somewhat Agree, Strongly Agree
The column values are realistic.	Strongly Disagree, Somewhat Disagree, Neither, Somewhat Agree, Strongly Agree
Can you tell this is a simulated repository?	Yes, Not Sure, No

Appendix

Table 6: Questions and response values included in the data plausibility human study.

Question Category	Question Type	Count
Repository Metadata	Readme	16
Repository Metadata	Title	13
Repository Metadata	Abstract	16
File Metadata	Extension	14
File Metadata	Count Rows	82
Directory Traversal	Prefix	57

Appendix

Table 7: The number of sampled questions used per question category and question type in the 500 questions included in the evaluation.

	Semantically Equivalent	Unambiguous	Fluent
GPT-OSS 20B	96%	98%	100%
Gemma 3 27B it	62%	68%	96%
Qwen3 4B Instruct	78%	80%	94%

Appendix

Table 8: Results of the paraphrase validity audit per paraphrase model. Each value shows the proportion of the 50 sampled question paraphrases that matches the column title, when compared against the templated original.

Explore similar work

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory
Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI
Sep 30, 2026Nishtha N. Vaidya, Stephan Grimm, Thomas Hubauer +1LLM Evaluation Synthetic Benchmark Generation
Siemens AG · Technical University of Munich

Jun 11, 2026J/K move · Enter open · S save

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

Pierre Beckmann, Marco Valentino, Andre Freitas

Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK+2

LLM Evaluation Scientific Reasoning Synthetic Benchmark Generation Structured Reasoning Scientific Reasoning in Language Models

Open paper AskPDF arXiv

Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on these in scientific settings is currently out of reach: scientific benchmarks built on human annotations are costly and lack mechanistic ground truth, while synthetic logical-reasoning benchmarks do not resemble real scientific documents. We introduce SciR, a benchmark that combines multi-paradigm reasoning with controllable scientific rendering, anchored on three paradigmatic scientific problems. Tasks are generated from formal objects (deduction tree, inductive rule hypothesis, causal graph) to guarantee verifiable answers, then rendered into multi-document scientific discourse via per-track domain-tuned genres. The construction lets us independently vary two difficulty axes: how hard it is to extract the key information needed for inference, and how hard the principled inference itself is. We test six models. Both axes hurt every model, and their effects compound. The rendering even hurts neurosymbolic pipelines, which hand inference to a verified solver. The two axes yield a per-model extraction-vs-inference profile: for instance, reasoning models like deepseek-r1 mostly surpass non-reasoning instruct models on the inference axis. To our knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty.

Abstract

Explore similar work

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI