Learning Scientific Exploration from Human Research Decision Trajectories

Xuchen Gong; Shane Gu; Haokun Liu; Dixi Yao; Chenhao Tan; Tian Li

cs.LGOct 5, 2026

Learning Scientific Exploration from Human Research Decision Trajectories

Authors: Xuchen Gong, Shane Gu, Haokun Liu, Dixi Yao, Chenhao Tan, Tian Li

Organizations: University of Chicago

Abstract

A key challenge in building AI systems for scientific research is enabling $\textit{scientific exploration}$ : the systematic process of investigating unknown phenomena or ideas to gain new knowledge through sequences of research decisions and actions. Yet this process is largely missing from existing scientific corpora; for example, research papers primarily record final outcomes rather than the trajectories that produced them. In this work, we introduce $\textbf{ResearchTrails}$ , a dataset of $\textbf{human research trajectories constructed from Git repositories}$ , where $\textbf{commit histories}$ serve as proxies for research exploration. We develop an automated and scalable pipeline that extracts structured research trajectories from repository commits, capturing successive changes to methods, experiments, and ablations. We characterize the resulting dataset and show that these trajectories contain meaningful signals about intermediate research decisions beyond what final papers reveal. We further demonstrate utilities of ResearchTrails in multiple use cases, including retrieving human research experience as external skills at test time and training models on research trajectories to improve generalization to new research decisions. Our results suggest a path toward AI systems that learn not only from the products of science, but from the evolving process of discovery itself.

Figures & tables

Figure 1 : Overview of ResearchTrails: learning scientific exploration from human research trajectories. Left: We extract research decisions from GitHub commit history. Right: We utilize these trajectories to improve LLMs’ capability of assisting with research exploration. For example, skills distilled from trajectories let GPT-5.6 Sol suggest more diagnostic and actionable interventions.

Figure 2 : Structure of a research trajectory (left) and a research decision (right).

Figure 3 : Workflow of extracting research decisions from GitHub repositories. Candidate paper/repository pairs are discovered and filtered before collecting repository code evidence. An LLM then processes that evidence chronologically, merges commits into decisions, with citations checked against the stored Git evidence. Only repositories with at least five validated decisions are annotated.

Panel of Table 1 — Table 1 : Count of paper/repository pairs throughout the workflow for NeurIPS 2022–2025 (left) and summaries of other dataset statistics (right), including the distributions of decision category , decision outcome , and trajectory lengths.

Figure 4 : Annotation example for Hu et al. (2026) . It plots a trajectory by category, showing how a project moves between building its method, testing it, and ablating it. Orange arcs connect a superseded decision to the one that replaced it.

Figure 5 : Recurring research patterns extracted by GPT-6 Astra that generalize across research projects.

Figure 8 : Sampling three next decisions based on previous decisions from held-out real trajectories. Upper: The decisions are generated with method ‘Skills’; Bottom: The decisions are generated with ‘Retrieve two demos’. The circled values after generated decisions are the decision similarity scores.

Figure 9 : User simulations where a user asks GPT-5.6 Sol (high reasoning) research queries. In both examples, the skills let Sol suggest smaller and more interpretable interventions, changing the suggestion from trying a remedy to determining which component needs repair .

Figure 10 : Test performance of two-stage training, including SFT cold-start (left and middle) and RL with next-decision rewards (right). We observe that SFT improves the probability of the next decision (left) whereas the generation quality plateaus at epoch 3, and the RL stage further improves generation quality (right). The error bars and shades indicate the score’s 95% confidence interval. Scores judged by Opus 5 are in Table 9 .

Figure 11 : Generated next decisions from Qwen3-8B before and after our two-stage training. The examples are from held-out real trajectories, with grey blocks indicating the annotated decisions. The circled values are decision similarity scores for the corresponding generated decisions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Output	What it records
Commit assessment	multiple , referencing $a_{1}$ and $a_{2}$
Decision $a_{1}$	The loss-function choice, citing its relevant code
Decision $a_{2}$	The evaluation choice, citing its relevant code

Appendix

Table 2: Example of mapping a commit to decisions, where we suppose one commit introduces a new loss function, adds an independent evaluation comparison, and reformats a utility file.

Assessment	Meaning	Count	Percentage
irrelevant	no research decision	66,712	56.6%
support	evidence for an existing one	17,640	15.0%
multiple	several decisions at once	16,544	14.0%
revise	changes an existing one	8,130	6.9%
new	begins a new one	5,130	4.4%
uncertain	held as an open thread	3,696	3.1%

Appendix

Table 3: Commit assessments for 599 annotated pairs for NeurIPS 2022–2025.

Function	Calls	Input Token / Call	Output Token / Call	Cost Percentage
commit_scan	8,543	143,042	9,175	99.0%
commit_finalize	708	18,918	1,353	0.5%
annotation	671	16,409	528	0.4%
Total	9,922			100%

Appendix

Table 4: LLM cost by stage for NeurIPS 2022–2025.

Actual (A) and predicted (P) decisions	C	O	S
A: Replace the optimizer with AdamW. P: Switch the training optimizer to AdamW.	2	2	2
A: Replace the optimizer with AdamW. P: Replace the optimizer.	2	2	0
A: Replace the optimizer with AdamW. P: Compare AdamW against the current optimizer before selecting one.	2	1	2
A: Replace the optimizer with AdamW. P: Replace the tokenizer with BPE.	0	2	0
A: Ablate the auxiliary loss. P: Permanently remove the auxiliary loss to simplify training.	2	1	0
A: Set batch size to 32. P: Set batch size to 128.	2	2	0

Appendix

Table 5: Examples of comparing research decisions across three dimensions. ‘C’ for Component , ‘O’ for Operation , and ‘S’ for Specification .

Figure 12 : Annotation example for Hu et al. (2026) (23 selected decisions). The annotation extracts methodological and experimental research decisions from the commit histories of its GitHub repository. Dates indicate the first repository evidence for each decision.

Figure 13 : Annotation example for Hu et al. (2026) (cont.).

Figure 14 : Annotation example for Tran et al. (2024) (30 selected decisions). The annotation extracts methodological and experimental research decisions from the commit histories of its GitHub repository. Dates indicate the first repository evidence for each decision.

Figure 15 : Annotation example for Tran et al. (2024) (cont.).

Figure 16 : Annotation example for Tran et al. (2024) . It plots a trajectory by category, showing how a project moves between building its method, testing it, and ablating it. Orange arcs connect a superseded decision to the one that replaced it.

Judge Model	Metric	Sample SD	Fleiss’ Kappa
GPT-5.6 Sol	Score	0.032	0.719
	Component	0.021	0.816
	Operation	0.012	0.800
Opus 5	Score	0.006	0.691
	Component	0.035	0.831
	Operation	0.040	0.778

Appendix

Table 6: Standard deviation and Fleiss’ Kappa ( Fleiss, 1971 ) of the judge scores measured on fixed 100 baseline predictions.

Judge Model	Predicted Decision	Intended Score	Score	Component	Operation
GPT-5.6 Sol	Same	4	4.00	2.00	2.00
	Paraphrase	4	4.00	2.00	2.00
	Adjacent	2	2.39	1.34	1.05
	Unrelated	0	0.10	0.04	0.06
Opus 5	Same	4	4.00	2.00	2.00
	Paraphrase	4	4.00	2.00	2.00

Appendix

Table 7: Judge’s scores match the intended scores when evaluating the actual decisions, their faithful paraphrases, adjacent decisions, and unrelated decisions.

Method	Prefix-1	Prefix-2	Prefix-3
Baseline	1.473 $\pm$ .088	1.555 $\pm$ .082	1.619 $\pm$ .049
All demos	1.537 $\pm$ .033	1.577 $\pm$ .049	1.602 $\pm$ .057
Random two demos	1.460 $\pm$ .062	1.505 $\pm$ .038	1.455 $\pm$ .110
Retrieve two demos	1.590 $\pm$ .033	1.687 $\pm$ .068	1.714 $\pm$ .044
Shuffled skill	1.575 $\pm$ .022	1.627 $\pm$ .071	1.555 $\pm$ .062
Final paper skill	1.764 $\pm$ .030	1.490 $\pm$ .089	1.445 $\pm$ .024

Appendix

Table 8: Decision similarity of the predicted next decision based on previous

\{1,2,3\}

decisions. Scores are judged by Opus 5 and averaged across evaluation trajectories, each with 3 runs. The conclusions from Table 7 apply to the results in this table: Skills is best at every prefix, Retrieval beats Random at every prefix, and intermediate decisions provide learning signals beyound the final papers.

Model	GPT-5.6 Sol	Opus 5
Qwen3-8B base	1.025	1.219
SFT epoch 3	1.199	1.333
GRPO epoch 7	1.505	1.674
GRPO epoch 10	1.510	1.637

Appendix

Table 9: Decision Similarity Score of Qwen3-8B at different stages judged by GPT-5.6 Sol and Opus 5.

Explore similar work

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows
Sep 10, 2026Aayam Bansal, Keertan BalajiAI for Science AI Agent Auditing
IdeaTrail: Full-Process Agent Trajectories for Scientific Ideation
Jul 11, 2026Hengquan GuoAI Agents for Scientific Discovery Scientific Hypothesis Generation
ShanghaiTech University
PreScience: A Dataset and Benchmark for Scientific Forecasting
Feb 24, 2026Anirudh Ajith, Amanpreet Singh, Jay DeYoung +7AI for Science Forecasting Benchmarks
Allen Institute for Artificial Intelligence, Seattle, WA, USA · Knowledge Lab, University of Chicago, Chicago, IL, USA · School of Computer Science and Engineering, Hebrew University of Jerusalem, Israel +1

Sep 10, 2026, cs.AIJ/K move · Enter open · S save

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

Aayam Bansal, Keertan Balaji

AI for Science AI Agent Auditing LLM Auditing AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose failure modes, or distinguish systematic reasoning from fortunate guessing. We present \textbf{OpenDiscoveryTrace}, a public dataset of 558 complete AI scientific agent trajectories that captures how models reason, not just what they produce. Each trajectory records a structured 9-field-per-step trace---including thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence---as models execute 124 scientific tasks spanning drug discovery, materials science, genomics, and scientific literature analysis. The dataset covers seven models: three frontier models (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro; 124 trajectories each, fully balanced across domains and difficulty levels) and four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B; 30 each), plus 60 live-retrieval variant trajectories. Pilot analysis on 363 LLM-judged trajectories reveals that process traces expose behavioral differences invisible to output-only evaluation: all three frontier models achieve comparable success rates (84--89%), yet Claude Opus 4.6 produces 30

\times

more errors than GPT-5.4 (2.5 vs. 0.08 per trajectory,

p < 0.0001

, Cliff's

\delta = 0.613

), with qualitatively different error profiles---66.7% tool misuse for Claude versus 83.6% reasoning errors for GPT-5.4. We define five benchmark tasks with baselines from logistic regression, random forests, LSTMs, and Transformer models. The dataset, trace schema, agent harness, and benchmark definitions are publicly available under CC BY 4.0 to support research on process-level evaluation, scientific agent auditing, and AI governance.

Abstract

Explore similar work

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

IdeaTrail: Full-Process Agent Trajectories for Scientific Ideation

PreScience: A Dataset and Benchmark for Scientific Forecasting