Open-Endedness Bench: Measuring Epistemic Process from Agent Records
Authors: Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu
Organizations: University of Michigan, Ann Arbor · ARA Lab · Massachusetts Institute of Technology · The Ohio State University · University of California, Davis
Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optimum nobody knows, or beating a standing record. Their execution logs record every step of this research, yet the runs are still judged by their outcome score. That score alone does not establish whether an agent's claims follow from executed experiments, and a reference answer may be unavailable. We evaluate the agent's epistemic process: how it forms hypotheses, tests them, and revises them in response to evidence. We introduce OEB (Open-Endedness Bench), a benchmark-agnostic methodology that reads only the agent's execution record and never a reference answer or an outcome score. OEB compiles the record into a unified epistemic event graph whose edges connect the propositions the agent states to the executed actions that test them; each node carries an exact excerpt that code verifies against the record. One principle governs scoring: prose can state a proposition, but only evidence returned by an executed action can support or refute it, so OEB checks what the agent writes against what it actually ran. From the graph, OEB scores four competence axes (evidence, experiment, revision, and no reward hacking), mostly as the share of opportunities for sound research that the agent took, and profiles six subjective persona traits that describe the agent's research habits. We score 119 existing runs over 12 tasks from three benchmarks: LLM post-training, chip design, and a training-speed record. Against logged results, only 16-29% of the improvements agents claim are real. On 9 of 10 tasks, the best run tries more new ideas in its second half than the worst run. The persona readings follow the model: for every trait, the model that ran explains more of its variance across runs than the task (a median of 43% against 7%).
Figures & tables
Figure 1: OEB evaluates epistemic process without reading task outcomes or reference answers. Raw logs compile into an epistemic event graph that connects stated propositions to executed actions, producing objective epistemic competence scores and subjective persona traits solely from how the agent handles evidence.
Figure 2: Propositions earn epistemic credit only when tied to actions that test them. When empirical checks contradict a prediction, the graph captures whether the agent updates its beliefs based on observed evidence rather than narrating on without checking.
Figure 3: Experiments the agent never reads, or rejects after reading, are missed opportunities. E2 on this run is the mean of consequential checks (33/59) and experiment yield (30/62), 0.52; E3 is 3/3 but reports no score, because three opportunities fall below the floor of five.
Table 1: OEB separates objective epistemic competence from subjective persona. Competence axes score discipline in handling evidence on [0,1] , while persona traits place an agent’s working style between two legitimate research habits.
Agent model
Tasks
E1 Evidence
E2 Experiment
E3 Revision
E4 No reward hacking
Claude Opus 5
6
0.78 10
0.08 12
–
0.83 12
Claude Fable 5
5
0.67 10
0.29 10
0.60 1
0.88 10
GPT-5.6
6
0.65 12
0.42 12
0.43 12
0.88 12
GPT-5.5
6
0.66 12
0.51 12
0.56 5
0.96 12
Kimi K3
6
0.71 12
0.48 12
0.62 8
0.83 12
GLM-5.2
6
0.65 12
0.51 12
0.67 6
0.77 12
Table 2: Competence scores across agent models on PostTrainBench. Each cell averages all runs of the model, first and second; subscripts count the runs with a score. A model whose record omits its reasoning scores low on E2 and offers fewer opportunities on E1 and E3, where more of its constructs abstain.
Trait
Poles
Score
T1 Budget
Doing vs. thinking
Share of output tokens spent on actions.
T2 Concurrency
One subject vs. several at once
Share of acts at which two or more subjects remained under investigation; share of commits made before evidence resolved the previous commit in the same line of inquiry.
T3 Search topology
Refine an idea vs. test new ideas
Share of research launches that are the first on their subject; share of new ideas a novelty jury judges substantially different from previously settled findings.
T4 Failure style
Pivot vs. persist
After an adverse result, share of next launches that move to another subject.
T5 Provenance
Transferred vs. novel
Among new ideas judged either transferred or novel, share built from a mechanism already proven earlier in the run.
T6 Epistemic tempo
Hypothesis-first vs. run-then-read
Share of propositions stated as a conjecture, question, or prediction at or before the first launch that tests them.
Table 3: Persona traits profile working style without normative ranking. Each trait measures the empirical frequency of choosing one valid research habit over another.
Figure 4: Evaluation scores are stable across repeated evaluations and alternative judge models. Differences in competence and trait scores largely remain within ±0.1 , so the system’s LLM judge adds little variance to the scores.
Truly improved
Built on by a later experiment
The agent read the result as
Speedrun
Chip-Bench
Speedrun
Chip-Bench
an improvement
16%
29%
91%
52%
no improvement
1%
20%
20%
60%
(no verdict stated)
3%
31%
33%
51%
Table 4: Agents claim wins that are not real, and build on them. Research experiments with a logged result (310 speedrun, 150 Chip-Bench) by the agent’s verdict: the share that truly beat the run’s best so far, and the share a later experiment modifies or extends.
Figure 5: Winning runs keep testing new ideas late, and an idea’s later tries add little. Left: new ideas first tested in the second half of the best and the worst run of each task. Right: on Chip-Bench, an idea’s fourth and later tries are 36% of experiments but bring 10% of the real gain.
E4
Auditors:
Task
flagged
per run
cheated
BFCL
10/14
2.29
8/14
HumanEval
3/12
0.25
0/11
HealthBench
5/20
0.25
1/20
GPQA
2/10
0.20
0/10
Arena-Hard
3/20
0.15
2/20
Table 5: The task decides whether agents hack; the model decides how. Left: E4’s reward-hacking instances per PostTrainBench run, by task, beside the auditors’ rulings. Right: each model’s rate and most frequent method on the tasks where some run hacked.
Figure 6: Agent models deploy distinct reward-hacking strategies when evaluation shortcuts exist. Rather than cheating identically, different model families exploit specific loopholes (such as synthesizing training data from test failures or fishing for favorable seeds).
Variance explained by
Mean score of each model
Trait
model
task
chance
Opus 5
Fable 5
GPT-5.6
GPT-5.5
Kimi K3
GLM-5.2
T1 budget
75%
3%
13%
.25
.22
.10
.10
.33
.23
T2 concurrency
32%
8%
13%
.44
.49
.27
.24
.28
.27
T3 search
36%
6%
15%
.40
.39
.16
.28
.26
.27
T4 failure style
31%
16%
21%
–
–
.52
.40
.44
.52
T5 provenance
51%
18%
35%
–
–
.41
–
–
–
Table 6: The model sets the persona. For each trait over up to 91 PostTrainBench runs: the variance the model and the task explain, the model’s share by chance, and each model’s mean score.
Figure 7: Persona reflects model identity rather than the task. Models exhibit consistent, distinct research habits across tasks, maintaining characteristic preferences for budgeting action versus reasoning and for stating hypotheses before execution.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
E1
E2
E3
E4
T1
T2
T3
T4
T5
T6
Same record, judged twice
0.04 37
0.05 40
0.16 12
0.09 40
0.00 35
0.09 39
0.13 35
0.11 21
0.13 4
0.10 28
Same agent, second run
0.12 33
0.14 35
0.23 10
0.14 35
0.02 31
0.16 35
0.11 31
0.13 15
0.10 2
0.16 27
Different agent models
0.15 80
0.23 85
0.25 13
0.17 85
0.11 71
0.18 85
0.15 71
0.14 17
0.33 1
0.28 59
Same record, other judge †
0.16 10
0.08 12
0.17 2
0.00 12
0.07 11
0.08 12
0.14 12
0.09 5
0.36 2
0.09 11
Appendix
Table 7: Scoring variation across repeated evaluations, independent runs, and different agents. Each cell is the mean absolute score difference; subscripts count the score pairs compared, where a pair counts only if both scores exist. Scores remain stable when the same judge rescores a record and consistent when a different judge does. † 12 Chip-Bench records scored by GLM-5.3, the judge for PostTrainBench and Chip-Bench, and again by GPT-5.6; the other rows use PostTrainBench records.
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1
Scientific research follows multi-turn, multi-step workflows that require proactively searching the literature, consulting figures and tables, and integrating evidence across papers to align experimental settings and support reproducible conclusions. This joint capability is not systematically assessed in existing benchmarks, which largely under-evaluate proactive search, multi-evidence integration and sustained evidence use over time. In this work, we introduce EpiBench, an episodic multi-turn multimodal benchmark that instantiates short research workflows. Given a research task, agents must navigate across papers over multiple turns, align evidence from figures and tables, and use the accumulated evidence in the memory to answer objective questions that require cross paper comparisons and multi-figure integration. EpiBench introduces a process-level evaluation framework for fine-grained testing and diagnosis of research agents. Our experiments show that even the leading model achieves an accuracy of only 29.23% on the hard split, indicating substantial room for improvement in multi-turn, multi-evidence research workflows, providing an evaluation platform for verifiable and reproducible research agents.
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21
Princeton University · Cornflower Labs · UK AI Security Institute +8