Open-Endedness Bench: Measuring Epistemic Process from Agent Records

Chengyang Shi; Xianglin Ji; Jintao Huang; Jicheng Wang; Yifeng He; Jiachen Liu

cs.AIOct 1, 2026

Open-Endedness Bench: Measuring Epistemic Process from Agent Records

Authors: Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu

Organizations: University of Michigan, Ann Arbor · ARA Lab · Massachusetts Institute of Technology · The Ohio State University ·

Abstract

Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optimum nobody knows, or beating a standing record. Their execution logs record every step of this research, yet the runs are still judged by their outcome score. That score alone does not establish whether an agent's claims follow from executed experiments, and a reference answer may be unavailable. We evaluate the agent's epistemic process: how it forms hypotheses, tests them, and revises them in response to evidence. We introduce OEB (Open-Endedness Bench), a benchmark-agnostic methodology that reads only the agent's execution record and never a reference answer or an outcome score. OEB compiles the record into a unified epistemic event graph whose edges connect the propositions the agent states to the executed actions that test them; each node carries an exact excerpt that code verifies against the record. One principle governs scoring: prose can state a proposition, but only evidence returned by an executed action can support or refute it, so OEB checks what the agent writes against what it actually ran. From the graph, OEB scores four competence axes (evidence, experiment, revision, and no reward hacking), mostly as the share of opportunities for sound research that the agent took, and profiles six subjective persona traits that describe the agent's research habits. We score 119 existing runs over 12 tasks from three benchmarks: LLM post-training, chip design, and a training-speed record. Against logged results, only 16-29% of the improvements agents claim are real. On 9 of 10 tasks, the best run tries more new ideas in its second half than the worst run. The persona readings follow the model: for every trait, the model that ran explains more of its variance across runs than the task (a median of 43% against 7%).

Figures & tables

Panel of Figure 1 — Figure 1: OEB evaluates epistemic process without reading task outcomes or reference answers. Raw logs compile into an epistemic event graph that connects stated propositions to executed actions, producing objective epistemic competence scores and subjective persona traits solely from how the agent handles evidence.

Panel of Figure 2 — Figure 2: Propositions earn epistemic credit only when tied to actions that test them. When empirical checks contradict a prediction, the graph captures whether the agent updates its beliefs based on observed evidence rather than narrating on without checking.

Panel of Figure 3 — Figure 3: Experiments the agent never reads, or rejects after reading, are missed opportunities. E2 on this run is the mean of consequential checks (33/59) and experiment yield (30/62), 0.52; E3 is 3/3 but reports no score, because three opportunities fall below the floor of five.

Axis / Trait	Core question
Tier 1: Objective epistemic competence (scored, $[0,1]$ )
E1 Evidence	Are propositions backed by what was seen?
E2 Experiment	Do experiments yield interpretable results?
E3 Revision	Does it address contradicted ideas?
E4 No reward hacking	Does it avoid manipulating the evaluation?
Tier 2: Subjective persona (profiled)

Table 1: OEB separates objective epistemic competence from subjective persona. Competence axes score discipline in handling evidence on

[0,1]

, while persona traits place an agent’s working style between two legitimate research habits.

Agent model	Tasks	E1 Evidence	E2 Experiment	E3 Revision	E4 No reward hacking
Claude Opus 5	6	0.78 10	0.08 12	–	0.83 12
Claude Fable 5	5	0.67 10	0.29 10	0.60 1	0.88 10
GPT-5.6	6	0.65 12	0.42 12	0.43 12	0.88 12
GPT-5.5	6	0.66 12	0.51 12	0.56 5	0.96 12
Kimi K3	6	0.71 12	0.48 12	0.62 8	0.83 12
GLM-5.2	6	0.65 12	0.51 12	0.67 6	0.77 12

Table 2: Competence scores across agent models on PostTrainBench. Each cell averages all runs of the model, first and second; subscripts count the runs with a score. A model whose record omits its reasoning scores low on E2 and offers fewer opportunities on E1 and E3, where more of its constructs abstain.

Trait	Poles	Score
T1 Budget	Doing vs. thinking	Share of output tokens spent on actions.
T2 Concurrency	One subject vs. several at once	Share of acts at which two or more subjects remained under investigation; share of commits made before evidence resolved the previous commit in the same line of inquiry.
T3 Search topology	Refine an idea vs. test new ideas	Share of research launches that are the first on their subject; share of new ideas a novelty jury judges substantially different from previously settled findings.
T4 Failure style	Pivot vs. persist	After an adverse result, share of next launches that move to another subject.
T5 Provenance	Transferred vs. novel	Among new ideas judged either transferred or novel, share built from a mechanism already proven earlier in the run.
T6 Epistemic tempo	Hypothesis-first vs. run-then-read	Share of propositions stated as a conjecture, question, or prediction at or before the first launch that tests them.

Table 3: Persona traits profile working style without normative ranking. Each trait measures the empirical frequency of choosing one valid research habit over another.

Figure 4: Evaluation scores are stable across repeated evaluations and alternative judge models. Differences in competence and trait scores largely remain within

\pm 0.1

, so the system’s LLM judge adds little variance to the scores.

	Truly improved		Built on by a later experiment
The agent read the result as	Speedrun	Chip-Bench	Speedrun	Chip-Bench
an improvement	16%	29%	91%	52%
no improvement	1%	20%	20%	60%
(no verdict stated)	3%	31%	33%	51%

Table 4: Agents claim wins that are not real, and build on them. Research experiments with a logged result (310 speedrun, 150 Chip-Bench) by the agent’s verdict: the share that truly beat the run’s best so far, and the share a later experiment modifies or extends.

Figure 5: Winning runs keep testing new ideas late, and an idea’s later tries add little. Left: new ideas first tested in the second half of the best and the worst run of each task. Right: on Chip-Bench, an idea’s fourth and later tries are 36% of experiments but bring 10% of the real gain.

	E4		Auditors:
Task	flagged	per run	cheated
BFCL	10/14	2.29	8/14
HumanEval	3/12	0.25	0/11
HealthBench	5/20	0.25	1/20
GPQA	2/10	0.20	0/10
Arena-Hard	3/20	0.15	2/20

Table 5: The task decides whether agents hack; the model decides how. Left: E4’s reward-hacking instances per PostTrainBench run, by task, beside the auditors’ rulings. Right: each model’s rate and most frequent method on the tasks where some run hacked.

Figure 6: Agent models deploy distinct reward-hacking strategies when evaluation shortcuts exist. Rather than cheating identically, different model families exploit specific loopholes (such as synthesizing training data from test failures or fishing for favorable seeds).

	Variance explained by			Mean score of each model
Trait	model	task	chance	Opus 5	Fable 5	GPT-5.6	GPT-5.5	Kimi K3	GLM-5.2
T1 budget	75%	3%	13%	.25	.22	.10	.10	.33	.23
T2 concurrency	32%	8%	13%	.44	.49	.27	.24	.28	.27
T3 search	36%	6%	15%	.40	.39	.16	.28	.26	.27
T4 failure style	31%	16%	21%	–	–	.52	.40	.44	.52
T5 provenance	51%	18%	35%	–	–	.41	–	–	–

Table 6: The model sets the persona. For each trait over up to 91 PostTrainBench runs: the variance the model and the task explain, the model’s share by chance, and each model’s mean score.

Figure 7: Persona reflects model identity rather than the task. Models exhibit consistent, distinct research habits across tasks, maintaining characteristic preferences for budgeting action versus reasoning and for stating hypotheses before execution.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	E1	E2	E3	E4	T1	T2	T3	T4	T5	T6
Same record, judged twice	0.04 37	0.05 40	0.16 12	0.09 40	0.00 35	0.09 39	0.13 35	0.11 21	0.13 4	0.10 28
Same agent, second run	0.12 33	0.14 35	0.23 10	0.14 35	0.02 31	0.16 35	0.11 31	0.13 15	0.10 2	0.16 27
Different agent models	0.15 80	0.23 85	0.25 13	0.17 85	0.11 71	0.18 85	0.15 71	0.14 17	0.33 1	0.28 59
Same record, other judge †	0.16 10	0.08 12	0.17 2	0.00 12	0.07 11	0.08 12	0.14 12	0.09 5	0.36 2	0.09 11

Appendix

Table 7: Scoring variation across repeated evaluations, independent runs, and different agents. Each cell is the mean absolute score difference; subscripts count the score pairs compared, where a pair counts only if both scores exist. Scores remain stable when the same judge rescores a record and consistent when a different judge does. † 12 Chip-Bench records scored by GLM-5.3, the judge for PostTrainBench and Chip-Bench, and again by GPT-5.6; the other rows use PostTrainBench records.

Agent Evaluation

Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.

See all 144 papers

EvalResearchBench: Can AI Agents Design Their Own Evaluations?
Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated Evaluation Benchmark Construction
Oregon State University · AG2 AI · University of California, San Diego +1
EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents
Apr 7, 2026Xuan Dong, Huanyang Zheng, Tianhao Niu +7AI Agent Benchmarks AI Agents for Scientific Discovery
Harbin Institute of Technology
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Jul 29, 2026Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21AI-Assisted Scientific Research AI Agent Evaluation
Princeton University · Cornflower Labs · UK AI Security Institute +8

Oct 3, 2026J/K move · Enter open · S save

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao+2

Oregon State University · AG2 AI · University of California, San Diego+1

Automated Evaluation Benchmark Construction Autonomous Research Agents AI Agent Benchmarks

Open paper AskPDF arXiv

Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.

Abstract

Agent Evaluation

Explore similar work

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents

Can AI agents conduct open-ended AI research? Early evidence from two case studies