Scientific discovery in data-rich domains is currently constrained by human bandwidth: the growth in the volume and complexity of real-world data far outpaces the rate at which researchers can read, reason, and synthesize. Recent LLM-based multi-agent systems have begun to automate portions of the research cycle, but they target hypothesis generation in settings where validation cannot itself be automated, and each run is one-shot, with no mechanism for findings to accumulate or steer subsequent inquiry. This paper introduces AutoKD, a multi-agent framework for autonomous knowledge discovery that is both computational and cumulative, allowing validated findings to persist and inform subsequent inquiry. Six coordinated LLM agents collaborate in an open-ended discovery loop, where accepted findings are stored in a persistent insight graph that serves as both long-term memory and an exploration-steering mechanism. We evaluate AutoKD on three diverse datasets from two perspectives: Open-ended Quality against published findings, and Conditioned Quality via literature-derived queries. Across both evaluation perspectives, AutoKD covers known findings and surfaces substantive discoveries that complement human-driven research. Our code is available at https://github.com/GeQinwen/AutoKD.
Figures & tables
Figure 1. High-level overview of AutoKD for autonomous knowledge discovery including insight graph storage allowing cross-round accumulation.
Figure 2. An overview of the AutoKD framework for autonomous agentic knowledge discovery. AutoKD first iteratively discovers and accumulates structured insights via the knowledge creation component, and then through a retrieval, synthesis, verification process enables knowledge extraction.
Figure 3. Proposed evaluation framework for autonomous knowledge discovery. Gold claims distilled from published papers support two complementary protocols: open-ended quality compares AutoKD insights with gold claims, while conditioned quality evaluates literature-derived queries through retrieval and synthesis over the insight graph.
Dataset
Rater
Int.
Surp.
Useful
Overall
WVS
Human
83.8
86.1
81.1
86.1
LLM
97.4
100.0
66.7
92.5
All
90.7
93.2
73.7
89.5
Amazon
Human
53.1
56.2
63.6
54.5
LLM
50.0
56.4
17.5
47.5
All
51.4
56.3
38.4
50.7
Table 1. AutoKD win rate (%, ties excluded) by dataset, dimension, and rater subset. Human and LLM results each aggregate four raters; All aggregates all eight raters.
Interesting
Surprising
Useful
Overall
Humans only (4 raters)
WVS
.012
.064
−.038
−.070
Amazon
.139
.186
.255
.202
SciSciNet
−.065
−.075
−.056
−.053
All
.098
.111
.111
.112
LLM judges only (4 raters)
Table 2. Inter-rater agreement (Fleiss’ κ ).
Case
AutoKD Finding (preferred)
Gold Counterpart
LLM Judges
Amazon (useful)
Longer reviews tend to appear on less popular books.
Longer reviews receive more helpful votes.
Disagree
WVS (interesting)
Higher social trust is negatively associated with discomfort with women earning more than their husbands.
Being in a relationship is positively associated with higher life satisfaction.
NIH- and NSF-funded papers are more likely to receive patent citations than non-funded papers.
Agree
Table 3. Representative cases where all four human evaluators preferred the AutoKD-generated finding over its gold-benchmark counterpart along the indicated dimension.
Figure 4. Variable-usage distribution for the full insight graph (blue), the score-only top-10 (orange), and the top-10 after diversity reranking (green). Variables are ordered by full-graph frequency.
Table 5. Discovery output per dataset. Insights are validated findings stored in the graph; relations are typed edges connecting them.
Variant
n
Estimands / insight ↑
Exact rediscovery ↓
Deep-test share ↑
Δ Depth
AutoKD
819
0.949
0.0%
28.9%
–
w/o persistence
1,021
0.672
16.7%
12.5%
−56.7%
w/o graph structure
828
0.944
0.0%
26.0%
−10.3%
global-only
785
0.941
0.0%
20.1%
−30.4%
Table 6. Ablation of cumulative discovery mechanisms on WVS at a matched 1,000-round budget. Δ Depth is the relative change in deep-test share from full AutoKD.
duplicate cosine threshold =0.92 ; retrieval order D > N > E > C
Orchestrator
mode weights (exp./global/ref./conf.) =0.30/0.15/0.40/0.30 ; gmin=12 ; max rounds =2,000
Appendix
Table 7. Agent-level hyperparameters for reported runs. W = WVS, A = Amazon, S = SciSciNet.
Figure 5. Worked example of the literature-grounded extraction pipeline on SciSciNet.
k
Amazon ( N=22 )
WVS ( N=198 )
SciSciNet ( N=87 )
0
1.94×104
1.54×107
1.29×106
1
3.70×105
3.00×109
1.08×108
2
3.34×106
2.91×1011
4.49×109
3
1.90×107
1.87×1013
1.23×1011
4
7.61×107
8.98×1014
2.48×1012
5
2.29×108
3.43×1016
3.98×1013
Appendix
Table 8. Total hypothesis counts across all four templates as a function of the number of controls k . N is the number of testable variables in each dataset.
Template
Estimator and reported effect
assoc
Pearson correlation for unadjusted numeric variables; otherwise OLS/WLS regression. A one-degree-of-freedom term is summarized by (partial) r ; a multi-level categorical predictor is tested jointly with Type-II ANOVA and summarized by partial η2 .
diff
For two groups without controls, Welch’s t -test with Cohen’s d is used for continuous outcomes, while low-cardinality ordered outcomes use Mann–Whitney U with rank-biserial r . More than two groups use Kruskal–Wallis with ϵ2 . With controls or weights, the comparison is expressed as an OLS/WLS model, using a contrast for two groups or a Type-II omnibus test for multiple groups.
interact
Regression of the form Y∼X∗Z+C , where C denotes optional controls. A single interaction coefficient is summarized by partial r ; interactions involving a multi-level categorical variable are tested jointly and summarized by partial η2 .
heterogeneity
Regression of the form Y∼X∗G+C , testing whether the X – Y association varies across groups G . Binary groups yield a single interaction contrast; multi-level groups use a Type-II omnibus interaction test.
Appendix
Table 9. Statistical estimators used by the deterministic hypothesis templates.
#
AutoKD system insight
Gold sub-claim
1
Solo-authored papers have less extreme novelty in their most atypical reference pair than co-authored papers.
NIH- or NSF-funded papers show higher novelty (more atypical reference combinations) than non-funded papers ( Yang, 2024 ) .
NIH-funded and NSF-funded papers are more likely to receive patent citations than non-funded papers ( Yang, 2024 ) .
3
The relationship between the number of fields a paper spans and its 3-year citation count depends on whether the paper is solo-authored.
The association between female first authorship and citation counts varies by academic field ( Shi and Wu, 2024 ) .
4
Papers with more 5-year citations tend to have later sleeping-beauty awakening years — the year a paper’s citation count finally starts surging, often decades after publication; a later year means a longer dormancy.
The negative relationship between a first author’s publication count and disruption (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) is stronger in STEM fields than in humanities or social sciences ( Li et al., 2024 ) .
5
Papers published longer ago earned fewer citations in their first three years after publication than more recent papers did — early citations are measured in a fixed three-year window, so this reflects that early-career citation rates have grown over time rather than older papers losing attention.
Papers with higher disruption scores (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) are less likely to receive patent citations; however, this largely reflects the metric’s tendency to score low-citation papers higher rather than a property of disruptive research itself ( Yang et al., 2025 ) .
6
Heavily cited papers have shorter sleeping-beauty burst durations than other papers — once a dormant paper’s citations start surging, the surge lasts less time before levelling off.
NIH- or NSF-funded papers do not differ meaningfully in disruption scores (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) from non-funded papers ( Yang, 2024 ) .
Appendix
Table 10. Complete SciSciNet CS comparison pairs between AutoKD-discovered insights and matched published gold claims.
#
AutoKD system insight
Gold sub-claim
1
Longer reviews tend to appear on less popular books.
Longer reviews receive more helpful votes ( Choi and Leon, 2020 ) .
2
Less popular books tend to have reviews with richer vocabulary (a higher proportion of unique words per review).
Reviews with extreme ratings (1-star or 5-star) receive more helpful votes than moderate-rated reviews ( Choi and Leon, 2020 ) .
3
The proportion of reviews containing questions has decreased over time.
Average review length in Amazon book reviews has declined over time for both 5-star and 1-star reviews ( Ziser et al., 2023 ) .
4
Books with more ’also bought’ recommendations — other products displayed on the book’s page — tend to have reviews with richer vocabulary (a higher proportion of unique words per review).
Reviewers who have posted many reviews receive fewer helpful votes per review than occasional reviewers do ( Choi and Leon, 2020 ) .
5
Reviews of verified purchases have different vocabulary richness (proportion of unique words per review) than non-verified reviews.
Reviews with a verified purchase badge receive more helpful votes than unverified reviews ( Choi and Leon, 2025 ) .
6
Books with more ’also bought’ recommendations — other products displayed on the book’s page — tend to have shorter reviews.
A book with more total reviews is associated with fewer helpful votes per individual review ( Choi and Leon, 2020 ) .
Appendix
Table 11. Complete Amazon Books comparison pairs between AutoKD-discovered insights and matched published gold claims.
#
AutoKD system insight
Gold sub-claim
1
Higher social trust is negatively associated with discomfort with women earning more than their husbands.
Being in a relationship is positively associated with higher life satisfaction ( Itani et al., 2024 ) .
2
Higher perceived corruption is positively associated with placing greater value on obedience as a child quality.
Males report higher justification of parents beating children than females ( Begum and Kavlu, 2023 ) .
3
More frequent religious service attendance is positively associated with stronger endorsement of nativist hiring priority when jobs are scarce.
Religious affiliation is positively associated with higher life satisfaction ( Itani et al., 2024 ) .
4
Stronger endorsement of male hiring priority when jobs are scarce is positively associated with greater confidence in churches.
Gender is not a significant predictor of life satisfaction ( Itani et al., 2024 ) .
5
Higher education level is positively associated with greater social trust.
Higher financial satisfaction is positively associated with higher life satisfaction ( Ng, 2026 ) .
6
Viewing immigrants’ impact more positively is negatively associated with placing importance on God in one’s life.
Higher education level is positively associated with lower justification of parents beating children ( Begum and Kavlu, 2023 ) .
Appendix
Table 12. Complete WVS comparison pairs between AutoKD-discovered insights and matched published gold claims.
Frameworks that use large language models for scientific discovery typically rely on a fixed, human-designed algorithm that decides what the model sees at each step, leaving the model only the role of proposer. The model knows nothing of the search beyond what it is shown. As models grow more capable, a question arises: does a search strategy chosen by a human before the run scale better than promoting the model from proposer to planner and letting it own the search? The Bitter Lesson suggests that choosing the strategy in advance is the kind of hand-designed structure that general methods eventually outscale. We introduce AgentDiscover, in which a coding agent plans the search using its context as working memory, runs experiments, and records every attempt in a database of ideas, candidates, and their relations. This database serves as the agent's long-term memory and is structured so that the selection rules of classical algorithms such as MAP-Elites and Monte Carlo tree search each reduce to a single query, which the agent is free to use, combine, or replace. A server maintains the database and steers the agent after every submission, keeping it on course over long runs. In our experiments, AgentDiscover is more cost-efficient than existing frameworks, reaching better scores at lower cost. On tasks in kernel engineering, biology, algorithm design, and mathematics, AgentDiscover outperforms prior discovery frameworks. Its programs would have placed first among human competitors in seven past AtCoder heuristic contests, and on eleven mathematical and systems optimization tasks it matches or exceeds every baseline that uses the same model. Our code is available at https://github.com/mhdfb/AgentDiscover.
Automated research with LLM agents has shown promising results in various domains. Through iterative refinement and scored verifiers, LLM agents are capable of discovering new ideas. There are different ways to scale automated research: let one agent iterate for longer, run several agents in parallel, or further let the parallel agents communicate and build on one another's candidates. However, we find that all three strategies eventually plateau as the agents accumulate history. Each agent becomes anchored to its own past attempts, making fewer and smaller edits. Beyond this point, additional budget brings little further improvement. In this paper, we show that the key to scaling discovery further is to forget. Periodically discarding the search state, including every agent's session and the candidate history, and restarting from the best candidate found so far, unblocks the search and pushes the same budget further. We implement forgetting in a team of agents with communication and evaluate it on four tasks spanning heuristic programming, kernel optimization and model training. The results show up to 15% improvement compared to parallel agents, and up to 39% compared to a single agent. Our results suggest that, for open-ended discovery, what an agent is allowed to remember matters as much as how much compute it is given.
Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.