AutoKD: Autonomous Knowledge Discovery

Qinwen Ge; Bo Ni; Haowei Fu; Ngoc N. Tran; Erik Blasch; Tyler Derr

cs.AISep 6, 2026

AutoKD: Autonomous Knowledge Discovery

Authors: Qinwen Ge, Bo Ni, Haowei Fu, Ngoc N. Tran, Erik Blasch, Tyler Derr

Organizations: Vanderbilt University · Air Force Research Laboratory

Abstract

Scientific discovery in data-rich domains is currently constrained by human bandwidth: the growth in the volume and complexity of real-world data far outpaces the rate at which researchers can read, reason, and synthesize. Recent LLM-based multi-agent systems have begun to automate portions of the research cycle, but they target hypothesis generation in settings where validation cannot itself be automated, and each run is one-shot, with no mechanism for findings to accumulate or steer subsequent inquiry. This paper introduces AutoKD, a multi-agent framework for autonomous knowledge discovery that is both computational and cumulative, allowing validated findings to persist and inform subsequent inquiry. Six coordinated LLM agents collaborate in an open-ended discovery loop, where accepted findings are stored in a persistent insight graph that serves as both long-term memory and an exploration-steering mechanism. We evaluate AutoKD on three diverse datasets from two perspectives: Open-ended Quality against published findings, and Conditioned Quality via literature-derived queries. Across both evaluation perspectives, AutoKD covers known findings and surfaces substantive discoveries that complement human-driven research. Our code is available at https://github.com/GeQinwen/AutoKD.

Figures & tables

Figure 1. High-level overview of AutoKD for autonomous knowledge discovery including insight graph storage allowing cross-round accumulation.

Figure 2. An overview of the AutoKD framework for autonomous agentic knowledge discovery. AutoKD first iteratively discovers and accumulates structured insights via the knowledge creation component, and then through a retrieval, synthesis, verification process enables knowledge extraction.

Figure 3. Proposed evaluation framework for autonomous knowledge discovery. Gold claims distilled from published papers support two complementary protocols: open-ended quality compares AutoKD insights with gold claims, while conditioned quality evaluates literature-derived queries through retrieval and synthesis over the insight graph.

Dataset	Rater	Int.	Surp.	Useful	Overall
WVS	Human	83.8	86.1	81.1	86.1
	LLM	97.4	100.0	66.7	92.5
	All	90.7	93.2	73.7	89.5
Amazon	Human	53.1	56.2	63.6	54.5
	LLM	50.0	56.4	17.5	47.5
	All	51.4	56.3	38.4	50.7

Table 1. AutoKD win rate (%, ties excluded) by dataset, dimension, and rater subset. Human and LLM results each aggregate four raters; All aggregates all eight raters.

Humans only (4 raters)
	Interesting	Surprising	Useful	Overall
WVS	.012	.064	$-.038$	$-.070$
Amazon	.139	.186	.255	.202
SciSciNet	$-.065$	$-.075$	$-.056$	$-.053$
All	.098	.111	.111	.112
LLM judges only (4 raters)

Table 2. Inter-rater agreement (Fleiss’

\kappa

Case	AutoKD Finding (preferred)	Gold Counterpart	LLM Judges
Amazon (useful)	Longer reviews tend to appear on less popular books.	Longer reviews receive more helpful votes.	Disagree
WVS (interesting)	Higher social trust is negatively associated with discomfort with women earning more than their husbands.	Being in a relationship is positively associated with higher life satisfaction.	Agree
SciSciNet (surprising)	Heavily cited papers span fewer broad academic fields.	NIH- and NSF-funded papers are more likely to receive patent citations than non-funded papers.	Agree

Table 3. Representative cases where all four human evaluators preferred the AutoKD-generated finding over its gold-benchmark counterpart along the indicated dimension.

Figure 4. Variable-usage distribution for the full insight graph (blue), the score-only top-10 (orange), and the top-10 after diversity reranking (green). Variables are ordered by full-graph frequency.

	MC	C	P
WVS	0.0	40.0	60.0
Amazon	40.0	0.0	60.0
SciSciNet	20.0	0.0	80.0
Avg.	20.0	13.3	66.7

Table 4. Gold-claim coverage by AutoKD (%). MC/C/P/NC = more-than covered/covered/partially covered/not covered.

	Amazon	WVS	SciSciNet (CS / All)
Insights	119	1,367	565 / 1,027
Relations	268	3,640	1,496 / 2,794

Table 5. Discovery output per dataset. Insights are validated findings stored in the graph; relations are typed edges connecting them.

Variant	$n$	Estimands / insight $\uparrow$	Exact rediscovery $\downarrow$	Deep-test share $\uparrow$	$\Delta$ Depth
AutoKD	819	0.949	0.0%	28.9%	–
w/o persistence	1,021	0.672	16.7%	12.5%	$-56.7\%$
w/o graph structure	828	0.944	0.0%	26.0%	$-10.3\%$
global-only	785	0.941	0.0%	20.1%	$-30.4\%$

Table 6. Ablation of cumulative discovery mechanisms on WVS at a matched 1,000-round budget.

\Delta

Depth is the relative change in deep-test share from full AutoKD.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Hyperparameters
Generator	batch size $B=10$ ; context anchors $=5$
Evaluator	$p_{\min}=0.05$ ; effect floor $e_{\min}=0.03$ ; replication split $=70/30$
Critic	insight threshold $\tau_{s}=0.55$ ; structural Jaccard $=0.80$ ; semantic threshold $=0.88$
Coder/Runner	timeout $=30$ s; max retries $=3$
Historian	duplicate cosine threshold $=0.92$ ; retrieval order D $>$ N $>$ E $>$ C
Orchestrator	mode weights (exp./global/ref./conf.) $=0.30/0.15/0.40/0.30$ ; $g_{\min}=12$ ; max rounds $=2{,}000$

Appendix

Table 7. Agent-level hyperparameters for reported runs. W = WVS, A = Amazon, S = SciSciNet.

Figure 5. Worked example of the literature-grounded extraction pipeline on SciSciNet.

$k$	Amazon ( $N=22$ )	WVS ( $N=198$ )	SciSciNet ( $N=87$ )
0	$1.94{\times}10^{4}$	$1.54{\times}10^{7}$	$1.29{\times}10^{6}$
1	$3.70{\times}10^{5}$	$3.00{\times}10^{9}$	$1.08{\times}10^{8}$
2	$3.34{\times}10^{6}$	$2.91{\times}10^{11}$	$4.49{\times}10^{9}$
3	$1.90{\times}10^{7}$	$1.87{\times}10^{13}$	$1.23{\times}10^{11}$
4	$7.61{\times}10^{7}$	$8.98{\times}10^{14}$	$2.48{\times}10^{12}$
5	$2.29{\times}10^{8}$	$3.43{\times}10^{16}$	$3.98{\times}10^{13}$

Appendix

Table 8. Total hypothesis counts across all four templates as a function of the number of controls

k

N

is the number of testable variables in each dataset.

Template	Estimator and reported effect
assoc	Pearson correlation for unadjusted numeric variables; otherwise OLS/WLS regression. A one-degree-of-freedom term is summarized by (partial) $r$ ; a multi-level categorical predictor is tested jointly with Type-II ANOVA and summarized by partial $\eta^{2}$ .
diff	For two groups without controls, Welch’s $t$ -test with Cohen’s $d$ is used for continuous outcomes, while low-cardinality ordered outcomes use Mann–Whitney $U$ with rank-biserial $r$ . More than two groups use Kruskal–Wallis with $\epsilon^{2}$ . With controls or weights, the comparison is expressed as an OLS/WLS model, using a contrast for two groups or a Type-II omnibus test for multiple groups.
interact	Regression of the form $Y\sim X*Z+C$ , where $C$ denotes optional controls. A single interaction coefficient is summarized by partial $r$ ; interactions involving a multi-level categorical variable are tested jointly and summarized by partial $\eta^{2}$ .
heterogeneity	Regression of the form $Y\sim X*G+C$ , testing whether the $X$ – $Y$ association varies across groups $G$ . Binary groups yield a single interaction contrast; multi-level groups use a Type-II omnibus interaction test.

Appendix

Table 9. Statistical estimators used by the deterministic hypothesis templates.

#	AutoKD system insight	Gold sub-claim
1	Solo-authored papers have less extreme novelty in their most atypical reference pair than co-authored papers.	NIH- or NSF-funded papers show higher novelty (more atypical reference combinations) than non-funded papers ( Yang, 2024 ) .
2	Heavily cited papers span fewer broad academic fields.	NIH-funded and NSF-funded papers are more likely to receive patent citations than non-funded papers ( Yang, 2024 ) .
3	The relationship between the number of fields a paper spans and its 3-year citation count depends on whether the paper is solo-authored.	The association between female first authorship and citation counts varies by academic field ( Shi and Wu, 2024 ) .
4	Papers with more 5-year citations tend to have later sleeping-beauty awakening years — the year a paper’s citation count finally starts surging, often decades after publication; a later year means a longer dormancy.	The negative relationship between a first author’s publication count and disruption (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) is stronger in STEM fields than in humanities or social sciences ( Li et al., 2024 ) .
5	Papers published longer ago earned fewer citations in their first three years after publication than more recent papers did — early citations are measured in a fixed three-year window, so this reflects that early-career citation rates have grown over time rather than older papers losing attention.	Papers with higher disruption scores (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) are less likely to receive patent citations; however, this largely reflects the metric’s tendency to score low-citation papers higher rather than a property of disruptive research itself ( Yang et al., 2025 ) .
6	Heavily cited papers have shorter sleeping-beauty burst durations than other papers — once a dormant paper’s citations start surging, the surge lasts less time before levelling off.	NIH- or NSF-funded papers do not differ meaningfully in disruption scores (a measure of how much a paper displaces prior work rather than building on it; higher = more disruptive) from non-funded papers ( Yang, 2024 ) .

Appendix

Table 10. Complete SciSciNet CS comparison pairs between AutoKD-discovered insights and matched published gold claims.

#	AutoKD system insight	Gold sub-claim
1	Longer reviews tend to appear on less popular books.	Longer reviews receive more helpful votes ( Choi and Leon, 2020 ) .
2	Less popular books tend to have reviews with richer vocabulary (a higher proportion of unique words per review).	Reviews with extreme ratings (1-star or 5-star) receive more helpful votes than moderate-rated reviews ( Choi and Leon, 2020 ) .
3	The proportion of reviews containing questions has decreased over time.	Average review length in Amazon book reviews has declined over time for both 5-star and 1-star reviews ( Ziser et al., 2023 ) .
4	Books with more ’also bought’ recommendations — other products displayed on the book’s page — tend to have reviews with richer vocabulary (a higher proportion of unique words per review).	Reviewers who have posted many reviews receive fewer helpful votes per review than occasional reviewers do ( Choi and Leon, 2020 ) .
5	Reviews of verified purchases have different vocabulary richness (proportion of unique words per review) than non-verified reviews.	Reviews with a verified purchase badge receive more helpful votes than unverified reviews ( Choi and Leon, 2025 ) .
6	Books with more ’also bought’ recommendations — other products displayed on the book’s page — tend to have shorter reviews.	A book with more total reviews is associated with fewer helpful votes per individual review ( Choi and Leon, 2020 ) .

Appendix

Table 11. Complete Amazon Books comparison pairs between AutoKD-discovered insights and matched published gold claims.

#	AutoKD system insight	Gold sub-claim
1	Higher social trust is negatively associated with discomfort with women earning more than their husbands.	Being in a relationship is positively associated with higher life satisfaction ( Itani et al., 2024 ) .
2	Higher perceived corruption is positively associated with placing greater value on obedience as a child quality.	Males report higher justification of parents beating children than females ( Begum and Kavlu, 2023 ) .
3	More frequent religious service attendance is positively associated with stronger endorsement of nativist hiring priority when jobs are scarce.	Religious affiliation is positively associated with higher life satisfaction ( Itani et al., 2024 ) .
4	Stronger endorsement of male hiring priority when jobs are scarce is positively associated with greater confidence in churches.	Gender is not a significant predictor of life satisfaction ( Itani et al., 2024 ) .
5	Higher education level is positively associated with greater social trust.	Higher financial satisfaction is positively associated with higher life satisfaction ( Ng, 2026 ) .
6	Viewing immigrants’ impact more positively is negatively associated with placing importance on God in one’s life.	Higher education level is positively associated with lower justification of parents beating children ( Begum and Kavlu, 2023 ) .

Appendix

Table 12. Complete WVS comparison pairs between AutoKD-discovered insights and matched published gold claims.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 287 papers

AgentDiscover: Autonomous Discovery with Minimal Search Scaffolding
Oct 4, 2026Mahdi Farahbakhsh, Ilan Sela, Fatemeh Doudi +4AI Agents for Scientific Discovery Automated Algorithm Discovery
Texas A&M University
Scaling Discovery Requires Forgetting
Oct 14, 2026Shangyin Tan, Ben Zhou, Lakshya A Agrawal +8AI Agents for Scientific Discovery LLM Agent Memory
UC Berkeley · Stanford · Bespoke Labs
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*

Oct 4, 2026J/K move · Enter open · S save

AgentDiscover: Autonomous Discovery with Minimal Search Scaffolding

Mahdi Farahbakhsh, Ilan Sela, Fatemeh Doudi, Vishnu Teja Kunde+3

Texas A&M University

AI Agents for Scientific Discovery Automated Algorithm Discovery Automated Heuristic Design Autonomous Scientific Discovery LLM Agent Planning Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Frameworks that use large language models for scientific discovery typically rely on a fixed, human-designed algorithm that decides what the model sees at each step, leaving the model only the role of proposer. The model knows nothing of the search beyond what it is shown. As models grow more capable, a question arises: does a search strategy chosen by a human before the run scale better than promoting the model from proposer to planner and letting it own the search? The Bitter Lesson suggests that choosing the strategy in advance is the kind of hand-designed structure that general methods eventually outscale. We introduce AgentDiscover, in which a coding agent plans the search using its context as working memory, runs experiments, and records every attempt in a database of ideas, candidates, and their relations. This database serves as the agent's long-term memory and is structured so that the selection rules of classical algorithms such as MAP-Elites and Monte Carlo tree search each reduce to a single query, which the agent is free to use, combine, or replace. A server maintains the database and steers the agent after every submission, keeping it on course over long runs. In our experiments, AgentDiscover is more cost-efficient than existing frameworks, reaching better scores at lower cost. On tasks in kernel engineering, biology, algorithm design, and mathematics, AgentDiscover outperforms prior discovery frameworks. Its programs would have placed first among human competitors in seven past AtCoder heuristic contests, and on eleven mathematical and systems optimization tasks it matches or exceeds every baseline that uses the same model. Our code is available at https://github.com/mhdfb/AgentDiscover.

Abstract

AI Agents for Scientific Discovery

Explore similar work

AgentDiscover: Autonomous Discovery with Minimal Search Scaffolding

Scaling Discovery Requires Forgetting

AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery