DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses

Jianyuan Zhong; Chenwei Wu; Zhaochen Yu; Xixian Jiang; Xuewei Li; Hao Xiao; Xilin Dai; Junyang Wang

cs.CLOct 16, 2026

DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses

Authors: Jianyuan Zhong, Chenwei Wu, Zhaochen Yu, Xixian Jiang, Xuewei Li, Hao Xiao, Xilin Dai, Junyang Wang,

Organizations: PhAI Labs · The Chinese University of Hong Kong · University of Michigan · National University of Singapore ·

Abstract

Scientific discovery requires developing worthwhile hypotheses into practical investigations, yet learning from this evolving research process remains challenging. We introduce an operational definition of valuable and testable research hypotheses: evidence-grounded claims about consequential uncertainties, coupled with feasible investigations, explicit decision criteria, and alternative explanations. Our DiscoveryRSI framework develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and recursively learns through three mechanisms: hypothesis credit optimization attributes scientific assessments to documented research contributions; harness replay identifies productive strategies and recurring failures to refine research workflows; and rediscovery applies accepted improvements to new investigations, generating further experience for subsequent learning. We also introduce DFM-Bench, which provides comprehensive assessment of hypothesis quality and cumulative research progress at the levels of hypotheses, revisions, development paths and research runs. Across four life-science domains, automated evaluation shows that hypothesis quality rises from 4.86 for the base model to 5.39 after two rounds of self-improvement, with the large gains in falsifiability and data resolvability. We further validate hypothesis quality through expert review, comparing our DiscoveryRSI-27B with other research agents. Preliminary human assessments show that DiscoveryRSI-27B scores higher than GPT-6 Luna on hypothesis importance and whether claims are plausible and evidence-bounded. DiscoveryRSI-27B achieved a 6.3% rate of hypothesis correspondence with research published after the model cutoff, largely outperform GPT-6 with 2.5% and Claude Opus 5 with 2.3%, while delivering greater human-rated improvement along research paths.

Figures & tables

Figure 1

Figure 2: A recorded research hypothesis and its planned investigation. This MIMIC-IV proposal concerns adults surviving a first ICU stay of at least 48 hours. (a) It asks whether repeated creatinine–urine output mismatch adds information about ICU readmission or in-hospital death within 48 hours after ICU discharge, during the same hospitalization. (b) B0–B2 compare clinical, current-mismatch, and repeated-mismatch logistic models; L1 adds hidden Markov model summaries, subject to data-support and stability checks. Models share a patient-level training, tuning, and test split. (c,d) Decision criteria and alternatives bound interpretation. No clinical outcome is asserted.

	Scientific assessment		Research-process assessment
Benchmark	Hypothesis merit	Contribution maturity	Evidence use	Revision	Development path	Whole-run allocation
AgentIdeaBench ( Mo et al., 2026 )	$\checkmark$	$\times$	$\circ$	$\times$	$\times$	$\times$
DiscoveryBench ( Majumder et al., 2024 )	$\circ$	$\times$	$\circ$	$\times$	$\times$	$\times$
BLADE ( Gu et al., 2025 )	$\times$	$\times$	$\circ$	$\times$	$\times$	$\times$
ScienceAgentBench ( Chen et al., 2024 )	$\times$	$\times$	$\circ$	$\times$	$\times$	$\times$
OEB ( Shi et al., 2026 )	$\times$	$\times$	$\checkmark$	$\checkmark$	$\circ$	$\circ$

Table 1: Evaluation coverage of selected scientific-discovery benchmarks. A check denotes explicit assessment, a circle related or partial assessment, and a cross no explicit assessment under our definitions. Recording intermediate actions alone does not establish coverage. Definitions and qualifications appear in Appendix D.6 .

Figure 3: Expert-supervised assessment of scientific hypotheses. (a) An anonymous MIMIC-IV proposal presents its scientific question, planned comparison, and claim limits. (b) Reviewers record an overall judgment, assess six anchored criteria, and identify design flaws; flaw details and comments are optional. The criterion list summarizes the rubric, while radio-option excerpts reproduce one saved assessment. Identifying metadata is omitted. The example illustrates the review procedure rather than an aggregate evaluation result.

Figure 4: Shared research process across life-science and battery domains. The process strip summarizes how evidence becomes a question, investigation, and evidence-bounded interpretation with iterative critique. The two rows show the four life-science domains and four battery domains used in the experiments.

Figure 5: Automated hypothesis quality across self-improvement rounds. (a) Top: example research tree from each round; color shows weighted quality across eight evaluation axes. Middle: mean hypothesis quality per round. Error bars show 95% confidence intervals and shading shows the interquartile range. Bottom: mean quality per domain with 95% confidence intervals. (b) Mean and axis-specific scores. All scores are produced by the automated grader using the weights in Appendix D .

Figure 6: Changes in research behavior across rounds. (a) Share of each action across all episodes in a round. (b) Share of campaigns taking each action at each episode position, sampled from six campaigns per round; blank cells indicate 0%. (c) Change in development-path scores from the base model to round 2.

Figure 7: Human assessment of hypothesis quality and revision progress. (a) Mean ratings across six criteria on a 1–4 scale. (b) Mean score change per revision. (c) Fractions of improved, unchanged, and worse revisions, in the same model order as panel (b).

Figure 8: Design changes and human ratings along a revision path. (a) A MIMIC-IV path and assessed siblings; node values are six-criterion means on a 1–4 scale. Numbered changes are explained below the tree. Dashed ochre nodes mark fatal flaws; the two final fatal siblings lack specified AKI thresholds. (b) Criterion ratings along the highlighted path.

Figure 9: Hypothesis inspiration measured through subsequent literature. The upper panel orders model groups by the proportion of generated hypothesis versions meeting the inspiration criterion. The lower panel reports the same rate across four domains. DiscoveryRSI-27B pools two production runs; Qwen original is omitted. The figure’s reference-actionability labels denote this retrospective inspiration measure.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Record type	Contents	Recorded relationships
Hypothesis versions	Claim, plan, criteria, alternatives, and parents.	Revision ancestry; restart context recorded separately.
Literature and evidence	Sources, excerpts, and retrieval metadata.	Origin, dates, versions, and content hashes.
Data dependencies	Files and inputs required for investigation.	Bindings to hypotheses, jobs, and tasks.
Supporting artifacts	Scripts, results, plots, and evidence notes.	Explicit hypothesis attachments with paths and hashes.
Drafts and notes	Working proposals and continuation notes.	Agent ownership and recorded publication links.
Scientific assessments	Validity, preferences, and Progress judgments.	Assessed objects, comparison pairs, and ancestors.

Appendix

Table 2: Persistent records supporting research and recursive improvement. Files retain material; explicit references retain provenance and relationships.

Quantity	Value	Function
Conservative-rating factor $\kappa$	$2$	Penalize uncertain preference means
Coverage constant $\nu$	$2$	Damp sparsely compared hypotheses
Uncertainty reference $\sigma_{\rm ref}$	$8.3333$	Fixed rating-scale reference
Weight interval $[q_{\min},q_{\max}]$	$[0.5,2]$	Bound eligible outcome weights
Progress supplement $\lambda_{p}$	$0.25$	Use recorded progress evidence
Coordination fraction $\gamma$	$0.2$	Reserve coordination share

Appendix

Table 12

Role	Collected	Training	Sequence tokens	Supervised tokens
Lead	3,423	634	98,357,399	977,931
Generation	2,502	861	96,414,270	2,437,985
Evolution	2,310	902	96,349,331	2,761,689
Critique	4,988	0	—	0
Compiler	2,056	0	—	0
Progress	513	0	—	0

Appendix

Table 4: Trajectory and token accounting for hypothesis-credit training. Training-sequence tokens include supervised targets and zero-loss context. Collected trajectories and retained training trajectories are counted separately.

Level	Object	Core scientific question	Outputs
Hypothesis	Frozen candidate plus evidence	Is this a worthwhile, grounded, testable research opportunity?	Eight-dimension profile; Q /10; unknowns and review coverage.
Revision	Actual parent–child pair	Did this step advance understanding, repair an important defect, or repeat prior work?	Substantive progress, necessary repair, repetition, insufficient evidence; numeric $\Delta Q$ separately.
Development path	Ordered hypothesis path	Does inquiry accumulate evidence and revise decisions coherently?	Seven-axis L1–L7 profile; complete, scope-matched composite /100.
Research run	Full research graph and decisions	Does the portfolio address meaningful opportunities and allocate effort well?	Independent T1–T7 profile; coverage, synthesis, allocation, and frontier diagnostics.
Scientific contribution	Deduplicated contribution family	What question, distinction, evidence, negative result, or uncertainty resolution was added?	Contribution ledger, evidence maturity, novelty scope, and adjudication status.

Appendix

Table 5: The five DFM-Bench evaluation objects. Scientific quality is evaluated at each level using evidence appropriate to that object. Delivery maturity is a separate axis.

Domain	Positive	Generated	Rate
Clinical and population health	13	557	2.3%
Therapeutic target prioritization	30	178	16.9%
Disease mechanisms	17	251	6.8%
Population multi-omics	14	190	7.4%
Total	74	1,176	6.3%

Appendix

Table 6: DiscoveryRSI-27B literature-inspiration counts. Numerators count hypothesis versions with positive judgments; denominators include all generated versions.

Scientific Hypothesis Generation

Scientific hypothesis generation develops methods for proposing testable explanations and research directions from literature, data, and experiments.

See all 89 papers

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher
Hypothesis-Driven Deep Research with Large Language Models: A Structured Methodology for Automated Knowledge Discovery
May 11, 2026Michael ChinDeep Research Agents AI-Assisted Scientific Research
Independent Researcher
Open-ended Scientific Discovery with Possibilistic Reasoning
Oct 8, 2026Anita Yang, Siu Lun Chau, Tomoya Wakayama +2Scientific Discovery Autonomous Scientific Discovery
Lattice Lab, Toyota Motor Corporation, Japan · Department of Computer Science, University of Tokyo, Japan · EPIC Lab, College of Computing & Data Science, Nanyang Technological University, Singapore +2

Sep 27, 2026J/K move · Enter open · S save

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Nan Huang, Mario Tapia-Pacheco, Kun Zhou, Yiming Huang+3

University of California San Diego · Independent Researcher

AI Agent Reliability Scientific Hypothesis Generation AI Agent Benchmarks Scientific Reasoning in Language Models Data Analysis Agents

Open paper AskPDF arXiv

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

Abstract

Scientific Hypothesis Generation

Explore similar work

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Hypothesis-Driven Deep Research with Large Language Models: A Structured Methodology for Automated Knowledge Discovery

Open-ended Scientific Discovery with Possibilistic Reasoning