Discovering what drives an outcome from data is central to science and often requires deciding what is worth investigating before a specific hypothesis is known. Yet most existing hypothesis generation settings substantially narrow this search by providing a research question, target relationship, or curated context. We introduce open-ended multiple hypothesis discovery, where a system receives only a large tabular dataset and an outcome and must discover multiple data-supported hypotheses without knowing in advance the relevant variables, functional forms, conditions, or number of underlying mechanisms. To study this problem, we introduce a benchmark built from structural causal models grounded in real-world data schemas, containing 351 planted hypotheses across 15 tasks and five domains, with 53--65 variables and 1M--10M rows per task. The causal model is hidden from the discovery method and retained only by the evaluator, allowing proposed hypotheses to be tested through interventions and separating observational discovery from causal validation. We further propose HypAgent, which treats the dataset as an interactive environment and performs iterative search by proposing candidate directions, triaging and refining promising hypotheses, and removing explained signal so that each accepted discovery redirects subsequent search. HypAgent achieves 0.47 precision, 0.51 recall, and 0.49 F1, compared with 0.35 F1 for the strongest baseline. Our analysis shows that many failures occur after a relevant variable has already been surfaced, suggesting that correctly characterizing a mechanism's functional form and conditions, rather than merely identifying relevant variables, remains a key challenge in open-ended discovery.
Figures & tables
Figure 1: Overview of Open-Ended Hypothesis Discovery. Left: Given only a tabular dataset D and outcome Y , the goal is to discover multiple unknown hypotheses. Middle: HypAgent iteratively proposes, triages, refines, and residualizes hypotheses so that each discovery guides the next search. Right: OpenHypBench evaluates proposed hypotheses through interventions on the hidden structural causal model.
Benchmark
Scale
Discovery setting
Open-ended evaluation
New targets
Causal
Context-based generation ( Qi et al., 2024 )
3K
Curated context
✗
✗
✗
Data-driven analysis ( Gu et al., 2024 )
∼ 100K
Prespecified question
✗
✗
✗
Hypothesis suites ( Liu et al., 2025a ; Majumder et al., 2025 )
Table 1: Comparison of existing hypothesis discovery benchmarks. Scale denotes the maximum number of rows in a benchmark dataset. New targets indicates that the evaluated data and target effects are newly generated rather than published targets.
Figure 3
Figure 3: Overall discovery performance on OpenHypBench. (a) Point estimates at each method’s reported operating point; time is mean end-to-end discovery time per task and excludes the shared evaluator. Confidence intervals are reported in Figure 6 in the appendix. (b) Recall after uniformly subsampling each method to a common budget of up to 25 proposals per task.
Figure 4: Recall across hypothesis archetypes with coral outlines marking the best-performing method for each archetype.
Figure 5: F1 as the number of rows, features, and planted hypotheses varies.
Configuration
P
R
F1
Impact
HypAgent
0.47
0.51
0.49
–
− Residualization
−0.15
−0.12
−0.13
68.8% increase in duplicates
− Triage
−0.08
−0.13
−0.11
Suboptimal direction in 25.5% of cases
− Refinement
−0.31
−0.36
−0.34
58.5% increase in unsupported hypotheses
Table 3: Ablation of HypAgent components across all 15 tasks. Ablated rows report changes relative to the full method.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Form
Definition and interventional signature
Count
Monotone threshold
T changes Y after crossing a threshold, e.g., ϕ(T)=wI[T>q] .
30
Non-monotone
The effect is strongest at both tails or within an interior band; the resulting curve contains at least one direction change.
47
Derived feature
The treatment is a function of multiple variables, U=g(T1,T2) , such as a ratio, product, or difference.
50
Interaction reversal
The direction of the effect of T changes across values of a moderator variable M .
71
Subpopulation-specific
The effect is active only under a condition c(M) .
57
Simpson’s reversal
The marginal association between T and Y has the opposite direction from the association within strata of M .
47
Appendix
Table 4: Functional forms in OpenHypBench. Counts are aggregated across all 15 tasks.
Task ID
Domain
Source dataset
URL
License
Rows
Outcome ( Y=1 )
Features
Hyp.
census_acs
Census
ACS PUMS ( U.S. Census Bureau, 2026a )
link
U.S. public domain
1M
Housing cost burden
63
23
census_adult
Census
UCI Adult ( Becker and Kohavi, 1996 )
link
CC BY 4.0
5M
High income
63
25
census_cps
Census
CPS ASEC ( U.S. Census Bureau, 2026b )
link
U.S. public domain
10M
Long commute
64
25
ecom_hm
E-commerce
H&M ( García Ling et al., 2022 )
link
Competition rules
5M
Customer churn
61
23
ecom_instacart
E-commerce
Instacart ( Risdal and Cukierski, 2017 )
link
Competition rules
1M
Product reordered
63
21
ecom_onlineretail
E-commerce
Online Retail II ( Chen, 2012 )
link
CC BY 4.0
10M
High lifetime value
62
24
Appendix
Table 5: Task list of OpenHypBench.
Figure 6: Precision, recall, and F1 at each method’s reported operating point. Whiskers show 95% cluster-bootstrap confidence intervals over tasks.
Figure 7: Failure modes among the 172 planted hypotheses missed by HypAgent.
Scientific discovery requires developing worthwhile hypotheses into practical investigations, yet learning from this evolving research process remains challenging. We introduce an operational definition of valuable and testable research hypotheses: evidence-grounded claims about consequential uncertainties, coupled with feasible investigations, explicit decision criteria, and alternative explanations. Our DiscoveryRSI framework develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and recursively learns through three mechanisms: hypothesis credit optimization attributes scientific assessments to documented research contributions; harness replay identifies productive strategies and recurring failures to refine research workflows; and rediscovery applies accepted improvements to new investigations, generating further experience for subsequent learning. We also introduce DFM-Bench, which provides comprehensive assessment of hypothesis quality and cumulative research progress at the levels of hypotheses, revisions, development paths and research runs. Across four life-science domains, automated evaluation shows that hypothesis quality rises from 4.86 for the base model to 5.39 after two rounds of self-improvement, with the large gains in falsifiability and data resolvability. We further validate hypothesis quality through expert review, comparing our DiscoveryRSI-27B with other research agents. Preliminary human assessments show that DiscoveryRSI-27B scores higher than GPT-6 Luna on hypothesis importance and whether claims are plausible and evidence-bounded. DiscoveryRSI-27B achieved a 6.3% rate of hypothesis correspondence with research published after the model cutoff, largely outperform GPT-6 with 2.5% and Claude Opus 5 with 2.3%, while delivering greater human-rated improvement along research paths.
Jianyuan Zhong, Chenwei Wu, Zhaochen Yu +20
PhAI Labs · The Chinese University of Hong Kong · University of Michigan +13
Data-driven mechanistic hypotheses are essential to scientific discovery because they explain how underlying processes produce observed phenomena. AI agents and AI scientists increasingly support scientific data analysis. However, their ability to turn empirical findings into mechanistic hypotheses remains insufficiently examined. To address this gap, we introduce MechHypoBench, the first benchmark for evaluating whether AI agents and AI scientists can generate such hypotheses from empirical data. It combines paper-derived mechanisms from 14 scientific fields with real-world datasets containing 17.98 million records. The construction retains the observational complexity of empirical data while providing a specified underlying mechanism. Agents analyze the observations and propose open-form hypotheses. We develop an evaluation framework that assesses open-form mechanistic hypotheses through their consequences under withheld conditions. Experiments with general agents and AI scientists reveal a substantial gap between generated hypotheses and the underlying mechanisms.
Xiaxun Xie, Qingqing Long, Meng Xiao +4
Computer Network Information Center, Chinese Academy of Sciences · National University of Singapore · Sichuan University
Autonomous scientific discovery systems offer the potential to accelerate research by automating the process of hypothesis generation and validation. However, current systems operate within constrained search spaces or require predefined research questions, limiting their capacity for true open-ended inquiry. Furthermore, while they generate hypotheses iteratively, they largely lack the ability to explicitly synthesize their own accumulated findings to uncover complex, interconnected phenomena. We introduce DiscoPER, an autonomous large language model-powered framework that conducts open-ended research by dynamically generating and executing code to explore datasets without pre-specified research objectives. To ensure rigorous scientific validity, every proposed discovery must pass statistical testing. To overcome the limitations of isolated search, our framework introduces a second-order reasoning mechanism that periodically analyzes its own accumulated discoveries. By treating prior discoveries as empirical data, DiscoPER identifies structural patterns, confounds, and epistemic gaps, actively redirecting hypothesis exploration toward uncharted regions of the search space. The search space is further expanded by incorporating tool use, enabling the system to explore hypotheses beyond structured metadata by seamlessly processing and extracting useful information from multimodal sources like images. Evaluated on iNatDisco, a new multimodal ecological knowledge benchmark with pattern-level ground truth obtained from peer-reviewed literature, DiscoPER recovers 8 of 9 known patterns with a 72.7% hypothesis support rate, outperforming both classical causal discovery and LLM-guided baselines. Ablations show that DiscoPER scales with more data, and confirms the benefits of second-order meta-reflection.
Bingchen Zhao, Sara Beery, Oisin Mac Aodha
University of Edinburgh · Massachusetts Institute of Technology