Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations
Organizations: Department of Computer Science Cornell University
Abstract
LLM agents are increasingly used for open-ended, data-driven scientific discovery, yet most of them evaluate hypotheses pointwise, testing an empirical claim only against its complement . When evidence rejects , the agent learns that one explanation is unlikely but not which alternative the evidence supports. To recover that alternative, we introduce Contrastive Discovery, which replaces the single hypothesis with a contrast set, a scientific question paired with mutually exclusive and collectively exhaustive competing explanations. Each experiment is scored by how its evidence redistributes belief across the set, and the question is resolved when belief concentrates on one explanation. Because contrast sets change only the unit an agent proposes and scores, an existing agent can use them to re-evaluate evidence it has already collected or to guide its search. Across six domains, of the pointwise candidates with the largest belief shifts are rejections. On the same evidence, contrast sets resolve of these candidates to a specific alternative. When contrast sets guide search, the agent resolves more questions than pointwise surprisal under the same experiment budget. In a blinded study, human experts chose the same explanation on of resolved questions and on of questions where the pointwise agent rejected its hypothesis and stopped. Together, these results support explicit competing explanations as a way for discovery agents to resolve more scientific questions from the same evidence and validation budget.
Figures & tables
| Questions | Experts agree | |
|---|---|---|
| Contrastive resolved | 51 | 40 (78.4%) |
| Contrastive unresolved | 9 | 3 (33.3%) |
| Pointwise confirmed | 32 | 23 (71.9%) |
| Pointwise rejected | 22 | 18 (81.8%) |
| Pointwise ambiguous | 6 | 2 (33.3%) |
| All items | 60 | 43 (71.7%) |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Setting | Pointwise | Contrastive [ ] |
|---|---|---|---|
| Conversation | Prior | 0.730 | 0.764 [0.822] |
| Posterior | 0.873 | 0.917 [0.893] | |
| Mortgage | Prior | 0.602 | 0.730 [0.786] |
| Posterior | 0.854 | 0.934 [0.894] | |
| Materials Project | Prior | 0.672 | 0.506 [0.780] |
| Posterior | 0.906 | 0.933 [0.897] |
| Diagnostic | Value |
|---|---|
| Contrast sets with complete prior/posterior elicitations | |
| Mean pairwise TVD, prior samples | |
| Mean pairwise TVD, posterior samples | |
| Posterior contrast sets with pairwise TVD below | |
| Posterior per-hypothesis SE of sample mean | |
| Posterior maximum-probability sample SD |
| Quantity | Count / estimate | Rate |
|---|---|---|
| Structurally MECE sampled contrast sets | ||
| Design-weighted population estimate | – | |
| 95% stratified bootstrap CI | – | |
| Overlap or redundancy failures | ||
| Inherently non-MECE mechanism failures |
| Judge | Provider | Raw pass | Design-weighted pass | Sets flagged |
|---|---|---|---|---|
| gpt-4o (same provider as proposer) | OpenAI | 96.9% | 98.4% | 12 |
| Claude Sonnet | Anthropic | 87.0% | 91.4% | 50 |
| Pairwise consistency | canonical | randomized | ratio |
|---|---|---|---|
| Prior pairwise TVD | |||
| Posterior pairwise TVD | |||
| Aggregate belief statistics | canonical | randomized | [95% CI] |
| Comparison | 95% CI | Precision | Recall | ||
|---|---|---|---|---|---|
| Support gate | |||||
| Rejection gate | |||||
| Joint two-sided gate |
| evidence that the gates are matched | resolved units at top-200 | ||||||
| contr. fires | point. fires | F1 | contrastive | pointwise | gain | ||
| 0.600 | 0.535 | 95.3% | 95.3% | 0.966 | 185.5 | 104.8 | +0.403 |
| 0.667 | 0.585 | 93.3% | 93.5% | 0.963 | 178.7 | 100.8 | +0.390 |
| 0.700 | 0.605 | 92.1% | 92.0% | 0.959 | 174.8 | 99.3 | +0.378 |
| 0.750 | 0.635 | 90.5% | 90.6% | 0.952 | 169.7 | 96.7 | +0.365 |
| 0.800 | 0.695 | 86.8% | 86.3% | 0.927 | 161.9 | 88.3 | +0.368 |
| Dataset | Domain | Rows | Cols. | Row unit |
| Materials Project | Materials science | 68,805 | 28 | Inorganic crystal |
| SDSS Galaxy (DR19) | Astronomy | 50,000 | 28 | Galaxy |
| BLADE datasets | ||||
| Mortgage | Finance / policy | 2,380 | 13 | Mortgage application |
| Conversation | Online discourse | 7,975 | 70 | Debate contribution |
| DiscoveryBench datasets | ||||
| of which resolved to | ||||
|---|---|---|---|---|
| Supported explanation removed | Declined or flagged | Unsupported | no effect | competing direction |
| Contrast set as generated | 46.6% | 53.4% | 50.9% | 2.5% |
| With “none of the above” | 99.4% | 0.6% | 0% | 0.6% |
| Pointwise | Contrastive | ||||||
|---|---|---|---|---|---|---|---|
| Dataset | |||||||
| Materials Project | 28 | 0.955 | 0.056 | 157 | 0.919 | 0.062 | – |
| BLADE Conversation | 70 | 0.905 | 0.033 | – | 0.863 | 0.038 | – |
| Requirements Engineering | 202 | 0.513 | 0.015 | – | 0.411 | 0.015 | – |
| SDSS Galaxy | 28 | 0.955 | 0.103 | 87 | 0.955 | 0.065 | 176 |
| BLADE Mortgage | 13 | 1.000 | 0.290 | 11 | 0.999 | 0.274 | 11 |
| Pointwise | Contrastive | |||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | ||||||||
| Materials Project | 2.43 | 0.031 | 0.094 | 0.135 | 2.45 | 0.064 | 0.092 | 0.130 |
| BLADE Conversation | 4.06 | 0.048 | 0.074 | 0.320 | 4.20 | 0.097 | 0.085 | 0.305 |
| Requirements Engineering | 2.41 | 0.023 | 0.024 | 0.505 | 2.34 | 0.135 | 0.080 | 0.415 |
| SDSS Galaxy | 3.41 | 0.089 | 0.113 | 0.135 | 3.21 | 0.150 | 0.135 | 0.135 |
| BLADE Mortgage | 3.71 | 0.170 | 0.211 | 0.065 | 3.93 | 0.235 | 0.255 | 0.065 |
| Setting | Value |
Large Language Model-Guided Scientific Discovery
Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.
See all 302 papers