Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations

Ziang Liu; James J. Kim; Yijia Dai; Jennifer J. Sun

cs.AINov 10, 2026

Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations

Authors: Ziang Liu, James J. Kim, Yijia Dai, Jennifer J. Sun

Organizations: Department of Computer Science Cornell University

Abstract

LLM agents are increasingly used for open-ended, data-driven scientific discovery, yet most of them evaluate hypotheses pointwise, testing an empirical claim $h$ only against its complement $\neg h$ . When evidence rejects $h$ , the agent learns that one explanation is unlikely but not which alternative the evidence supports. To recover that alternative, we introduce Contrastive Discovery, which replaces the single hypothesis with a contrast set, a scientific question paired with mutually exclusive and collectively exhaustive competing explanations. Each experiment is scored by how its evidence redistributes belief across the set, and the question is resolved when belief concentrates on one explanation. Because contrast sets change only the unit an agent proposes and scores, an existing agent can use them to re-evaluate evidence it has already collected or to guide its search. Across six domains, $89\%$ of the pointwise candidates with the largest belief shifts are rejections. On the same evidence, contrast sets resolve $73\%$ of these candidates to a specific alternative. When contrast sets guide search, the agent resolves $67\%$ more questions than pointwise surprisal under the same experiment budget. In a blinded study, $38$ human experts chose the same explanation on $78\%$ of resolved questions and on $82\%$ of questions where the pointwise agent rejected its hypothesis and stopped. Together, these results support explicit competing explanations as a way for discovery agents to resolve more scientific questions from the same evidence and validation budget.

Figures & tables

Figure 1: A contrast set turns a rejection into a resolution. A hypothesis generator proposes a contrast set of three competing explanations ( $H_{1}$ – $H_{3}$ ) for how X relates to Y, and coder, analyst, reviewer, and reviser agents run an experiment that produces evidence. Bars show elicited belief in each explanation before and after the same evidence. Contrastive Discovery updates belief over all three explanations and moves the lead from $H_{1}$ (0.64) to $H_{2}$ (0.81), so the question resolves to $H_{2}$ and $H_{1}$ is ruled out. Pointwise discovery updates belief only in $H_{1}$ (0.64 to 0.07), so it records a rejection but leaves the question unresolved. See § 3 and § 4 for details.

Figure 2: Pointwise discovery masks resolvable alternatives behind rejection. Left: pointwise proposals show prior collapse, with most mass in the “maybe true” bin under each of three belief elicitors (columns), while explanations in contrast sets span a broader range of prior beliefs. Right: among the top-50 hypotheses ranked by pointwise surprisal, 89% are rejected under pointwise evaluation, but re-evaluating them with a post-hoc Contrastive companion on the same evidence resolves 73% of all 50 to a specific alternative. On the Contrastive side, confirmed marks resolutions to the explanation pointwise confirmed, and ambiguous marks unresolved sets. Per-dataset breakdowns over each run’s 200 experimental validations are in Figure 10 .

Figure 3: Contrastive Discovery resolves more scientific questions across top-

K

budgets. Shaded gap marks corrective resolutions, which pointwise baselines record only as rejections. The dashed curve counts confirmative resolutions only, and legend values are resolved counts at

K{=}200

. Curves are means over datasets and bands are

\pm 1

standard error across the six datasets. The number of ruled-out alternatives is shown in Appendix Q.2 .

Figure 4: Contrastive ranking, evaluation, and discovery-time construction all contribute to resolved-question yield. Across resolution rate (left) , realized information gain (IG; the contrastive belief shift

\kappa_{t}

, not bounded by

1

) (middle) , and margin gain (right) , Full Contrastive performs best at nearly every budget. Contrastive (“Cont.” in the legend) ranking improves resolution even under pointwise evaluation, Contrastive evaluation produces a larger step change by allowing evidence to resolve over alternatives, and combining both gives stronger updates than either alone. Since pointwise evaluation has no explicit runner-up, margin gain is defined only for contrastively evaluated variants. Curves are means and bands are

\pm 1

standard error across the six datasets.

Questions	$n$	Experts agree
Contrastive resolved	51	40 (78.4%)
Contrastive unresolved	9	3 (33.3%)
Pointwise confirmed	32	23 (71.9%)
Pointwise rejected	22	18 (81.8%)
Pointwise ambiguous	6	2 (33.3%)
All items	60	43 (71.7%)

Table 1: Experts chose the explanation Contrastive Discovery identified on most resolved questions. The same 60 items are grouped by the Contrastive outcome and the pointwise verdict.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	Pointwise	Contrastive $r$ [ $1{-}\mathrm{TVD}$ ]
Conversation	Prior	0.730	0.764 [0.822]
	Posterior	0.873	0.917 [0.893]
Mortgage	Prior	0.602	0.730 [0.786]
	Posterior	0.854	0.934 [0.894]
Materials Project	Prior	0.672	0.506 [0.780]
	Posterior	0.906	0.933 [0.897]

Appendix

Table 2: Per-dataset cross-elicitor agreement, mean across the three elicitor pairs. Pointwise columns (

n{=}90

hypotheses per dataset,

n{=}540

pooled): mean pairwise Spearman

\rho

on per-hypothesis mean-belief scores. Contrastive columns (

n{\approx}30

sets per dataset,

n{=}177

pooled after dropping

M{=}2

sets from the Pearson statistic): mean pairwise per-set Pearson

r

on the

M

-cell partition allocation, with mean

1{-}\mathrm{TVD}

total-variation similarity bracketed as a robustness check (interpretable as fraction of probability mass agreeing across raters; computed on all aligned sets including

M{=}2

M{=}2

sets (

\sim

1.7% of Contrastive sets) are degenerate for per-set Pearson and dropped from that statistic only. Bonferroni and Benjamini–Hochberg corrections applied over 24 tests (two methods

\times

prior and posterior

\times

6 datasets, treating each per-dataset cell of this table as the unit of significance claim); all cells significant at

p<0.05

under both corrections.

Figure 5: Within-contrast-set dispersion of the

n=5

belief samples. Prior elicitations are more dispersed, while posterior elicitations are consistently sharper after conditioning on evidence.

Diagnostic	Value
Contrast sets with complete $n=5$ prior/posterior elicitations	$3{,}213$
Mean pairwise TVD, prior samples	$0.125$
Mean pairwise TVD, posterior samples	$0.065$
Posterior contrast sets with pairwise TVD below $0.10$	$85.5\%$
Posterior per-hypothesis SE of sample mean	$0.0165$
Posterior maximum-probability sample SD	$0.043$

Appendix

Table 3: Stability diagnostics for the

n=5

contrastive belief elicitor. Prior allocations are more dispersed than posterior allocations, as expected before evidence is observed. The downstream quantities used by the pipeline remain stable under bootstrap and drop-one sensitivity.

Quantity	Count / estimate	Rate
Structurally MECE sampled contrast sets	$372/384$	$96.9\%$
Design-weighted population estimate	–	$98.4\%$
95% stratified bootstrap CI	–	$[97.3\%,99.3\%]$
Overlap or redundancy failures	$9/384$	$2.3\%$
Inherently non-MECE mechanism failures	$3/384$	$0.8\%$

Appendix

Table 4: Structural MECE audit summary under the primary

4/5

rule.

Judge	Provider	Raw pass	Design-weighted pass	Sets flagged
gpt-4o (same provider as proposer)	OpenAI	96.9%	98.4%	12
Claude Sonnet	Anthropic	87.0%	91.4%	50

Appendix

Table 5: Structural MECE pass rate on the same 384-set sample under the same rule, by judge.

Pairwise consistency	canonical	randomized	ratio
Prior pairwise TVD	$0.077$	$0.042$	$1.83\times$
Posterior pairwise TVD	$0.031$	$0.025$	$1.24\times$
Aggregate belief statistics	canonical	randomized	$\Delta$ [95% CI]
$\mathbb{E}[\mathrm{KL}]$	$0.987$	$0.975$	$-0.012$ $[-0.332,+0.284]$
$\mathbb{E}[\mathrm{KL}/\log_{2}M]$	$0.554$	$0.568$	$+0.014$ $[-0.167,+0.181]$
$\mathbb{E}[\max p^{\text{post}}]$	$0.874$	$0.873$	$-0.001$ $[-0.046,+0.045]$

Appendix

Table 6: Effect of hypothesis-order randomization on belief elicitation. The top block measures pairwise consistency across five displayed hypothesis orderings; lower TVD means greater order robustness. The bottom block compares aggregate belief statistics in two parallel Contrastive runs that differ only in whether hypothesis order is randomized. KL is measured in bits, and

\mathrm{KL}/\log_{2}M

normalizes by the number of explanations

M

in the contrast set.

\Delta

is the randomized minus the canonical mean, with a bootstrap 95% CI.

Comparison	$n$	$F_{1}$	95% CI	Precision	Recall
Support gate	$1{,}543$	$0.937$	$[0.928,0.947]$	$0.915$	$0.961$
Rejection gate	$1{,}672$	$0.896$	$[0.884,0.907]$	$0.981$	$0.824$
Joint two-sided gate	$3{,}215$	$0.934$	$[0.927,0.940]$	$0.973$	$0.898$

Appendix

Table 7: Paired calibration between pointwise and contrastive thresholds. Precision is the fraction of pointwise positives that also satisfy the corresponding contrastive gate; recall is the fraction of contrastive positives recovered by the pointwise gate. Confidence intervals are bootstrap 95% intervals.

		evidence that the gates are matched			resolved units at top-200
$\tau$	$\gamma$	contr. fires	point. fires	F1	contrastive	pointwise	gain
0.600	0.535	95.3%	95.3%	0.966	185.5	104.8	+0.403
0.667	0.585	93.3%	93.5%	0.963	178.7	100.8	+0.390
0.700	0.605	92.1%	92.0%	0.959	174.8	99.3	+0.378
0.750	0.635	90.5%	90.6%	0.952	169.7	96.7	+0.365
0.800	0.695	86.8%	86.3%	0.927	161.9	88.3	+0.368

Appendix

Table 8: Threshold robustness of the reported gain, recomputed through the exact top-

K{=}200

estimator of the headline result. Each contrastive threshold

\tau

is paired with the pointwise threshold

\gamma

of matched firing rate; the middle columns show how closely each pair is matched on the

1{,}543

paired elicitations where both gates score the same evidence. Gain is the difference as a fraction of the budget. † At

\tau{=}0.95

the positive-set overlap falls to

0.567

, so that row is exploratory.

Dataset	Domain	Rows	Cols.	Row unit
Materials Project	Materials science	68,805	28	Inorganic crystal
SDSS Galaxy (DR19)	Astronomy	50,000	28	Galaxy
BLADE datasets
Mortgage	Finance / policy	2,380	13	Mortgage application
Conversation	Online discourse	7,975	70	Debate contribution
DiscoveryBench datasets

Appendix

Table 9: Summary of experimental datasets. The benchmark portfolio spans heterogeneous scientific and social domains and covers a broad range of sample sizes, enabling evaluation across different levels of statistical power, feature spaces, and domain structure.

Figure 6: Parent–descendant heritability across MCTS lags. Spearman rank correlation between a parent’s score and its descendant’s score at increasing tree distance, pooled across 6 datasets and the 200 experimental validations of each tree. Solid lines are point estimates; shaded regions are 95% percentile cluster-bootstrap CIs over runs. Left: self_value (the UCT-propagated reward, the mechanism-defining metric). Pointwise (gray) stays near zero at lags 1 and 2 and rises at deeper lags. Contrastive (red, ours) shows a positive lag-1 trend (

\rho=+0.077

) that is not significant after BH-FDR correction (

q=0.28

). Right: surprisal (the logged belief change without clipping or

M

-normalization; a robustness check). Contrastive lag-1 is FDR-significant (

\rho=+0.131

q=0.046

\star

marks

q\leq 0.10

) and decays monotonically toward zero by lag 4, as local inheritance would predict. Pointwise shows no lag-1 heritability on either metric; its slow rise at deeper lags is consistent with UCT’s preferential extension of high-value lineages enriching deep-lag pair pools (selection bias, not mechanism-relevant signal). The weaker self_value result is consistent with construction-induced compression (13.7% of nodes hit the clipping ceiling) of the same underlying signal that the surprisal panel resolves more cleanly.

			of which resolved to
Supported explanation removed	Declined or flagged	Unsupported	no effect	competing direction
Contrast set as generated	46.6%	53.4%	50.9%	2.5%
With “none of the above”	99.4%	0.6%	0%	0.6%

Appendix

Table 10: Outcomes on the 163-question primary panel when the supported explanation is removed. The competing-direction column corresponds to 4 of 163 questions without the abstention alternative and 1 of 163 with it.

		Pointwise			Contrastive
Dataset	$C$	$\alpha$	$k$	$t_{95}$	$\alpha$	$k$	$t_{95}$
Materials Project	28	0.955	0.056	157	0.919	0.062	–
BLADE Conversation	70	0.905	0.033	–	0.863	0.038	–
Requirements Engineering	202	0.513	0.015	–	0.411	0.015	–
SDSS Galaxy	28	0.955	0.103	87	0.955	0.065	176
BLADE Mortgage	13	1.000	0.290	11	0.999	0.274	11

Appendix

Table 11: Cumulative column-coverage saturation across the six datasets. For each (dataset, condition), we fit

\mathrm{Coverage}(t)=\alpha\,(1-e^{-kt})

to the permuted-mean cumulative-unique-columns trajectory and report the asymptotic coverage fraction

\alpha

, the rate constant

k

(per hypothesis), and the smallest

t

at which the observed curve reaches 95% column coverage. “–” marks a curve that never reaches 95% within the run (all such cells have

\alpha<0.95

C

is the dataset’s column count.

Figure 7: Cumulative column coverage across six datasets. Both methods are shown at the same budget of

N=200

hypotheses. Coverage at hypothesis

t

is averaged over

20

random orderings of each run’s hypotheses, so the curve does not depend on the order in which the search produced them. Solid lines show the observed coverage; dashed lines extend the fitted saturating-exponential curve until it reaches

99\%

of its asymptote; dotted horizontal lines mark the fitted asymptotes

\alpha

in matching colors; the gray dashed line marks full coverage. Values of

\alpha

k

, and

t_{95}

are in Table 11 . Pointwise reaches a higher asymptote than Contrastive on five of six datasets and the same asymptote on SDSS Galaxy; the largest gap is on Requirements Engineering (

\alpha=0.51

for pointwise vs.

0.41

for Contrastive ). In the legend, C1 is pointwise and C2 is Contrastive .

Figure 8: Cumulative column coverage under the natural MCTS creation order. Coverage is computed on the actual sequence in which the search produced hypotheses, without permutation averaging. Solid lines, dashed extrapolations, dotted per-condition asymptotes, and the gray full-coverage reference follow the same conventions as Figure 7 . Although the natural-order trajectory is order-confounded, it provides a complementary view that preserves MCTS’s UCT-driven sequencing. Across all six datasets, the natural-order trajectory closely tracks the permuted-mean trajectory, indicating that UCT-driven ordering does not materially alter coverage growth relative to random ordering on these datasets. Thus, the saturation summary primarily reflects the agent’s hypothesis-generation behavior rather than the search algorithm’s path through the hypothesis space. In the legend, C1 is pointwise and C2 is Contrastive .

Figure 9: Sibling versus non-sibling column-set overlap across datasets and conditions. Each marker represents one dataset–condition cell. The

y

-axis shows the pair-pooled mean Jaccard similarity,

\bar{J}_{\mathrm{sib}}

, between touched-column sets of sibling pairs, defined as nodes sharing the same MCTS parent. The

x

-axis shows the corresponding mean Jaccard similarity for non-sibling pairs,

\bar{J}_{\mathrm{non\text{-}sib}}

. The diagonal

y=x

marks the null pattern of no sibling anchoring: siblings are no more column-similar than randomly selected node pairs. Under pointwise , points lie close to the diagonal: children of the same MCTS parent are no more column-similar than random pairs, so the pointwise proposer does not push siblings toward shared variables. Under Contrastive , sibling similarity exceeds non-sibling similarity on three of six datasets (Conversation, Requirements Engineering, SDSS), and the sibling-to-non-sibling ratio is higher than under pointwise on five of six, so the Contrastive proposer tends to concentrate consecutive expansions on the same variables. The largest asymmetry is on Requirements Engineering:

\bar{J}_{\mathrm{sib}}=0.135

versus

\bar{J}_{\mathrm{non\text{-}sib}}=0.080

under Contrastive , compared with

0.023

versus

0.024

under pointwise . This is also the dataset with the largest coverage gap between the two methods (Table 11 ). Per-cell values are in Table 12 . In the legend, C1 is pointwise and C2 is Contrastive .

	Pointwise				Contrastive
Dataset	$\overline{\|C_{i}\|}$	$\bar{J}_{\mathrm{sib}}$	$\bar{J}_{\mathrm{non\text{-}sib}}$	$\Delta_{\mathrm{new}}$	$\overline{\|C_{i}\|}$	$\bar{J}_{\mathrm{sib}}$	$\bar{J}_{\mathrm{non\text{-}sib}}$	$\Delta_{\mathrm{new}}$
Materials Project	2.43	0.031	0.094	0.135	2.45	0.064	0.092	0.130
BLADE Conversation	4.06	0.048	0.074	0.320	4.20	0.097	0.085	0.305
Requirements Engineering	2.41	0.023	0.024	0.505	2.34	0.135	0.080	0.415
SDSS Galaxy	3.41	0.089	0.113	0.135	3.21	0.150	0.135	0.135
BLADE Mortgage	3.71	0.170	0.211	0.065	3.93	0.235	0.255	0.065

Appendix

Table 12: Per-node breadth and inter-node similarity across the six datasets.

\overline{|C_{i}|}

is the mean per-node touched-set size (over non-empty sets);

\bar{J}_{\mathrm{sib}}

and

\bar{J}_{\mathrm{non\text{-}sib}}

are pair-pooled mean Jaccard similarities over sibling and non-sibling pairs in the first

N{=}200

-node window;

\Delta_{\mathrm{new}}

is the average number of never-before-seen columns added per node in creation order.

Figure 10: Section 5.1 , per-dataset breakdowns using all 200 pointwise experimental validations per run. Each pointwise node is confirmed, ambiguous, or rejected under its own gates; rejected nodes are further split by what the post-hoc contrastive companion does with the same evidence (resolves to an alternative, rebounds to the focal hypothesis, or leaves the question unresolved).

Figure 11: Section 5.1 , outcomes within the pointwise -rejected slice. Fraction of pointwise -rejected nodes that the contrastive companion resolves to an alternative (73% pooled over each run’s 200 experimental validations), rebounds to the focal hypothesis (under 1%), or leaves unresolved, using the same evidence.

Figure 12: Section 5.1 , outcomes within the pointwise -ambiguous slice. Where pointwise evaluation reached no verdict, the companion resolves 21% of nodes to an alternative and 4% to the focal hypothesis; most remain unresolved, so the companion does not manufacture resolutions where the evidence is weak.

Figure 13: Contrastive Discovery rules out alternatives on more questions as the top-

K

budget grows. Pointwise curves count rejected hypotheses, the Contrastive curve counts questions with at least one alternative ruled out, and legend values are counts at

K{=}200

Figure 14: Ablations on decisive-unit rate and winner-switch rate. Winner-switch rate is the fraction of top-

K

nodes whose prior-leading explanation differs from the posterior-leading explanation.

Panel of Figure 15 — Figure 15: Ablations on cumulative pointwise surprisal: (a) each contrast set projected to one member under different selection rules; (b) the pointwise generator under different prompts.

Setting

Value

Appendix

Table 13: Search and agent settings used in the experiments. These settings are held fixed across compared methods.

Large Language Model-Guided Scientific Discovery

Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.

See all 302 papers

Evidence-Informed LLM Beliefs for Continual Scientific Discovery
Jun 28, 2026Dhruv Agarwal, Reece Adamson, Andrew McCallum +3Scientific Hypothesis Generation Belief Updating in LLMs
University of Massachusetts Amherst · Allen Institute for AI
SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
Oct 27, 2026Yusheng Zhao, Bowei He, Xue Liu +1AI Agents for Scientific Discovery Scientific Agent Benchmarks
MBZUAI · McGill University · Shanghai AI Lab
DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher

Jun 28, 2026J/K move · Enter open · S save

Evidence-Informed LLM Beliefs for Continual Scientific Discovery

Dhruv Agarwal, Reece Adamson, Andrew McCallum, Peter Clark+2

University of Massachusetts Amherst · Allen Institute for AI

Scientific Hypothesis Generation Belief Updating in LLMs Scientific Reasoning in Language Models Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Open-ended scientific discovery with large language models (LLMs) increasingly operates as a long-horizon loop of hypothesis search and verification, where a reward signal guides which hypotheses to test next. A notable recent example is AutoDiscovery, which uses "Bayesian surprise" - the belief shift an LLM undergoes after observing evidence for a hypothesis - as both a discovery metric and a reward for search. We first observe that AutoDiscovery treats surprisal as a static quantity, while surprisal in human reasoning is non-stationary - it is defined relative to beliefs that evolve with experience, a prerequisite for continual scientific discovery. We address this mismatch with evidence-informed LLM beliefs: priors updated with evidence from previous hypotheses to compute non-stationary surprisal for new hypotheses. We compare in-context belief-updating mechanisms and find that embedding-based retrieval-augmented generation over prior discoveries best anticipates eventual posteriors, identifying 37.5% of static surprisals as spurious. We then modify search to avoid these spurious rewards and prioritize hypotheses that remain surprising under non-stationary beliefs. Concretely, we introduce two complementary changes to the original search procedure: belief-update filtering and diversity maximization. Across five discovery domains, our method increases accumulated non-stationary surprisal by 30.62% on average compared to the original search procedure, demonstrating that continual scientific discovery with LLMs requires not only better belief measurement but also search procedures that avoid redundancy and encourage diversity.

Abstract

Large Language Model-Guided Scientific Discovery

Explore similar work

Evidence-Informed LLM Beliefs for Continual Scientific Discovery

SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?