SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces

Harshit Joshi; Priyank Shethia; Monica S. Lam

cs.AIOct 13, 2026

SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces

Authors: Harshit Joshi, Priyank Shethia, Monica S. Lam

Organizations: Stanford University, USA

Abstract

Test-time scaling has enabled LLMs to solve increasingly complex reasoning, discovery, and optimization tasks. However, existing approaches often gather observations while seeking evidence for current hypotheses, then generate new hypotheses conditioned on those hypotheses and their associated evidence. This couples observation discovery to existing hypotheses, favoring local refinements over exploration of unknown hypotheses regions. Inspired by the hypothetico-deductive method (Hempel, 1945) and dual-space models of scientific discovery (Klahr & Dunbar, 1988), we introduce SHELDON, a scientific reasoning harness that separately represents and jointly explores hypothesis and observation spaces. By actively seeking new observations along with new hypotheses, SHELDON can discover observations that its current hypotheses do not explain. These unexplained observations expose gaps in its understanding, motivate new investigations, and guide the generation of hypotheses beyond refinements of existing ones. Across diagnosis, abstract reasoning, and optimization benchmarks, SHELDON outperforms evaluated baselines by up to 26.7 points. Additional analysis shows that SHELDON generates up to 4.5x more novel solutions.

Figures & tables

Figure 1: SHELDON jointly explores hypothesis and observation spaces, and outperforms strong baselines across tasks with and without explicit scoring functions. (a) Conceptual illustration of dual-space reasoning; edges represent refinement of hypotheses. (b) Improvements on a system optimization task. (c) Performance across diverse investigation tasks.

Figure 2: Conceptual illustration of evolution of the hypothesis space across successive rounds driven by discovery of newer observations. (a) Early rounds: Limited observational coverage constrains exploration to a small subset of the hypothesis space, producing potentially suboptimal solutions. (b) Middle rounds: New observations helps uncovers newer hypotheses regions guiding the search toward higher-performing solutions. (c) Later rounds: Broader observational coverage supports the exploration of multiple regions, allowing distinct solution strategies to emerge and achieve comparable performance. Blue bars represent observed regions, while hatched bars indicate unexplored regions. Green and orange contours denote higher- and lower-performing regions of the hypothesis space, respectively. Solid arrows indicate iterative solution improvement, and dashed arrows represent observation space-driven transitions.

Figure 4: This ORCA-Bench incident contains concurrent catalog and payment failures. In round 1, investigating hypothesis $h_{2}$ leads to the observation ( $o_{7}$ ) that Charge errors begin at 20:35, whereas $h_{6}$ leads to the observation ( $o_{14}$ ) Invalid token . In round 2, Explore combines these observations to form a new hypothesis ( $h_{10}$ ) about paymentFailure , showing how the observation space leads to new hypotheses. On the other hand, hypothesis ( $h_{15}$ ) paymentUnreachable and ( $h_{13}$ ) productCatalogFailure are obtained by refining hypotheses from round 1.

Benchmark	Case material	Tools	Hypothesis space	Observation space	Metric
ORCA Bench: Find the cause of a service fault	Telemetry, service code, incident report	Container shell; Grafana API	Fault explanations: service, fault type, and failure mechanism	Metrics, logs, traces, code, and incident timeline	GT
Diagnosis-Arena: Diagnose a patient	Patient record: history, exam, tests, and notes	EviDx record readers	Candidate diagnoses and their predicted symptoms and test results	History, labs, imaging, pathology, and treatment response	GT
ARC-AGI-2: Infer a grid transformation rule	Example grid pairs, test grid, seed program	Python sandbox	Candidate rules: symmetry, movement, replacement, encoding, and tiling	Grid changes, object shapes, patterns across examples, and synthetic tests	GT
QNN topology: Optimise a two-qubit circuit	Circuit seed and training harness	Python sandbox; evaluator	Circuit design explanations: gate count, connections, rotation axes, and parameters	Circuit scores, gate-removal tests, design variants, and repeated runs	SF
ADRS: Optimise a systems program	Program seed, evaluator, workload	Python sandbox; evaluator	Algorithm explanations: greedy methods, grouping, allocation, search, and scheduling	Program scores, validity checks, test workloads, edge cases, and timing	SF
DOCK-STRING: Design a drug-like molecule	Target and scored molecule library	Docking, property, and similarity tools; Python	Molecule design explanations: scaffold, chemical groups, size, and rigidity	Docking scores, molecule properties, similar molecules, and chemical variants	SF

Table 1: Details of benchmark tasks, case material, tools, how hypothesis and observation space could look like, and if the benchmark contains a Ground Truth (GT) or a Scoring Function (SF)

Panel of Table 2 — Table 2: Performance on diagnostic and reasoning benchmarks across different agent and optimization paradigms. Cell shading indicates the top three performing methods within each benchmark, with darker shading denoting higher rank. EvoX requires a scoring function, and GoS requires benchmark-specific sub-agent prompts.

Panel of Table 3 — Table 3: Molecular optimization. Yellow cells beat the best molecule in the DOCKSTRING library.

Panel of Table 4 — Table 4: Performance across ADRS system-level optimization task benchmarks. “Mean” reports the average score over 3 runs. “Cost to best” reports the cost in USD required to obtain the best-performing solution; lower is better. The darkest cell marks the best result. “Human best” is the best human-designed method as reported by Liu et al. (2026a) , measured on their hardware.

Panel of Table 5 — Table 5: SHELDON ablation. Bold denotes the change relative to the full SHELDON configuration.

Panel of Figure 5 — Figure 5: Best score against model spend for each method on the five ADRS tasks.

Panel of Figure 8 — Figure 8: Search trajectories of SHELDON on transaction scheduling and LLM_SQL: best score versus cumulative spend. Open circles mark annotated improvements, filled circles the final scores. The largest LLM_SQL gain follows the gated combination of canonical row ordering with conditional grouping ( $h_{39}\rightarrow h_{44}$ ).

Figure 9: SHELDON on an ARC-AGI-2 task. $h_{1}$ and $h_{2}$ are refuted, $h_{19}$ is generated from an observation, and $h_{31}$ combines earlier hypotheses and their observations into the submitted answer.

Panel of Figure 10 — Figure 10: Hull share of $k$ randomly drawn programs (300 draws per $k$ ) in the shared PCA frame of each task; thin lines are runs, bold lines medians. Panel titles give SHELDON’s ratio to the strongest baseline at $k{=}20$ . A higher hull share means more novel programs generated.

Panel of Figure 12 — Figure 12: Origins and refinement of hypotheses. (a) Number of identified regions versus the number containing at least one hypothesis. (b) Number of hypotheses from distinct regions that improve upon the previous best. (c) Distribution of hypothesis origins; $\star$ marks tasks whose final best hypothesis originates from exploration. (d) Distribution of hypotheses by number of refinements.

Panel of Figure 13 — Figure 13: Exploration on DiagnosisArena and ORCA-Bench. (a) Distinct ideas investigated per task by SHELDON and GoS. (b) Share of each round’s new hypotheses placed in a region opened after round 1 (solid) and of its observations filed in a region receiving its first observation that round (dashed). (c) Distinct hypothesis regions judged by each comparative-analysis observation.

Panel of Figure 14 — Figure 14: Observations as shared evidence. (a) Hypotheses judged per observation, by the stage that filed it. (b) Share of observations that refute at least one hypothesis. (c) ARC-AGI-2 hypotheses born after round 1, by whether their rationale cites an observation and where that observation came from.

Panel of Figure 15 — Figure 15: Observations and exploration. (a) Share of each round’s observations filed in a region opened that round. (b) Distinct hypothesis regions judged by each comparative-analysis observation on ARC-AGI-2. (c) Observation regions opened empty, and the subset filled in a later round.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 16: The two-qubit QNN circuit topology discovered by SHELDON on the QNN task (held-out accuracy 1.00): ten gates at depth six, with seven trainable rotations. Each qubit first receives an $R_{x}$ rotation, a CNOT entangles $q_{0}$ and $q_{1}$ , a second layer of rotations follows ( $R_{x}$ on both qubits, then $R_{y}$ on $q_{0}$ and $R_{x}$ on $q_{1}$ ), a reverse CNOT re-entangles them, and the circuit closes with $R_{x}$ on $q_{0}$ and a Hadamard on $q_{1}$ . Inputs are angle-encoded before the ansatz and the evaluator fits the rotation angles.

Figure 17: Molecular optimization on the six DOCKSTRING targets: $-\mathrm{Vina}$ (binding, higher is better) against QED (drug-likeness) for every molecule docked by SHELDON (blue), with its Pareto front (dark blue), the best molecule by $s$ (star), and the Pareto front of the 260k-molecule DOCKSTRING library (grey). Hollow markers are molecules that were already in the library and are excluded from the fronts. Dotted and dashed lines are the level sets $s=\mathrm{const}$ through SHELDON’s and EvoX’s best molecules, respectively; everything above a line beats that score.

Panel of Figure 18 — Figure 18: Ratio of SHELDON’s median to the strongest baseline’s median on each breadth metric, log scale; the baseline is named beside each point. Blue: SHELDON is wider; grey: a baseline is.

	DiagnosisArena		ORCA-Bench
Method	Cost	Time	Cost	Time
ReAct	0.02	0.3	0.54	4.2
Codex	0.06	0.8	0.73	4.5
RLM	0.09	1.7	1.56	2.7
GoS	2.79	29.1	7.54	55.4
SHELDON	2.91	14.0	6.90	22.0

Appendix

Table 6: Cost ($) and wall-clock time (minutes) per task, on the 50 DiagnosisArena cases and the 30 ORCA-Bench incidents.

Parameter	Value	Meaning
mode	session	a round’s deciding steps run as turns of one conversation ( session )
plan	on	separate the problem from the answer specification before round 1
rounds	8	round cap
n_initial	5	hypotheses proposed by Explore in round 1
n_explore	3	hypotheses proposed by Explore in each later round
select_k	5	hypotheses selected for CandidateAnalysis per round (round 2 on)

Appendix

Table 7: Method hyperparameters.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 144 papers

Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
Jul 17, 2026Tianyun Zhong, Wangyi Jiang, Wei Wang +15LLM Evaluation Scientific Hypothesis Generation
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · 3Alibaba Group
Open-ended Scientific Discovery with Possibilistic Reasoning
Oct 8, 2026Anita Yang, Siu Lun Chau, Tomoya Wakayama +2Scientific Discovery Autonomous Scientific Discovery
Lattice Lab, Toyota Motor Corporation, Japan · Department of Computer Science, University of Tokyo, Japan · EPIC Lab, College of Computing & Data Science, Nanyang Technological University, Singapore +2
Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks
Jun 12, 2026Li Xia, Baoxun WangScientific Hypothesis Generation Large Language Model-Guided Scientific Discovery
School of Economics and Management, Tsinghua University · Platform & Content Group, Tencent

Jul 17, 2026J/K move · Enter open · S save

Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

Tianyun Zhong, Wangyi Jiang, Wei Wang, Xuanang Chen+14

University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · 3Alibaba Group

LLM Evaluation Scientific Hypothesis Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured. We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discriminative, and testable hypothesis spaces from inconclusive evidence, including anomalous observations and fragmented records, to guide subsequent investigation. To evaluate this capability, we introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets. To construct HypoData at scale, we propose Retrospective Context Regression, a Forge--Audit pipeline that reconstructs pre-conclusion contexts from completed expert documents by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate. Because PHD admits multiple valid outputs, HypoEval combines bidirectional pairwise judgments with Bradley--Terry--Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. Experiments on 15 frontier LLMs reveal clear capability stratification and model-dependent effects of structured analytical skills, with gains for several lower-performing models on HypoArena but regressions for other systems, including a top-performing model. Compared with absolute rubric scoring, arena evaluation resolves finer-grained differences among models, with aggregated rankings showing strong agreement with human experts and an independent judge. Together, these results support treating PHD as a distinct target for evaluating how LLMs formulate investigative directions when final conclusions are withheld. Our code and data are publicly available at github.com/SKYLENAGE-AI/HypoArena and github.com/SKYLENAGE-AI/HypoArena.

Abstract

Scientific Reasoning in Language Models

Explore similar work

Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

Open-ended Scientific Discovery with Possibilistic Reasoning

Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks