Selecting LLM-Generated Hypotheses Raises Truth Beyond What an LLM Panel Expects: A Pre-registered Study on NAS-Bench-201 and HPOBench
Organizations: Independent Researcher, USA · Santa Clara University, USA
Abstract
LLM research-ideation systems aim for hypotheses that are both true and non-obvious, yet neither property can normally be checked. We build a testbed in which both can. Generators propose conditional interaction claims about two fully tabulated benchmarks, NAS-Bench-201 and HPOBench; every claim is verified exactly against the tables, and its non-obviousness is scored against a calibrated blind prior elicited from other LLMs. Excess truth measures how much more often a run's claims hold than this panel expects. Across a series of pre-registered stages with Claude and GPT generators, our main positive result is a simple two-stage step found by a harness optimizer: draft a large pool of claims, then select a subset in a separate call. For Sonnet on NAS-Bench-201, this step raised excess truth in every pre-registered test, including against a same-pool ablation that keeps only likely-true claims, without lowering precision. The gain comes both from credit for true claims and from avoiding near-miss false ones. In post hoc checks it held under every calibration, panel and truth rule we tried, and it exceeded random and type-filter selection from the same pools. Because kept claims recur across runs, the evidence weakens when runs and claims are resampled jointly, and it is not significant against Sonnet's own prior. Two secondary findings are also pre-registered. GPT makes real use of the results of cheap probes, beating shuffled and withheld controls, though not more efficiently than asking for more claims. Optimizing the harness for an LLM judge raised judge scores while sharply cutting precision for every generator.
Figures & tables
| Claim | Evidence | Status |
|---|---|---|
| Stage-3 selection step raises at equal or higher precision (Sonnet, NB201) | Stages 13–15 | Pre-registered; supported in all three tests |
| robust to calibration, panel and truth definition | App. D | Post hoc; holds under all 14 calibrations, 8 panels and 6 truth definitions ( 1–3 pp) |
| robust to resampling the recurring propositions | App. F | Post hoc; yes under the Claude and Claude+GPT panels ( ); not under Sonnet’s own prior ( – ) |
| beats random and claim-type selection from its own pools | App. C | Post hoc; yes ( , ); gain concentrated in about 10 propositions |
| gain is credit for true claims, not only avoided errors | App. F | Post hoc; credit of ; with propositions resampled – |
| a selector given the scoring panel’s priors does better | App. C | Post hoc; yes ( against the harness’s ); the harness has no access to them |
| Stage | What it tested |
|---|---|
| 1 | Screen of structural harness knobs (NB201, Sonnet, 72 runs) |
| 2, 3 | Harness optimization with three fitness functions (judge score, number of true claims, sv ); 3 reran the sv optimizer after a bug fix |
| 4, 4b | Measurement pilots (8 and 16 runs per arm) |
| 5 | 24-run tests on NB201: judge-optimized harness and measurement (Sonnet, Opus) |
| 6–8 | Measurement controls for Opus on NB201: simulated noise and a cheap proxy (6), shuffled or withheld values (7), designing without results (8) |
| 9 | Replication on HPOBench (Sonnet, Opus) |
| Comparison | Stage | Prec. | sv 3 | [90% CI] | Calibrations | Panels | Truth rules | |
|---|---|---|---|---|---|---|---|---|
| Stage-3 harness | – | .83 | 5.17 | |||||
| vs direct generation | 13 | .79 | 3.54 | [ , ] | 14/14 | 8/8 | 6/6 | |
| vs new direct runs | 15 | .74 | – | [ , ] | 14/14 | – | 6/6 | |
| vs likely-true ablation | 14 | .82 | 3.58 | [ , ] | 14/14 | – | 6/6 |
| Bench. | Comparison (stage) | Tokens | [90% CI] | Prec. ratio | Verdict | Calibrations | ||
|---|---|---|---|---|---|---|---|---|
| NB201 | Pooled direct (13) | 21.2k | 10.0 | [ , ] | .71 | Supported | 12/14 | |
| Pooled + stage-3 rule (14) | – | 10.0 | [ , ] | .68 | Supported | 12/14 | ||
| 40-claim call (15) | 11k | 9.8 | [ , ] | .61 | Supported | 12/14 | ||
| Shuffled values (15) | – | 10.0 | [ , ] | 2.64 | Supported | 14/14 | ||
| Withheld values (15) | – | 10.0 | [ , ] | .72 | Supported | 11/14 | ||
| HPO | Pooled direct (13) | 20.4k | 7.5 | [ , ] | .98 | Supported | 12/14 |
| Bench. | Gen. | Harness | Prec. | Judge | sv 3 | |
|---|---|---|---|---|---|---|
| NB201 | Sonnet | Direct | .79 | – | 3.54 | |
| Judge-opt. | .43 | – | 4.25 | |||
| Opus | Direct | .87 | 8.25 | 6.96 | ||
| Judge-opt. | .28 | 8.88 | 2.75 | |||
| GPT | Direct | .69 | 8.04 | 4.08 | ||
| Judge-opt. | .26 | 8.63 | 2.58 |
| Self-cond. | Removes edge | Other | |
|---|---|---|---|
| Base rate of truth (all claims) | .275 | .173 | .067 |
| Share, Sonnet direct / judge-opt. | .68 / .03 | .28 / .23 | .03 / .74 |
| Share, Opus direct / judge-opt. | .31 / .00 | .67 / .17 | .03 / .82 |
| Opus judge, total | GPT judge | |||||||
|---|---|---|---|---|---|---|---|---|
| Bench. | Gen. | Original | No “why” (13) | Retest (14) | “Why” effect | total | novelty | quality |
| NB201 | Sonnet | – | ||||||
| Opus | ||||||||
| GPT | ||||||||
| HPO | Sonnet | |||||||
| Opus | ||||||||
| NB201 | HPO | |
|---|---|---|
| Mean , H/S/O/G | 21/25/46/54 | 22/28/40/49 |
| Lower share of true claims | .57 | .54 |
| Lower share of sv -eligible true claims | .00 | .00 |
| Equivalent wordings, opposite labels | – | 41–60/60 |
| Three-way , H/S/O/G | 70/68/47/24 | 68/69/53/38 |
| True lower claims, prior (three-way) | .92 | .95 |
| Corrected, Claude panel | GPT added to panel | ||||||
|---|---|---|---|---|---|---|---|
| Bench. | Gen. | Orig. sv | Mix part | sv 3 [90% CI] | sv 3 | ||
| NB201 | Sonnet | [ , ] | |||||
| Opus | [ , ] | ||||||
| GPT | [ , ] § | – | – | ||||
| HPO | Sonnet | [ , ] | |||||
| Opus | [ , ] | ||||||
| Bench. | Gen. | Conflict claims (share) | True-cell share | Precision: conflict / agree |
|---|---|---|---|---|
| NB201 | Sonnet | 86 (.37) | .30 | .24 / .56 |
| Opus | 12 (.05) | .25 | .17 / .78 | |
| HPO | Sonnet | 21 (.09) | .57 | .48 / .88 |
| Opus | 17 (.07) | .76 | .76 / .89 |
| Stage | Pre-registered prediction | Verdict |
|---|---|---|
| 1 | Structural knobs (retrieval, roles, challenge, selection) shift sv by /run | Inconclusive |
| 1 | Judge novelty does not track sv ( ) | Supported ( ; novelty vs prior ) |
| 2 | Judge fitness raises the judge score without raising sv | Supported ( ; precision .78 .29) |
| 2 | Validity fitness plays safe (prior , fewer distinct claims) | Supported (prior ) |
| 2 | sv fitness beats the other two | Refuted (final harness = direct generation) |
| 3 | No harness beats direct generation on sv | Supported (the sv rerun after a bug fix regressed on fresh runs); post hoc, sv 3 agrees, but favors that rerun harness (8 runs; confirmed in stage 13) |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Pools | Selection | /run [5%, 95% of draws] | Prec. | Kept baseline | Paired: runs kept , |
|---|---|---|---|---|---|
| Stage-3 harness | Rule (kept) | .825 | |||
| Random | [ , ] | .745 | 24/24, | ||
| Other first | [ , ] | .739 | 22/24, | ||
| Non-self first | [ , ] | .646 | |||
| Other, then removes-edge | [ , ] | .648 | |||
| Ablation | Likely-true (kept) | .824 |
| Contribution of top | after dropping the top from both arms | |||||
|---|---|---|---|---|---|---|
| Comparison | 5 | 10 | ||||
| Direct (stage 5) | ( ) | (.006) | (.38) | |||
| New direct (stage 15) | ( ) | (.001) | (.02) | |||
| Likely-true ablation | ( ) | (.005) | (.53) | |||
| Bench. | Comparison | Def. | Prior | +Type | CV | Prop. | Raw | Sig./14 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| NB201 | Stage-3 harness vs direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 |
| vs new direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 | |
| vs likely-true ablation | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 | |
| GPT measure vs direct | ∗ | ∗ | ∗ | ∗ | ∗ | 12 | ||||
| vs pooled direct | ∗ | ∗ | ∗ | ∗ | ∗ | 12 | ||||
| vs pooled + stage-3 rule | ∗ | ∗ | ∗ | ∗ | ∗ | 12 |
| Bench. | Comparison | Def. | Prior | +Type | CV | Prop. | Raw | Sig./14 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| NB201 | Stage-3 harness vs direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 |
| vs new direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 | |
| vs likely-true ablation | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 14 | |
| GPT measure vs direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 13 | |||
| vs pooled direct | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 13 | |||
| vs pooled + stage-3 rule | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | 13 |
| Stage-3 harness (Sonnet) | GPT measure-first | Judge-opt. | |||||
|---|---|---|---|---|---|---|---|
| Bench. | Truth rule | True (rate) | vs direct / new direct | Prec. | LOO / own | Prec. | bound met |
| NB201 | all, | 419 (.241) | ∗ / ∗ | .90 .88 | ∗ / ∗ | .85 .62 × | 0/3 |
| all, | 288 (.166) | ∗ / ∗ | .89 .86 | ∗ / ∗ | .79 .58 × | 0/3 | |
| all, (main) | 198 (.114) | ∗ / ∗ | .79 .83 | ∗ / ∗ | .69 .50 × | 0/3 | |
| all, ‡ | 108 (.062) | ∗ / ∗ | .59 .63 | ∗ / ∗ | .49 .32 × | 0/3 | |
| 2 of 3, | 339 (.195) | ∗ / ∗ | .86 .90 | ∗ / ∗ | .76 .62 | 0/3 | |
| Bench. | Comparison | One member | Two members | Three members |
|---|---|---|---|---|
| NB201 | Stage-3 harness (Sonnet) | H ∗ , O ∗ , G ∗ | H+O ∗ , H+G ∗ , O+G ∗ | H+O+G ∗ |
| GPT measure | H , S , O ∗ | H+S , H+O ∗ , S+O ∗ | H+S+O ∗ | |
| Opus measure | H , S , G ∗ | H+S , H+G ∗ , S+G ∗ | H+S+G ∗ | |
| Sonnet measure | H , O , G | H+O , H+G , O+G | H+O+G | |
| HPO | GPT measure | H , S ∗ , O ∗ | H+S ∗ , H+O ∗ , S+O ∗ | H+S+O ∗ |
| Opus measure | H , S ∗ , G ∗ | H+S , H+G , S+G ∗ | H+S+G ∗ |
| Leave-one-out Claude panel | Generator’s own ratings | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Bench. | Comparison | Credit | Cost avoided | true | Credit | Cost avoided | Prec. ratio | ||
| NB201 | Stage-3 harness vs direct | ∗ | ∗ | 0.58/0.70 | ∗ | ∗ | 1.05 | ||
| vs new direct | ∗ | ∗ | 0.58/0.69 | ∗ | ∗ | 1.12 | |||
| vs likely-true ablation | ∗ | ∗ | 0.58/0.73 | ∗ | ∗ | 1.00 | |||
| HPO | GPT measure-first vs direct | ∗ | ∗ | 0.69/0.77 | ∗ | 1.05 | |||
| vs pooled direct | ∗ | 0.69/0.78 | ∗ | 0.98 | |||||
| Bench. | Comparison | Prec. | Prec. (comparison) | Ratio | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|---|---|---|---|---|---|---|---|---|
| NB201 | Stage-3 harness vs direct | 0.83 | 0.79 | 1.05 | yes | yes | yes | yes | yes |
| vs new direct | 0.83 | 0.74 | 1.12 | yes | yes | yes | yes | yes | |
| vs likely-true ablation | 0.83 | 0.82 | 1.00 | yes | yes | yes | yes | yes | |
| GPT measure-first vs direct | 0.50 | 0.69 | 0.72 | yes | yes | – | – | – | |
| vs pooled direct | 0.50 | 0.70 | 0.71 | yes | yes | – | – | – | |
| vs pooled + stage-3 rule | 0.50 | 0.73 | 0.68 | yes | – | – | – | – |
| Truth | Panel | Comparison | runs only | crossed | credit, crossed ( ) | cost avoided, crossed ( ) | ||
|---|---|---|---|---|---|---|---|---|
| , all (main) | Claude LOO | vs direct | [ , ] | [ , ] | .015 | [ , ] (.054) | [ , ] (.086) | |
| vs new direct | [ , ] | [ , ] | .004 | [ , ] (.028) | [ , ] (.039) | |||
| vs ablation | [ , ] | [ , ] | .024 | [ , ] (.045) | [ , ] (.137) | |||
| Claude+GPT LOO | vs direct | [ , ] | [ , ] | .009 | [ , ] (.020) | [ , ] (.102) | ||
| vs new direct | [ , ] | [ , ] | .002 | [ , ] (.013) | [ , ] (.051) | |||
| vs ablation | [ , ] | [ , ] | .004 | [ , ] (.005) | [ , ] (.145) |
| Panel | Exact | Noise | 2 noise | 12-ep. proxy | Withheld | Shuffled | Design only |
|---|---|---|---|---|---|---|---|
| Claude leave-one-out (H+S) | 0.19 | 0.14 | 0.27 | 0.71 | 0.73 | 1.65 | 1.23 |
| H+S+G | 0.90 ∗ | 0.50 | 0.45 | 0.24 | 0.46 | 0.76 | 0.95 |
| G only | 1.25 ∗ | 0.76 ∗ | 0.72 ∗ | 0.65 ∗ | 0.30 | 0.63 | 0.77 |
| Opus’s own ratings | 0.91 ∗ | 0.48 | 0.26 | 0.27 | 0.30 | 0.86 | 1.02 |
| Precision (direct .87) | .75 | .71 | .69 | .65 | .79 | .58 | .69 |
| Date (UTC) | claude-haiku-5-5 | claude-sonnet-5-5 | claude-opus-5-5 |
|---|---|---|---|
| 2026-10-20 | 766 | 384 | 240 |
| 2026-10-21 | 455 | 1,636 | 1,150 |
| 2026-10-22 | 35 | 34 | 110 |
| 2026-10-23 | – | 96 | 576 |
| 2026-10-24 | – | 24 | – |
| Document | Created (UTC) | Last modified (UTC) | SHA-256 |
|---|---|---|---|
| PREREG_s1.md | 2026-10-21 00:56 | 2026-10-21 00:56 | 9d7125b6b956db218b5252ec4a8eda318ef03de5d72a94c941e89ec0aaa57112 |
| PREREG_s2.md | 2026-10-21 03:56 | 2026-10-21 03:56 | ff825cd04b8ccb9a6c0d06804584b5ff21e384014d4e64d7af32010abe1de50c |
| PREREG_s3.md | 2026-10-21 04:53 | 2026-10-21 04:53 | 07f52c56b97a846b20fa78be1e0fd31159aae5197448627d0e4fd3cd32aa005a |
| PREREG_s4.md | 2026-10-21 05:37 | 2026-10-21 05:52 | 2a9c42cdb788d5c2c19c21657929bc4277b1065e97d3d1431db16b10c7551ee5 |
| PREREG_s5.md | 2026-10-21 06:04 | 2026-10-21 06:04 | 52d567d9091c092c341e06b02dacbb3bf97bb55ab83936566b3fa88b2a448840 |
| PREREG_s6.md | 2026-10-21 07:29 | 2026-10-21 07:29 | 2c2f7d9899d4210bbef90f91694a6d10ff27716c047862f458398dbfbde0f493 |
| Distinct claims per run | Leave-one-out Claude panel | Generator’s own ratings | |||||
|---|---|---|---|---|---|---|---|
| Bench. | Comparison | First arm | Second arm | ( ) | ( ) | ||
| NB201 | Stage-3 harness vs direct | 10.00 | 10.00 | ( ) | ( ) | ||
| vs new direct | 10.00 | 10.00 | ( ) | ( ) | |||
| vs likely-true ablation | 10.00 | 9.96 | ( ) | ( ) | |||
| GPT measure-first vs direct | 10.00 | 10.00 | ( ) | ( ) | |||
| vs pooled direct | 10.00 | 10.00 | ( ) | ( ) | |||
| Precision | Share of claims and precision within type: judge-opt. / direct | |||||||
| Bench. | Gen. | Judge-opt. | Direct | Type-adjusted | Type | Share | Precision | |
| NB201 | Sonnet | .43 | .79 | .36 | Self-condition | .03 / .68 | .57 / .87 | 7 / 164 |
| Removes an edge | .23 / .28 | .49 / .65 | 55 / 68 | |||||
| Other | .74 / .03 | .40 / .25 | 178 / 8 | |||||
| NB201 | Opus | .28 | .86 | .71 | Self-condition | .00 / .31 | – / .84 | 0 / 73 |
| Removes an edge | .17 / .67 | .52 / .89 | 42 / 158 | |||||
| Bench. | Gen. | Direct | Measure-first | Judge-opt. | Pooled direct |
|---|---|---|---|---|---|
| NB201 | Sonnet | 5.2k | 30.9k | 33.3k | – |
| Opus | 5.9k | 32.6k | 35.1k | – | |
| GPT | 3.5k | 21.0k | 25.2k | 21.2k | |
| HPO | Sonnet | 5.2k | 27.3k | 32.2k | – |
| Opus | 5.4k | 36.5k | 34.4k | – | |
| GPT | 3.6k | 17.4k | 24.2k | 20.4k |
| Gen. | Stage | Harness | prec. | sv 3 | Holm | Holm | ||
|---|---|---|---|---|---|---|---|---|
| Sonnet | 1 | L9 cell (0,1,1,1) | 8 | .09 | 1.50 | 1 | 2.02 | 1 |
| 1 | L9 cell (0,2,2,2) | 8 | .45 | 1.12 | 1 | 1.15 | 1 | |
| 1 | L9 cell (1,0,1,2) | 8 | .03 | 2.00 | .343 | 1.08 | .405 | |
| 1 | L9 cell (1,1,2,0) | 8 | .51 | 2.12 | 1 | 1.43 | 1 | |
| 1 | L9 cell (1,2,0,1) | 8 | .01 | 0.75 | 1 | 1.25 | 1 | |
| 1 | L9 cell (2,0,2,1) | 8 | .03 | 0.12 | 1 | 1.09 | 1 |
| Gen. | Panel | AUC | direct | Gen. | Panel | AUC | direct | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| NB201 | |||||||||||
| Sonnet | H | .67 | 3.74 | 1.97 | 1 | Opus | H+S+G | .90 | 1.36 | 0.90 | .008 |
| Sonnet | H+O | .87 | 1.10 | 0.57 | .954 | Opus | G | .91 | 1.03 | 1.25 | .001 |
| Sonnet | O | .88 | 1.61 | 0.90 | .995 | Opus | S+G | .91 | 1.46 | 0.84 | .015 |
| Sonnet | H+G | .90 | 0.72 | 0.30 | .823 | GPT | H | .67 | 3.55 | 0.56 | .909 |
| Sonnet | G | .91 | 0.57 | 0.17 | .693 | GPT | S | .71 | 3.25 | 0.12 | .613 |
| Full calibration | Prior-only calibration | ||||||
|---|---|---|---|---|---|---|---|
| Bench. | Gen. | Holm | Holm | ||||
| NB201 | Sonnet | 2.05 | 1 | 1 | 1.90 | 1 | 1 |
| Opus | 0.91 | .007 | .021 | 0.67 | .029 | .088 | |
| GPT | 2.69 | .001 | .001 | 2.46 | .001 | .001 | |
| HPO | Sonnet | 0.25 | .115 | .230 | 0.30 | .897 | 1 |
| Opus | 0.48 | .004 | .017 | 0.47 | .009 | .038 | |
Large Language Model-Guided Scientific Discovery
Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.
See all 312 papersBefore the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
Language models lack systematicity in generating and updating hypotheses
powers of 2'' and by numbers up to 20.'' We probe the concepts LLMs hypothesize from the observed examples and use their behavioral patterns to understand how they generate and update hypotheses. We measure the LLM posterior over hypotheses in three tasks (prediction, evaluation, and generation) and compare LLM behavior with a Bayesian model and with humans. LLMs favor narrower hypotheses than humans and show a robust evaluation--generation gap: in evaluation they select hypotheses more consistent with the observed examples, whereas in generation they propose narrower hypotheses that are more often rules. They also update poorly: as examples accumulate, their accuracy declines while their confidence grows. Thinking reduces, and can even reverse, the gap; it also broadens the hypotheses LLMs favor, raising accuracy while lowering confidence. Whereas prior studies have shown that LLMs are good hypothesis proposers, our results and further comparisons with the Bayesian model show that LLM inductive reasoning lacks coherence across the three tasks, suggesting that LLMs do not maintain a stable representation shared across tasks that make the same demand. Our results highlight a limitation of LLMs in inductive inference, particularly in settings such as science, where hypotheses must go beyond the data.