Verifier strength does not buy discovery quality: a pre-registered two-domain test of LLM-based automated research, with post-hoc evidence pointing to proposer-pool composition
Authors: Peter Chen, Ni Zhang, Siyuan Zhang
Organizations: Shenyang Institute of Technology, Shenyang, China
Automated research pipelines pair an LLM proposer with a verifier on an implicit bet: discovery quality rises with verification strength. Under a pre-registered protocol we test this bet in two real-data domains -- job-shop scheduling and vehicular edge offloading on Beijing T-Drive trajectories -- across 100 matched LLM/random pairs per domain and three verifier tiers. The manipulation binds -- a 429-case mutation battery shows scores rising across tiers -- yet the paired LLM-over-random advantage does not scale with it (A, one-sided p=0.1115; D, p=0.9994), and in D shrinks under the strict tier (margin contrast -0.040, p=0.0013). The two lower gates promote identical candidates in 100/100 and 99/100 pairs, so the strict order is untestable when they coincide. Constructively, LLM proposals carry a real paired advantage over random controls, but a post-hoc union-pool analysis shows the fastest path to better discovery is widening the pool, not tightening the gate: mixing the two pools adds +1.52 holdout-value units in domain A (one-sided Wilcoxon, n=100, p=0.0002) -- evidence that pool composition, not verification strength, is the lever. Criteria, ledgers and scripts are archived on Zenodo (DOI: 10.5281/zenodo.23308866).
Figures & tables
symbol
meaning
P / V
proposer (LLM under iterative feedback) / verifier
S - , S 0 , S +
weak / standard / strict verifier tier (Sec. 3.2 )
J(c)
frozen holdout score of candidate c
J(G)
mean of J over a group G of promoted rules
Δ(⋅)
paired LLM-over-random margin of a tier
MS(⋅)
mutation score of a tier (Sec. 4 )
Table 1: Notation summary.
domain
n
MS(S - )
MS(S 0 )
MS(S + )
A
119
0.000 [0.000, 0.031]
0.0168 [0.005, 0.059]
0.1092 [0.065, 0.178]
D
310
0.3484 [0.298, 0.403]
0.3548 [0.304, 0.410]
0.3613 [0.310, 0.416]
Table 2: Manipulation check (E1-D2): mutation score per tier (fraction of battery mutants killed), with Wilson 95% CIs. Populations disclosed before unblinding (protocol v1.2).
domain
Δ (S - )
Δ (S 0 )
Δ (S + )
SD
p
A
+0.28
+0.28
+0.38
– (floor)
0.1115
D
+0.10
+0.10
+0.06
0.211/0.211/0.193
0.9994
Table 3: Main result. Δ (tier) = holdout value of the LLM-promoted rule minus that of the random control (frozen scoring). p : one-sided Wilcoxon, S + vs. S - (the frozen criterion). Δ scales are domain-specific (different objectives) and are not comparable across rows. SD: per-tier margin standard deviation – floor-free in domain D ( 0.19 – 0.22 , matching the bootstrap CIs in Table 4 ); in domain A the raw SDs are meaningless because of the −109 floor contamination, so floor-free means on n=98 pairs (Section 5.2, sensitivity analysis) are reported instead. S 0≡ S - degeneracy makes the S 0 -vs-S - and S + -vs-S 0 contrasts untestable (identical promotions).
domain
Δ (S - )
Δ (S 0 )
Δ (S + )
A (n=98)
+0.367[−1.15,1.85]
+0.367[−1.18,1.88]
+0.469[−1.08,2.02]
D (n=97)
+0.102[+0.060,0.145]
+0.102[+0.061,0.145]
+0.062[+0.024,0.100]
Table 4: Per-tier LLM-over-random margins with bootstrap 95% CIs (post-hoc, computed by descriptive_stats.py ; not part of the frozen criteria). Domain A is reported on the n=98 floor-free pairs; domain D on its n=97 valid pairs.
domain
identical
diff pairs
S + wins
mean diff
A (n=100)
100
14
10
+0.714
D (n=100)
99
23
1
−0.066
Table 5: Tier degeneracy and stress-test behavior. “identical”: pairs where S - and S 0 promote the same candidate. “diff pairs”: pairs where S + and S 0 promote different candidates, with S + wins and mean holdout difference among those pairs.
Figure 1: Group contrast J(S+)−J(S−) against measured capacity utilization ( J : the frozen holdout score of a promoted rule, Section 3.1; the figure legend abbreviates tasks-per-vehicle as “tpv”). The contrast crosses zero: negative in the idle confirmatory regime (4%), positive on both extension days, peaking at medium utilization (22–23%). Script-generated from tension_gradient.json and the frozen tier-1 ledgers.
Pre-registration is increasingly proposed as a trust mechanism for machine-learning research involving LLM-generated content, but its canonical form assumes a frozen protocol can be executed unchanged. Software-intensive studies violate this assumption: the protocol is partly implemented, and implementations carry defects. We report a three-case study of protocol repairs ("patches") from a program where an LLM proposes executable rules that a programmatic verifier gates, across production scheduling, incentive design, and task assignment. Each protocol required exactly one defect-repair patch: three events in total. None modified a numbered judgment criterion, its threshold, or the scoring rule, and none altered already collected data; two changed pre-registered design parameters before any data were collected, each with a recorded rationale. We propose a two-axis taxonomy of patches (defect class: environmental / feasibility / correctness, x timing: pre-data / post-data), introduce a verifiable non-impact claim for post-data repairs, and distill a nine-item checklist; all nine items were exercised by at least one patch event -- an observation of use, not a validation of sufficiency. The credibility of verifier-bound, LLM-proposer studies depends less on whether patches occur than on whether they are auditable.
LLM research-ideation systems aim for hypotheses that are both true and non-obvious, yet neither property can normally be checked. We build a testbed in which both can. Generators propose conditional interaction claims about two fully tabulated benchmarks, NAS-Bench-201 and HPOBench; every claim is verified exactly against the tables, and its non-obviousness is scored against a calibrated blind prior elicited from other LLMs. Excess truth measures how much more often a run's claims hold than this panel expects. Across a series of pre-registered stages with Claude and GPT generators, our main positive result is a simple two-stage step found by a harness optimizer: draft a large pool of claims, then select a subset in a separate call. For Sonnet on NAS-Bench-201, this step raised excess truth in every pre-registered test, including against a same-pool ablation that keeps only likely-true claims, without lowering precision. The gain comes both from credit for true claims and from avoiding near-miss false ones. In post hoc checks it held under every calibration, panel and truth rule we tried, and it exceeded random and type-filter selection from the same pools. Because kept claims recur across runs, the evidence weakens when runs and claims are resampled jointly, and it is not significant against Sonnet's own prior. Two secondary findings are also pre-registered. GPT makes real use of the results of cheap probes, beating shuffled and withheld controls, though not more efficiently than asking for more claims. Optimizing the harness for an LLM judge raised judge scores while sharply cutting precision for every generator.
Shanglin Yang, Zhan Shi
Independent Researcher, USA · Santa Clara University, USA
Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optimize an objective. In this work, we study the relationship between the population of previous iterates and eventual discovery success. We generalize past work on harness design to develop a suite of 12 harnesses called 'Modular' and characterize their performance across 5 diverse discovery tasks, finding that discovery success is brittle and sensitive to harness design. We uncover mode collapse, characterized by a dramatic drop in the diversity of iterates, as a common failure mode. We find that popular state-of-the-art harnesses and diversity-inducing harness interventions, which aim to prolong this collapse, yield inconsistent gains. Our results instead uncover that the performance of early discoveries is predictive of eventual success. We therefore propose a universally applicable intervention that performs an initial stage of parallel exploration in order to initialize subsequent iterative optimization. Our method provides consistent gains across many harnesses and target applications, confirming the importance of initialization in LLM-driven discovery.
Mansi Sakarvadia, Marco Ciccone, Colin Raffel
University of Chicago · Vector Institute · University of Toronto