Artificial intelligence (AI) scientists pursue discovery through a chain of hypotheses, fitting candidate models to existing observations and often choosing experiments where those models disagree most. Yet when early data miss a narrow feature, model selection favors simple curves that agree across the unmea- sured gap, so disagreement-guided search has no reason to look where the new phenomenon lies. When we tested two frontier AI scientists, Claude Sonnet 5.5 and GPT-6.1-Sol, across 32 closed-loop trials with a narrow peak omitted from the starting data, both sampled inside the peak region less often than random search. Surprisingly, the agent that followed model disagreement more closely missed the peak in all 32 trials, whereas discovery succeeded when experiments probed the unmeasured gap early and followed up across the anomaly. Asking the right question, rather than testing only where current models disagree, is therefore key to AI discovery in a chain of hypotheses.
Figures & tables
Figure 1: How unmeasured gaps trap a closed-loop chain of hypotheses. a , Study design and blind-spot mechanism. Left: over 12 rounds, the agent proposes candidate equations for existing data and chooses two new conditions per round (each measured twice). Right: when starting data (open circles) miss a narrow peak region W (shaded band), smooth curves fitted to the background differ near x2=0 but agree across the unmeasured gap, steering disagreement-based experiments away from W . b , Example of a trapped trial (GPT-6.1-Sol). Top: chosen x2 values across Rounds 1–12 never enter W . Middle and bottom: by Round 4, the top three equations agree within 0.75σ across W and differ mainly at x2=0 ( 2.9σ ), leaving the agent on a smooth curve without the peak (final error E=4.60 ). c , Example of a successful recovery (Claude Sonnet 5.5). Top: after a weak edge point in Round 5 ( 1.6σ ), a strong measurement inside W in Round 9 ( x2=3.0 , 15.7σ ) reveals the jump, prompting a peak term in Round 10 and follow-up measurements across W . Bottom: the final equation matches the true curve ( E=0.13 ).
Figure 2: Both AI scientists measure inside the hidden peak less often than simple search baselines. a , Percentage of the 32 paired trials entering W by each round, compared with two search strategies that fit no equations: interval bisection along x2 (repeatedly measuring the midpoint of the widest unmeasured interval) and random sampling. Shaded bands show 95% confidence intervals; the vertical dashed line marks Round 6 ( p=0.002 for Sonnet 5.5 and p=8×10−13 for GPT-6.1-Sol below random sampling). b , Final prediction error E=RMSE/σ on held-out test points across all 32 paired trials (horizontal bars, medians). Trials below the dashed line ( E≤1.05 ) are solved ( 7/32 for Sonnet 5.5 versus 0/32 for GPT-6.1-Sol; p=0.016 ). Smooth curves without the peak fall inside the gray band ( E=3.9 – 4.8 ). c , Where each agent’s top three equations disagreed most during rounds before entering W ( 251 rounds for Sonnet 5.5; 348 for GPT-6.1-Sol). Disagreement concentrated near x2≤1 and rarely fell inside W ( 95% bootstrap confidence intervals). d , Ratio of equation disagreement at the chosen experiment to the maximum disagreement anywhere in the space (per-trial median across n=31 shared trials; 1 = testing at maximum disagreement). Gray lines connect the same trial across agents, and horizontal bars mark medians ( p=9×10−10 ; one trial excluded because Sonnet 5.5 entered W in Round 1).
Figure 3: Recovering the hidden peak requires a multi-step chain of hypotheses after entering W . a , Anomaly strength and timing across the 32 Sonnet 5.5 trials that entered W (out of 64 total runs). Each point shows the strongest peak signal (in units of σ ) measured in W before a peak term entered the top-ranked equation (or through Round 12 if none did), plotted against the first round reaching ≥10σ . b , Percentage of the 32 entered Sonnet 5.5 trials solved by the number of distinct x2 positions measured inside W ( 1 , 2 , or ≥3 positions; p=2×10−7 for ≥3 versus 1 – 2 positions; error bars, 95% confidence intervals). c , Distribution of all 768 measured x2 values across the 32 paired trials for Sonnet 5.5 (top) and GPT-6.1-Sol (bottom) in 0.25 -unit bins; the shaded band spans all 32 peak regions W . d , Percentage of the 32 paired trials entering W by Round 12 when W contained an integer ( 23 trials) versus lay between consecutive integers ( 9 trials; error bars, 95% confidence intervals).