Large language models (LLMs) are increasingly applied across the scientific discovery pipeline, from synthesizing literature to generating hypotheses to informing experimental design. A unifying principle behind these tasks is the need to continuously update beliefs as evidence accumulates, for which Bayesian inference provides a natural framework. However, existing work assumes that LLMs cannot carry out principled Bayesian belief updating in context, instead relying on simplifications and heuristics. To challenge this assumption, we present a principled evaluation of Bayesian reasoning capabilities in LLMs, analyzing both the model's internal consistency as well as its ability to recover an externally determined posterior. Specifically, we build an evaluation framework spanning qualitative epistemological examples to probe probabilistic coherence, quantitative benchmarks of Bayesian inference capabilities, and analyses of the impact of non-Bayesian updates on downstream task performance. Across different scenarios, we find that LLMs generally violate Bayes' rule when updating their beliefs in context. However, when equipped with modern reasoning abilities and code execution capabilities, their performance drastically improves, enabling even the close approximation of analytically intractable posteriors. Our evaluation shows that current LLMs with access to appropriate tools could already be used to make complex belief updates in scientific discovery, enabling rapid acceleration of the efficiency and scale of AI-powered scientific discovery.
Figures & tables
Figure 1: Overview of our evaluation framework for In-Context Bayesian Inference We present our three axes of evaluation, using example task: ”Who will win the World Cup - Brazil, France, Germany, or Italy?”. Left: Probabilistic Coherence, evaluated using Dutch-Book scenarios in the same chat and across conversations to find violations of probability axioms. Middle: Posterior Fidelity, as KL divergence between LLM posterior and Bayesian oracle. Right: Downstream Task Performance, assessing how non-Bayesian updates affect success in Experimental Design.
Figure 2: Share of elicited probability judgments that are Dutch Bookable under cross-conversation and within-dialogue elicitation. While beliefs elicited across different conversations frequently violate normalization (N), monotonicity (M), and extensionality (E), stronger LLMs are mostly coherent in the within-dialogue setting.
Figure 3: Posterior fidelity of non-reasoning LLMs on 20Q and ProteinWorld - simple . On both environments, the average KL(pora-pask∥pLLMk) by number of experiments k takes on large values after only a few experiments have been executed, showing that LLM belief updates diverge substantially from those presented by the Bayesian oracle with same evidence.
Figure 4: Downstream task performance of different non-reasoning LLMs on 20Q and ProteinWorld - simple . On both environments, the average pora-actk(θ⋆)−pLLMk(θ⋆) by number of experiments k rises to above 30 % for all LLMs, showing that the non-Bayesian belief updates lead to substantially decreased performance in experimental design.
Figure 5: Impact of native reasoning on in-context Bayesian inference for modern open-source LLMs on 20Q and ProteinWorld - simple . We display, by number of executed experiments k , with reasoning enabled and disabled: Left: Posterior fidelity - average KL(pora-pask∥pLLMk) , Right: Downstream task performance - average posterior-mass gap pora-actk(θ⋆)−pLLMk(θ⋆) . Reasoning models show greatly reduced losses compared to their non-reasoning counterparts, providing evidence that the flexible compute budget introduced through reasoning is effective in improving belief update quality and even leads to rivaling oracle performance.
Figure 6: Impact of code execution on in-context Bayesian inference for Gemini-2.5-Pro on ProteinWorld - hard . We display, by number of executed experiments k , a baseline with reasoning only against reasoning + code execution: Left: Posterior fidelity, Right: Downstream task performance. Access to a Python environment drives KL divergence and performance gap close to 0 , showing that the model can use tools to approximate an analytically intractable posterior effectively.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Difficulty
Capabilities
Model
Dominant behavior
simple
—
Gemma-4-31B-it
Correct posterior mode, miscalibrated mass
simple
reasoning
Gemma-4-31B-it
Exact closed-form posterior; heavy redundant re-verification
hard
reasoning
Gemini-2.5-Pro
Correct model and correct diagnosis of intractability; hallucinated Monte Carlo output
hard
reasoning †
Gemini-2.5-Pro
Analytic approximation with numerical over-simplification
hard
reasoning + code
Gemini-2.5-Pro
Correct model, correct Monte Carlo implementation, oracle-level posterior
Appendix
Table 3: Regimes examined in this section and the dominant qualitative behavior in each. † denotes the variant in which simulation is explicitly suppressed via the system prompt (App. B.3 ).
Gemma-4-31B-it
Hypothesis
no reasoning
reasoning
Oracle
Xylocorein
0.643
0.967
0.967
Thalassogen
0.000
0.000
0.000
Nebulase
0.357
0.033
0.033
Cryovitin
0.000
0.000
0.000
Appendix
Table 4: Posterior pLLM1 compared to pora-pas1 on the ProteinWorld - simple instance in App. A.1 after ξ1=ProtFold-X Analysis , y1=−1.4657 , using a Uniform prior. Without reasoning the mode is correct, but the mass is misallocated; with reasoning the closed-form posterior is recovered exactly.
Gemini-2.5-Pro
Hypothesis
reasoning
reasoning †
reasoning + code
Oracle
Chronocortin
0.102
0.546
0.372
0.360
Glycorectin
0.008
0.454
0.214
0.216
Luminase
0.043
0.000
0.148
0.162
Somatoplexin
0.847
0.000
0.266
0.261
Appendix
Table 5: Posterior pLLM2 compared to pora-pas2 on the ProteinWorld - hard instance in App. A.2 after ξ1=Kinetic Interaction Capture System , y1=−0.3303 , ξ2=Post-Translational Cascade Mapping , y2=0 , using a Uniform prior. † denotes the variant with simulation suppressed via the system prompt. Neither reasoning-only variant approaches the oracle.
Figure 7: Downstream task performance on 20Q : classification accuracy by iteration for BED-LLM ( Choudhury et al., 2026 ) and the naive baseline with reasoning, evaluated on state-of-the-art open-source reasoning models.
Violation
LLM
Prompt A
PLLM(A)
Prompt B
PLLM(B)
Normalization
Gemini-2.5-Pro (reasoning: low)
Will a Democrat win the 2028 US presidential election?
50%
Will a candidate who is NOT a Democrat win the 2028 US presidential election?
55%
Monotonicity
Gemini-2.5-Pro (reasoning: low)
Will Russia control at least 70% of Donbass at the end of October 2026?
60%
Will Russia control at least 75% of Donbass at the end of October 2026?
80%
Extensionality
Gemini-2.5-Pro (reasoning: low)
Will there be at least 1,000 deaths due to direct conflict between India and Pakistan before 2027?
3%
Will the death toll from direct conflict between India and Pakistan reach four figures before 2027?
10%
Normalization
Gemini-2.5-Pro (reasoning: low)
Will the People’s Republic of China blockade or quarantine Taiwan before September 1, 2026?
35%
Will the People’s Republic of China NOT blockade or quarantine Taiwan before September 1, 2026?
88%
Monotonicity
Gemini-2.5-Pro (reasoning: low)
Will Nvidia’s stock price close below $150 on any day in 2026?
80%
Will Nvidia’s stock price close below $140 on any day in 2026?
Open-ended scientific discovery with large language models (LLMs) increasingly operates as a long-horizon loop of hypothesis search and verification, where a reward signal guides which hypotheses to test next. A notable recent example is AutoDiscovery, which uses "Bayesian surprise" - the belief shift an LLM undergoes after observing evidence for a hypothesis - as both a discovery metric and a reward for search. We first observe that AutoDiscovery treats surprisal as a static quantity, while surprisal in human reasoning is non-stationary - it is defined relative to beliefs that evolve with experience, a prerequisite for continual scientific discovery. We address this mismatch with evidence-informed LLM beliefs: priors updated with evidence from previous hypotheses to compute non-stationary surprisal for new hypotheses. We compare in-context belief-updating mechanisms and find that embedding-based retrieval-augmented generation over prior discoveries best anticipates eventual posteriors, identifying 37.5% of static surprisals as spurious. We then modify search to avoid these spurious rewards and prioritize hypotheses that remain surprising under non-stationary beliefs. Concretely, we introduce two complementary changes to the original search procedure: belief-update filtering and diversity maximization. Across five discovery domains, our method increases accumulated non-stationary surprisal by 30.62% on average compared to the original search procedure, demonstrating that continual scientific discovery with LLMs requires not only better belief measurement but also search procedures that avoid redundancy and encourage diversity.
Dhruv Agarwal, Reece Adamson, Andrew McCallum +3
University of Massachusetts Amherst · Allen Institute for AI
Modern AI systems are being deployed in complex domains such as medicine, science, and law, where it is important that they not only produce correct answers, but also represent and update uncertain beliefs about the world as new evidence arrives. We introduce the novel technique of studying LLMs as information processing rules and utilize the information processing gap to study the internal (in)consistencies of how LLMs update their probabilistic beliefs from evidence. Our extensive experiments evaluate multiple approaches in which LLMs can incorporate evidence into their beliefs. Some of these approaches produce (nearly) Bayesian updates; others seem to use a learned heuristic. Surprisingly, the non-Bayesian heuristic updates often outperform exact Bayesian computation in terms of downstream task performance -- indicating the LLMs' probabilistic models of the world are misspecified. Lastly, we show how our measure can provide diagnostics to identify issues with LLM-powered inferential systems.
Chacha Chen, Matthew Jörke, Adam Goliński +4
1Apple · 2Stanford University · †Work done while an intern at Apple. +1
Scientific model discovery requires proposing candidate models of an unknown system and designing experiments that discriminate among them. However, LLM-based approaches typically rely on in-context learning to predict outcomes and select experiments, lacking explicit model evaluation to systematically guide hypothesis search, while Bayesian approaches enable principled model comparison but are confined to a predefined candidate set which results in limited search spaces. To this end, we introduce BMD, which closes the loop between LLM-driven hypothesis generation and Bayesian experimental design. Specifically, an LLM proposes model families, while a numerical engine fits their parameters, weighs them by model evidence, and averages their predictions. Then, BMD chooses the next experimental conditions by maximizing the joint expected information gain over families and their parameters, while feeding the updated beliefs and residual diagnostics back to the LLM to guide its next proposals. Across four scientific discovery benchmarks, BMD achieves lowest final prediction error than in-context learning and LLM-based scientific agents while recovering exact functional forms more often, with ablations supporting numerical feedback and the joint objective.