Despite their extensive pretraining knowledge and strong reasoning capabilities, large language models (LLMs) struggle to adaptively gather information in experimental design tasks. We study how to amortize sequential Bayesian experimental design (BED) into an LLM by explicitly training the reasoning that guides action selection using information-gain rewards. Our approach specifies the experimental generative model independently of the LLM's own beliefs, allowing us to optimize expected information gain under a model of our choosing. We investigate the design choices for amortization, including alternative generative-model constructions, reward estimation, hypothesis-space specification, multi-step objectives, and computational costs relative to training- and inference-time BED baselines. On Twenty Questions, the resulting policies improve information acquisition and task performance, outperform inference- and training-time BED baselines at substantially lower computational cost, and generalize to unseen categories. We further demonstrate positive transfer from Twenty Questions training to Medi-Q, a real-world clinical dialogue benchmark. These results suggest that information gathering can be learned as a generalizable reasoning skill in LLMs.
Figures & tables
Method
Policy
Prior p(θ)
Likelihood p(y∣θ,x)
Objective
Contrastive set C
Training targets
Kobalczyk et al. (2025) Choudhury et al. (2025)
π(x∣h)†
LLM candidates + consistency filtering
LLM response model
EIG (inference)
Sampled at each step
—
Blau et al. (2022) Foster et al. (2021)
π(x∣h)
Specified prior
Known / simulated
sPCE (training)
Sampled for each game
Matches prior
Hartmann et al. (2026)
π(x∣h)
LLM candidates + consistency filtering
LLM response model
EIG (training)
Sampled at each step
LLM-induced
AR-BED (ours)
π(z,x∣h)
Fixed prior over C
External oracle
EIG (training)
Fixed C
ptrain(θ)⊂C
AR-BEDDirect (§ 5.1 )
π(x∣h)
—
—
EIG (training)
Fixed C
ptrain(θ)⊂C
Misspecified priors (§ 5.3.1 )
π(z,x∣h)
—
—
EIG (training)
Restricted coverage
Within restricted C
Table 1: Policy structure and information-value estimation across prior methods and AR-BED. Policy notation is schematic: h is the interaction history and z an explicit reasoning trace (Eq. 5 ); parameter and time indices are omitted. † Candidate proposals are followed by inference-time EIG selection. Indented rows denote ablations; shading marks changes and — denotes an unchanged prior or likelihood. Misspecification restricts the coverage of C and the training targets, while retaining a fixed prior over C . Section references point to experimental analyses.
Section
Question
§ 5.1
Does amortizing BED into a reasoning LLM improve general information acquisition?
§ 5.2
How does our approach compare to existing inference- and training-time BED approaches?
§ 5.3
How do choices for estimating and optimizing information gain affect performance?
§ 5.4
Does a policy trained for information acquisition in one task transfer to a different task?
Table 2
Figure 1: Amortizing BED into reasoning improves information acquisition and transfer. Bars and bands show mean ± SEM over three evaluation repeats.
Setting
Model
Time
Evaluation
BED-LLM
2376 s (61.1 × )
(game)
AR-BED 1
38.9 s
Training
ASIG
136.3 s (2.9 × )
(step)
AR-BED 1
47.3 s
Table 2: AR-BED1 is more efficient at both inference and training time.
Figure 2: Generalization across held-out animal subgroups. Solve rate over turns for held-out mammals and non-mammals. Bands show mean ± SEM over three evaluation repeats.
Figure 3: Comparison to BED baselines. AR-BED1 outperforms inference-time BED-LLM and training-time ASIG on most domains; bars and bands show mean ± SEM over evaluation repeats.
Dermatology (ID, n=95 )
Other Specialties (OOD, n=92 )
Model
Accuracy (%) ↑
Correct-answer likelihood (%) ↑
Brier score ↓
Accuracy (%) ↑
Correct-answer likelihood (%) ↑
Brier score ↓
Base
90.5 ± 3.0
76.0 ± 2.2
0.165 ± 0.031
84.8 ± 3.8
72.8 ± 2.8
0.227 ± 0.042
AR-BED1
94.7 ± 2.3
79.2 ± 1.9
0.120 ± 0.026
90.2 ± 3.1
78.1 ± 2.4
0.157 ± 0.035
Table 3: Performance on MediQ after training on Twenty Questions. Each policy conducts five question–answer turns with a fixed GPT-5 Patient; a separate GPT-5 decider predicts the final answer from the resulting history. Values are means ±1 SEM across cases
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Training-distribution size and coverage affect acquisition and transfer.
Figure 5: Expected rewards generally improve learning over realized rewards. One-step reasoning policies trained with the same 200-animal generative model, differing in reward estimator. Expected rewards improve EIG per question and solve rate on Animals, Plants, and Countries, but not Celebrities. Bars and bands show mean ± SEM over three evaluation repeats.
Figure 6: A controlled confirmation of non-myopic learning in Cube-NM. Left: probability of selecting the initially uninformative context test during training. Center: total EIG under greedy acquisition. Right: Bayesian MAP solve rate. The two-step objective learns to acquire context before disease information, improving later acquisition. These are matched single-seed, no-reasoning policies evaluated on all 96 disease–context pairs; the decoder is Bayesian, not an LLM diagnosis.
Figure 7: Multi-step training can improve target-attribute prediction during questioning. Accuracy on annotated target questions from TwentyQuestions ( Wang et al., 2025 ) , grouped by answer-frequency difficulty and aggregated across groups. Each animal-trained policy generates its own object-game histories; probes are scored without reasoning or access to the target identity. Curves and bands show mean ± SEM over three evaluation seeds. Later-turn averages use the games that reach that turn, rather than carrying forward predictions from completed games.
Figure 8: Immediate and downstream information objectives can select different questions. Tie-aware agreement between the candidate maximizing one-step EIG and that maximizing estimated EIG over two to five questions, by history length in the GPT-5 animal-game simulation. Agreement increases at later turns but remains below one.
Figure 9: Lookahead selection trades immediate for later support reduction. Mean active hypotheses after one, two, or three questions when the first question is selected by estimated EIG1 , EIG2 , or EIG3 , followed by GPT-5 continuations. Labels show differences from one-step selection. This is a candidate-selection diagnostic, not a comparison of the trained AR-BED policies.
Figure 10: Multi-step training improves mean downstream acquisition on Twenty Questions. Cumulative expected information gain divided by evaluation horizon, relative to AR-BED1 , from 180 matched held-out animal-game states. The evaluation horizons are one, two, and three questions, independently of the training horizon K . Differences are paired by evaluation repeat; error bars show SEM over three repeats.
Large language models (LLMs) exhibit strong reasoning and world-knowledge capabilities, yet often struggle to gather information effectively across the multi-turn interactions required in sequential decision-making settings. We introduce Amortised Sequential Information Gathering (ASIG), a fine-tuning approach that amortises Bayesian Experimental Design (BED) into LLM policies via a multi-turn extension of Group Relative Policy Optimisation with an Expected Information Gain reward. Evaluated on the 20 Questions task, ASIG more than doubles the success rate of the 7B base model and reduces inference cost by over 25× relative to BED-LLM, a competitive inference-time baseline. Applied to MediQ, a medical diagnosis benchmark unseen during training, ASIG improves information-seeking performance at the 7B scale, suggesting that the learned strategies can transfer out of distribution. Our findings show that amortising BED into LLM policies provides an effective and computationally efficient approach to sequential information gathering.
Jakob Hartmann, James Harvey, Jhonathan Navott +5
University of Oxford, Oxford, United Kingdom · 2Ellison Institute of Technology, Oxford, United Kingdom
Large Language Models (LLMs) excel at static reasoning tasks, yet their performance often degrades in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in selecting questions that reduce uncertainty while incorporating responses that may be ambiguous or only partially informative. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialog planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED maintains a belief distribution over hypotheses, anticipates possible answers, and propagates expected information gain through a simulated conversation tree. Across two structured entity-deduction benchmarks, CA-BED yields an average 21.8% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It achieves these gains with an average increase of only 1.8 conversational turns compared to direct prompting.
Large language models (LLMs) are increasingly used to drive scientific discovery. Understanding how LLMs make decisions from new data and memory of the literature is vital before trusting them to design experiments under tight experimental budgets. However, their decision strategies are invisible in the current performance scores used to evaluate research agents. Here, we benchmark five frontier LLMs in a Bayesian Optimization setting against published statistical baselines on seven combinatorial datasets spanning protein engineering, reaction optimization, molecular design, peptide self-assembly, and catalysis. Performance is paired with direct measurements of model beliefs and actions, enabling highly resolved behavior analysis. A prompt ablation that progressively strips context separates memorization from chemical reasoning and from bare categorical optimization. Prior chemical knowledge helps in expectation, but with high variance and occasionally even harms performance. No configuration tested decisively beats a mean statistical baseline across domains. Belief-movement and Martingale diagnostics, corrected here for a measurement-noise bias that mislabels rational agents as irrational, show that models overreact to incoming data rather than entrenching on their priors in the contexts studied here. Interestingly, while LLM actions are exploitative, models sincerely intend to explore and consistently act on that intent. This failure is a competence gap arising from context-stickiness. Removing in-context history restores exploration, indicating that priors and data must be decoupled to achieve effective LLM-driven discovery.