Standard Bayesian experimental design (BED) methods assume the experimenter alone will act on the data, choosing the best action given their own beliefs and goals. In practice, data are often acted on by external parties whose procedures the experimenter cannot change and may not know, particularly when the data are released publicly. To extend BED to such settings, we introduce usage-aware experimental design, a framework in which an adaptive design policy is trained to minimise the usage risk: the experimenter's expected loss when one or more external rules act on the collected data. These rules may be non-Bayesian or black-box, and when several may apply, we either average over them or take the worst case. We show that, unlike in standard BED, more informative data can be harmful under such rules, so the usage risk cannot in general be expressed as an expected uncertainty reduction; our framework thus strictly generalises standard BED. We then use it to tackle a range of problems that are difficult to address with existing approaches.
Figures & tables
Test applied
Design
Designed for
Bonferroni
Simes
Bayes-act design
Bayes act
0.00
0.00
Single stage
Bonferroni
52.27
53.23
Simes
52.22
53.29
Adaptive (ours)
Bonferroni
57.79
57.57
Simes
57.59
57.76
Table 1: Expected utility (millions of euros; higher is better) of designs optimised for one test (rows) and evaluated under each test (columns). Adaptive entries average 5 training seeds. Standard errors of paired per-trial differences between adaptive entries and the DP or single-stage entries are below 0.05 . The Bayes-act design, the best single-stage design under a Bayes-act regulator, runs no trial.
Figure 1: Accounting for the consumer’s rule changes both sampling and stopping. (a) Share of the first 5 inspections sent to each zone; shaded bands mark the zones emphasised by the consumer’s prior. (b) Fraction of lots still being inspected: the usage-aware policy inspects for longer because the consumer requires more evidence. Both panels pool 5 training seeds with 104 evaluation lots each.
Figure 2: Image acquisition for frozen classifiers. Results average over three independently trained classifier pairs, both classifiers within each pair, and three policy-training seeds; bands show one standard deviation across the three per-pair averages.
Policy
Loss ( ↓ )
Random (full budget)
0.524±0.006
RL-BOED (full budget)
0.432±0.015
Action-BED (optional stopping)
0.648±0.008
Action-BED (forced full budget)
0.414±0.022
Usage-aware policy (ours)
0.265±0.006
Table 2: Mean total loss under the sceptical consumer, averaged over five training seeds and 10,000 evaluation lots each, with the standard deviation across seeds. The loss includes decision error and the cost of destroyed units.
Figure 3: Bayesian ranking and selection. (a) Usage risk against the DM’s caution κ . (b) Usage risk split into Bayes risk Bℓ and incompatibility Cℓ at κ=3 . (c) Risk reduction against the number of shared candidates for the DMs. Average : the average policy vs. the better specialist on the average risk ( 9 ). Minimax : the minimax policy vs. the best randomisation of specialists on the worst-case risk ( 10 ). All panels average 105 simulated episodes; standard errors are below 0.002 in (a, b), and 95% paired confidence intervals extend at most 0.003 from the estimates in (c).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Classifier Fu , qω
Adaptive policy fϕ
Static policy
Input
(R4,C4) , 2×28×28
(Rt−1,Ct−1) , 2×28×28
none
Conv 1
32 channels, ReLU, max-pool
24 channels, SiLU, average-pool
Conv 2
64 channels, ReLU, max-pool
32 channels, SiLU, average-pool
Context
(t/T,bt/Λ) appended to the flattened features
Dense 1
128 units, ReLU
128 units, SiLU
Output
10 class logits
196 block logits
4×196 logits ηt
Appendix
Table 3: Network architectures. Convolutions are 3×3 with unit stride and same padding, followed by 2×2 pooling of stride 2 . Weights use LeCun-normal initialisation and zero biases, except the policy’s output layer, which is initialised at zero so that every policy starts from the uniform allocation.
Classifiers
Adaptive policies, Action-BED
Static policies
Training images
40,000
10,000
10,000
Optimiser
AdamW
AdamW
AdamW
Peak learning rate
10−3
10−3
0.03
Weight decay
10−4
10−4
0
Batch size
256
256
256
Epochs
30
120
120
Appendix
Table 4: Training hyperparameters. All values are shared across budgets, classifiers and seeds. They were set to common defaults without tuning, except the policy epoch count, which we raised to 120 after training curves of preliminary runs showed that all policies had converged by then. The learning rate follows a cosine decay to zero over training. After every epoch the model is scored on the validation images and the parameters with the lowest validation cross-entropy are kept.
Bayesian experimental design (BED) has traditionally been based on maximising expected uncertainty reductions from prior to posterior. A major shortfall of this approach is that it leads to doubly intractable objectives that are difficult to optimise, while customising them to particular downstream tasks of interest can also be difficult. Following first principles decision theory, we demonstrate that BED can alternatively be formulated in terms of an expected future loss (EFL) on downstream actions, providing a simple and naturally task-driven framework. Critically, we then show that all such EFLs can be rearranged into singly intractable objectives that can be jointly optimised with respect to both the design policy and a downstream action policy using stochastic gradients, an approach we refer to as ACTION-BED. This formulation further sidesteps the need for any explicit posterior or marginal likelihood estimation and is naturally implicit, requiring only the ability to sample from the joint model over model parameters and data, and evaluate the downstream loss function. It thus allows design policies to be learned more effectively, efficiently, and simply than existing methods, while providing easy customisation to different downstream tasks and losses.
Bayesian experimental design (BED) is a principled framework for data-efficient design of sequential experiments. However, existing BED methods are unable to adapt to dynamic constraints inherent in real-world tasks due to budget limitations, varying costs, or physical constraints that restrict how designs evolve over time. In this paper, we introduce a novel approach to BED that enables constrained optimization of experimental designs by combining offline pre-training of an amortized policy and a posterior network with online multi-step lookahead planning using scenario trees. We empirically demonstrate that our method yields substantially more informative design sequences than existing methods across a range of constrained BED tasks, while incurring only a modest additional computational overhead.
Yujia Guo, Daolang Huang, Xinyu Zhang +3
ELLIS Institute Finland · Department of Computer Science, Aalto University, Finland · Department of Computer Science, University of Manchester, UK
Scientific experiments are often designed to maximize information gain, yet in many applications the primary objective is to support reliable downstream decision-making. Existing decision-aware experimental design and active learning methods typically assume well-specified outcome models and implicitly rely on the stability of the optimal decision under real-world perturbations. In practice, however, experimental outcomes are frequently influenced by hidden or weakly modeled effects, which can substantially alter decision optimality and lead to misleading conclusions. We study sequential adversarially robust decision-aware experimental design, where data acquisition has to take into account information gain against plausible worst-case unexpected effects, modeled here as variation in adversarial variables. Building on Bayesian decision theory, we formalize an adversarially robust optimal decision under this setting and derive a principled Bayesian experimental design criterion. The criterion explicitly targets decision stability rather than nominal optimality. Experiments on synthetic and real-world scientific datasets show that conventional decision-aware design can converge rapidly to high confidence yet fragile decisions, while our robustness-aware approach yields decisions that are significantly more stable and reliable under adversarial variation.
Department of Computer Science, The University of Manchester, UK · ELLIS Institute Finland · Department of Computer Science, Aalto University, Espoo, Finland