Designing Experiments for External Decision Makers

Sahel Iqbal; Tom Rainforth

stat.MLOct 15, 2026

Designing Experiments for External Decision Makers

Authors: Sahel Iqbal, Tom Rainforth

Organizations: University of Oxford

Abstract

Standard Bayesian experimental design (BED) methods assume the experimenter alone will act on the data, choosing the best action given their own beliefs and goals. In practice, data are often acted on by external parties whose procedures the experimenter cannot change and may not know, particularly when the data are released publicly. To extend BED to such settings, we introduce usage-aware experimental design, a framework in which an adaptive design policy is trained to minimise the usage risk: the experimenter's expected loss when one or more external rules act on the collected data. These rules may be non-Bayesian or black-box, and when several may apply, we either average over them or take the worst case. We show that, unlike in standard BED, more informative data can be harmful under such rules, so the usage risk cannot in general be expressed as an expected uncertainty reduction; our framework thus strictly generalises standard BED. We then use it to tackle a range of problems that are difficult to address with existing approaches.

Figures & tables

		Test applied
Design	Designed for	Bonferroni	Simes
Bayes-act design	Bayes act	$0.00$	$0.00$
Single stage	Bonferroni	$52.27$	$53.23$
	Simes	$52.22$	$53.29$
Adaptive (ours)	Bonferroni	$57.79$	$57.57$
	Simes	$57.59$	$57.76$

Table 1: Expected utility (millions of euros; higher is better) of designs optimised for one test (rows) and evaluated under each test (columns). Adaptive entries average

5

training seeds. Standard errors of paired per-trial differences between adaptive entries and the DP or single-stage entries are below

0.05

. The Bayes-act design, the best single-stage design under a Bayes-act regulator, runs no trial.

Panel of Figure 1 — Figure 1: Accounting for the consumer’s rule changes both sampling and stopping. (a) Share of the first $5$ inspections sent to each zone; shaded bands mark the zones emphasised by the consumer’s prior. (b) Fraction of lots still being inspected: the usage-aware policy inspects for longer because the consumer requires more evidence. Both panels pool $5$ training seeds with $10^{4}$ evaluation lots each.

Panel of Figure 2 — Figure 2: Image acquisition for frozen classifiers. Results average over three independently trained classifier pairs, both classifiers within each pair, and three policy-training seeds; bands show one standard deviation across the three per-pair averages.

Policy	Loss ( $\downarrow$ )
Random (full budget)	$0.524\pm 0.006$
RL-BOED (full budget)	$0.432\pm 0.015$
Action-BED (optional stopping)	$0.648\pm 0.008$
Action-BED (forced full budget)	$0.414\pm 0.022$
Usage-aware policy (ours)	$\mathbf{0.265\pm 0.006}$

Table 2: Mean total loss under the sceptical consumer, averaged over five training seeds and 10,000 evaluation lots each, with the standard deviation across seeds. The loss includes decision error and the cost of destroyed units.

Panel of Figure 3 — Figure 3: Bayesian ranking and selection. (a) Usage risk against the DM’s caution $\kappa$ . (b) Usage risk split into Bayes risk $B_{\ell}$ and incompatibility $C_{\ell}$ at $\kappa=3$ . (c) Risk reduction against the number of shared candidates for the DMs. Average : the average policy vs. the better specialist on the average risk ( 9 ). Minimax : the minimax policy vs. the best randomisation of specialists on the worst-case risk ( 10 ). All panels average $10^{5}$ simulated episodes; standard errors are below $0.002$ in (a, b), and $95\%$ paired confidence intervals extend at most $0.003$ from the estimates in (c).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Classifier $F_{u}$ , $q_{\omega}$	Adaptive policy $f_{\phi}$	Static policy
Input	$(R_{4},C_{4})$ , $2\times 28\times 28$	$(R_{t-1},C_{t-1})$ , $2\times 28\times 28$	none
Conv 1	$32$ channels, ReLU, max-pool	$24$ channels, SiLU, average-pool
Conv 2	$64$ channels, ReLU, max-pool	$32$ channels, SiLU, average-pool
Context		$(t/T,\,b_{t}/\Lambda)$ appended to the flattened features
Dense 1	$128$ units, ReLU	$128$ units, SiLU
Output	$10$ class logits	$196$ block logits	$4\times 196$ logits $\eta_{t}$

Appendix

Table 3: Network architectures. Convolutions are

3\times 3

with unit stride and same padding, followed by

2\times 2

pooling of stride

2

. Weights use LeCun-normal initialisation and zero biases, except the policy’s output layer, which is initialised at zero so that every policy starts from the uniform allocation.

	Classifiers	Adaptive policies, Action-BED	Static policies
Training images	$40{,}000$	$10{,}000$	$10{,}000$
Optimiser	AdamW	AdamW	AdamW
Peak learning rate	$10^{-3}$	$10^{-3}$	$0.03$
Weight decay	$10^{-4}$	$10^{-4}$	$0$
Batch size	$256$	$256$	$256$
Epochs	$30$	$120$	$120$

Appendix

Table 4: Training hyperparameters. All values are shared across budgets, classifiers and seeds. They were set to common defaults without tuning, except the policy epoch count, which we raised to

120

after training curves of preliminary runs showed that all policies had converged by then. The learning rate follows a cosine decay to zero over training. After every epoch the model is scored on the validation images and the parameters with the lowest validation cross-entropy are kept.

Bayesian Experimental Design

Bayesian experimental design uses probabilistic models to choose measurements that improve learning or support better decisions.

See all 63 papers

Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives
Jun 22, 2026Tom Rossa, Angus Phillips, Tom RainforthBayesian Decision Theory Bayesian Experimental Design
Department of Statistics University of Oxford
Constrained Bayesian Experimental Design via Online Planning
May 26, 2026Yujia Guo, Daolang Huang, Xinyu Zhang +3Bayesian Experimental Design Sequential Experimental Design
ELLIS Institute Finland · Department of Computer Science, Aalto University, Finland · Department of Computer Science, University of Manchester, UK
Robust Bayesian Decision Making under Adversarial Uncertainty
Jul 9, 2026Haripriya Harikumar, Sammie Katt, Yasir Zubayr Barlas +1Adversarial Robustness Bayesian Decision Theory
Department of Computer Science, The University of Manchester, UK · ELLIS Institute Finland · Department of Computer Science, Aalto University, Espoo, Finland

Jun 22, 2026J/K move · Enter open · S save

Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives

Tom Rossa, Angus Phillips, Tom Rainforth

Department of Statistics University of Oxford

Bayesian Decision Theory Bayesian Experimental Design Decision-Making under Uncertainty

Open paper AskPDF arXiv

Bayesian experimental design (BED) has traditionally been based on maximising expected uncertainty reductions from prior to posterior. A major shortfall of this approach is that it leads to doubly intractable objectives that are difficult to optimise, while customising them to particular downstream tasks of interest can also be difficult. Following first principles decision theory, we demonstrate that BED can alternatively be formulated in terms of an expected future loss (EFL) on downstream actions, providing a simple and naturally task-driven framework. Critically, we then show that all such EFLs can be rearranged into singly intractable objectives that can be jointly optimised with respect to both the design policy and a downstream action policy using stochastic gradients, an approach we refer to as ACTION-BED. This formulation further sidesteps the need for any explicit posterior or marginal likelihood estimation and is naturally implicit, requiring only the ability to sample from the joint model over model parameters and data, and evaluate the downstream loss function. It thus allows design policies to be learned more effectively, efficiently, and simply than existing methods, while providing easy customisation to different downstream tasks and losses.

Abstract

Bayesian Experimental Design

Explore similar work

Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives

Constrained Bayesian Experimental Design via Online Planning

Robust Bayesian Decision Making under Adversarial Uncertainty