Thompson sampling is a widely used strategy for sequential decision-making under uncertainty, offering a principled way to balance exploration and exploitation. This strategy involves maximizing a target objective under a single sample drawn from the current belief over unknown quantities. Conventionally, this optimization is performed once the belief has been inferred from data, and it becomes computationally burdensome when many Thompson samples are required, especially in continuous action spaces. In this work, we introduce an approach that amortizes this process ahead of time, before any data is collected, so that an approximation of the full Thompson sampling distribution is available immediately once observations arrive. We employ this approach to solve the multi-round setting, involving sequential data acquisition and Thompson-sample recommendation across rounds. Beyond accelerating Thompson sampling, our approach enables efficient estimation of expected information gain about the optimal action. This can be used to select more informative Thompson samples or to obtain maximally informative designs with optimization. Combining this estimate of expected information gain with immediate regret supports information-directed sampling (IDS), which can outperform Thompson sampling by explicitly balancing these two quantities. We demonstrate our approach on two simulator-based design problems in synthetic biology, where the simulators have intractable likelihoods.
Figures & tables
Figure 1: We model foreign gene insertion. The excessive gene induction (designable parameter) can lead to an overworked host. This can decrease the expression of the gene and slow down its growth. The model can be used to predict the behaviour of the system but a lack of knowledge results in uncertainty.
Figure 2: Left: Regret for each step for 1000 rollouts each for different sample of θ∼p(θ) . Right: Thompson sample distributions for example context hexample=(0.61,1.95) using both methods.
Figure 3: Selection objectives for example history context hexample=(0.61,1.95) .
Figure 4: Comparison of action-selection strategies. Final denotes the last step, at which the immediate regret strategy was always used.
Approach
Criterion
Final
Cumulative
Selection
Immediate regret
0.105±0.007
0.955±0.050
One-step lookahead
0.014±0.001
3.098±0.046
Information gain
0.013±0.001
3.122±0.046
IDS
0.012±0.001
0.879±0.035
Optimization
Immediate regret
0.111±0.009
0.969±0.049
One-step lookahead
0.007±0.001
6.086±0.089
Table 1: Monte Carlo estimates of expected final and cumulative regret, reported as mean ± standard error (SEM). Actions a1 – a3 are chosen using the specified strategy; the final action is selected by minimising expected immediate regret. Italics identify the best-performing methods within each selection group (Selection or Optimization), and bold identifies the best-performing methods overall. Highlighted methods include the lowest-mean method and those not significantly different from it under paired t -tests with Holm correction ( α=0.05 ).
Figure 5: Cumulative regret for 1000 rollouts for GFP case study. The shading indicates standard deviation rescaled by 51 to aid presentation.
Figure 6: Insertion of three repressing genes (i.e. the repressilator ) is modelled. Under suitable conditions this results in oscillatory expression. The model not only has uncertain parameters; even with fixed parameters returns highly stochastic prediction of the outcome due to the molecular noise.
Figure 7: Monte Carlo estimate of the regret across stages for unfiltered Thompson sampling in the repressilator case study.
Figure 8: Regret for each acquisition strategy: final regret (left) and cumulative regret (right).
Criterion
Step 1
Step 2
Step 3
Final
Cumulative
Thompson sampling
0.685±0.077
0.177±0.038
0.181±0.038
0.078±0.024
1.112±0.072
Immediate regret
0.479±0.079
0.172±0.030
0.103±0.026
0.079±0.023
0.854±0.075
Information gain
0.716±0.074
0.639±0.072
0.552±0.081
0.051±0.018
2.048±0.081
IDS
0.434±0.067
0.151±0.026
0.134±0.031
0.090±0.025
0.833±0.075
Table 2: Estimated regret for the repressilator case study, reported as mean ± standard error. The specified strategy is used in Steps 1–3; the final action (Step 4) is selected by minimising expected immediate regret for all strategies.
Approach
Criterion
Final regret
Selection
Immediate regret
0.1697±0.0317
Information gain
0.1052±0.0236
IDS
0.1767±0.0536
Optimization
Immediate regret
0.1445±0.0311
Information gain
0.1179±0.0257
IDS
0.1600±0.0283
Table 3: Two-step case study: each strategy selects the first action, followed by an exploitative final action. Final regret is reported as mean ± SEM.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Step
Criterion
Mean ± SEM
Median
a2 (selection on)
Information gain
1.190±0.041
0.685
One-step lookahead
1.094±0.018
1.155
IDS
0.646±0.020
0.527
Immediate regret
0.540±0.029
0.222
vanilla
0.676±0.031
0.281
a3 (selection off)
Information gain
0.091±0.008
0.011
Appendix
Table A1: Regret by step and strategy (mean ± SEM and median) for example context.
Figure A1: Plot presenting the results in Table A1 .
Step
Mean ± SEM
Median
First action (prior optimization)
0.651±0.028
0.416
2nd (posterior flow)
0.211±0.016
0.090
2nd (action flow)
0.259±0.018
0.122
3rd (posterior flow)
0.108±0.013
0.055
3rd (action flow)
0.136±0.014
0.061
4th (posterior flow)
0.101±0.012
0.060
Appendix
Table A3: Estimated (MC) expected regret across four stages, with standard error of the mean (SEM) and median regret.