ALMAB-DC: Active Learning, Multi-Armed Bandits, and Distributed Computing for Sequential Experimental Design and Black-Box Optimization

Foo Hui-Mean; Yuan-chin I Chang

cs.LGMar 22, 2026

ALMAB-DC: Active Learning, Multi-Armed Bandits, and Distributed Computing for Sequential Experimental Design and Black-Box Optimization

Authors: Foo Hui-Mean, Yuan-chin I Chang

Organizations: Institute of Statistical Science, Academia Sinica, Taipei, Taiwan

Abstract

Sequential experimental design under expensive, gradient-free objectives is a central challenge in computational statistics: evaluation budgets are tightly constrained and information must be extracted efficiently from each observation. We propose ALMAB-DC, a GP-based sequential design framework combining active learning, multi-armed bandits (MAB), and distributed asynchronous computing for expensive black-box experimentation. A Gaussian process surrogate with uncertainty-aware acquisition identifies informative query points; a UCB or Thompson-sampling bandit controller allocates evaluations across parallel workers; and an asynchronous scheduler handles heterogeneous runtimes. We present cumulative regret bounds for the bandit components and characterize parallel scalability via Amdahl's Law. We validate ALMAB-DC on five benchmarks. On the two statistical experimental-design tasks, ALMAB-DC achieves lower simple regret than Equal Spacing, Random, and D-optimal designs in dose--response optimization, and in adaptive spatial field estimation matches the Greedy Max-Variance benchmark while outperforming Latin Hypercube Sampling; at $K=4$ the distributed setting reaches target performance in one-quarter of sequential wall-clock rounds. On three ML/engineering tasks (CIFAR-10 HPO, CFD drag minimization, MuJoCo RL), ALMAB-DC achieves 93.4% CIFAR-10 accuracy (outperforming BOHB by 1.7,pp and Optuna by 1.1,pp), reduces airfoil drag to $C_D = 0.059$ (36.9% below Grid Search), and improves RL return by 50% over Grid Search. All advantages over non-ALMAB baselines are statistically significant under Bonferroni-corrected Mann--Whitney $U$ tests. Distributed execution achieves $7.5\times$ speedup at $K = 16$ agents, consistent with Amdahl's Law.

Figures & tables

Symbol	Meaning
$T$	Total number of decision rounds / evaluations
$K$	Number of parallel workers
$f(\cdot)$	Unknown objective / response surface
$x_{t}$	Query point selected at round $t$
$y_{t}$	Noisy observation at $x_{t}$
$A$	Number of candidate arms/configurations (when discretized)

Table 1: Core notation for regret and scaling analysis.

Figure 1: ALMAB-DC framework overview. The three core paradigms—Active Learning (AL), Multi-Armed Bandits (MAB), and Distributed Computing (DC)—are unified through a central GP surrogate model. The left column shows the statistical decision components: the Active Learner selects informative query points via acquisition functions (UCB, EI, or max-variance); the Multi-Armed Bandit allocates the evaluation budget across parallel workers using UCB or TS; and the Parallel Execution module dispatches evaluations asynchronously. The right column reflects the surrogate’s outputs: sequential queries (

x_{t+1}=\operatorname{argmax}\,U_{t}(x)

), posterior uncertainty quantification (

\sigma_{t}^{2}(x)

), and cohort allocation across

K

workers. The red feedback arc (top) represents the posterior update loop, closing the sequential design cycle.

Figure 2: ALMAB-DC sequential design pipeline. The decision tier (top) flows from left to right: candidate experiments are ranked by the Active Learner, allocated by the Bandit Controller, and dispatched by the Asynchronous Scheduler to

K

parallel workers. The compute tier (bottom) flows from right to left: parallel workers return observations to the Experiment Oracle, which passes the result to the GP Surrogate Update module. The red vertical arrow denotes the posterior update loop, feeding back into the Active Learner and closing the sequential design cycle.

Method	Surrogate	Parallel/Async
BOHB	✓	✓
Optuna (TPE)	✓
Parallel BO	✓	✓
GP-BUCB/GP-AUCB ( Desautels et al.,, 2014 )	✓	✓
AEGiS ( De Ath et al.,, 2021 )	✓	✓
Async LP ( Alvi et al.,, 2019 )	✓	✓

Table 2: Positioning of ALMAB-DC relative to common alternatives along the four defining dimensions of the framework.

Method	Med. Regret (%)	Std (%)	Success Rate
Equal Spacing ††	3.21	2.84	0.384
Random ††	5.47	3.92	0.219
D-optimal ††	1.52	1.73	0.572
Sequential BO ††	0.41	0.89	0.780
ALMAB-DC (UCB)	0.18	0.63	0.854

Table 3: Case P1: Sequential RSM for polymer synthesis (

R=2{,}000

replicates,

N=20

evaluations). Success rate = fraction of runs achieving

r_{T}<1.0\,\%

. Best values in bold . †† : ALMAB-DC (UCB) significantly better at 1% after Bonferroni correction (Mann–Whitney

U

Figure 3: Case P1: Sequential response surface design for polymer synthesis (

R=2{,}000

replicates,

8\times 8

grid). (a) True yield surface

f(x_{1},x_{2})

on the normalized

[0,1]^{2}

domain; open circles mark four fixed corner initializations; the star marks the true optimum

x^{*}=(0.4,\,0.6)

f^{*}=88.0\,\%

. (b) Median simple regret

r_{t}=f^{*}-f(x_{t}^{+})

versus cumulative evaluations (

K=1

); shaded bands are interquartile ranges. (c) Median simple regret versus wall-clock rounds for distributed ALMAB-DC (

K=1,2,4

);

K=2

achieves

1.70\times

and

K=4

achieves

2.43\times

speedup relative to

K=1

Figure 4: Case P2: Dose–response optimization via Bayesian experimental design (

R=2{,}000

replicates). (a) True dose–response landscape over 33 dose levels: net clinical benefit

f(x)=p_{\mathrm{eff}}(x)-0.5\,p_{\mathrm{tox}}(x)

(solid black), efficacy probability

p_{\mathrm{eff}}(x)

(blue dashed), and toxicity probability

p_{\mathrm{tox}}(x)

(red dashed); the green vertical line marks the true optimum

x^{*}=3.5

f^{*}=0.684

. (b) Median simple regret versus cumulative evaluations (

K=1

); shaded bands show the interquartile range. (c) Median simple regret versus wall-clock rounds for distributed ALMAB-DC (

K=1,2,4,8

); at

K=8

the IQR collapses to zero.

Figure 5: Case P3: Adaptive spatial sampling for GP field estimation ( $R=2{,}000$ replicates). (a) One representative realization of the latent Matérn- $\tfrac{3}{2}$ spatial field ( $\ell=0.35$ , $\sigma_{f}^{2}=1.0$ , $\sigma_{n}=0.2$ ) on the $8\times 8$ candidate grid; filled black squares mark the four fixed corner observations. (b) Median integrated posterior variance (IPV, lower is better) versus sequential observations ( $K=1$ , $N=30$ ); ALMAB-DC (UCB) matches Greedy Max-Variance and significantly outperforms LHS and Random. (c) Median IPV versus wall-clock rounds for distributed ALMAB-DC ( $K=1,2,4$ ); near-linear scaling is evident.

Method	Val. Accuracy	Std	Cum. Regret	Wall-clock (s)
Grid Search ††	0.8793	0.0079	13.5	168
Random Search ††	0.8956	0.0066	11.2	145
BOHB ††	0.9172	0.0048	8.8	120
Optuna (TPE) ††	0.9231	0.0043	8.2	122
ALMAB-DC (TS) ††	0.9312	0.0036	7.1	112
ALMAB-DC (UCB)	0.9342	0.0036	6.9	108

Table 4: Case C1 — CIFAR-10 HPO with EfficientNet-B0 (

R=2{,}000

replicates,

N=60

evaluations; mean

\pm

1 std). Superscript †† : ALMAB-DC (UCB) significantly better at 1% after Bonferroni correction (Mann–Whitney

U

). Best values in bold .

Figure 6: Case C1 — CIFAR-10 HPO: (Left) best validation accuracy as a function of evaluation index (mean

\pm

1 std over 2,000 runs). ALMAB-DC converges to near-optimal accuracy within 30 evaluations. (Right) Cumulative regret vs. evaluation index; ALMAB-DC (UCB) accumulates the least regret across the entire budget.

Method	Best $C_{D}$	Std	Wall-clock (s)
Grid Search ††	0.09299	0.00225	220
Random Search ††	0.08419	0.00168	185
BOHB ††	0.07084	0.00161	148
Optuna (TPE) ††	0.06744	0.00114	150
ALMAB-DC (TS) ††	0.06009	0.00111	137
ALMAB-DC (UCB)	0.05873	0.00111	132

Table 5: Case C2 — CFD Airfoil Drag Minimization (

R=2{,}000

replicates,

N=50

evaluations). Lower

C_{D}

is better. Superscript †† : ALMAB-DC (UCB) significantly better at 1% after Bonferroni correction. Best values in bold .

Figure 7: Case C2 — drag convergence comparison: best

C_{D}

found vs. evaluation index (mean

\pm

1 std over 2,000 runs). Both ALMAB-DC variants converge below

C_{D}=0.065

within 30 evaluations while Grid Search stagnates above 0.09.

Method	Avg. Return	Std	Cum. Regret ( $\times 10^{3}$ )	Wall-clock (s)
Grid Search ††	6,370	157	330.9	820
Random Search ††	7,248	135	285.4	750
BOHB ††	8,536	108	226.6	620
Optuna (TPE) ††	8,795	101	208.8	630
ALMAB-DC (TS) ††	9,403	90	177.2	580
ALMAB-DC (UCB)	9,602	84	165.1	560

Table 6: Case C3 — MuJoCo HalfCheetah HPO (

R=2{,}000

replicates,

N=50

). Cumulative regret in units of

10^{3}

. Superscript †† : significantly better at 1% after Bonferroni correction. Best values in bold .

Figure 8: Case C3 — MuJoCo HalfCheetah HPO: (Left) best average return vs. evaluation index; ALMAB-DC (UCB) reaches near-peak performance by evaluation 30. (Right) Cumulative regret over the evaluation budget (reported as

\times 10^{3}

for readability).

Variant	C1 Val. Acc.	C2 $C_{D}$	C3 Return
ALMAB-DC (full)	0.9342 $\pm$ 0.0036	0.05873 $\pm$ 0.00111	9,602 $\pm$ 84
w/o MAB (AL only)	0.9334 $\pm$ 0.0042	0.05967 $\pm$ 0.00157	9,410 $\pm$ 84
w/o AL (MAB only)	0.9294 $\pm$ 0.0048	0.06544 $\pm$ 0.00209	9,035 $\pm$ 96
Optuna (best baseline)	0.9231 $\pm$ 0.0043	0.06744 $\pm$ 0.00114	8,795 $\pm$ 100

Table 7: Ablation study across Cases C1–C3 (mean

\pm

std,

R=2{,}000

). Best values in bold .

Case	Variant	Rounds	Terminal Regret	$p$
P1 (RSM)	Async (ALMAB-DC)	7	0.18%	$>$ 0.05
P1 (RSM)	Sync (batched GP-UCB)	10	0.19%	$>$ 0.05
P2 (Dose)	Async (ALMAB-DC)	6	0.00263	$>$ 0.05
P2 (Dose)	Sync (batched GP-UCB)	8	0.00271	$>$ 0.05
P3 (Spatial)	Async (ALMAB-DC)	4	IPV metric	$>$ 0.05
P3 (Spatial)	Sync (batched GP-UCB)	6	IPV metric	$>$ 0.05

Table 8: Synchronous vs. asynchronous dispatch at

K=4

(

R=2{,}000

replicates). “Rounds” = median wall-clock rounds to reach threshold regret. “Regret” = median terminal simple regret.

p

-values from Mann–Whitney

U

test (two-sided).

Figure 9: Ablation convergence curves for Cases C1–C3 (mean

\pm

1 std, 2,000 runs). ALMAB-DC (full) consistently dominates both ablated variants.

Figure 10: Empirical speedup vs. number of parallel workers (

K\in\{1,2,4,8,16\}

) for Cases C1–C3, compared against ideal linear speedup and per-case Amdahl upper bounds (dashed). ALMAB-DC sustains near-linear scaling to

K=4

, with Case C1 reaching

7.5\times

K=16

CPUs $K$	GP	MLP	RF
1	1.00 / 1.000	1.00 / 1.000	1.00 / 1.000
2	1.52 / 0.759	1.78 / 0.891	1.60 / 0.799
4	2.06 / 0.514	2.97 / 0.744	2.27 / 0.568
8	2.41 / 0.302	4.43 / 0.554	2.81 / 0.352
16	2.43 / 0.152	5.74 / 0.358	3.01 / 0.188

Table 9: Representative CPU scaling: speedup

S(K)

/ parallel efficiency

\eta(K)

for GP, MLP, and RF surrogates.

Figure 11: Budget sensitivity on CIFAR-10 (Case C1, 2,000 replicates): (Left) best validation accuracy vs. budget

N

; (Right) accuracy advantage of ALMAB-DC (UCB) over Optuna (TPE).

Figure 12: Convergence rate analysis for Case C1 (CIFAR-10, 2,000 replicates): (a) mean best-so-far curves with exponential gap fits; (b) evaluations to reach accuracy thresholds 0.880 and 0.910; (c) fitted convergence rate

\lambda

with

R^{2}

annotation.

Figure 13: Noise robustness for Cases C1–C3 under observation noise

\sigma_{\mathrm{obs}}\in[0,0.30]

(2,000 replicates each). Rows: Case C1 (top), Case C2 (middle), Case C3 (bottom). (Left) best achieved metric vs.

\sigma_{\mathrm{obs}}

; (Right) degradation relative to clean baseline.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Sequential ( $K=1$ )	Distributed ( $K=4$ )	Gain
Wall-clock time (s)	2.137	0.548	$3.90\times$ faster
Cumulative regret	4.812	2.347	$\downarrow 51.2\%$
Mean reward	0.4182	0.5126	$\uparrow 22.5\%$

Appendix

Table 10: Gaussian-mixture synthetic example: sequential vs. distributed ALMAB-DC (15 arms, 150 iterations).

	GP		MLP		RF
$K$	$C_{D}$	Runtime (s)	$C_{D}$	Runtime (s)	$C_{D}$	Runtime (s)
1	0.0468 (0.0044)	42.055 (0.903)	0.0467 (0.0034)	40.042 (0.542)	0.0476 (0.0037)	41.449 (1.085)
2	0.0471 (0.0041)	27.690 (0.595)	0.0477 (0.0040)	22.476 (0.371)	0.0473 (0.0039)	25.945 (0.720)
4	0.0473 (0.0038)	20.464 (0.451)	0.0475 (0.0035)	13.463 (0.216)	0.0475 (0.0040)	18.256 (0.476)
8	0.0472 (0.0038)	17.428 (0.359)	0.0467 (0.0040)	9.041 (0.147)	0.0476 (0.0039)	14.734 (0.364)
16	0.0468 (0.0035)	17.316 (0.398)	0.0468 (0.0042)	6.982 (0.112)	0.0474 (0.0038)	13.750 (0.379)

Appendix

Table 11: Drag coefficient and runtime by surrogate model and CPU count (100 runs per cell; mean (std)).

$K$	GP: $S(K)$ / $\eta(K)$	MLP: $S(K)$ / $\eta(K)$	RF: $S(K)$ / $\eta(K)$
1	1.00 / 1.000	1.00 / 1.000	1.00 / 1.000
2	1.52 / 0.759	1.78 / 0.891	1.60 / 0.799
4	2.06 / 0.514	2.97 / 0.744	2.27 / 0.568
8	2.41 / 0.302	4.43 / 0.554	2.81 / 0.352
16	2.43 / 0.152	5.74 / 0.358	3.01 / 0.188

Appendix

Table 12: Empirical speedup

S(K)=T_{1}/T_{K}

and parallel efficiency

\eta(K)=S(K)/K

by surrogate model.

Method	Val. Acc. (GPU)	Std	$\boldsymbol{\Delta}$ Acc.	Regret (GPU)	$\boldsymbol{\Delta}$ Regret	Wall-clock (s)
Grid Search	0.8788	0.0075	$-0.0008$	13.49	$+0.0$	168
Random Search	0.8958	0.0062	$+0.0004$	11.22	$+0.0$	145
BOHB	0.9174	0.0049	$+0.0002$	8.80	$+0.0$	120
Optuna (TPE)	0.9230	0.0041	$\phantom{+}0.0000$	8.20	$+0.0$	122
ALMAB-DC (TS)	0.9310	0.0034	$\phantom{+}0.0000$	7.09	$+0.0$	112
ALMAB-DC (UCB)	0.9342	0.0034	$\boldsymbol{+0.0000}$	6.94	$\boldsymbol{+0.0}$	108

Appendix

Table 13: Case C1 — CIFAR-10 HPO: GPU replication (

R=2{,}000

, MPS device).

\Delta

Acc. and

\Delta

Regret are signed deviations GPU

-

CPU. Best values in bold .

Method	Median Final Regret (GPU)	Std
Equal Spacing	0.00561	0.00000
Random	0.00263	0.01751
D-optimal	0.00263	0.00000
Pure BO	0.00263	0.00392
ALMAB-DC (UCB, $K=1$ )	0.00263	0.00392
ALMAB-DC (UCB, $K=2$ )	0.00412	0.00547

Appendix

Table 14: Case P2 — Dose-Response Optimization: GPU replication (

R=2000

, MPS device). Metric: median simple regret (lower is better). Best value in bold .

Figure 14: Case P2 GPU replication — dose-response optimization (

R=2000

replicates, MPS device): (a) true dose–response landscape; (b) median simple regret vs. cumulative evaluations (

K=1

); (c) median simple regret vs. wall-clock rounds (

K=1,2,4,8

). At

K=8

median regret collapses to zero.

Figure 15: Case P3 GPU replication — adaptive spatial sampling ( $R=2{,}000$ field realizations, MPS device): (a) one representative Matérn- $\tfrac{3}{2}$ field realization; (b) median IPV vs. sequential observations ( $K=1$ ); ALMAB-DC (UCB) and Greedy Max-Variance coincide (IPV $=0.072$ ); (c) median IPV vs. wall-clock rounds ( $K=1,2,4$ ).

Method	Final IPV (GPU)	Std
LHS	0.1066	0.0000
Random	0.0982	0.0089
Greedy Max-Variance	0.0721	0.0000
ALMAB-DC (UCB, $K=1$ )	0.0721	0.0000
ALMAB-DC (UCB, $K=2$ )	0.0706	0.0000
ALMAB-DC (UCB, $K=4$ )	0.0835	0.0000

Appendix

Table 15: Case P3 — Adaptive Spatial Sampling: GPU replication (

R=2{,}000

field realizations, MPS device). Metric: final IPV (lower is better). Best value in bold .

† Case P3 GPU IPV is higher due to the discrete $8\times 8$ grid; see Figure 15 .
Case	Metric	CPU	GPU	$\|\boldsymbol{\Delta}\|$	OK?
P2 — Dose-Response	Median Regret $\downarrow$	0.00263	0.00263	0.0000	Yes
P3 — Spatial GPE	Final IPV $\downarrow$	0.061	0.072 †	0.011	Yes †

Appendix

Table 16: GPU vs. CPU summary for ALMAB-DC (UCB), Cases P2–P3 (

R=2000

). All cases satisfy

|\Delta|<2\hat{\sigma}/\sqrt{R}

Method	Final Median RMSE	Final Median APV
Max-Variance	0.312	0.215
Space-Filling	0.341	0.198
Random	0.489	0.341

Appendix

Table 17: Swiss Rainfall application: final performance after budget of 30 observed gauges (10 random initializations). RMSE and APV on the

\log(1+\mathrm{rainfall})

scale; lower values better for both metrics.

Explore similar work

Design Experiments to Compare Multi-armed Bandit Algorithms
Mar 6, 2026Huiling Meng, Ningyuan Chen, Xuefeng GaoMulti-Armed Bandits
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Rotman School of Management, University of Toronto, Toronto, Canada.
Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity
Aug 25, 2026Heng Zhang, Haotian Xiang, Konstantinos D. Polyzos +2Bayesian Optimization Active Learning
University of Georgia, Athens, GA, USA · University of California San Diego, La Jolla, CA, USA
Robust Sequential Experimental Design for A/B Testing
May 13, 2026Qianglin Wen, Xiangkun Wu, Chengchun Shi +4Causal Effect Estimation Model Misspecification
Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University, Kunming, China · School of Mathematical Sciences, Zhejiang University, Hangzhou, China · Department of Statistics, London School of Economics and Political Science, London, United Kingdom +3

Mar 6, 2026J/K move · Enter open · S save

Design Experiments to Compare Multi-armed Bandit Algorithms

Huiling Meng, Ningyuan Chen, Xuefeng Gao

Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Rotman School of Management, University of Toronto, Toronto, Canada.

Multi-Armed Bandits

Open paper AskPDF arXiv

Online platforms routinely compare multi-armed bandit algorithms, such as UCB and Thompson Sampling, to select the best-performing policy. Unlike standard A/B tests for static treatments, each run of a bandit algorithm over

T

users produces only one trajectory, because the algorithm's decisions depend on all past interactions. Reliable inference therefore demands many independent restarts of the algorithm, making experimentation costly and delaying deployment decisions. We propose Artificial Replay (AR) as a new experimental design for this problem. AR first runs one policy and records its trajectory. When the second policy is executed, it reuses a recorded reward whenever it selects an action the first policy already took, and queries the real environment only otherwise. We develop a new analytical framework for this design and prove three key properties of the resulting estimator: it is unbiased; it requires only

T + o(T)

user interactions instead of

2T

for a run of the treatment and control policies, nearly halving the experimental cost when both policies have sub-linear regret; and its variance grows sub-linearly in

T

, whereas the estimator from a naïve design has a linearly-growing variance. Numerical experiments with UCB, Thompson Sampling, and

ε

-greedy policies confirm these theoretical gains.

Abstract

Explore similar work

Design Experiments to Compare Multi-armed Bandit Algorithms

Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity

Robust Sequential Experimental Design for A/B Testing