In-Context Pure Exploration in Continuous Decision Spaces

Alessio Russo; Yin-Ching Lee; Ryan Welch; Aldo Pacchiano

Figure 5: Induction diagram used in the proof of Lemma 8 .

Figure 6: Sequential encoder and query-pooling readout used by the inference network

I_{\phi}

, the critic

Q_{\psi}

, and the learned TD3 actors.

Figure 7: Binary search: visualization of how C-ICPE explores in

6

dimensions. From left to right: the query action (left), posterior mean (middle), and posterior standard deviation (right) along an exploration trajectory in the noisy binary search problem.

Figure 8: Effect of Ackley function’s parameters on output range across multiple dimensions: (a) range vs

b

; (b) range vs

c

.

Figure 9: Visualization of how C-ICPE explores in the max-value estimation problem. From left to right: the query action (left), posterior mean (middle), and posterior standard deviation (right) along an exploration trajectory in the GP value estimation problem. Darker samples are queried earlier in the trajectory, while lighter samples are queried later.

Environment	$d$	$\varepsilon$	$t_{\max}$
$\varepsilon$ -Best-Arm	15	0.005	150
GP-value estimation	1	0.2	60
Geochemical	2	0.15	150

Appendix

Table 1: Sample budget

t_{\max}

for setting where

t_{max}\neq 100

Figure 10: Results for Binary Search Problem with fixed confidence

\delta=0.1

and

N=100

across different dimensions at the most hardest

(\varepsilon,\sigma)

setting: (top) survival function of

\tau

; (bottom) inference uncertainty convergence.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.1$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.025$	$\sigma=0.05$
6	C-ICPE-TD3	$0.895$ [.886,.902]	$0.898$ [.886,.908]	$0.903$ [.893,.914]	0.901 [.889,.912]
	C-ICPE-TS	0.904 [.895,.912]	0.907 [.899,.913]	0.909 [.899,.920]	$0.900$ [.893,.909]
	C-ICPE-TTPS	$0.897$ [.890,.905]	$0.903$ [.893,.913]	$0.900$ [.893,.907]	$0.898$ [.889,.905]
	C-ICPE-uniform	$0.901$ [.894,.908]	$0.904$ [.893,.914]	$0.854$ [.835,.874]	$0.688$ [.666,.709]
8	C-ICPE-TD3	$0.903$ [.897,.911]	$0.898$ [.890,.907]	$0.901$ [.891,.912]	$0.899$ [.890,.906]

Appendix

Table 2: Binary search: accuracy (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.1$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.025$	$\sigma=0.05$
6	C-ICPE-TD3	10.4 [10.2,10.6]	$15.5$ [15.2,15.9]	$13.3$ [12.5,14.2]	$16.3$ [15.7,16.8]
	C-ICPE-TS	$11.1$ [11.0,11.3]	15.3 [15.2,15.4]	12.0 [11.9,12.0]	14.4 [14.3,14.6]
	C-ICPE-TTPS	$11.3$ [11.0,11.6]	$16.2$ [16.1,16.3]	$12.4$ [12.1,12.6]	$15.1$ [14.9,15.4]
	C-ICPE-uniform	$36.3$ [36.0,36.7]	$54.1$ [52.4,56.9]	$60.0$ [58.2,62.3]	$57.3$ [56.1,58.1]
8	C-ICPE-TD3	11.8 [11.7,11.9]	17.3 [17.0,17.5]	$15.8$ [15.0,16.8]	$19.7$ [19.1,20.2]

Appendix

Table 3: Binary search: sample complexity (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

Figure 11: Results for

\epsilon

-Best-Arm Identification Problem with fixed confidence

\delta=0.005

and

N=100/150

across different dimensions at the most hardest

(\varepsilon,\sigma)

setting: (top) survival function of

\tau

; (bottom) inference uncertainty convergence.

$d$	Method	$\varepsilon=0.02$		$\varepsilon=0.005$
		$\sigma=0.005$	$\sigma=0.01$	$\sigma=0.00125$	$\sigma=0.0025$
5	C-ICPE-TD3	$0.925$ [.907,.943]	$0.915$ [.905,.926]	$0.898$ [.890,.907]	$0.902$ [.893,.912]
	C-ICPE-TS	$0.939$ [.912,.958]	$0.939$ [.929,.947]	$0.908$ [.890,.930]	$0.904$ [.891,.917]
	C-ICPE-TTPS	$0.934$ [.922,.944]	$0.938$ [.930,.944]	$0.902$ [.891,.911]	$0.900$ [.893,.909]
	C-ICPE-uniform	$0.936$ [.925,.946]	$0.931$ [.919,.943]	$0.922$ [.914,.928]	$0.927$ [.918,.935]
	LT&S round-robin	1.000 [1.000,1.000]	1.000 [1.000,1.000]	1.000 [1.000,1.000]	1.000 [1.000,1.000]

Appendix

Table 4:

\epsilon

-best arm: accuracy (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

$d$	Method	$\varepsilon=0.02$		$\varepsilon=0.005$
		$\sigma=0.005$	$\sigma=0.01$	$\sigma=0.00125$	$\sigma=0.0025$
5	C-ICPE-TD3	$7.4$ [7.3,7.5]	$7.5$ [7.4,7.5]	$9.4$ [9.2,9.6]	$9.4$ [9.3,9.5]
	C-ICPE-TS	$7.3$ [7.2,7.5]	$7.6$ [7.5,7.6]	$8.9$ [8.6,9.1]	$9.0$ [8.8,9.2]
	C-ICPE-TTPS	$7.3$ [7.1,7.4]	$7.5$ [7.4,7.5]	$8.7$ [8.6,8.9]	$8.8$ [8.7,9.0]
	C-ICPE-uniform	$8.3$ [8.2,8.4]	$8.2$ [8.0,8.4]	$10.0$ [9.9,10.1]	$10.1$ [10.0,10.3]
	LT&S round-robin	5.0 [5.0,5.0]	5.0 [5.0,5.0]	5.0 [5.0,5.0]	5.0 [5.0,5.0]

Appendix

Table 5:

\epsilon

-best arm: sample complexity (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

Figure 12: Results for Ackley function with fixed confidence

\delta=0.1

and

N=100

across different dimensions at the most hardest

(\varepsilon,\sigma)

setting: (top) survival function of

\tau

; (bottom) inference uncertainty convergence.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.1$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.025$	$\sigma=0.05$
3	C-ICPE-TD3	$15.3$ [14.6,15.9]	$18.9$ [18.2,19.6]	$18.9$ [18.2,19.8]	$20.6$ [19.9,21.3]
	C-ICPE-TS	14.9 [14.5,15.5]	$18.8$ [18.1,19.7]	17.5 [16.9,18.0]	$19.3$ [18.5,20.2]
	C-ICPE-TTPS	$15.4$ [14.8,16.1]	18.7 [18.1,19.4]	$18.0$ [17.3,18.7]	19.2 [18.6,19.9]
	C-ICPE-uniform	$37.6$ [34.4,40.2]	$47.9$ [46.3,49.4]	$50.5$ [48.6,52.2]	$51.7$ [49.8,53.7]
4	C-ICPE-TD3	$25.9$ [25.0,26.7]	$32.8$ [31.8,33.9]	$31.5$ [30.5,32.6]	$33.9$ [33.1,34.7]

Appendix

Table 6: Ackley: sample complexity (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.1$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.025$	$\sigma=0.05$
3	C-ICPE-TD3	0.915 [.902,.927]	$0.904$ [.894,.914]	$0.898$ [.888,.906]	$0.908$ [.899,.918]
	C-ICPE-TS	$0.896$ [.886,.904]	$0.899$ [.890,.909]	$0.899$ [.887,.906]	$0.907$ [.895,.917]
	C-ICPE-TTPS	$0.902$ [.894,.913]	0.907 [.894,.919]	0.911 [.900,.920]	0.912 [.899,.924]
	C-ICPE-uniform	$0.902$ [.886,.915]	$0.905$ [.890,.918]	$0.906$ [.894,.914]	$0.890$ [.868,.910]
	TPE	$0.098$ [.072,.124]	$0.136$ [.106,.166]	$0.016$ [.005,.027]	$0.028$ [.014,.042]

Appendix

Table 7: Ackley: accuracy (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

Figure 13: Results for GP value estimation with fixed confidence

\delta=0.1

and

N=100

across different dimensions at the most hardest

(\varepsilon,\sigma)

setting: (a) survival function of

\tau

; (b) inference uncertainty convergence.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.15$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.0375$	$\sigma=0.075$
1	C-ICPE-TD3	0.906 [.878,.928]	$0.910$ [.883,.932]	0.920 [.890,.942]	$0.930$ [.903,.948]
	C-ICPE-uniform	$0.902$ [.889,.914]	0.927 [.915,.938]	$0.901$ [.887,.916]	0.931 [.915,.946]
	Uniform bin	$0.130$ [.100,.160]	$0.176$ [.143,.209]	$0.230$ [.193,.267]	$0.226$ [.189,.263]
	Uniform top 5%	$0.648$ [.606,.690]	$0.672$ [.631,.713]	$0.762$ [.725,.799]	$0.696$ [.656,.736]

Appendix

Table 8: GP value estimation: accuracy (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

$d$	Method	$\varepsilon=0.2$		$\varepsilon=0.15$
		$\sigma=0.05$	$\sigma=0.1$	$\sigma=0.0375$	$\sigma=0.075$
1	C-ICPE-TD3	13.7 [12.3,15.4]	22.7 [21.8,23.8]	39.9 [36.6,43.4]	43.7 [41.3,45.8]
1	C-ICPE-uniform	$31.7$ [30.6,32.8]	$41.0$ [39.8,42.1]	$41.6$ [39.5,43.9]	$49.0$ [46.7,51.2]

Appendix

Table 9: GP value estimation: sample complexity (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

Figure 14: Robustness to prior misspecification on the 20D noisy binary search problem (

\varepsilon=0.1

and

\sigma=0.05

). Each heatmap reports the mean accuracy and the confidence intervals under varying Beta prior hyperparameters

(\alpha,\beta)\in\{0.5,1,3,5,7\}

. The white box indicates the matched prior (

\alpha=\beta=1

) during training.

Figure 15: Robustness to prior misspecification on the 15D

\varepsilon

-best-arm identification problem (

\varepsilon=0.005

and

\sigma=0.0025

). Each heatmap reports the mean accuracy and the confidence intervals under varying Beta prior hyperparameters

(\alpha,\beta)\in\{0.5,1,3,5,7\}

. The white box indicates the matched prior (

\alpha=\beta=1

) during training.

Figure 16: Robustness to prior misspecification on the 5D Ackley function (

\varepsilon=0.1

and

\sigma=0.05

). Each heatmap reports the mean accuracy and the confidence intervals under varying Beta prior hyperparameters

(\alpha,\beta)\in\{0.5,1,3,5,7\}

. The white box indicates the matched prior (

\alpha=\beta=1

) during training.

Figure 17: Robustness to prior misspecification on the GP value estimation problem (

\varepsilon=0.15

and

\sigma=0.075

). Each heatmap reports the mean accuracy and the confidence intervals under varying Beta prior hyperparameters

(\alpha,\beta)\in\{0.5,1,3,5,7\}

. The white box indicates the matched prior (

\alpha=\beta=1

) during training.

Figure 18: Example copper concentration in a 2D region in the geochemical exploration task. Red regions indicate concentration of copper within

\epsilon

of the maximum value.

Figure 19: Left: Normalized log-copper concentration in the Kingman region (southeastern California / southern Nevada). Each point is a soil sample from the USGS National Geochemical Survey [ , 2004 ] ; the red star marks the location of peak GP posterior mean. Right: ASTER satellite image of the Mountain Pass rare earth mine (35.5 ∘ N, 115.5 ∘ W) within this region, acquired March 28, 2010. The mine area is visible as the light-colored open pit in the upper center of the image. Credit: NASA/GSFC/METI/ERSDAC/JAROS, and U.S./Japan ASTER Science Team [ NASA/GSFC/METI/ERSDAC/JAROS, and U.S./Japan ASTER Science Team, 2010 ] .

Figure 20: All 17 geographic regions used in the geochemical experiment. Each panel shows soil sample locations colored by normalized log-copper concentration. Regions are split into training and evaluation sets; evaluation regions have spatial structure not seen during meta-training.

Figure 21: ELBO training curves for sparse variational GP fits across all 17 regions (1000 Adam iterations,

M=500

inducing points). All regions converge smoothly, indicating well-behaved GP fits.

		$\varepsilon=0.2$	$\varepsilon=0.15$
		$\sigma=0$	$\sigma=0$
2	C-ICPE-TD3	0.913 [.894,.931]	$0.916$ [.904,.927]
	C-ICPE-TS	$0.913$ [.894,.930]	0.925 [.905,.944]
	TPE	$0.438$ [.394,.482]	$0.526$ [.482,.570]
	CMA-ES	$0.484$ [.440,.528]	$0.514$ [.470,.558]
	GP-logEI	$0.564$ [.520,.608]	$0.724$ [.685,.763]

Appendix

Table 10: Geochem: accuracy (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

		$\varepsilon=0.2$	$\varepsilon=0.15$
		$\sigma=0$	$\sigma=0$
2	C-ICPE-TD3	$23.9$ [21.6,26.2]	$44.4$ [42.4,46.5]
2	C-ICPE-TS	22.0 [20.0,23.9]	38.7 [34.2,43.5]

Appendix

Table 11: Geochem: sample complexity (mean and 95% CI) for every

(d,\varepsilon,\sigma)

configuration.

Figure 22: Results for geochemical problem with fixed confidence

\delta=0.1

and

N=150

across different dimensions at the most hardest

\varepsilon

setting: (a) survival function of

\tau

; (b) inference uncertainty convergence.

Figure 23: Robustness to prior misspecification on the Geochemical exploration (

\varepsilon=0.15

). Each heatmap reports the mean accuracy and the confidence intervals under varying Beta prior hyperparameters

(\alpha,\beta)\in\{0.5,1,3,5,7\}

. The white box indicates the matched prior (

\alpha=\beta=1

) during training.

In-Context Pure Exploration in Continuous Decision Spaces

Abstract

Figures & tables

Appendix

Sequential Hypothesis Testing

ICMAPE: In-Context Multiagent Pure Exploration

RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits

Active Context Selection Improves Simple Regret in Contextual Bandits

Abstract

Sequential Hypothesis Testing

Explore similar work

ICMAPE: In-Context Multiagent Pure Exploration

RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits

Active Context Selection Improves Simple Regret in Contextual Bandits