Bayesian -Explorer for Minimax Model Estimation in MDPs
Organizations: University of Southern California
Abstract
We study active exploration for minimax transition estimation in unknown Markov decision processes (MDPs). Frequentist -Explorer~\citep{gu2026kappa} addresses this problem through a parameterized family of exploration objectives, but requires confidence-bound constructions and joint optimization over plausible transition kernels and occupancy measures, introducing a computational burden that limits its practical applicability. We develop Bayesian -Explorer, which simplifies this procedure by placing Dirichlet priors on the transition distributions. Under the posterior mean estimator, the posterior expected squared error equals the total posterior variance, providing a closed-form exploration signal. The algorithm plans under the posterior mean model, eliminating confidence-set optimization. Under the stated assumptions, this simpler planning procedure retains the surrogate convergence rate of frequentist -Explorer. We further bound the maximum posterior variance relative to the optimal asymptotic minimax allocation benchmark. Experiments on Pendulum, Mountain Car, and Taxi demonstrate lower worst-case transition-estimation errors than the tested baselines.
Figures & tables
RL Exploration
Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.
See all 235 papers