cs.LG · 2607.21637 Copy arXiv ID · Jul 21, 2026 Save Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning Authors: Julian G. Soltes
Organizations: Regis University, USA
Abstract This paper explores the efficacy of quasi-Monte Carlo (QMC) weight initialization for meta-reinforcement learning within modern benchmark environments. Various sampling methods are used to bound a population-based search and aggregate an optimal prior from a baseline set of tasks. The QMC meta-priors show improvements in training convergence compared to modern orthogonal (SB3) defaults when extrapolated to similar unseen continuous control environments. In dissimilar tasks, the orthogonal orientation was globally superior for an unbiased search.
Explore similar work Jun 13, 2026 · Octave Oliviers, Glenn Vinnicombe Monte Carlo Autonomous Exploration Strategies
Jul 1, 2026 · Michael Y. Li, Anthony Zhan, Kanishk Gandhi +2 Monte Carlo Test-Time Scaling
Jun 13, 2026 · cs.LG J/K move · Enter open · S save
Octave Oliviers, Glenn Vinnicombe
Department of Engineering · University of Cambridge · Cambridge, UK
The asymptotic behaviour of Monte Carlo Exploring Starts (MCES) is a long-standing open question in reinforcement learning, even in the tabular setting. We investigated the convergence properties of tabular MCES by constructing examples in which the algorithm converges to suboptimal solutions. This paper presents new counterexamples for both initial-visit and first-visit MCES and gives a convergence-restoring modification for the initial-visit case. We show that stable suboptimal solutions may exist for initial-visit MCES with sample-average updates even when greedy actions are updated more often than non-greedy actions on average. However, by scaling learning rates inversely to update frequencies on a state-by-state basis, convergence to optimality is guaranteed. Unlike previous uniformisation methods, this modification is applicable to large-scale problems that require approximating the estimated value function. We then extend the example to show that sample-average first-visit MCES may also converge to suboptimal solutions. This largely settles a fundamental open problem and shows that exploring starts alone do not guarantee convergence to optimality. More broadly, these results highlight that convergence depends critically on the relative size and frequency of updates applied to different actions, making the choice of learning rates and the balance between exploration and exploitation central to the analysis of MCES and the implementation of scalable Monte Carlo control methods.