Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence

Shengbo Wang

cs.LGMar 3, 2026

Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence

Authors: Shengbo Wang

Organizations: Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California

Abstract

We study reinforcement learning in infinite-horizon discounted Markov decision processes with continuous state spaces, where data are generated online from a single trajectory under a Markovian behavior policy. To avoid maintaining an infinite-dimensional, function-valued estimate, we propose the novel Q-Measure-Learning, which learns a signed empirical measure supported on visited state-action pairs and reconstructs an action-value estimate via kernel integration. The method jointly estimates the stationary distribution of the behavior chain and the Q-measure through coupled stochastic approximation, leading to an efficient weight-based implementation with $O(n)$ memory and $O(n)$ computation cost per iteration. Under uniform ergodicity of the behavior chain, we prove almost sure sup-norm convergence of the induced Q-function to the fixed point of a kernel-smoothed Bellman operator. We also bound the approximation error between this limit and the optimal $Q^*$ as a function of the kernel bandwidth. To assess the performance of our proposed algorithm, we conduct RL experiments in a two-item inventory control setting.

Figures & tables

q(a)\leftarrow\dfrac{\sum_{k=1}^{n}\mathbf{W}_{k}\kappa((X_{n+1},a),Z_{k-1})}{\sum_{k=0}^{n}\mathbf{u}_{k}\kappa((X_{n+1},a),Z_{k})}

Algorithm 1 Q-Measure-Learning for finite

\mathbb{A}

Figure 1: Left: estimated discounted return of the greedy policy induced by $q_{n}$ . Right: $Q^{*}$ estimation error, where the curves exhibit stabilization/decrease consistent with convergence.

Figure 2: Greedy policy induced by $q_{n}$ compared with the DP policy on a grid.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Visitation count under the behavior policy: the entire state space is well-covered.

Panel of Figure 4 — Figure 4: Q-Measure estimate $q_{n}(\cdot,a)$ versus the DP benchmark, $a=(0,2)$ and $(0,5)$ .

Panel of Figure 5 — Figure 5: Partial visitation of inventory states: the behavior policy explores only the bottom-left corner of the state space. The Q-measure estimates of $Q^{*}$ are accurate in the bottom-left region, but are off in the top-right region.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
Aug 23, 2026Shengbo WangQ-Learning Function Approximation in RL
Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California
Commit to the Bit: Reactive Reinforcement Learning Done Right
May 27, 2026Onno Eberhard, Claire Vernade, Michael MuehlebachReinforcement Learning Partially Observable RL
Max Planck Institute for Intelligent Systems, Tübingen, Germany · University of Tübingen · University of Technology Nuremberg.
Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation
May 1, 2026Andrzej Ruszczynski, Tiangang ZhangMarkov Decision Processes Reinforcement Learning
Department of Management Science and Information Systems, Rutgers University

Aug 23, 2026J/K move · Enter open · S save

Q-Learning with Stable Infinite-Dimensional Linear Function Approximation

Shengbo Wang

Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California

Q-Learning Function Approximation in RL Linear Function Approximation Stochastic Approximation

Open paper AskPDF arXiv

Q-learning with linear function approximation can be unstable because an arbitrary approximation architecture need not preserve the Bellman contraction. We develop a stable infinite-dimensional linear function approximation framework for Q-learning from a single Markovian behavior-policy trajectory. The learning variable is a coefficient field

θ\in C(\mathbb L)

on a compact latent metric space

(\mathbb L,ρ)

. The framework uses a reconstruction operator that maps

θ

to a continuous Q-function and a compression operator that maps Bellman updates back to latent coordinates. Nonexpansiveness of both operators induces a contractive latent Bellman map on

C(\mathbb L)

, with a unique fixed point

θ^*

whose reconstruction approximates the optimal Q-function up to representation error. We propose two stochastic approximation (SA) algorithms and establish their sup-norm convergence bounds with a leading term of order

\widetilde O(n^{-1/2})

. The infinite-dimensional formulation provides a powerful abstraction for identifying the structures that govern statistical difficulty. Smoothness of the compression map in

ρ

is inherited by

θ^*

and the SA iterates, allowing uniform estimation errors to be controlled through covering numbers of

(\mathbb L,ρ)

rather than the dimension of

C(\mathbb L)

. Remarkably, the SA algorithms we propose are agnostic to the choice of

ρ

, and thus can automatically adapt to both the smoothness and the geometry. We further illustrate the framework through Q-measure-learning with linear density approximation and output-layer neural weight training under a frozen pretrained network.

Abstract

Reinforcement Learning

Explore similar work

Q-Learning with Stable Infinite-Dimensional Linear Function Approximation

Commit to the Bit: Reactive Reinforcement Learning Done Right

Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation