Online Statistical Inference of Constant Sample-averaged Q-Learning

Saunak Kumar Panda; Tong Li; Ruiqi Liu; Yisha Xiang

stat.MLMar 27, 2026

Online Statistical Inference of Constant Sample-averaged Q-Learning

Authors: Saunak Kumar Panda, Tong Li, Ruiqi Liu, Yisha Xiang

Organizations: Department of Industrial Engineering, University of Houston · Department of Mathematics and Statistics, Texas Tech University

Abstract

Reinforcement learning algorithms have been widely used for decision-making tasks in various domains. However, the performance of these algorithms can be impacted by high variance and instability, particularly in environments with noise or sparse rewards. In this paper, we propose a framework to perform statistical online inference for a sample-averaged Q-learning approach. We adapt the functional central limit theorem (FCLT) for the modified algorithm under some general conditions and then construct confidence intervals for the Q-values via random scaling. We conduct experiments to perform inference on both the modified approach and its traditional counterpart, Q-learning using random scaling and report their coverage rates and confidence interval widths on two problems: a grid world problem as a simple toy example and a dynamic resource-matching problem as a real-world example for comparison between the two solution approaches.

Figures & tables

		$n$ = 2000	$n$ = 6000	$n$ = 10000
$\eta=0.1,\sigma=2$
Cov. for $\mathbf{Q}^{*}$ (%)	RS for Vanilla QL	99.0 (0.054)	97.5 (0.021)	96.0 (0.0153)
Cov. for $\mathbf{Q}^{*}$ (%)	RS for Sample-averaged QL	99.0 (0.075)	99.0 (0.025)	99.0 (0.0158)
Cov. for $\mathbb{E}_{\mathbf{Q}_{\eta}}\mathbf{Q}$ (%)	RS for Vanilla QL	99.0 (0.054)	98.0 (0.021)	97.5 (0.0153)
Cov. for $\mathbb{E}_{\mathbf{Q}_{\eta}}\mathbf{Q}$ (%)	RS for Sample-averaged QL	99.0 (0.075)	99.0 (0.025)	99.0 (0.0158)
Length	RS for Vanilla QL	0.735	0.285	0.21

Table 1: Statistical Inference for Q-learning for Grid World Problem

		$n$ = 500	$n$ = 1000	$n$ = 2000
$\eta=0.1,\sigma=2$
Cov. for $\mathbf{Q}^{*}$ (%)	RS for Vanilla QL	99.9	99.9	99.9
Cov. for $\mathbf{Q}^{*}$ (%)	RS for Sample-averaged QL	99.9	99.9	99.9
Cov. for $\mathbb{E}_{\mathbf{Q}_{\eta}}\mathbf{Q}$ (%)	RS for Vanilla QL	99.9	99.9	99.9
Cov. for $\mathbb{E}_{\mathbf{Q}_{\eta}}\mathbf{Q}$ (%)	RS for Sample-averaged QL	99.9	99.9	99.9
Length	RS for Vanilla QL	299.4	199.5	113.8

Table 2: Statistical Inference for Q-learning for Dynamic Matching Problem

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Online Inference for Quantile Distributional Reinforcement Learning
Aug 13, 2026Zijie Cheng, Yang Peng, Zhihua ZhangDistributional RL Reinforcement Learning
School of Mathematical Sciences, Peking University · Yau Mathematical Sciences Center, Tsinghua University
Statistical Inference for Policy Evaluation with Temporal Difference Learning
Oct 21, 2024Weichen Wu, Gen Li, Yuting Wei +1Parameter Estimation Confidence Region Estimation
The Voleon Group, New York, NY, USA · Department of Statistics, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. +1
Non-asymptotic Convergence of Average-reward Q-learning with Options
Oct 14, 2026Kintan Saha, Ahana Deb, Anders Jonsson +1Q-Learning Average-Reward RL
Indian Institute of Science · Universitat Pompeu Fabra · Univ. Lille, Inria, CNRS- CRIStAL, Centrale Lille

Aug 13, 2026J/K move · Enter open · S save

Online Inference for Quantile Distributional Reinforcement Learning

Zijie Cheng, Yang Peng, Zhihua Zhang

School of Mathematical Sciences, Peking University · Yau Mathematical Sciences Center, Tsinghua University

Distributional RL Reinforcement Learning Temporal-Difference Learning

Open paper AskPDF arXiv

In this paper, we study online statistical inference for quantile-based distributional policy evaluation, which aims to characterize the distribution of discounted cumulative rewards under a given policy. To obtain a finite-dimensional representation of the return distribution, we consider the fixed point

\bmη_m

of the quantile-projected distributional Bellman equation, parameterized by the quantile vector

\bmθ_m

. We develop two online inference procedures for

\bmθ_m

, covering synchronous and asynchronous sampling from a generative model as well as Markovian data. The first method uses the trajectory of quantile temporal-difference learning (QTD). We establish functional central limit theorems for the partial sums of QTD iterates and construct a recursively computable random-scaling statistic. The resulting confidence intervals have asymptotically exact coverage at deterministic horizons, with length of order

T^{-1/2}

in probability. The second method constructs anytime-valid confidence intervals directly from samples. Exploiting the monotonicity of the quantile Bellman map, we develop a computationally explicit coordinate-shaving algorithm with finite-sample simultaneous coverage. Under positive limiting state-visit frequencies, the resulting confidence intervals have lengths of order

\sqrt{\log\log T/T}

, which we establish as classwise optimal through a matching lower bound. Together, these results provide complementary guarantees for online inference in quantile distributional reinforcement learning, combining asymptotically exact fixed-horizon inference with finite-sample anytime validity.

Abstract

Reinforcement Learning

Explore similar work

Online Inference for Quantile Distributional Reinforcement Learning

Statistical Inference for Policy Evaluation with Temporal Difference Learning

Non-asymptotic Convergence of Average-reward Q-learning with Options