Reinforcement Learning as (Discrete) Potential Theory

Christopher Connolly

cs.LGAug 17, 2026

Reinforcement Learning as (Discrete) Potential Theory

Authors: Christopher Connolly

Organizations: Computer Science Laboratory SRI International

Abstract

Reinforcement learning (RL) theory fundamentally depends on probability theory through the Markov chain. There is a deep connection between probability theory and potential theory. This paper reviews that connection and explores the potential-theoretic viewpoint for core reinforcement learning representations and algorithms under a fixed-policy assumption. This viewpoint may offer a path for improved sample efficiency and formal constraints that can be applied to RL. When the fixed-policy assumption is relaxed, the linear potential theory framework can be naturally extended to the nonlinear case.

Figures & tables

Figure 1: Discrete harmonic value function on an 80x80 grid domain where absorbing states are shown with P=1 and P=0. In this example, values at grid edges are allowed to float.

Figure 2: A distributed reward function for a Poisson-style RL problem.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Mathematical methods of reinforcement learning
Jul 8, 2026Denis Belomestny, Alexander Gasnikov, Egor Gladin +5Markov Decision Processes Reinforcement Learning
Duisburg-Essen University, HSE University · Innopolis University, MIPT, HSE University · HSE University +1
PAC Guarantees for Reinforcement Learning: Sample Complexity, Coverage, and Structure
Mar 1, 2026Joshua SteierFunction Approximation in RL Reinforcement Learning
Theoretical Foundations of $\max$ @ $k$ Reinforcement Learning
Jul 20, 2026Riccardo Poiani, Martino Bernasconi, Andrea CelliReinforcement Learning Policy Learning
Bocconi University

Jul 8, 2026J/K move · Enter open · S save

Mathematical methods of reinforcement learning

Denis Belomestny, Alexander Gasnikov, Egor Gladin, Alexey Naumov+4

Duisburg-Essen University, HSE University · Innopolis University, MIPT, HSE University · HSE University+1

Markov Decision Processes Reinforcement Learning Function Approximation in RL

Open paper AskPDF arXiv

Reinforcement learning (RL) is increasingly grounded in tools from probability, optimization, and operator theory. This survey organizes the mathematical structures that underpin the design and analysis of modern algorithms in RL. We begin from Markov decision processes (MDPs) and the Bellman operators, emphasizing contraction mappings, monotonicity, and fixed-point theory that yield convergence guarantees and rates for value and policy iteration, and temporal-difference schemes. We then develop the optimization perspective: stochastic approximation and martingale methods, convex duality and the role of regularization linking mirror/proximal methods. Function approximation is treated through linear and non-linear settings, covering stabilization, error decomposition, and sample-complexity via concentration inequalities for dependent data and mixing processes. We further cover off-policy evaluation/learning, constrained RL and constrained MDPs (CMDPs). Throughout we unify algorithmic templates under common operator and variational lenses, highlighting both finite-sample bounds and asymptotic results. Our presentation is intended to provide a unified mathematical entry point for researchers in probability, optimization, and statistics interested in reinforcement learning.

Abstract

Reinforcement Learning

Explore similar work

Mathematical methods of reinforcement learning

PAC Guarantees for Reinforcement Learning: Sample Complexity, Coverage, and Structure

Theoretical Foundations of max⁡\maxmax@kkk Reinforcement Learning

Theoretical Foundations of $\max$ @ $k$ Reinforcement Learning