Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient

Lijun Bo; Yijie Huang; Chenhao Lu

math.OCOct 7, 2026

Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient

Authors: Lijun Bo, Yijie Huang, Chenhao Lu

Abstract

We develop a continuous-time deterministic policy gradient framework for reinforcement learning with reflected state dynamics, where the state process is governed by a controlled reflected stochastic differential equation on a hypercube. Under suitable regularity assumptions, we establish the connection between the value function and the Neumann Bellman equation, introduce an advantage-rate function that yields a deterministic policy gradient formula, and prove the martingale characterization theorem. Motivated by these theoretical results, we propose a continuous-time deep deterministic policy gradient algorithm for reflected stochastic systems, in which the Neumann boundary condition is imposed via either soft penalization or hard architectural constraint. We further quantify the discrepancy between the ideal continuous-time dynamics and the discretely sampled exploratory dynamics executed in practice, showing that the error decays as the time grid is refined and exploration noise vanishes. Our experiments on reservoir control problems illustrate the effectiveness of the RL framework, highlighting that boundary-aware methods substantially reduce Neumann boundary residuals and enhance learning stability.

Figures & tables

\displaystyle a_{t_{k}}=\Pi_{A}\left(\mu_{\phi}(t_{k},x_{t_{k}})+\eta_{t_{k}}\right),~\eta_{k}\sim\mathcal{N}(0,\sigma_{\mathrm{exp}}^{2}I_{d}).

Algorithm 1 CT-DDPG-R on the hypercube

Panel of Figure 1 — Figure 1 : Returns and boundary residual

Panel of Figure 2 — Figure 2 : Value functions and policies under different constraints

Figure 4 : Policy surfaces without boundary constraint at $t=0.5$ .

Figure 5 : Policy surfaces under soft constraint at $t=0.5$ .

Figure 6 : Policy surfaces under hard constraint at $t=0.5$ .

Figure 7 : Finite-difference reference policy at $t=0.5$ .

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Jun 2, 2026Saket Tiwari, Tejas Kotwal, George KonidarisSoft Actor-Critic Offline Reinforcement Learning
Department of Computer Science Brown University · Department of Applied Mathematics Brown University
Policy Gradient for Continuous-Time Robust Markov Decision Processes
Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaMarkov Decision Processes Policy Gradient
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR) · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR) · College of Computing and Data Science, Nanyang Technological University
Monotone Neural Policy Iteration for High-Dimensional First-Order Hamilton--Jacobi--Bellman Equations
Date pendingMinseok Kim, Yeongjong Kim, Namkyeong Cho +1Hamilton-Jacobi Reachability Model-Based Reinforcement Learning
Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Department of Mathematics, Pohang University of Science and Technology, Pohang, Republic of Korea. · Department of Mathematical Finance and Big Data, Gachon University, Seongnam, Republic of Korea +1

Jun 2, 2026, cs.LGJ/K move · Enter open · S save

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Saket Tiwari, Tejas Kotwal, George Konidaris

Department of Computer Science Brown University · Department of Applied Mathematics Brown University

Soft Actor-Critic Offline Reinforcement Learning

Open paper AskPDF arXiv

We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.

Abstract

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Policy Gradient for Continuous-Time Robust Markov Decision Processes

Monotone Neural Policy Iteration for High-Dimensional First-Order Hamilton--Jacobi--Bellman Equations