Action-Driven Processes for Continuous-Time Control

Ruimin He; Shaowei Lin

stat.MLOct 30, 2025

Action-Driven Processes for Continuous-Time Control

Authors: Ruimin He, Shaowei Lin

Organizations: Ukusan Pte Ltd, Singapore

Abstract

At the heart of reinforcement learning are actions -- decisions made in response to observations of the environment. Actions are equally fundamental in the modeling of stochastic processes, as they trigger discontinuous state transitions and enable the flow of information through large, complex systems. In this paper, we unify the perspectives of stochastic processes and reinforcement learning through action-driven processes, and illustrate their application to spiking neural networks. Leveraging ideas from control-as-inference, we show that minimizing the Kullback-Leibler divergence between a policy-driven true distribution and a reward-driven model distribution for a suitably defined action-driven process is equivalent to maximum entropy reinforcement learning.

Figures & tables

Panel of Figure 1 — Figure 1: A circuit model of an integrate-and-fire network

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Jun 2, 2026Saket Tiwari, Tejas Kotwal, George KonidarisSoft Actor-Critic Offline Reinforcement Learning
Department of Computer Science Brown University · Department of Applied Mathematics Brown University
Discretizing Reward Models
Jun 19, 2026Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4Responses
1Carnegie Mellon University · 2Meta Superintelligence Labs
Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
Dec 1, 2025Sebastian Sanokowski, Kaustubh Patil, Majid KhadivDiffusion-Based Reinforcement Learning Methods Markov Decision Processes
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich · Practical Project Student Exchange Program, Technical University Munich · MIT World Peace University

Jun 2, 2026, cs.LGJ/K move · Enter open · S save

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Saket Tiwari, Tejas Kotwal, George Konidaris

Department of Computer Science Brown University · Department of Applied Mathematics Brown University

Soft Actor-Critic Offline Reinforcement Learning

Open paper AskPDF arXiv

We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.

Abstract

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Discretizing Reward Models

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning