RL Control

RL: Reinforcement Learning

Follow this topic Ask about this topic

Reinforcement learning control studies how agents can learn actions that steer dynamic systems toward goals while managing costs, constraints, and uncertainty. Common approaches include actor-critic methods such as PPO, SAC, and TD3; model-based learning with predictive control; and offline learning from logged data, often combined with safety filters or physical models. Work spans simulated benchmarks and applications such as robotics, traffic, buildings, power grids, and fluid flow, with evaluation increasingly testing hardware transfer and changing conditions. A central tension is that high simulated reward or average performance does not ensure safe, reliable behavior under real-world disturbances, limited data, or distribution shift.

Latest papers

276 papers34 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

Mar 15, 2026

Adapting Critic Match Loss Landscape Visualization to Off-policy Reinforcement Learning

This work extends an established critic match loss landscape visualization method from online to off-policy reinforcement learning (RL), aiming to reveal the optimization geometry behind critic learning. Off-policy RL differs from stepwise online actor-critic learning in its replay-based data flow and target computation. Based on these two structural differences, the critic match loss landscape visualization method is adapted to the Soft Actor-Critic (SAC) algorithm by aligning the loss evaluation with its batch-based data flow and target computation, using a fixed replay batch and precomputed critic targets from the selected policy. Critic parameters recorded during training are projected onto a principal component plane, where the critic match loss is evaluated to form a 3-D landscape with an overlaid 2-D optimization path. Applied to a spacecraft attitude control problem, the resulting landscapes are analyzed both qualitatively and quantitatively using sharpness, basin area, and local anisotropy metrics, together with temporal landscape snapshots. Comparisons between convergent SAC, divergent SAC, and divergent Action-Dependent Heuristic Dynamic Programming (ADHDP) cases reveal distinct geometric patterns and optimization behaviors under different algorithmic structures. The results demonstrate that the adapted critic match loss visualization framework serves as a geometric diagnostic tool for analyzing critic optimization dynamics in replay-based off-policy RL-based control problems.

Jingyi Liu, Jian Guo, Eberhard Gill

Delft University of Technology, Delft, 2629 HS, the Netherlands

Loss Landscape Geometry RL Control Off-Policy RL

Mar 15, 2026

Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms

Reinforcement learning has proven its power on various occasions. However, its performance is not always guaranteed when system dynamics change. Instead, it largely relies on users' empirical experience. For reinforcement learning algorithms with an actor-critic structure, the critic neural network reflects the approximation and optimization process in the RL algorithm. Analyzing the performance of the critic neural network helps to understand the mechanism of the algorithm. To support systematic interpretation of such algorithms in dynamic control problems, this work proposes a critic match loss landscape visualization method for online reinforcement learning. The method constructs a loss landscape by projecting recorded critic parameter trajectories onto a low-dimensional linear subspace. The critic match loss is evaluated over the projected parameter grid using fixed reference state samples and temporal-difference targets. This yields a three-dimensional loss surface together with a two-dimensional optimization path that characterizes critic learning behavior. To extend analysis beyond visual inspection, quantitative landscape indices and a normalized system performance index are introduced, enabling structured comparison across different training outcomes. The approach is demonstrated using the Action-Dependent Heuristic Dynamic Programming algorithm on cart-pole and spacecraft attitude control tasks. Comparative analyses across projection methods and training stages reveal distinct landscape characteristics associated with stable convergence and unstable learning. The proposed framework enables both qualitative and quantitative interpretation of critic optimization behavior in online reinforcement learning.

Jingyi Liu, Jian Guo, Eberhard Gill

Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands

RL Control Actor-Critic Methods

Mar 15, 2026

Physics-Informed Policy Optimization via Analytic Dynamics Regularization

Reinforcement learning (RL) has achieved strong performance in robotic control; however, state-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity and often produce physically inconsistent actions. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. In this paper, we introduce a novel physics-informed RL framework, called PIPER, that seamlessly integrates physical constraints directly into neural policy optimization with analytical soft physics constraints. At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor's objective. This residual, extracted from a robot's simulator description, subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms. Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy, establishing a new paradigm for efficient and physically consistent robotic control.

Namai Chandra, Liu Mohan, Zhihao Gu +1

Electronic Systems, IIT Madras, India · EmPACT Lab, Nanyang Technological University, Singapore

Physics-Informed RL RL Control Robotic Control

Mar 13, 2026

Reinforcement Learning for Elliptical Cylinder Motion Control Tasks

The control of devices with limited input always bring attention to solve by research due to its difficulty and non-trival solution. For instance, the inverted pendulum is benchmarking problem in control theory and machine learning. In this work, we are focused on the elliptical cylinder and its motion under limited torque. The inspiration of the problem is from untethered magnetic devices, which due to distance have to operate with limited input torque. In this work, the main goal is to define the control problem of elliptic cylinder with limited input torque and solve it by Reinforcement Learning. As a classical baseline, we evaluate a two-stage controller composed of an energy-shaping swing-up law and a local Linear Quadratic Regulator (LQR) stabilizer around the target equilibrium. The swing-up controller increases the system's mechanical energy to drive the state toward a neighborhood of the desired equilibrium, a linearization of the nonlinear model yields an LQR that regulates the angle and angular-rate states to the target orientation with bounded input. This swing-up + LQR policy is a strong, interpretable reference for underactuated system and serves a point of comparison to the learned policy under identical limits and parameters. The solution shows that the learning is possible however, the different cases like stabilization in upward position or rotating of half turn are very difficult for increasing mass or ellipses with a strongly unequal perimeter ratio.

Pawel Marczewski, Paulina Superczynska, Jakub Bernat +1

Institute of Computing Science Poznan University of Technology Poznan, 61-138 · Institute of Automatic Control and Robotics Poznan University of Technology Poznan, 61-138

RL Control Feedback Control Constrained RL

Mar 12, 2026

Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.

Taeho Lee, Donghwan Lee

School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Adversarial MARL Reinforcement Learning Robust RL Adversarial Training RL Control Disturbance Rejection

Mar 12, 2026

Sim-to-reality adaptation for Deep Reinforcement Learning applied to an underwater docking application

Deep Reinforcement Learning (DRL) offers a robust alternative to traditional control methods for autonomous underwater docking, particularly in adapting to unpredictable environmental conditions. However, bridging the "sim-to-real" gap and managing high training latencies remain significant bottlenecks for practical deployment. This paper presents a systematic approach for autonomous docking using the Girona Autonomous Underwater Vehicle (AUV) by leveraging a high-fidelity digital twin environment. We adapted the Stonefish simulator into a multiprocessing RL framework to significantly accelerate the learning process while incorporating realistic AUV dynamics, collision models, and sensor noise. Using the Proximal Policy Optimization (PPO) algorithm, we developed a 6-DoF control policy trained in a headless environment with randomized starting positions to ensure generalized performance. Our reward structure accounts for distance, orientation, action smoothness, and adaptive collision penalties to facilitate soft docking. Experimental results demonstrate that the agent achieved a success rate of over 90% in simulation. Furthermore, successful validation in a physical test tank confirmed the efficacy of the sim-to-reality adaptation, with the DRL controller exhibiting emergent behaviors such as pitch-based braking and yaw oscillations to assist in mechanical alignment.

Alaaeddine Chaarani, Narcis Palomeras, Pere Ridao

Authors with the Faculty of Computer Science, Computer Vision and Robotics Research Group (vicorob) Universitat de Girona, 17003 Girona, Spain.

Underwater Robotics RL Control Sim-to-Real Transfer Robotics Simulation

Mar 12, 2026

Hybrid Energy-Aware Reward Shaping: A Unified Lightweight Physics-Guided Methodology for Policy Optimization

Deep reinforcement learning for continuous control often suffers from high variance, low energy efficiency, and poor generalization under distribution shift, as purely data-driven exploration ignores available physical structure. This paper proposes Hybrid Energy-Aware Reward Shaping (H-EARS), which encodes dominant energy terms -- assumed known a priori -- directly as reward potentials at O(n) per-step computation. H-EARS decomposes the shaping potential into task-oriented and energy-based components, supplemented by an action regularization term that deliberately modifies the optimization objective to enforce energy-efficient control. A complete theoretical foundation is established: functional independence of shaping and regularization, energy-based gradient enrichment under positive-definite Hessian conditions, convergence guarantees under function approximation, and approximate potential error bounds. Across four continuous control benchmarks and four baseline algorithms, H-EARS achieves consistent gains in convergence speed, policy stability, and final performance. High-fidelity vehicle simulations validate applicability in safety-critical settings under extreme road conditions.

Qijun Liao, Jue Yang, Yiting Kang +3

School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China · Jiangsu XCMG Construction Machinery Research Institute Co., Ltd., Jiangsu 221000, China

RL Control Reinforcement Learning Reward Shaping Reward Design for RL Physics-Informed RL

Mar 11, 2026

Muscle Synergy Priors Enhance Biomechanical Fidelity in Predictive Musculoskeletal Locomotion Simulation

Human locomotion emerges from high-dimensional neuromuscular control, making predictive musculoskeletal simulation challenging. We present a physiology-informed reinforcement-learning framework that constrains control using muscle synergies. We extracted a low-dimensional synergy basis from inverse musculoskeletal analyses of a small set of overground walking trials and used it as the action space for a muscle-driven three-dimensional model trained across variable speeds, slopes and uneven terrain. The resulting controller generated stable gait from 0.7-1.8 m/s and on

\pm

6

^{\circ}

grades and reproduced condition-dependent modulation of joint angles, joint moments and ground reaction forces. Compared with an unconstrained controller, synergy-constrained control reduced non-physiological knee kinematics and kept knee moment profiles within the experimental envelope. Across conditions, simulated vertical ground reaction forces correlated strongly with human measurements, and muscle-activation timing largely fell within inter-subject variability. These results show that embedding neurophysiological structure into reinforcement learning can improve biomechanical fidelity and generalization in predictive human locomotion simulation with limited experimental data.

Ilseung Park, Eunsik Choi, Jangwhan Ahn +1

Carnegie Mellon University · Department of Physical Education, Seoul National University, Seoul, Republic of Korea · Seoul National University +2

Musculoskeletal Modeling RL Control

Mar 9, 2026

Posterior Sampling Reinforcement Learning with Gaussian Processes for Continuous Control: Sublinear Regret Bounds for Unbounded State Spaces

We analyze the Bayesian regret of the Gaussian process posterior sampling reinforcement learning (GP-PSRL) algorithm. Posterior sampling is a heuristic for decision-making under uncertainty that has been used to develop successful algorithms for a variety of continuous control problems. However, theoretical work on GP-PSRL is limited. All known regret bounds either have a sub-optimal growth rate, require strong smoothness assumptions, or fail to properly account for the fact that the set of possible system states is unbounded. Through a recursive application of the Borell-Tsirelson-Ibragimov-Sudakov inequality, we show that, with high probability, the states actually visited by the algorithm are contained within a ball of near-constant radius. We then use the chaining method to control the regret suffered by GP-PSRL under weak smoothness conditions. Our main result is a Bayesian regret bound of the order

\widetilde{\mathcal{O}}(H\sqrt{γ_TT})

, where

H

is the horizon,

T

is the number of time steps and

γ_T

is the expected information gain. With this result, we resolve the limitations with prior theoretical work on PSRL, and provide the theoretical foundation and tools for analyzing PSRL in complex settings.

Hamish Flynn, Joe Watson, Ingmar Posner +1

Carnegie Mellon University · University of Oxford · Technical University of Darmstadt +2

RL Control Regret Minimization in RL Reinforcement Learning Bayesian RL Gaussian Processes Posterior Sampling

Mar 8, 2026

Reinforcement learning-based dynamic cleaning scheduling framework for solar energy system

Advancing autonomous green technologies in solar photovoltaic (PV) systems is key to improving sustainability and efficiency in renewable energy production. This study presents a reinforcement learning (RL)-based framework to autonomously optimize the cleaning schedules of PV panels in arid regions, where soiling from dust and other airborne particles significantly reduces energy output. By employing advanced RL algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), the framework dynamically adjusts cleaning intervals based on uncertain environmental conditions. The proposed approach was applied to a case study in Abu Dhabi, UAE, demonstrating that PPO outperformed SAC and traditional simulation optimization (Sim-Opt) methods, achieving up to 13% cost savings by dynamically responding to weather uncertainties. The results highlight the superiority of flexible, autonomous scheduling over fixed-interval methods, particularly in adapting to stochastic environmental dynamics. This aligns with the goals of autonomous green energy production by reducing operational costs and improving the efficiency of solar power generation systems. This work underscores the potential of RL-driven autonomous decision-making to optimize maintenance operations in renewable energy systems. In future research, it is important to enhance the generalization ability of the proposed RL model, while also considering additional factors and constraints to apply it to different regions.

Heungjo An

Department of Industrial Engineering, Kumoh National Institute of Technology, Gumi City, South Korea

RL Control Power Systems

Mar 7, 2026

Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.

Chenyang Miao

Xi’an Jiaotong University

Reinforcement Learning RL Control Robotic RL Data-Efficient Robot Learning

Mar 6, 2026

How to Model Your Crazyflie Brushless

The Crazyflie quadcopter is widely recognized as a leading platform for nano-quadcopter research. In early 2025, the Crazyflie Brushless was introduced, featuring brushless motors that provide around 50% more thrust compared to the brushed motors of its predecessor, the Crazyflie 2.1. This advancement has opened new opportunities for research in agile nano-quadcopter control. To support researchers utilizing this new platform, this work presents a dynamics model of the Crazyflie Brushless and identifies its key parameters. Through simulations and hardware analyses, we assess the accuracy of our model. We furthermore demonstrate its suitability for reinforcement learning applications by training an end-to-end neural network position controller and learning a backflip controller capable of executing two complete rotations with a vertical movement of just 1.8 meters. This showcases the model's ability to facilitate the learning of controllers and acrobatic maneuvers that successfully transfer from simulation to hardware. Utilizing this application, we investigate the impact of domain randomization on control performance, offering valuable insights into bridging the sim-to-real gap with the presented model. We have open-sourced the entire project, enabling users of the Crazyflie Brushless to swiftly implement and test their own controllers on an accurate simulation platform.

Alexander Gräfe, Christoph Scherer, Wolfgang Hönig +1

Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Germany · Intelligent Multi-Robot Coordination Lab, TU Berlin, Germany

Quadrotor Control Robotics Simulation Robot Dynamics Identification Sim-to-Real Transfer RL Control

Mar 4, 2026

Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control

Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. However, the behavior of offline RL algorithms under genuinely stochastic dynamics -- inherent to wireless systems due to fading, noise, and traffic mobility -- remains insufficiently understood. We address this gap by evaluating Bellman-based (Conservative Q-Learning), sequence-based (Decision Transformers), and hybrid (Critic-Guided Decision Transformers) offline RL methods in an open-access stochastic telecom environment (mobile-env). Our results show that Conservative Q-Learning consistently produces more robust policies across different sources of stochasticity, making it a reliable default choice in lifecycle-driven AI management frameworks. Sequence-based methods remain competitive and can outperform Bellman-based approaches when sufficient high-return trajectories are available. These findings provide practical guidance for offline RL algorithm selection in AI-driven network control pipelines, such as O-RAN and future 6G functions, where robustness and data availability are key operational constraints.

Nicolas Helson, Pegah Alizadeh, Anastasios Giovanidis

Ericsson Research, Massy, France

Conservative RL Offline RL RL Control Robust RL Decision Transformer

Mar 2, 2026

State-Action Inpainting Diffuser for Continuous Control with Delay

Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.

Dongqi Han, Wei Wang, Enze Zhang +1

Microsoft Research Asia · University of Pennsylvania · Shanghai Jiaotong University

Delayed RL Reinforcement Learning RL Control Model-Based RL Model-Free RL

Feb 27, 2026

RF-Agent: Automated Reward Function Design via Language Agent Tree Search

Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at https://github.com/deng-ai-lab/RF-Agent.

Ning Gao, Xiuhui Zhang, Xingyu Jiang +3

Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Reward Design for RL RL Control LLM-Guided RL

Feb 25, 2026

Two-Stage Active Distribution Network Voltage Control via LLM-RL Collaboration: A Hybrid Knowledge-Data-Driven Approach

The growing integration of distributed photovoltaics (PVs) into active distribution networks (ADNs) has exacerbated operational challenges, making it imperative to coordinate diverse equipment to mitigate voltage violations and enhance power quality. Although existing data-driven approaches have demonstrated effectiveness in the voltage control problem, they often require extensive trial-and-error exploration and struggle to incorporate heterogeneous information, such as day-ahead forecasts and semantic-based grid codes. Considering the operational scenarios and requirements in real-world ADNs, in this paper, we propose a hybrid knowledge-data-driven approach that leverages dynamic collaboration between a large language model (LLM) agent and a reinforcement learning (RL) agent to achieve two-stage voltage control. In the day-ahead stage, the LLM agent receives coarse region-level forecasts and generates scheduling strategies for on-load tap changer (OLTC) and shunt capacitors (SCs) to regulate the overall voltage profile. Then in the intra-day stage, based on accurate node-level measurements, the RL agent refines terminal voltages by deriving reactive power generation strategies for PV inverters. On top of the LLM-RL collaboration framework, we further propose a self-evolution mechanism for the LLM agent and a pretrain-finetune pipeline for the RL agent, effectively enhancing and coordinating the policies for both agents. The proposed approach not only aligns more closely with practical operational characteristics but also effectively utilizes the inherent knowledge and reasoning capabilities of the LLM agent, significantly improving training efficiency and voltage control performance. Comprehensive comparisons and ablation studies demonstrate the effectiveness of the proposed method.

Xu Yang, Chenhui Lin, Xiang Ma +4

Sichuan Energy Internet Research Institute, Tsinghua University, Chengdu 610213, China. · State Grid Zhejiang Electric Power Co., Ltd., Jinhua Power Supply Company, Jinhua 321017, China.

Power Systems RL Control Language Model-Based Control

Feb 25, 2026

Learning End-to-End Control for Omnidirectional Aerial Motion on Overactuated Tilt-rotor Quadrotors

While reinforcement learning (RL) has been successfully applied to conventional quadrotors for agile and robust flight, whether actuator-level RL can be reliably deployed on tilt-rotor aerial robots remains an open question, as the hybrid actuation coupling brushless rotors with rotational joints introduces a substantially harder sim-to-real gap. In this work, we propose an end-to-end RL framework for omnidirectional motion control on overactuated tilt-rotor quadrotors, directly mapping target poses to joint and rotor commands. The learning framework combines actuator-level simulation with an asymmetric actor-critic architecture for 6D pose-reaching. For reliable sim-to-real transfer on the hybrid actuation, we integrate system identification with minimal yet physically grounded domain randomization. The trained policy is deployed zero-shot on real hardware and evaluated across waypoint hovering, external disturbances, payload variation and trajectory tracking, together with simulated traversal of allocation-singular configurations. The policy is compared with a state-of-the-art NMPC baseline: NMPC attains lower steady-state position error, whereas the RL policy offers a more uniform orientation error across evaluations, transitions between poses faster, and requires less onboard computation.

Wentao Zhang, Zhaoqi Ma, Jinjie Li +8

DRAGON Lab, Department of Mechanical Engineering, The University of Tokyo, Tokyo, 113-8656, Japan · School of Computing and Data Science, The University of Hong Kong, 999077, HKSAR · Autonomous Systems Lab(ASL), ETH Zurich, 8092 Zurich, Switzerland

Quadrotor Control RL Control Sim-to-Real Transfer

Feb 20, 2026

Learning to Tune Pure Pursuit in Autonomous Racing: Joint Lookahead and Steering-Gain Control with PPO

Pure Pursuit (PP) is widely used in autonomous racing for real-time path tracking due to its efficiency and geometric clarity, yet performance is highly sensitive to how key parameters-lookahead distance and steering gain-are chosen. Standard velocity-based schedules adjust these only approximately and often fail to transfer across tracks and speed profiles. We propose a reinforcement-learning (RL) approach that jointly chooses the lookahead Ld and a steering gain g online using Proximal Policy Optimization (PPO). The policy observes compact state features (speed and curvature taps) and outputs (Ld, g) at each control step. Trained in F1TENTH Gym and deployed in a ROS 2 stack, the policy drives PP directly (with light smoothing) and requires no per-map retuning. Across simulation and real-car tests, the proposed RL-PP controller that jointly selects (Ld, g) consistently outperforms fixed-lookahead PP, velocity-scheduled adaptive PP, and an RL lookahead-only variant, and it also exceeds a kinematic MPC raceline tracker under our evaluated settings in lap time, path-tracking accuracy, and steering smoothness, demonstrating that policy-guided parameter tuning can reliably improve classical geometry-based control.

Mohamed Elgouhary, Amr S. El-Wakeel

The authors are with the Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA

Autonomous Racing Adaptive Control RL Control Feedback Control

Feb 20, 2026

Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.

Zhaoming Xie, Kevin Karol, Jessica Hodgins

Robotics AI Institute, Cambridge, Massachusetts, USA

RL Control Robot Imitation Learning Jacobian Regularization

Feb 16, 2026

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the

Q

-function collapses to the value function

V

, eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function

q

. However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle

V

and

q

into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates:

q

is learned from diffusion generators on

V

, and

V

is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarter

-

nearly doubling the second-best method.

Minh Nguyen

Google · University of Texas at Austin

Reinforcement Learning Continuous-Time RL RL Control Actor-Critic Methods

Jan 30, 2026

Agile Reinforcement Learning through Separable Neural Architecture and Applications

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Spline-based architectures such as Kolmogorov-Arnold Networks (KANs) have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks) for RL. SPAN adapts the KHRONOS framework with a learnable preprocessing layer. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, offline settings (Minari/D4RL) and a real-world datacenter HVAC control application. SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Despite incurring a per-step evaluation overhead of 1.2-1.8x, SPAN's superior convergence reliability yields an expected total training cost 1.3-6.3x lower than MLP baselines when accounting for convergence failures. In the HVAC application, SPAN reduces energy consumption in 9 of 12 months relative to MLP while simultaneously achieving a 1.1-3.4x reduction in thermal comfort violations across the evaluation year, demonstrating generalization to real-world engineering control. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high-performance alternative for learning efficient policies in resource-limited settings.

Rajib Mostakim, Reza T. Batley, Sourav Saha

Department of Mechanical Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

RL Control Neural Network Approximation Theory Sample-Efficient RL Heating, Ventilation, and Air Conditioning Control

Dec 19, 2025

The HydroGym Reinforcement Learning Platform for Fluid Dynamics

Modeling and controlling fluids is critical across science and engineering. Effective flow control can increase lift, reduce drag, enhance mixing, and attenuate noise, potentially unlocking new technologies. Yet controlling fluids is hard: the dynamics are high-dimensional, nonlinear, and multiscale. While reinforcement learning (RL) has recently succeeded in robotics and protein folding through shared benchmarks, fluid dynamics has resisted such progress: each controller is typically tuned to a single geometry and operating point, making results hard to accumulate, transfer, and compare. We introduce HydroGym, a solver-independent RL platform for flow control, and show that standardized infrastructure unlocks transferable control intelligence across flow regimes. HydroGym provides 61+ validated environments spanning laminar to turbulent flows, with systematic Reynolds number progressions up to Re=400,000 and Mach number variations in 2D and 3D. It supports diverse backends, including finite-volume, spectral-element, finite-element, lattice-Boltzmann, and fully differentiable solvers for gradient-enhanced optimization. Across environments, RL agents consistently discover robust control principles, such as boundary-layer manipulation, acoustic-feedback disruption, and wake reorganization, yielding drag reductions exceeding 90% in canonical configurations. Critically, we demonstrate zero-shot transfer: agents trained only on a simplified channel flow achieve 38% friction-drag reduction on an unseen 3D wing section at chord Reynolds number Re=200,000 reducing exploration costs by four orders of magnitude versus direct on-wing optimization. This suggests RL agents uncover essential physics rather than configuration-specific patterns, pointing toward generalizable control. HydroGym offers extensible, scalable community infrastructure for fluid dynamics, machine learning, and control research.

Christian Lagemann, Sajeda Mokbel, Miro Gondrum +18

Department of Mechanical Engineering, University of Washington, Seattle, WA 98195, United States · AI Institute in Dynamic Systems, University of Washington, Seattle, WA 98195, United States · Chair of Fluid Mechanics and Institute of Aerodynamics, RWTH Aachen University, Aachen, Germany +9

RL Control RL Benchmarks Reinforcement Learning Computational Fluid Dynamics

Dec 17, 2025

Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes

We study reinforcement learning for controlled diffusion processes with unbounded continuous state spaces, bounded continuous actions, and polynomially growing rewards: settings that arise naturally in finance, economics, and operations research. To overcome the challenges of continuous and high-dimensional domains, we introduce a model-based algorithm that adaptively partitions the joint state-action space. The algorithm maintains estimators of drift, volatility, and rewards within each partition, refining the discretization whenever estimation bias exceeds statistical confidence. This adaptive scheme balances exploration and approximation, enabling efficient learning in unbounded domains. Our analysis establishes regret bounds that depend on the problem horizon, state dimension, reward growth order, and a newly defined notion of zooming dimension tailored to unbounded diffusion processes. The bounds recover existing results for bounded settings as a special case, while extending theoretical guarantees to a broader class of diffusion-type problems. Finally, we validate the effectiveness of our approach through numerical experiments, including applications to high-dimensional problems such as multi-asset mean-variance portfolio selection.

Hanqing Jin, Renyuan Xu, Yanzhao Yang

Mathematical Institute, University of Oxford · Department of Management Science and Engineering, Stanford University

RL Control Reinforcement Learning Stochastic Optimal Control Model-Based RL

Dec 9, 2025

Sim2Swim: Zero-Shot Velocity Control for Agile AUV Maneuvering in 3 Minutes

Holonomic autonomous underwater vehicles (AUVs) have the hardware ability for agile maneuvering in both translational and rotational degrees of freedom (DOFs). However, due to challenges inherent to underwater vehicles, such as complex hydrostatics and hydrodynamics, parametric uncertainties, and frequent changes in dynamics due to payload changes, control is challenging. Performance typically relies on carefully tuned controllers targeting unique platform configurations, and a need for re-tuning for deployment under varying payloads and hydrodynamic conditions. As a consequence, agile maneuvering with simultaneous tracking of time-varying references in both translational and rotational DOFs is rarely utilized in practice. To the best of our knowledge, this paper presents the first general zero-shot sim2real deep reinforcement learning-based (DRL) velocity controller enabling path following and agile 6DOF maneuvering with a training duration of just 3 minutes. Sim2Swim, the proposed approach, inspired by state-of-the-art DRL-based position control, leverages domain randomization and massively parallelized training to converge to field-deployable control policies for AUVs of variable characteristics without post-processing or tuning. Sim2Swim is extensively validated in pool trials for a variety of configurations, showcasing robust control for highly agile motions.

Lauritz Rismark Fosso, Herman Biørn Amundsen, Marios Xanthidis +1

SINTEF Ocean, Trondheim, Norway

RL Control Sim-to-Real Transfer Underwater Robotics

Dec 1, 2025

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Diffusion models provide an expressive framework for sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion-based policies by introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs). DA-MDPs interpret each reverse-diffusion transition as an individual reinforcement-learning decision, while only the final denoised action is executed in the environment. Our DA-MDPs follow from a principled derivation based on the variational-inference formulation of ME-RL. By augmenting policy and target trajectories with intermediate diffusion variables, we obtain a tractable reverse-KL upper bound via the data-processing inequality. This bound decomposes across denoising transitions, yielding diffusion-augmented variants of soft rewards, value functions, and local policy objectives. This provides a general framework for adapting ME-RL algorithms to diffusion policies while differentiating through only one diffusion transition at a time. We instantiate the framework with PPO, REPPO, and a maximum-entropy extension of WPO. Experiments demonstrate improved continuous-control performance, benefits from additional diffusion steps, and memory-efficient training. On the StackCube and PushT manipulation tasks, DA-MDP methods learn alternative successful strategies from the same initial state and achieve higher success rates and generally higher success-weighted mode entropy than the Gaussian ME-RL baseline. We also demonstrate successful training when using action chunking.

Sebastian Sanokowski, Kaustubh Patil, Majid Khadiv

Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich · Practical Project Student Exchange Program, Technical University Munich · MIT World Peace University

RL Control Diffusion Policy Maximum Entropy RL RL for Diffusion Models Policy Learning

Nov 4, 2025

A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms

This paper presents a quantitative comparison between centralised and distributed multi-agent reinforcement learning (MARL) architectures for controlling a soft robotic arm modelled as a Cosserat rod in simulation. Using PyElastica and the OpenAI Gym interface, we train both a global Proximal Policy Optimisation (PPO) controller and a Multi-Agent PPO (MAPPO) under identical budgets. Both approaches are based on the arm having

n

number of controlled sections. The study systematically varies

n

and evaluates the performance of the arm to reach a fixed target in three scenarios: default baseline condition, recovery from external disturbance, and adaptation to actuator failure. Quantitative metrics used for the evaluation are mean action magnitude, mean final distance, mean episode length, and success rate. The results show that there are no significant benefits of the distributed policy when the number of controlled sections

n\le4

. In very simple systems, when

n\le2

, the centralised policy outperforms the distributed one. When

n

increases to

4< n\le 12

, the distributed policy shows a high sample efficiency. In these systems, distributed policy promotes a stronger success rate, resilience, and robustness under local observability and yields faster convergence given the same sample size. However, centralised policies achieve much higher time efficiency during training as it takes much less time to train the same size of samples. These findings highlight the trade-offs between centralised and distributed policy in reinforcement learning-based control for soft robotic systems and provide actionable design guidance for future sim-to-real transfer in soft rod-like manipulators.

Linxin Hou, Qirui Wu, Zhihang Qin +3

Department of Electrical and Computer Engineering, National University of Singapore · Department of Computer Science, National University of Singapore · Department of Mechanical Engineering, National University of Singapore

RL Control Robotic Policy Evaluation Decentralized MARL Soft Robotics Multi-Agent Reinforcement Learning

Oct 27, 2025

TARC: Time-Adaptive Robotic Control

Most robotic systems rely on fixed-frequency discrete-time controllers, creating a trade-off between the efficiency of low-frequency control and the responsiveness of high-frequency feedback. As a result, systems typically default to high control rates for robustness, at the cost of wasted inference and unnecessary actuation. Addressing this, we introduce Time-Adaptive Robotic Control (TARC), a reinforcement learning framework in which the policy jointly predicts a control action and its duration of application. TARC learns temporally extended actions by optimizing task performance under soft or hard constraints on the number of control switches, enabling adaptive modulation of control rates. We evaluate TARC on two robotic hardware platforms: a high-speed RC car and the Unitree Go1 quadruped, and on a vision-language action model in simulation, where each query incurs a costly transformer forward pass. Across all settings, TARC matches the performance of high-frequency discrete-time controllers while operating at less than half their control frequency. Unlike fixed-rate controllers, TARC adapts its control frequency online, allocating high-frequency feedback only when required.

Arnav Sukhija, Lenart Treven, Jin Cheng +3

ETH Zurich

Robotic Control RL Control Adaptive Control Real-Time Robot Control

May 5, 2025

Graph Neural Network-Based Reinforcement Learning for Controlling Biological Networks - the GATTACA Framework

Cellular reprogramming, the artificial transformation of one cell type into another, has been attracting increasing research attention due to its therapeutic potential for complex diseases. However, identifying effective reprogramming strategies through classical wet-lab experiments is hindered by lengthy time commitments and high costs. In this study, we explore the use of deep reinforcement learning (DRL) to control Boolean network models of complex biological systems, such as gene regulatory and signalling pathway networks. We formulate a novel control problem for Boolean network models under the asynchronous update mode, specifically in the context of cellular reprogramming. To solve it, we devise GATTACA, a scalable computational framework. To facilitate scalability of our framework, we consider previously introduced concept of a pseudo-attractor and improve the procedure for effective identification of pseudo-attractor states. We then incorporate graph neural networks with graph convolution operations into the artificial neural network approximator of the DRL agent's action-value function. This allows us to leverage the available knowledge on the structure of a biological system and to indirectly, yet effectively, encode the system's modelled dynamics into a latent representation. Experiments on several large-scale, real-world biological networks from the literature demonstrate the scalability and effectiveness of our approach.

Andrzej Mizera, Jakub Zarzycki

Faculty of Mathematics, Informatics and Mechanics, University of Warsaw, Banacha 2, 02-097 Warsaw, Poland · IDEAS Research Institute, Królewska 27, 00-060 Warsaw, Poland · IDEAS NCBR Sp. z o.o., Chmielna 69, 00-801 Warsaw, Poland

RL Control Asynchronous RL

Apr 17, 2025

TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.

Siow Meng Low, Ze Gong, Akshat Kumar

School of Computing and Information Systems, Singapore Management University, Singapore · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

RL Control Credit Assignment in RL Step-Level Credit Assignment in RL Constrained RL Safe RL

Feb 20, 2025

Sample Complexity of Linear Quadratic Regulator Without Initial Stability

Inspired by REINFORCE, we introduce a novel receding-horizon algorithm for the Linear Quadratic Regulator (LQR) problem with unknown dynamics. Unlike prior methods, our algorithm avoids reliance on two-point gradient estimates while maintaining the same order of sample complexity. Furthermore, it eliminates the restrictive requirement of starting with a stable initial policy, broadening its applicability. Beyond these improvements, we introduce a refined analysis of error propagation through the contraction of the Riccati operator under the Riemannian distance. This refinement leads to a better sample complexity and ensures improved convergence guarantees.

Amirreza Neshaei Moghaddam, Alex Olshevsky, Bahman Gharesifard

RL Control Sample Complexity Feedback Control Linear Quadratic Regulator Sample-Efficient RL