Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift
Authors: Shaifalee Saxena, Rafael Fierro, Alexander Scheinker
Organizations: Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM 87106, USA · Los Alamos National Lab, Los Alamos, NM 87547, USA
Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important for reliable robot planning and control in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking (ES) to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches. Crucially, under reasonable assumptions, we prove finite time RL-based convergence of a robotic arm to an object after which bounded ES achieves finite-time convergence to a bounded velocity moving target. We provide analytic formulas for both convergence times, and an ability to tune the convergence times by our choice of gains.
Figures & tables
Hyperparameter
Value
State dimension ns
28
Action dimension na
4
Discount factor γ
0.99
Actor learning rate
1×10−4
Critic learning rate
1×10−4
Replay buffer size ∣D∣
106 transitions
TABLE I : DDPG training hyperparameters
Fig. 1 : DDPG training success rates for Fetch push and pick-and-place. The dotted gray curves denote the raw success rates, while the solid black curves denote the smoothed training trends. Each training epoch consists of 100 episodes.
Fig. 2 : Architecture of the ES-DRL controller. A supervisor selects binary β∈{0,1} based on the contact flag. ES is initialized from DRL (dotted).
Fig. 3 : Robotic manipulation over spatially varying frictional surfaces. Left: Environment with three friction patches, μ=0.8,1.2,1.5 . Middle: For a fixed goal, ES lacks a good initial pushing direction, RL fails in the high-friction region, and ES-DRL drives the block toward the goal. Right: For a time-varying goal, RL fails early, whereas ES-DRL tracks the goal.
Fig. 4 : 3D tracking of a time-varying goal. The RL-only controller (top) fails to track the goal, whereas ES-DRL (bottom) maintains substantially closer tracking.
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany