Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion

Xinyu Cui; Fei Han; Hang Xu; Yongcheng Zeng; Luoyang Sun; Ruizhi Zhang; Jian Zhao; Haifeng Zhang

cs.ROMar 4, 2026

Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion

Authors: Xinyu Cui, Fei Han, Hang Xu, Yongcheng Zeng, Luoyang Sun, Ruizhi Zhang, Jian Zhao, Haifeng Zhang,

Organizations: FSI lab, Westlake University, Hangzhou, 310030, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Zhejiang University, Hangzhou, 310027, China. ·

Abstract

Bio-inspired aquatic propulsion offers high thrust and maneuverability but is prone to destabilizing forces such as lift fluctuations, which are further amplified by six-degree-of-freedom (6-DoF) fluid coupling. We formulate quadrupedal swimming as a constrained optimization problem that maximizes forward thrust while minimizing destabilizing fluctuations. Our proposed framework, Accelerated Constrained Proximal Policy Optimization with a PID-regulated Lagrange multiplier (ACPPO-PID), enforces constraints with a PID-regulated Lagrange multiplier, accelerates learning via conditional asymmetric clipping, and stabilizes updates through cycle-wise geometric aggregation. Initialized with imitation learning and refined through on-hardware towing-tank experiments, ACPPO-PID produces control policies that transfer effectively to quadrupedal free-swimming trials. Results demonstrate improved thrust efficiency, reduced destabilizing forces, and faster convergence compared with state-of-the-art baselines, underscoring the importance of constraint-aware safe RL for robust and generalizable bio-inspired locomotion in complex fluid environments.

Figures & tables

Fig. 1: Proposed framework: imitation learning initializes a periodic gait from predefined motions while sensor feedback is Kalman‑filtered; safe RL fine‑tuning accelerates on‑hardware convergence under stability constraints; and the resulting one‑cycle paddle is transferred to diagonal limb pairs with a half‑cycle phase offset to enable smooth and stable free‑swimming.

Parameter	$A_{\theta^{H}}(rad)$	$A_{\theta^{K}}(rad)$	$f(Hz)$	$\phi$	$\theta^{H}_{0}/\theta^{K}_{0}$
Range	$[{\pi}/6,{\pi}/3]$	$[{\pi}/12,{\pi}/4]$	$[0.3,0.6]$	$[0,\pi]$	$[{\pi}/4,{5\pi}/4]$

TABLE I: Experimental Parameter Ranges

Fig. 2: The training curves (reward and average cost) of the baselines over 400 episodes, each method is trained in three random seeds. The gray line marks the cost limit.

Fig. 3: Ablations removing the cycle‑level objective, conditional high clipping, or imitation learning quantify each component’s contribution to stability and data efficiency. Each ablation is trained in three random seeds. The gray line marks the cost limit.

Algorithm	Episode Reward	Average Cost
CPPO-PID	$39.58\pm\pm 3.93$	$0.249\pm\pm 0.021$
CPPO-PID-H	$37.10\pm\pm 10.84$	$0.261\pm\pm 0.023$
PPO-Penalty	$43.46\pm\pm 3.91$	$0.314\pm\pm 0.021$
PPO (w/o Cost)	$48.58\pm\pm 5.15$	$0.391\pm\pm 0.039$
Ours (ACPPO-PID)	$45.19\pm\pm 3.83$	$0.235\pm\pm 0.019$
Ours (w/o Cycle Loss)	$43.35\pm\pm 5.92$	$0.271\pm\pm 0.027$

TABLE II: Single limb evaluated in a tow tank

Fig. 4: Forward swimming performance under two representative gaits per algorithm, averaged over three trials. ACPPO-PID achieves the highest scores with lower variance, while CPPO-PID, PPO, and BF show reduced thrust and greater sensitivity to gait differences.

Fig. 5: Comparison of $F_{x}^{\text{mean}}$ , $F_{z}^{\text{mean}}$ , and $F_{z}^{\text{var}}$ across algorithms, showing ACPPO-PID achieves strong thrust with reduced lift fluctuations.The right panel illustrates the maximum displacement achieved in this experiment under parameterized motion, standard PPO training, and our proposed ACPPO-PID.

Fig. 6: Comparison of limb motion patterns between our constrained method (ACPPO-PID, top) and unconstrained PPO (bottom) during one locomotion cycle. Key differences are highlighted: calf movement during pre-paddle phase (red) and hip movement during paddle phase (yellow).

Constrained RL

Constrained reinforcement learning optimizes task performance while keeping safety risks, resource use, or other costs within explicit limits.

See all 215 papers

Learning Control as Enabling Layer for Embodied Intelligence Research explored with Soft Robotic Swimming in diverse Flow Speeds
Jun 9, 2026Fabian Schwab, Federico Allione, Bingcheng Wang +5Robotic Control Robust Control
1Engineering Sciences Department, Swiss Federal Laboratories for Materials Science and Technology, Switzerland · Institute for NeuroInformatics, University of Zurich, Switzerland · Department of Electronic Engineering, University of Rome Tor Vergata, Italy
Constraint-Aware Diffusion Priors for High-Fidelity and Versatile Quadruped Locomotion
May 9, 2026Jianhui Chen, Ruixin Zhan, Liu Liu +2Sim-to-Real Transfer Constrained RL
Amap, Alibaba Group
ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers
Mar 20, 2026Vrushabh Zinage, Narek Harutyunyan, Eric Verheyden +2Robust RL Reinforcement Learning
California Institute of Technology · Brown University

Jun 9, 2026J/K move · Enter open · S save

Learning Control as Enabling Layer for Embodied Intelligence Research explored with Soft Robotic Swimming in diverse Flow Speeds

Fabian Schwab, Federico Allione, Bingcheng Wang, Mohamed El Arayshi+4

1Engineering Sciences Department, Swiss Federal Laboratories for Materials Science and Technology, Switzerland · Institute for NeuroInformatics, University of Zurich, Switzerland · Department of Electronic Engineering, University of Rome Tor Vergata, Italy

Robotic Control Robust Control Robot Locomotion Iterative Learning Control Soft Robotics

Open paper AskPDF arXiv

Soft robots are valuable robophysical platforms for studying body-caudal undulatory locomotion, but their compliant bodies are difficult to control precisely under changing hydrodynamic loading. Conventional proportional-integral-derivative (PID) feedback stabilizes periodic undulation in static water, but can accumulate flow-dependent tracking delay and increasing inter-trial variability when environmental flow becomes non-trivial. Here, we evaluate whether augmenting PID control with a Linear Repetitive Learning Estimation Scheme (PID-LRLES) recovers tracking accuracy and repeatability under dynamic flow. The LRLES generalizes classical integral action from constant to periodic, non-constant references, while using a stable transfer-function realization whose poles have negative real parts to avoid the long-term instability issues of classical repetitive control. Closed-loop experiments were carried out in a recirculating flow tank at five bulk flow speeds spanning 0 to 32.6 cm s^-1, using an embedded soft capacitive bending sensor at a 1 kHz control-loop rate. With controller gains tuned once in static water and then held fixed across all conditions, PID-LRLES tracked the periodic bending-envelope reference more closely than the PID baseline and significantly reduced the inter-trial spread of the per-trial RMSE (paired Wilcoxon signed-rank test, p = 1.8 x 10^-4, n = 25). Embedded soft proprioception and cycle-to-cycle learning act as complementary contributors to robustness: the sensor exposes the periodic hydrodynamic bias in body deformation, while the learning term absorbs it over recent oscillation cycles. By reducing flow-dependent control-induced variability, the approach provides an enabling layer for future robophysical studies seeking to isolate the effects of morphology, sensing, and environmental flow on aquatic locomotion.

Abstract

Constrained RL

Explore similar work

Learning Control as Enabling Layer for Embodied Intelligence Research explored with Soft Robotic Swimming in diverse Flow Speeds

Constraint-Aware Diffusion Priors for High-Fidelity and Versatile Quadruped Locomotion

ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers