A Unified Dynamics Framework for Reinforcement Learning and Classical Control of a Six-DOF Pipeline-Tracking ROV in NVIDIA Isaac Sim

Cheng Siong Chin; M. Venkateshkumar; Jianhua Zhang

cs.ROOct 4, 2026

A Unified Dynamics Framework for Reinforcement Learning and Classical Control of a Six-DOF Pipeline-Tracking ROV in NVIDIA Isaac Sim

Authors: Cheng Siong Chin, M. Venkateshkumar, Jianhua Zhang

Organizations: Faculty of Science, Agriculture, and Engineering, Newcastle University Singapore, Singapore 828608, Singapore · Amrita School of Engineering, Coimbatore, Amrita Vishwa Vidyapeetham, India · School of Information and Control Engineering, Qingdao University of Technology, Qingdao 266525, China

Abstract

Reinforcement learning controllers for underwater vehicles are usually trained against one physics representation and deployed against another, so reported performance does not always describe behavior outside training. This paper presents a pipeline-tracking architecture for a six-degree-of-freedom remotely operated vehicle (ROV) in which one Universal Scene Description (USD) scene supplies the real BlueROV2-Heavy mass, added-mass, damping, buoyancy, and thruster parameters to both halves of the system: a vectorized NumPy implementation of Fossen's marine-craft equations, and an interactive NVIDIA Isaac Sim deployment applying the identical equations as PhysX forces at every step. The Coriolis-centripetal term is the primary dynamics model in both branches; a controlled ablation on PPO and TRPO shows that including it does not destabilize either algorithm and modestly improves tracking, about 19 percent lower standoff RMS error for TRPO. Five reinforcement learning algorithms, PPO, soft actor-critic, TD3, DDPG, and TRPO, are trained against one environment, reward, and randomized evaluation harness through a checkpoint-compatibility layer scoring any policy with the same code. The pipeline is extended with six classical baselines, PID, sliding-mode, fuzzy logic, feedback linearization, model predictive control, and an adaptive neuro-fuzzy inference system, driven by the same guidance geometry and thruster allocation as the learned policies. Under Coriolis-enabled dynamics, PPO, TRPO, and feedback linearization reach the strongest combination of 100 percent success and competitive accuracy; PID, fuzzy control, and the neuro-fuzzy baseline also reach 100 percent success with looser tracking; DDPG and TD3 each show a specific, explainable failure mode rather than a general weakness of off-policy learning; and classical control remains a strong baseline against the best learned policies.

Figures & tables

Fig. 2: Reference reinforcement-learning pipeline for a vehicle simulated in NVIDIA Omniverse / Isaac Sim, with an optional Isaac Lab training loop and an optional ROS 2 bridge to a physical vehicle (reference diagram).

Fig. 1: System architecture. A single USD scene supplies vehicle and environment parameters to a vectorized training branch, shared across the five compared reinforcement-learning algorithms and the six classical baselines, and an interactive NVIDIA Isaac Sim deployment branch. A trained policy checkpoint or a classical controller’s tuned parameters is the only artifact that crosses between them, via a checkpoint-compatibility layer that lets any controller be flown and scored identically.

Parameter	Value	Degree of freedom
Rigid-body mass, m	11.5 kg	All
Added mass, surge (m_added,1)	-5.5 kg	Surge, u
Added mass, sway (m_added,2)	-12.7 kg	Sway, v
Added mass, heave (m_added,3)	-14.57 kg	Heave, w
Added mass, roll (m_added,4)	-0.12 kg m 2	Roll rate, p
Added mass, pitch (m_added,5)	-0.12 kg m 2	Pitch rate, q

TABLE I: Rigid-body mass and added-mass parameters

Degree of freedom	D_lin	D_quad	Units
Surge, u	-4.03	-18.18	N per m/s; N per (m/s) 2
Sway, v	-6.22	-21.66	N per m/s; N per (m/s) 2
Heave, w	-5.18	-36.99	N per m/s; N per (m/s) 2
Roll rate, p	-0.07	-1.55	N $\cdot$ m per rad/s; N $\cdot$ m per (rad/s) 2
Pitch rate, q	-0.07	-1.55	N $\cdot$ m per rad/s; N $\cdot$ m per (rad/s) 2
Yaw rate, r	-0.07	-1.55	N $\cdot$ m per rad/s; N $\cdot$ m per (rad/s) 2

TABLE II: Linear and quadratic damping coefficients

Parameter	Value
Weight, W	112.8 N
Buoyancy, B	114.8 N
Net weight-minus-buoyancy, W_net = W - B	-2.0 N (positively buoyant)

TABLE III: Weight, buoyancy, and net restoring force

Fig. 3: BlueROV2-Heavy thruster force/torque diagram. See URL for more details: https://bluerobotics.com/introducing-bluerov2-heavy/

Fig. 4: This project’s thruster layout (indices T1-T8), drawn from the scene-construction script’s actual geometry: T1-T4 horizontal in an X pattern at 45 degrees, T5-T8 vertical, one per hull corner.

Parameter	Value
Number of thrusters	8 (T1-T4 horizontal, T5-T8 vertical)
Maximum per-thruster force, T_max	40 N
Normalized command range, u_i	[-1, 1] per thruster
Allocation matrix, A	6 $\times$ 8, read from the USD scene (Table IV-B)

TABLE IV: Thruster allocation parameters

Thruster	u	v	w	p	q	r
T1	0.707	-0.707	0	0.060	0.060	-0.189
T2	0.707	0.707	0	-0.060	0.060	0.189
T3	-0.707	-0.707	0	0.060	-0.060	0.189
T4	-0.707	0.707	0	-0.060	-0.060	-0.189
T5	0	0	-1	-0.218	0.120	0
T6	0	0	1	-0.218	-0.120	0

TABLE IV-B: Thruster allocation matrix A (per-thruster contribution to surge u, sway v, heave w, roll p, pitch q, yaw r)

Group	Dims	Contents
Position	3	x, y, z
Orientation	6	sin and cos of $\varphi$ , $\theta$ , $\psi$
Body velocity	6	$\nu$ = [u, v, w, p, q, r]
Route progress	2	Along-track distance s; distance to goal
Cross-track error	2	Horizontal cross_h; vertical cross_v
Pipe standoff	1	3D distance to pipeline centerline

TABLE V: Observation vector (25 dimensions)

Parameter	Value	Meaning
k_progress	48	Reward for along-track progress
k_track, k_track_rate	20, 0.5	Penalty for 3D standoff error (1.2 m target) and its rate
k_cross_h, cross_h_target	15, 0.2 m	Horizontal cross-track penalty and target
k_heading	1.5	Reward for facing along the pipeline
k_energy	1.0	Thruster-effort penalty
completion_bonus	100	Bonus for reaching the end

TABLE VI: Reward weights and safety limits (identical for all five reinforcement-learning algorithms)

Hyperparameter	Value
Total timesteps (final run)	20,004,864
Parallel environments	64
Rollout length	256 steps
Curriculum fraction (short scene first)	0.35
Learning rate	3 $\times$ 10 -4 , Adam
PPO epochs per update	10

TABLE VII: PPO training hyperparameters (final accepted run)

Hyperparameter	SAC	TD3	DDPG	TRPO
Total env-steps	1,802,240 (early-stopped at 90%)	8,994,816	2,998,272	20,004,864
Discount, $\gamma$	0.95	0.99	0.99	0.95
Action / exploration noise	entropy-driven + 0.15 rollout noise	$\sigma$ = 0.20	$\sigma$ = 0.10	N/A
Target entropy	-1.8 (floor log $\alpha$ $\geq$ ln 0.018)	N/A	N/A	N/A
Action-penalty coefficient	N/A	N/A	N/A	0.01
Curriculum	Disabled	Disabled	Disabled	Enabled (0.35 fraction)

TABLE VIII: Training configuration, final accepted run

Parameter	Value
Controlled channels	cross_h (sway), cross_v (heave/standoff), yaw, roll, pitch, surge
Cruise speed (surge setpoint)	1.825 m/s
Standoff target (heave setpoint)	1.2 m above the pipeline centerline (1.0 m hard floor, Section V-C)
Roll / pitch setpoint	0 rad (active levelling)
Heading guidance	Line-of-sight, look-ahead point 1.5 m along the route
Maximum episode length	1,040 steps (Section V-A)

TABLE IX: Shared guidance and control settings, all six classical and model-based baselines

Channel	k_p	k_i	k_d
cross_h	0.9	0.05	0.6
cross_v	0.9	0.05	0.6
yaw	1.1	0.02	0.35
roll	1.4	0.0	0.30
pitch	1.4	0.0	0.30
surge	0.6	0.10	0.0

TABLE X: PID gains, per channel

Channel	$\lambda$ (surface slope)	K (switching gain)	$\varphi$ (boundary-layer width)
cross_h	0.5	0.35	0.6
cross_v	0.5	0.35	0.6
yaw	0.6	0.35	0.8
roll	0.8	0.45	0.5
pitch	0.8	0.45	0.5
surge	0.4	0.35	0.6

TABLE XI: Sliding-mode control parameters, per channel

Channel	G_e (error scale)	G_ $\dot{e}$ (error-rate scale)	G_u (output scale)
cross_h	1.2	0.8	1.0
cross_v	1.2	0.8	1.0
yaw	1.5	0.6	1.2
roll	2.0	0.8	1.5
pitch	2.0	0.8	1.5
surge	1.0	0.0	0.8

TABLE XII: Fuzzy-logic scaling gains, per channel

e \ $\dot{e}$	NB	NS	Z	PS	PB
NB	-1.0	-1.0	-1.0	-0.5	0.0
NS	-1.0	-1.0	-0.5	0.0	0.5
Z	-1.0	-0.5	0.0	0.5	1.0
PS	-0.5	0.0	0.5	1.0	1.0
PB	0.0	0.5	1.0	1.0	1.0

TABLE XIII: Fuzzy-logic rule table (shared across all six channels): consequent y_ij for error i and error-rate j

Channel	k_p (1/s 2 )	k_d (1/s)
cross_h	3.0	2.5
cross_v	3.0	2.5
yaw	4.0	2.0
roll	6.0	2.5
pitch	6.0	2.5
surge	2.0	0.0

TABLE XIV: Feedback-linearization gains, per channel

Parameter	Value
Prediction horizon, H	20 steps
Samples per CEM iteration, K	256
CEM iterations	2
Elite fraction	0.1 (minimum 4 elites; 25 in practice)
Sampling noise, initial standard deviation	0.5
Action bounds	[-1, 1] per thruster (8 thrusters)

TABLE XV: Model predictive control parameters

Channel	c_e (MF 1, 2, 3)	$\sigma$ _e (MF 1, 2, 3)	c_ $\dot{e}$ (MF 1, 2, 3)	$\sigma$ _ $\dot{e}$ (MF 1, 2, 3)
cross_h	-1.189, -0.837, 0.763	0.535, 0.605, 0.693	-1.130, 0.389, 1.155	0.753, 0.705, 0.535
cross_v	-1.012, -0.585, 1.083	0.733, 0.668, 0.777	-1.165, 1.263, 1.048	0.634, 0.312, 0.693
yaw	-1.055, -0.520, 0.751	1.021, 0.048, 1.010	-2.145, -0.610, 1.623	0.292, 1.500, 1.528
roll	-1.242, -0.595, 1.098	0.665, 0.368, 0.748	-1.013, 0.828, 1.023	0.886, 0.376, 0.839
pitch	-1.316, -0.726, 1.230	0.534, 0.318, 0.567	-1.107, 0.059, 1.205	0.824, 0.588, 0.703
surge	-1.722, -0.252, 2.165	0.484, 0.735, 0.580	-0.435, 0.000, 0.435	1.494, 1.001, 1.494

TABLE XVI: ANFIS trained membership-function parameters, per channel (centre c, width

\sigma

; three membership functions each for e and

\dot{e}

)

Channel	r1	r2	r3	r4	r5	r6	r7	r8	r9
cross_h	-1.009	-0.798	-0.373	-1.067	-0.233	0.280	-0.345	0.583	0.626
cross_v	-0.665	-0.023	-0.215	-0.400	0.097	-0.081	0.495	0.843	1.170
yaw	-1.816	-2.479	-0.467	-0.900	-0.821	0.297	-0.159	0.710	2.162
roll	-0.262	0.115	0.118	-0.329	0.076	0.022	-0.049	0.227	0.351
pitch	-0.537	-0.215	0.160	-0.496	-0.235	0.174	-0.125	0.398	0.729
surge	0.545	0.579	0.545	0.520	0.505	0.520	1.091	0.877	1.091

TABLE XVII: ANFIS trained rule consequents, per channel (rule r = 3i + j, i indexing the error membership function, j the error-rate membership function)

Algorithm	Coriolis term	Success rate	Flip rate	Corridor-gap violation	Standoff RMSE (m)
PPO	Without	100%	0%	0%	0.152
PPO	With	100%	0%	0%	0.146
TRPO	Without	100%	0%	0%	0.176
TRPO	With	100%	0%	0%	0.142

TABLE XVIII: Coriolis-term ablation, PPO and TRPO (30-episode randomized evaluation)

Fig. 5: Coriolis-term ablation for PPO and TRPO: (a) success rate and (b) standoff root-mean-square error, with and without the Coriolis-centripetal term.

Controller	Type	Success	95% CI	Flip	Gap viol.	RMSE (m)	Std (m)	Mean XT (m)	Max XT (m)	Head. align.	Action mag.
PPO	RL	100%	—	0%	0%	0.146	0.140	1.238	1.908	0.935	0.330
TD3	RL	0%	0–11%	0%	0%	0.055	0.052	1.218	1.908	0.987	0.138
DDPG	RL	63.3%	45.5–78.1%	30%	0%	0.375	0.239	1.489	1.908	0.475	0.758
TRPO	RL	100%	—	0%	0%	0.142	0.140	1.224	1.908	0.833	0.230
PID	Classical	100%	—	0%	0%	0.215	0.192	1.298	1.914	0.864	0.409
SMC	Classical	100%	—	0%	0%	0.293	0.173	1.437	1.915	0.684	0.414

TABLE XIX: Reinforcement learning versus classical and model-based control, Coriolis-enabled dynamics (30-episode randomized evaluation)

Fig. 6: Success rate for all ten evaluated controllers under Coriolis-enabled dynamics, reinforcement-learning policies and classical/model-based baselines colour-coded separately.

Fig. 7: Standoff root-mean-square tracking error for the same ten controllers.

Fig. 8: Flip rate and corridor-gap-violation rate (safety metrics) for the same ten controllers.

Fig. 9: Three-dimensional standoff distance to the pipeline versus time, canonical episode, for the eight controllers that complete the route, against the 1.2 m target setpoint and the 1.0 m hard floor of Table IX.

Fig. 10: Position (x, y, z) and attitude (roll, pitch, yaw) versus time, canonical episode, for the same eight controllers; a circular marker denotes success at the pipeline end.

Fig. 11: Flown trajectories against the pipeline centerline, canonical episode, with the vehicle rendered to scale at the start, 25%, 50%, 75%, and end waypoints: (a) top view (x-y), (b) side view (x-z), (c) front view (y-z).

Fig. 12: The same flown trajectories of Fig. 11 in an oblique three-dimensional perspective.

Fig. 13: The same eight flown trajectories as Fig. 11, plotted as centerlines only, over the full route from start to end.

Fig. 14: Body-frame linear velocity (surge u, sway v, heave w) and angular velocity (roll rate p, pitch rate q, yaw rate r) versus time, canonical episode, for the same eight controllers.

Metric	PPO	TD3	DDPG	TRPO
Episode time (s)	6.9	52.0	1.1	8.6
Along-track progress (of 10 m route)	9.70 m	5.00 m	0.38 m	9.71 m
Mean \|thrust\| (normalized)	0.330	0.138	0.758	0.230
Energy (normalized thrust 2 $\cdot$ s)	11.8	15.7	5.9	9.4
Energy per metre travelled	1.2	3.1	n/a (negligible progress)	1.0
Steps to 90% deterministic success	10,485,760	never reached	never reached	14,745,600

TABLE XX: Additional quantitative indicators, canonical episode and training logs (PPO, TD3, DDPG, TRPO)

Setting	PPO	TD3	DDPG	TRPO
Learning rate	3 $\times$ 10 -4 (actor and critic, Adam)	3 $\times$ 10 -4 (actor and critic, Adam)	3 $\times$ 10 -4 (actor and critic, Adam)	Policy: none (natural gradient); value 1 $\times$ 10 -3 (Adam)
Discount factor, $\gamma$	0.99 (GAE $\lambda$ = 0.95)	0.99	0.99	0.95 (GAE $\lambda$ = 0.95)
Clipping / trust-region parameter	Clip ratio 0.2	Target-noise clip 0.5 (Polyak $\tau$ = 0.005)	Polyak $\tau$ = 0.005	Max KL 0.01 (trust region), no ratio clip
Batch size	4,096 minibatch (16,384-sample rollout: 64 environments $\times$ 256 steps)	256 (replay buffer 1M)	256 (replay buffer 1M)	16,384 full rollout (value minibatch 4,096)
Updates per rollout / step	10 epochs per rollout	8 gradient updates per environment step	8 gradient updates per environment step	1 conjugate-gradient step (10 CG iterations) + 10 value epochs
Network	MLP, 2 $\times$ 256, tanh (actor and critic)	MLP, 2 $\times$ 256, tanh (actor, twin critics)	MLP, 2 $\times$ 256, tanh (actor, critic)	MLP, 2 $\times$ 256, tanh (actor and critic)

TABLE XXI: Consolidated hyperparameters and algorithm-specific settings (PPO, TD3, DDPG, TRPO)

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 274 papers

Oct 8, 2026

Sim-to-Real RL for ASVs using SysID

Autonomous Surface Vehicles (ASVs) operating in dynamic marine environments require robust control policies for tasks such as path following and station keeping, making reinforcement learning (RL) a promising alternative to classical controllers. However, existing ASV simulators rarely support parallel environments for RL training. Such existing simulators require accurate hydrodynamic modeling from computational fluid dynamics solvers or towing tank tests for setting hydrodynamic parameters to address the sim-to-real gap. To address these challenges, we present an ASV simulator and accompanying pipeline that enables training policies starting from unknown vehicle dynamics. Our framework uses only a CAD model and brief set of open-water field trajectories for approximating and refining both hydrodynamic and thruster parameters. Real-world deployments on a BlueBoat ASV demonstrate successful zero-shot sim-to-real transfer in path following and station-keeping tasks without prior hydrodynamic and propeller information.

Cody Sheltraw, Tsimafei Lazouski, Maani Ghaffari +1

University of Michigan, Ann Arbor, MI 48109, USA.

Sim-to-Real Transfer Robotics Simulation RL Control Robot Navigation

Ask PDF

Jul 10, 2026

CORL-AUV: CFD Oriented Reinforcement Learning for Autonomous Underwater Vehicles

Precise control and positioning of autonomous underwater vehicles (AUVs) is critical for sampling, maintenance, and survey applications. Reinforcement learning (RL) offers rapid controller development while handling a range of deployment parameters via domain randomization (DR). However, DR is limited by the underlying simulation's capacity to model real physics such as drag, which is a large contributor to sim-to-real gaps. Computational fluid dynamics (CFD) provides high-fidelity drag models but has significant computational cost. Thus, in this paper we train surrogate approximations of CFD data of a given vehicle, providing drag estimates 700,000x faster than solving for drag at each time step with CFD within the RL training pipeline. We deploy a Proximal Policy Optimization (PPO) policy zero-shot on a 6-DOF AUV in which policy training is performed on these surrogate drag models (SDMs), providing analysis on zero-shot transfer, reward shaping sensitivity, and physical parameter perturbations. On 15 identical segments in the ocean, we observe median gains of 19% in mean tracking error and 31% in control effort compared to the equivalent inertia box model. Our SDM based RL controller better predicts zero-shot transfer and reaches the most waypoints across all three reward shaping choices that we evaluate. Finally, we add 0.907 kg to the vehicle perturbing the mass, center-of-mass, and center-of-buoyancy. The policy reaches 100% of waypoints only when the SDM drag model is used; the policy fails to reach any of the 15 waypoints when the other drag models are used.

Steven Roche, Milo Van Mooy, Nathan McGuire +3

MIT-WHOI Joint Program, MA 02543, USA · WHOI, MA 02543, USA · Massachusetts Institute of Technology, Cambridge, MA 02319, USA

RL Control RL for Robotics Surrogate Modeling Sim-to-Real Transfer Computational Fluid Dynamics

Ask PDF

Aug 9, 2026

Knowledge-Distilled End-to-End Reinforcement Learning for Smooth 6-DOF Thrust Control and Rapid Adaptation to Ocean Currents in Remotely Operated Vehicles

With the continuous improvement of computational capabilities, end-to-end reinforcement learning has been rapidly developed for remotely operated vehicles control. Nevertheless, existing end-to-end reinforcement-learningbased methods still face challenges in achieving optimal control under oceancurrent disturbances. In particular, there remains a lack of a unified control framework that can simultaneously achieve low steady-state tracking error, rapid transient response, energy-efficient operation, and smooth controlforce outputs under disturbances. To address the issue, this paper proposes the thrust smoothness rapid current adaptation proximal policy optimization (TSRCA-PPO) method which learns a near-optimal strategy by a twostage distillation learning framework. The core innovations of this work lie in the reward-function design and the privileged multi-encoder architecture. Ablation studies validate the effectiveness of each module. Simulation results demonstrate that the proposed TSRCA-PPO method consistently outperforms the conventional cascaded P-PID controller across all evaluation metrics. Specifically, TSRCA-PPO reduces the steady-state position error, steady-state attitude error, settling time, energy index, and thrustsmoothness index to 42.7%, 76.5%, 10.6%, 93.5%, and 15.9% of the corresponding P-PID values, respectively.

Tiankuang Wen, Huiping Li, Gang Liu +1

Northwestern Polytechnical University, Xi’an 710129, China · Wuhan Second Ship Design and Research Institute, Wuhan 430205, China

Robotic RL Proximal Policy Optimization Adaptive Control

Ask PDF

Abstract

RL Control

Explore similar work

Sim-to-Real RL for ASVs using SysID

CORL-AUV: CFD Oriented Reinforcement Learning for Autonomous Underwater Vehicles

Knowledge-Distilled End-to-End Reinforcement Learning for Smooth 6-DOF Thrust Control and Rapid Adaptation to Ocean Currents in Remotely Operated Vehicles