ManeuverNet: A Soft Actor-Critic Framework for Precise Maneuvering of Double-Ackermann-Steering Robots with Optimized Reward Functions
Authors: Kohio Deflesselle, Mélodie Daniel, Aly Magassouba, Miguel Aranda, Olivier Ly
Organizations: Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, F-33400 Talence, France. · School of Computer Science, University of Nottingham, UK. · Instituto de Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, 50018 Zaragoza, Spain.
Autonomous control of double-Ackermann-steering robots is essential in agricultural applications, where robots must execute precise and complex maneuvers within a limited space. Classical methods, such as the Timed Elastic Band (TEB) planner, can address this problem, but they rely on parameter tuning, making them highly sensitive to changes in robot configuration or environment and impractical to deploy without constant recalibration. At the same time, end-to-end deep reinforcement learning (DRL) methods often fail due to unsuitable reward functions for non-holonomic constraints, resulting in sub-optimal policies and poor generalization. To address these challenges, this paper presents ManeuverNet, a DRL framework tailored for double-Ackermann systems, combining Soft Actor-Critic with CrossQ. Furthermore, ManeuverNet introduces four specifically designed reward functions to support maneuver learning. Unlike prior work, ManeuverNet does not depend on expert data or handcrafted guidance. We extensively evaluate ManeuverNet against both state-of-the-art DRL baselines and the TEB planner. Experimental results demonstrate that our framework substantially improves maneuverability and success rates, achieving more than a 40% gain over DRL baselines. Moreover, ManeuverNet effectively mitigates the strong parameter sensitivity observed in the TEB planner. In real-world trials, ManeuverNet achieved up to a 90% increase in maneuvering trajectory efficiency, highlighting its robustness and practical applicability.
Figures & tables
Fig. 1 : Maneuver handling (left figure) with 4WS robots in DRL is challenging because it requires current reward loss (circled in red in the right figure), making classical approaches sub-optimal.
Fig. 2 : DASMR rotating around an instantaneous center of rotation (ICR).
Fig. 3 : Comparison between the shape of our reward functions (top) vs. classic state-of-the-art reward functions (bottom). X is the longitudinal axis, Y is the lateral axis, and the robot’s center position is at the origin. Each point on the heatmaps represents the reward for a Xd in the same position. Thus, the robot reaches Xd when ∥Xd∥<dth .
Rewards
Formulation
RHS
−(Δx)2+(c⋅(Δy±max{0,∣Δy∣−∣Δx∣}))2
RES
−(Δx)2+(cΔy)2
RCh
−max{∣Δx∣,∣Δy∣}
RCl
\begin{cases}\tan^{-1}\left(\frac{\Delta y}{\Delta x}\right)\times ce^{d_{\text{th}}-d}&\text{ if \Delta y>d_{\text{th}} }\\ -d&\text{otherwise}\\ \end{cases}
TABLE I : DASMR reward functions where Δx and Δy are the X and Y components of (Xd−Xc) and c a weighting parameter.
Parameter
Value
Nb. layers
2
Actor Hidden size
256
Critic Hidden size
1024
αA=αC
1e-3
Replay buffer size
1,000,000
Batch size N
256
TABLE II : SAC and CrossQ parameters used on SBX during training and testing.
Fig. 4 : Environment setup in simulation and real-world settings. The red square denotes the goal space. The blue square represents the robot’s workspace. In real-world settings, both spaces coincide to impose stricter constraints on navigation and positioning.
Fig. 5 : An example of a maneuver executed by ManeuverNet in both simulation and real-world settings is shown in the first two rows. Another example, showcasing multi-terrain performance, is presented in the final row of figures. The red dot represents Xd .
Approach
Approach
Test Unseen: seed = 10
Type
SR ↑
AE( σ ) ↓
SPL ↑
Analytical
TEB [ 12 ]
100
0.05 (0.05)
0.79
DRL
Standard DRL [ 32 ]
45
0.25 (0.17)
0.35
FastRLap [ 22 ]
00
3.79 (2.11)
0.00
ManeuverNet (ours)
85
0.15 (0.04)
0.70
TABLE III : Benchmarking results for an unseen seed and dth=15 cm, with the success rate (SR) in %, the average error (AE) (the standard deviation σ ) in m, and the average SPL.
Rew.
dth
Seen
Unseen 1
Unseen 2
(cm)
SR ↑
AE( σ ) ↓
SPL ↑
SR ↑
AE( σ ) ↓
SPL ↑
SR ↑
AE( σ ) ↓
SPL ↑
RCar
15
02
0.19(0.62)
0.00
02
0.17(0.75)
0.02
02
0.17(0.62)
0.02
10
02
0.19(0.63)
0.00
02
0.18(0.76)
0.02
02
0.17(0.63)
0.02
RExp
15
04
0.53(0.20)
0.03
04
0.55(0.25)
0.03
06
0.55(0.26)
0.04
10
02
0.53(0.24)
0.01
02
0.56(0.24)
0.01
00
0.57(0.24)
0.00
RHER
15
55
0.25(0.21)
0.47
50
0.28(0.25)
0.40
55
0.30(0.26)
0.45
TABLE IV : ManeuverNet simulation results for different reward functions on seen, unseen 1, and unseen 2 seeds, with the success rate (SR) in %, the average error (AE) (the standard deviation σ ) in m, and the average SPL.
Approach
Target ID
#Param.
1
2
3
4
5
6
SPL ↑ [Time ↓ ]
TEB [ 12 ]
0.21 [9]
0.31 [4]
0.63 [7]
0.93 [4]
0.28 [7]
0.37 [5]
>10
ManeuverNet
0.40 [4]
0.32 [4]
1.00 [3]
0.99 [3]
0.41 [5]
0.37 [5]
0
Δ+ ( % )
90 [56]
03 [00]
60 [57]
06 [25]
46 [29]
00 [00]
−
TABLE V : Real-world comparison of SPL and [navigation time] (in seconds) across six target positions. The final column reports the number of parameters requiring manual tuning for real-world deployment. Δ+ denotes the performance improvement of ManeuverNet relative to the TEB planner.
Robust deployment of deep reinforcement learning (DRL) policies on real robots remains challenging due to discrepancies between simulation and real-world dynamics. We address this issue in the context of maneuvering with double-Ackermann-steering mobile robots, which introduce additional constraints due to their non-holonomic nature. Building upon the DRL framework ManeuverNet, we extend its objective from position control to full pose control, resulting in a more challenging task. We further investigate the impact of actuation-related uncertainties on policy transfer. The use of simplified actuation models during training of the extended policy can lead to poor generalization, shown by a success rate drop from 100% in PyBullet to 25% in Gazebo under stricter evaluation conditions. To address this limitation, we adopt a sim-to-sim-to-real approach, where actuation effects observed in Gazebo are incorporated into the PyBullet training environment. Using multi-environment DRL with SAC and CrossQ, we learn policies that remain robust despite modeling inaccuracies. This approach can significantly reduce the performance gap across simulators, achieving up to 92% success rate in Gazebo and maintaining 69% under stricter thresholds, with successful transfer to a real robot without additional tuning.
Oussama Zaim, Mélodie Daniel, Aly Magassouba +2
Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, F-33400 Talence, France. · School of Computer Science, University of Nottingham, UK. · Instituto de Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, 50018 Zaragoza, Spain.
We present a modular framework to benchmark new and existing methods for trajectory planning and control in high-acceleration maneuvers that push autonomous driving to the limits. Our framework includes time-optimal raceline generation, online time-optimal velocity replanning, geometric path tracking controllers, and a new model-structured neural network (MS-NN) to learn the inverse dynamics for steering control. We deploy our framework on a 1:10-scale RoboRacer platform, using two circuits. Through several ablations with cautious and aggressive racelines, we study the performance of single modules and their combinations. We show that our MS-NN significantly improves tracking accuracy, decreases steering oscillations, and is physically interpretable. Moreover, online velocity replanning improves lap times by compensating for execution errors, and enables the vehicle to safely reach higher speeds and accelerations. To support future research, our code, datasets, videos and results are publicly available at https://roboracer-benchmark.github.io/planning_control_benchmark/.
Mattia Piccinini, Patrick Zambiasi, Aniello Mungiello +3
Professorship of Autonomous Vehicle Systems, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI). · Avilus GmbH, Germany. · Department of Information Technology and Electrical Engineering (DIETI), University of Naples Federico II, Naples 80125, Italy. +1
Drones equipped with overhead manipulators offer unique capabilities for inspection, maintenance, and contact-based interaction. However, the motion of the drone and its manipulator is tightly linked, and even small attitude changes caused by wind or control imperfections shift the end-effector away from its intended path. This coupling makes reliable tracking difficult and also limits the direct use of learning-based arm controllers that were originally designed for fixed-base robots. These effects appear consistently in our tests whenever the UAV body experiences drift or rapid attitude corrections. To address this behavior, we develop a reinforcement-learning (RL) framework with a transformer-based double deep Q learning (DDQN), with the core idea of using an adaptive beam-search planner that applies a short-horizon beam search over candidate control sequences using the learned critic as the forward estimator. This allows the controller to anticipate the end-effector's motion through simulated rollouts rather than executing those actions directly on the actual model, realizing a software-in-the-loop (SITL) approach. The lookahead relies on value estimates from a Transformer critic that processes short sequences of states, while a DDQN backbone provides the one-step targets needed to keep the learning process stable. Evaluated on a 3-DoF aerial manipulator under identical training conditions, the proposed meta-adaptive planner shows the strongest overall performance with a 10.2% reward increase, a substantial reduction in mean tracking error (from about 6% to 3%), and a 29.6% improvement in the combined reward-error metric relative to the DDQN baseline. Our method exhibits elevated stability in tracking target tip trajectory (by maintaining 5 cm tracking error) when the drone base exhibits drifts due to external disturbances, as opposed to the fixed-beam and Transformer-only variants.