Residual policy learning (RPL), in which a learned policy refines a static base policy using deep reinforcement learning (DRL), has shown strong performance across various robotic applications. Its effectiveness is particularly evident in autonomous racing, a domain that serves as a challenging benchmark for real-world DRL. However, deploying RPL-based controllers introduces system complexity and increases inference latency. We address this by introducing an extension of RPL named attenuated residual policy optimization (α-RPO). Unlike standard RPL, α-RPO yields a standalone neural policy by progressively attenuating the base policy, which initially serves to bootstrap learning. Furthermore, this mechanism enables a form of privileged learning, where the base policy is permitted to use sensor modalities not required for final deployment. We design α-RPO to integrate seamlessly with PPO, ensuring that the attenuated influence of the base controller is dynamically compensated during policy optimization. We evaluate α-RPO by building a framework for 1:10-scaled autonomous racing around it. In both simulation and zero-shot real-world transfer to Roboracer cars, α-RPO not only reduces system complexity but also improves driving performance compared to baselines - demonstrating its practicality for robotic deployment. Our code is available at: https://github.com/raphajaner/arpo_racing.
Figures & tables
Fig. 1 : We test our proposed α -RPO method by learning competitive real-world racing behavior with 1:10-scaled autonomous Roboracer cars. Compared to classical RPL, α -RPO attenuates the contribution of the base policy during training, improving final performance while yielding a standalone neural policy at inference time for efficient deployment.
Fig. 2 : Learning curves with the episodic values of the return during training, showing both the fraction of the return corresponding only to the lap progress ( left ) and the total sum ( right ). Additionally, the cumulative number of crashes during training is shown ( mid ). The agents are trained for 2.5M interaction steps on 15 different maps.
Racetrack
Total race time [↓]
Collisions per lap [↓]
Maximum speed [↑]
FTG
Stanley
DRL
BCD
RPL
α -RPO
FTG
Stanley
DRL
BCD
RPL
α -RPO
FTG
Stanley
DRL
BCD
RPL
α -RPO
Train
Abu Dhabi
67.54
50.41
48.50
46.29
48.16
44.02
0.00
0.00
0.03
0.00
0.09
0.00
4.32
5.06
4.81
4.72
5.15
5.15
London
71.04
55.72
55.73
50.05
52.57
47.58
0.00
0.00
0.20
0.00
0.04
0.00
5.03
4.81
5.00
4.84
5.10
5.29
Mexico City
55.78
41.15
37.51
37.54
37.78
35.53
0.04
0.00
0.00
0.00
0.00
0.00
4.78
5.21
5.10
4.92
5.37
5.50
New York
71.88
63.08
54.49
54.00
56.17
51.95
0.00
0.00
0.00
0.00
0.00
0.00
4.58
4.45
5.08
4.95
4.89
5.34
Paris
51.40
39.74
41.08
35.88
35.78
34.13
0.00
0.00
0.19
0.00
0.00
0.00
4.48
5.32
5.25
5.00
5.41
5.50
TABLE I : Simulated racing results, showing a subset of 7 training and 6 testing racetracks. The Munich racetrack is an SLAM-based map of our own real-world racetrack. A race finishes when 5 laps are completed. The acronym BCD is short for the BC+DRL method. Full results including all maps and DRL+SSL are shown in Appendix C .
Fig. 3 : Learning curves showing the progress-return for our ablation studies. We ablate the synchronization trick ( left ), longer α -schedules ( mid ), and design aspects of the DNN ( right ).
Fig. 4 : Qualitative comparison of trajectories ( top ) and speed profiles ( bottom ) on three racetracks, showing a single flying lap. Markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Racetrack
Total race time in s [ ↓ ]
DRL
DRL *
α -RPO
α -RPO *
Toronto
59.95
59.37
54.90
52.49
Madrid
50.30
47.40
46.59
43.41
Munich
31.23
29.43
28.52
26.35
TABLE II : Results for fine-tuned agents of the single best training seed, denoted by * . Fine-tuning continues training only on the specific racetrack for a further 0.5 M interaction steps.
Fig. 5 : Real-world trajectories of α -RPO on the Munich racetrack for 5 laps ( top ), including a varaition with two obstacles placed; current speed shown as color bar. Positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown ( bottom ). Markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Metric
Method
FTG
DRL
α -RPO
α -RPO *
Finishing time in s [ ↓ ]
40.2
34.4
30.5
28.1
Difference to simulation in s [ ↓ ]
+2.1
+3.2
+2.0
+1.7
Maximum speed in ms−1 [ ↑ ]
3.1
4.8
5.0
5.4
TABLE III : Real-world lap times on the Munich track, using the model corresponding to the best seed of each method. Finish times cover races with 5 full laps. Reported values are estimated offline from rosbag data.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Training Hyperparameter
Total steps
2.5e 6
Discount γ
0.99
Learning rate (LR)
1.5e -4
GAE λ
0.95
LR schedule
cos
Value coefficient
0.5
Trajectory length
1024
Max. gradient norm
0.5
Clip ϵ
0.2
Update epochs
5
Batch size
512
Initial logσR,θ
-1.0
Appendix
TABLE IV : Hyperparameters for PPO backend as used in AR-PO, RPL, and DRL.
Vehicle Parameter
Vehicle mass in kg
3.3
Inertia in kgm2
0.0627
Vehicle width in m
0.27
Vehicle length in m
0.51
Distance front axle in m
0.1625
Distance rear axle in m
0.1625
CG height in m
0.02
Steering angle in rad
±0.45
Max. velocity in ms−1
8.0
Min. velocity in ms−1
0.0
Max. acceleration in ms−2
4.9
Max. braking in ms−2
- 3.7
Appendix
TABLE V : Simulation parameters for the single-track vehicle model.
Table 11
Fig. 7 : Visualization of the synthetically-generated racetracks used in this work. The shape is inspired by real-world Roboracer competition, including various challenging curvy and high-speed sections. Each racetrack’s width varies between a plus-or-minus ±20% of the mean width, which is randomly selected between 1.5m and 1.8m . The racetracks are named after famous cities from around the globe; however, there is no actual affiliation with the names. The last row shows the 5 racetracks only used for testing purposes and are not used for training.
Fig. 8 : Visualization of the Munich racetrack, based on a SLAM mapping of our real-world testbed.
Table 14
Fig. 9 : Qualitative comparison of trajectories and speed profiles on three training racetracks of α -RPO against Stanley ( left ), DRL ( mid ), and RPL ( right ). A single flying lap is shown, where markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Fig. 10 : Qualitative comparison of trajectories and speed profiles on three testing racetracks of α -RPO against Stanley ( left ), DRL ( mid ), and RPL ( right ). A single flying lap is shown, where markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Fig. 11 : Real-world trajectories for FTG ( left ) and DRL ( right ) agents on the Munich racetrack for 5 laps; current speed shown as color bar. Positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown. Markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Fig. 12 : Trajectories on the Munich racetrack for 5 laps, comparing the α -RPO agent to the finetuned α -RPO * version in simulation ( left ) and showing its real-world behavior ( right ). It can be seen that α -RPO * shows superior performance, particularly in the left-right section at the top, by cutting it straight with higher speed. The real-world trajectory shows the current speed as a color bar; positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown. Markers indicate a specific POI on the racetracks: + marks a high-speed section (red), − indicates the section with minimal speed (blue), while F is the finish line (yellow).
Fig. 13 : Proposed DNN architecture. The input to the LidarEncoder is the stacked, processed LiDAR observations Lt′ , and the StateEncoder input is the normalized vehicle state st′ , which is the subset of st without Lt . The FusionBlock outputs the embeddings zt′ , which are fed into the three independent heads: PolicyHead , LogStdHead , and CriticHead .
Layer Type
Parameter Count
%
Kernel
Input Size
Output Size
└ Encoder
–
–
–
[B, 4, 1081]
[B, 128]
└ LidarPreprocessor
–
–
–
[B, 4, 1081]
[B, 4, 512]
└ AvgPool1d
–
–
[2]
[B, 4, 1024]
[B, 4, 512]
└ LidarEncoder
–
–
–
[B, 4, 512]
[B, 128]
└ Conv1d
416
0.12%
[3]
[B, 4, 512]
[B, 32, 256]
└ SiLU
–
–
–
[B, 32, 256]
[B, 32, 256]
Appendix
TABLE VIII : Summary with parameters counts per layer of the DNN model.
A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.
Joel Siegert, Edoardo Ghignone, Michele Magno
Center for Project-Based Learning, D-ITET, ETH Zurich
Pure Pursuit (PP) is widely used in autonomous racing for real-time path tracking due to its efficiency and geometric clarity, yet performance is highly sensitive to how key parameters-lookahead distance and steering gain-are chosen. Standard velocity-based schedules adjust these only approximately and often fail to transfer across tracks and speed profiles. We propose a reinforcement-learning (RL) approach that jointly chooses the lookahead Ld and a steering gain g online using Proximal Policy Optimization (PPO). The policy observes compact state features (speed and curvature taps) and outputs (Ld, g) at each control step. Trained in F1TENTH Gym and deployed in a ROS 2 stack, the policy drives PP directly (with light smoothing) and requires no per-map retuning. Across simulation and real-car tests, the proposed RL-PP controller that jointly selects (Ld, g) consistently outperforms fixed-lookahead PP, velocity-scheduled adaptive PP, and an RL lookahead-only variant, and it also exceeds a kinematic MPC raceline tracker under our evaluated settings in lap time, path-tracking accuracy, and steering smoothness, demonstrating that policy-guided parameter tuning can reliably improve classical geometry-based control.
Mohamed Elgouhary, Amr S. El-Wakeel
The authors are with the Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA
Pure Pursuit (PP) is a widely used path-tracking algorithm in autonomous vehicles due to its simplicity and real-time performance. However, its effectiveness is sensitive to the choice of lookahead distance: shorter values improve cornering but can cause instability on straights, while longer values improve smoothness but reduce accuracy in curves. We propose a hybrid control framework that integrates Proximal Policy Optimization (PPO) with the classical Pure Pursuit controller to adjust the lookahead distance dynamically during racing. The PPO agent maps vehicle speed and multi-horizon curvature features to an online lookahead command. It is trained using Stable-Baselines3 in the F1TENTH Gym simulator with a KL penalty and learning-rate decay for stability, then deployed in a ROS2 environment to guide the controller. Experiments in simulation compare the proposed method against both fixed-lookahead Pure Pursuit and an adaptive Pure Pursuit baseline. Additional real-car experiments compare the learned controller against a fixed-lookahead Pure Pursuit controller. Results show that the learned policy improves lap-time performance and repeated lap completion on unseen tracks, while also transferring zero-shot to hardware. The learned controller adapts the lookahead by increasing it on straights and reducing it in curves, demonstrating effectiveness in augmenting a classical controller by online adaptation of a single interpretable parameter. On unseen tracks, the proposed method achieved 33.16 s on Montreal and 46.05 s on Yas Marina, while tolerating more aggressive speed-profile scaling than the baselines and achieving the best lap times among the tested settings. Initial real-car experiments further support sim-to-real transfer on a 1:10-scale autonomous racing platform
Mohamed Elgouhary, Amr S. El-Wakeel
Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA.