Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimization

Raphael Trumpp; Denis Hoornaert; Mirco Theile; Marco Caccamo

cs.ROMar 13, 2026

Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimization

Authors: Raphael Trumpp, Denis Hoornaert, Mirco Theile, Marco Caccamo

Organizations: Technical University of Munich, Munich, Germany

Abstract

Residual policy learning (RPL), in which a learned policy refines a static base policy using deep reinforcement learning (DRL), has shown strong performance across various robotic applications. Its effectiveness is particularly evident in autonomous racing, a domain that serves as a challenging benchmark for real-world DRL. However, deploying RPL-based controllers introduces system complexity and increases inference latency. We address this by introducing an extension of RPL named attenuated residual policy optimization ( $α$ -RPO). Unlike standard RPL, $α$ -RPO yields a standalone neural policy by progressively attenuating the base policy, which initially serves to bootstrap learning. Furthermore, this mechanism enables a form of privileged learning, where the base policy is permitted to use sensor modalities not required for final deployment. We design $α$ -RPO to integrate seamlessly with PPO, ensuring that the attenuated influence of the base controller is dynamically compensated during policy optimization. We evaluate $α$ -RPO by building a framework for 1:10-scaled autonomous racing around it. In both simulation and zero-shot real-world transfer to Roboracer cars, $α$ -RPO not only reduces system complexity but also improves driving performance compared to baselines - demonstrating its practicality for robotic deployment. Our code is available at: https://github.com/raphajaner/arpo_racing.

Figures & tables

Fig. 1 : We test our proposed $\alpha$ -RPO method by learning competitive real-world racing behavior with 1:10-scaled autonomous Roboracer cars. Compared to classical RPL, $\alpha$ -RPO attenuates the contribution of the base policy during training, improving final performance while yielding a standalone neural policy at inference time for efficient deployment.

Panel of Figure 2 — Fig. 2 : Learning curves with the episodic values of the return during training, showing both the fraction of the return corresponding only to the lap progress ( left ) and the total sum ( right ). Additionally, the cumulative number of crashes during training is shown ( mid ). The agents are trained for 2.5M interaction steps on 15 different maps.

Racetrack		Total race time $[\downarrow]$						Collisions per lap $[\downarrow]$						Maximum speed $[\uparrow]$
		FTG	Stanley	DRL	BCD	RPL	$\alpha$ -RPO	FTG	Stanley	DRL	BCD	RPL	$\alpha$ -RPO	FTG	Stanley	DRL	BCD	RPL	$\alpha$ -RPO
Train	Abu Dhabi	67.54	50.41	48.50	46.29	48.16	44.02	0.00	0.00	0.03	0.00	0.09	0.00	4.32	5.06	4.81	4.72	5.15	5.15
	London	71.04	55.72	55.73	50.05	52.57	47.58	0.00	0.00	0.20	0.00	0.04	0.00	5.03	4.81	5.00	4.84	5.10	5.29
	Mexico City	55.78	41.15	37.51	37.54	37.78	35.53	0.04	0.00	0.00	0.00	0.00	0.00	4.78	5.21	5.10	4.92	5.37	5.50
	New York	71.88	63.08	54.49	54.00	56.17	51.95	0.00	0.00	0.00	0.00	0.00	0.00	4.58	4.45	5.08	4.95	4.89	5.34
	Paris	51.40	39.74	41.08	35.88	35.78	34.13	0.00	0.00	0.19	0.00	0.00	0.00	4.48	5.32	5.25	5.00	5.41	5.50

TABLE I : Simulated racing results, showing a subset of 7 training and 6 testing racetracks. The Munich racetrack is an SLAM-based map of our own real-world racetrack. A race finishes when 5 laps are completed. The acronym BCD is short for the BC+DRL method. Full results including all maps and DRL+SSL are shown in Appendix C .

Panel of Figure 4 — Fig. 3 : Learning curves showing the progress-return for our ablation studies. We ablate the synchronization trick ( left ), longer $\alpha$ -schedules ( mid ), and design aspects of the DNN ( right ).

Panel of Figure 5 — Fig. 4 : Qualitative comparison of trajectories ( top ) and speed profiles ( bottom ) on three racetracks, showing a single flying lap. Markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Racetrack	Total race time in s [ $\downarrow$ ]
Racetrack	DRL	DRL *	$\alpha$ -RPO	$\alpha$ -RPO *
Toronto	59.95	59.37	54.90	52.49
Madrid	50.30	47.40	46.59	43.41
Munich	31.23	29.43	28.52	26.35

TABLE II : Results for fine-tuned agents of the single best training seed, denoted by * . Fine-tuning continues training only on the specific racetrack for a further 0.5 M interaction steps.

Panel of Figure 7 — Fig. 5 : Real-world trajectories of $\alpha$ -RPO on the Munich racetrack for 5 laps ( top ), including a varaition with two obstacles placed; current speed shown as color bar. Positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown ( bottom ). Markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Metric	Method
Metric	FTG	DRL	$\alpha$ -RPO	$\alpha$ -RPO *
Finishing time in $\mathrm{s}$ [ $\downarrow$ ]	40.2	34.4	30.5	28.1
Difference to simulation in $\mathrm{s}$ [ $\downarrow$ ]	+2.1	+3.2	+2.0	+1.7
Maximum speed in $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ [ $\uparrow$ ]	3.1	4.8	5.0	5.4

TABLE III : Real-world lap times on the Munich track, using the model corresponding to the best seed of each method. Finish times cover races with 5 full laps. Reported values are estimated offline from rosbag data.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Training Hyperparameter
Total steps	2.5e 6	Discount $\gamma$	0.99
Learning rate (LR)	1.5e -4	GAE $\lambda$	0.95
LR schedule	$\cos$	Value coefficient	0.5
Trajectory length	1024	Max. gradient norm	0.5
Clip $\epsilon$	0.2	Update epochs	5
Batch size	512	Initial $\log\sigma_{R,\theta}$	-1.0

Appendix

TABLE IV : Hyperparameters for PPO backend as used in AR-PO, RPL, and DRL.

Vehicle Parameter
Vehicle mass in $\mathrm{kg}$	3.3	Inertia in $\mathrm{kg}\text{\,}{\mathrm{m}}^{2}$	0.0627
Vehicle width in $\mathrm{m}$	0.27	Vehicle length in $\mathrm{m}$	0.51
Distance front axle in $\mathrm{m}$	0.1625	Distance rear axle in $\mathrm{m}$	0.1625
CG height in $\mathrm{m}$	0.02	Steering angle in $\mathrm{rad}$	$\pm 0.45$
Max. velocity in $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$	8.0	Min. velocity in $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$	0.0
Max. acceleration in $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$	$4.9$	Max. braking in $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$	- $3.7$

Appendix

TABLE V : Simulation parameters for the single-track vehicle model.

Fig. 7 : Visualization of the synthetically-generated racetracks used in this work. The shape is inspired by real-world Roboracer competition, including various challenging curvy and high-speed sections. Each racetrack’s width varies between a plus-or-minus $\pm 20\text{\,}\mathrm{\%}$ of the mean width, which is randomly selected between $1.5\text{\,}\mathrm{m}$ and $1.8\text{\,}\mathrm{m}$ . The racetracks are named after famous cities from around the globe; however, there is no actual affiliation with the names. The last row shows the 5 racetracks only used for testing purposes and are not used for training.

Fig. 8 : Visualization of the Munich racetrack, based on a SLAM mapping of our real-world testbed.

Table 14

Panel of Figure 15 — Fig. 9 : Qualitative comparison of trajectories and speed profiles on three training racetracks of $\alpha$ -RPO against Stanley ( left ), DRL ( mid ), and RPL ( right ). A single flying lap is shown, where markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Panel of Figure 16 — Fig. 10 : Qualitative comparison of trajectories and speed profiles on three testing racetracks of $\alpha$ -RPO against Stanley ( left ), DRL ( mid ), and RPL ( right ). A single flying lap is shown, where markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Panel of Figure 17 — Fig. 11 : Real-world trajectories for FTG ( left ) and DRL ( right ) agents on the Munich racetrack for 5 laps; current speed shown as color bar. Positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown. Markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Panel of Figure 18 — Fig. 12 : Trajectories on the Munich racetrack for 5 laps, comparing the $\alpha$ -RPO agent to the finetuned $\alpha$ -RPO * version in simulation ( left ) and showing its real-world behavior ( right ). It can be seen that $\alpha$ -RPO * shows superior performance, particularly in the left-right section at the top, by cutting it straight with higher speed. The real-world trajectory shows the current speed as a color bar; positions are estimated offline from rosbag data using SLAM. The agent’s actions command (cmd) and measured speed (meas.) are also shown. Markers indicate a specific POI on the racetracks: $+$ marks a high-speed section (red), $-$ indicates the section with minimal speed (blue), while $\mathrm{F}$ is the finish line (yellow).

Fig. 13 : Proposed DNN architecture. The input to the LidarEncoder is the stacked, processed LiDAR observations

L^{\prime}_{t}

, and the StateEncoder input is the normalized vehicle state

s^{\prime}_{t}

, which is the subset of

s_{t}

without

L_{t}

. The FusionBlock outputs the embeddings

z^{\prime}_{t}

, which are fed into the three independent heads: PolicyHead , LogStdHead , and CriticHead .

Layer Type	Parameter Count	%	Kernel	Input Size	Output Size
$\llcorner$ Encoder	–	–	–	[B, 4, 1081]	[B, 128]
$\llcorner$ LidarPreprocessor	–	–	–	[B, 4, 1081]	[B, 4, 512]
$\llcorner$ AvgPool1d	–	–	[2]	[B, 4, 1024]	[B, 4, 512]
$\llcorner$ LidarEncoder	–	–	–	[B, 4, 512]	[B, 128]
$\llcorner$ Conv1d	416	0.12%	[3]	[B, 4, 512]	[B, 32, 256]
$\llcorner$ SiLU	–	–	–	[B, 32, 256]	[B, 32, 256]

Appendix

TABLE VIII : Summary with parameters counts per layer of the DNN model.

Explore similar work

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform
Jul 27, 2026Joel Siegert, Edoardo Ghignone, Michele MagnoBackpropagation Continual Robot Learning
Center for Project-Based Learning, D-ITET, ETH Zurich
Learning to Tune Pure Pursuit in Autonomous Racing: Joint Lookahead and Steering-Gain Control with PPO
Feb 20, 2026Mohamed Elgouhary, Amr S. El-WakeelAutonomous Racing Adaptive Control
The authors are with the Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA
Dynamic Lookahead Distance via Reinforcement Learning-Based Pure Pursuit for Autonomous Racing
Mar 30, 2026Mohamed Elgouhary, Amr S. El-WakeelAutonomous Racing RL Control
Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA.

Jul 27, 2026J/K move · Enter open · S save

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform

Joel Siegert, Edoardo Ghignone, Michele Magno

Center for Project-Based Learning, D-ITET, ETH Zurich

Backpropagation Continual Robot Learning Autonomous Racing Generalization in RL Continual RL

Open paper AskPDF arXiv

A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.

Abstract

Explore similar work

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform

Learning to Tune Pure Pursuit in Autonomous Racing: Joint Lookahead and Steering-Gain Control with PPO

Dynamic Lookahead Distance via Reinforcement Learning-Based Pure Pursuit for Autonomous Racing