Does Knowledge About Perceptual Uncertainty Help an Agent in Automated Driving?

Natalie Grabowsky; Annika Mütze; Joshua Wendland; Nils Jansen; Matthias Rottmann

cs.CVFeb 17, 2025

Does Knowledge About Perceptual Uncertainty Help an Agent in Automated Driving?

Authors: Natalie Grabowsky, Annika Mütze, Joshua Wendland, Nils Jansen, Matthias Rottmann

Organizations: Institute of Mathematics Technical University of Berlin Berlin, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Faculty of Computer Science Ruhr University Bochum Bochum, Germany

Abstract

Agents in real-world scenarios like automated driving deal with uncertainty in their environment, in particular due to perceptual uncertainty. Although, reinforcement learning is dedicated to autonomous decision-making under uncertainty these algorithms are typically not informed about the uncertainty currently contained in their environment. On the other hand, uncertainty estimation for perception itself is typically directly evaluated in the perception domain, e.g., in terms of false positive detection rates or calibration errors based on camera images. Its use for deciding on goal-oriented actions remains largely unstudied. In this paper, we investigate how an agent's behavior is influenced by an uncertain perception and how this behavior changes if information about this uncertainty is available. Therefore, we consider a proxy task, where the agent is rewarded for driving a route as fast as possible without colliding with other road users. For controlled experiments, we introduce uncertainty in the observation space by perturbing the perception of the given agent while informing the latter. Our experiments show that an unreliable observation space modeled by a perturbed perception leads to a defensive driving behavior of the agent. Furthermore, when adding the information about the current uncertainty directly to the observation space, the agent adapts to the specific situation and in general accomplishes its task faster while, at the same time, accounting for risks.

Figures & tables

Panel of Figure 1 — Fig. 1 : An illustration of the main idea of our experiments in the CARLA driving simulator. An agent perceives its environment through a semantic segmentation mask in a bird’s eye view, which we perturb in a controlled manner. This corresponds to perfectly quantifiable perceptual uncertainties, which we can provide to the agent. In our study, we investigate whether the agent benefits from this uncertainty information.

Panel of Figure 2 — Fig. 2 : We consider four scenarios differing in perturbation of the observation space as well as the availability of the uncertainty information. Green color indicates the scenario-dependent components of the observation space.

Panel of Figure 3 — Fig. 3 : Observation space setup. The observation space dimension increases when the uncertainty is added. We distinguish between three components: 1) Vision component – consisting of a flattened and gray-scaled semantic segmentation image; 2) Non-visual component - containing, e.g., velocity; 3) Uncertainty component – containing the one-hot encoded uncertainty information.

Panel of Figure 4 — Fig. 4 : A summary of the perceptual perturbations used in our experiments. Four different perturbations are used for simulating uncertainty in perception. The last case stands for a correct perception. We use the abbreviations for visible (V), agent/ego-vehicle (E) and invisible (X).

Fig. 6 : Left: Reward progression of the training and validation for scenario 1 - correct perception without uncertainty information. Right: Distance to the front vehicle during one episode.

Panel of Figure 6 — Fig. 7 : Left: Human driving behavior in our simulator settings. Right: An example of the agent’s driving behavior during the MPC case with fixed time steps for switching the case.

Fig. 8 : Left: Reward progression when training in scenario 2 with visual perturbation but no uncertainty information. Right: Reward progression when training in scenario 3 with uncertainty information.

Panel of Figure 8 — Fig. 9 : The agent’s behavior in terms of distance traveled, time steps per episode elapsed, frequency of brake action and, the distance to the front vehicle. These are presented for the perturbation case (indicated in the titles) and the experiments 1–3.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Case	Trav. dist. [m]	Brake [%]	Time [steps]	Dist. front veh. [m]	Vel. [tr. di./time]
VEXV	$\;43.80\pm\;2.03$	$15\pm\;\,\,2$	$\;\;327\pm\;\;\;\;43$	$24.90\pm 0.67$	$0.1339$
VEXX	$\;43.76\pm\;1.95$	$17\pm\;\,\,3$	$\;\;329\pm\;\;\;\;44$	$24.82\pm 0.61$	$0.1330$
XEXX	$\;43.42\pm\;3.36$	$17\pm\;\,\,3$	$\;\;329\pm\;\;\;\;46$	$24.90\pm 0.61$	$0.1320$
VEVV	$141.28\pm 20.75$	$57\pm 12$	$3227\pm 1667$	$22.99\pm 8.75$	$0.0438$
MPC	$\;\;53.00\pm 12.51$	$22\pm\;\,\,9$	$\;\;552\pm\;\;289$	$22.70\pm 2.27$	$0.0960$

Appendix

TABLE I : Quantitatively values for the behavior analysis of Experiment 1 – Correct perception without uncertainty information. Trav. dist. shows the traveled distance in meters. Brake describes the frequency how often the action was brake. Time describes the average length of an validation episode in time steps. The distance to the next front vehicle is illustrated in dist. front veh.. The last column describes the average velocity per time step.

Case	Trav. dist. [m]	Brake [%]	Time [steps]	Dist. front veh. [m]	Vel [tr. di./time]
VEXV	$\;\;98.49\pm 18.20$	$58\pm\;\;2$	$1654\pm\;\;484$	$\;\;31.71\pm\;\;3.14$	$0.0596$
VEXX	$103.46\pm 21.88$	$62\pm\;\;8$	$2202\pm 1673$	$\;\;32.36\pm\;\;7.01$	$0.0470$
XEXX	$\;\;45.90\pm 12.28$	$95\pm\;\;1$	$7415\pm\;\;652$	$109.94\pm 15.23$	$0.0062$
VEVV	$136.80\pm 22.59$	$62\pm\;\;9$	$3343\pm 1737$	$\;\;29.15\pm\;\;8.37$	$0.0409$
MPC	$132.75\pm 22.72$	$70\pm 10$	$3818\pm 2009$	$\;\;38.39\pm 11.66$	$0.0348$

Appendix

TABLE II : Quantitatively values for the behavior analysis of Experiment 2 – Perturbed perception without uncertainty information. Trav. dist. shows the traveled distance in meters. Brake describes the frequency how often action was brake. Time describes the average length of an validation episode in time steps. The distance to the next front vehicle is illustrated in dist. front veh.. The last column describes the average velocity per time step.

Case	Reward	Done	Trav. dist.	Brake	Time	Dist. front. veh.	Vel
VEXV	$2.5212$ ( $\nearrow$ )	$\,\;\;\;\,\,\,3.33$ ( $\nearrow$ )	$2.2486$ ( $\nearrow$ )	$3.8667$ ( $\nearrow$ )	$5.0581$ ( $\nearrow$ )	$1.2735$ ( $\nearrow$ )	$0.4451$ ( $\searrow$ )
VEXX	$2.7199$ ( $\nearrow$ )	$\;\,\,\,\,10.00$ ( $\nearrow$ )	$2.3643$ ( $\nearrow$ )	$3.6471$ ( $\nearrow$ )	$6.6930$ ( $\nearrow$ )	$1.3038$ ( $\nearrow$ )	$0.3534$ ( $\searrow$ )
XEXX	$0.9750$ ( $\rightarrow$ )	$\,\,\,\,\;\;\;1.67$ ( $\nearrow$ )	$1.0571$ ( $\rightarrow$ )	$5.5882$ ( $\nearrow$ )	$1.0359$ ( $\nearrow$ )	$4.4153$ ( $\nearrow$ )	$0.0470$ ( $\searrow$ )
VEVV	$0.9312$ ( $\rightarrow$ )	$-15.00$ ( $\searrow$ )	$0.9683$ ( $\rightarrow$ )	$1.0877$ ( $\rightarrow$ )	$1.0359$ ( $\rightarrow$ )	$1.2679$ ( $\rightarrow$ )	$0.9338$ ( $\searrow$ )
MPC	$3.0799$ ( $\nearrow$ )	$\,\,\;\,\,58.33$ ( $\nearrow$ )	$2.5047$ ( $\nearrow$ )	$3.1818$ ( $\nearrow$ )	$6.0167$ ( $\nearrow$ )	$1.6912$ ( $\nearrow$ )	$0.3625$ ( $\searrow$ )

Appendix

TABLE III : Comparison between Experiment 2 and Experiment 1. The arrows show in direction of perturbed perception in comparison to correct perception. If

\nearrow

the value of perturbed perception is larger than the mean plus the standard deviation of the correct one, else if

\searrow

the values are smaller and if

\rightarrow

the results are in the same range at least in the standard deviation range. The values are defined as the ratio between the behavior of the agent trained on perturbed perception and the agent trained on the correct perception. The value for Done is the difference between the perturbed perception and correct perception, measured in percentage points.

Case	Trav. dist. [m]	Brake [%]	Time [steps]	Dist. front veh. [m]	Vel. [tr.di/time]
VEXV	$\;\;93.88\pm 11.66$	$53\pm\;\;8$	$1656\pm 1112$	$30.47\pm\;\;8.13$	$0.0567$
VEXX	$\;\;96.33\pm 16.32$	$52\pm\;\;3$	$1532\pm\;\;353$	$28.66\pm\;\;2.48$	$0.0629$
XEXX	$\;\;80.12\pm 27.62$	$83\pm 15$	$6003\pm 2465$	$77.57\pm 25.70$	$0.0133$
VEVV	$124.62\pm 32.35$	$57\pm\;\;9$	$2546\pm 1610$	$24.43\pm\;\;7.81$	$0.0489$
MPC	$115.93\pm 28.64$	$57\pm 11$	$2516\pm 1799$	$29.35\pm\;\;8.32$	$0.0489$

Appendix

TABLE IV : Quantitatively values for the behavior analysis of Experiment 3 – Perturbed perception with uncertainty information. The agent is trained on the observation space including perturbed perception and uncertainty information. Trav. dist. shows the traveled distance in meters. Brake describes the frequency how often the action is brake. Time describes the average length of a validation episode in time steps. The distance to the next front vehicle is illustrated in dist. front veh.. The last column describes the average velocity per time step.

Case	Reward	Done	Trav. dist.	Brake	Time	Dist. front veh.	Vel.
VEXV	$0.9455$ ( $\rightarrow$ )	$\;\;-3.33$ ( $\searrow$ )	$0.9532$ ( $\rightarrow$ )	$0.9138$ ( $\rightarrow$ )	$1.0012$ ( $\rightarrow$ )	$0.9609$ ( $\rightarrow$ )	$0.9513$ ( $\searrow$ )
VEXX	$0.9060$ ( $\rightarrow$ )	$\;\;-8.33$ ( $\searrow$ )	$0.9311$ ( $\rightarrow$ )	$0.8387$ ( $\nearrow$ )	$0.6957$ ( $\rightarrow$ )	$0.8857$ ( $\nearrow$ )	$1.3383$ ( $\nearrow$ )
XEXX	$2.0267$ ( $\nearrow$ )	$\;\;-1.67$ ( $\searrow$ )	$1.7451$ ( $\nearrow$ )	$0.8737$ ( $\rightarrow$ )	$0.8096$ ( $\rightarrow$ )	$0.7056$ ( $\nearrow$ )	$2.1252$ ( $\nearrow$ )
VEVV	$0.9045$ ( $\rightarrow$ )	$-10.00$ ( $\searrow$ )	$0.9110$ ( $\rightarrow$ )	$0.9194$ ( $\rightarrow$ )	$0.7616$ ( $\rightarrow$ )	$0.8381$ ( $\rightarrow$ )	$1.1956$ ( $\nearrow$ )
MPC	$0.8387$ ( $\rightarrow$ )	$-25.00$ ( $\searrow$ )	$0.8733$ ( $\rightarrow$ )	$0.8143$ ( $\rightarrow$ )	$0.6590$ ( $\rightarrow$ )	$0.7645$ ( $\rightarrow$ )	$1.1164$ ( $\nearrow$ )

Appendix

TABLE V : Comparison between Experiment 3 and Experiment 2. The arrows point in direction of the perturbed perception with uncertainty information in comparison to perturbed perception without uncertainty information. I.e., if

\nearrow

the value of perturbed perception with perceptual uncertainty information is significantly larger than without, else if

\searrow

the values are smaller and if

\rightarrow

the results are in the same range at least in the standard deviation range. The values are the ratio between the perturbed perception concatenated with uncertainty information and the perturbed perception without uncertainty information in the observation space. The column Done is the difference between these two validations, measured in percentage point.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

Breaking the Epistemic Trap: Active Perception Under Compound Uncertainty
May 26, 2026Chayan Banerjee, Ethan GoanActive Perception Constrained RL
School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, Australia.
UECP: Uncertainty-Enhanced Collaborative Perception
Jun 22, 2026Kang Yang, Tianci Bu, Peng Wang +3Uncertainty Quantification Autonomous Driving Perception
Renmin University of China, Beijing, China · College of Systems Engineering, National University of Defense Technology, Changsha, China · Intelligent Shipping Center, China Waterborne Transport Research Institute, Beijing, China
Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies
Oct 14, 2026Sajad Marvi, Julian Schmidt, Alessandro Canevaro +2Uncertainty-Aware Exploration Autonomous Driving Planning
Mercedes-Benz AG, Germany. · Department of Computer Science, University of Freiburg, Germany.

May 26, 2026J/K move · Enter open · S save

Breaking the Epistemic Trap: Active Perception Under Compound Uncertainty

Chayan Banerjee, Ethan Goan

School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, Australia.

Active Perception Constrained RL RL Exploration Safe RL Epistemic Uncertainty

Open paper AskPDF arXiv

Deploying reinforcement learning in safety critical domains, from autonomous vehicles to medical decision support, is constrained by failures arising when systems encounter unfamiliar conditions. We argue that the fundamental bottleneck is not individual challenges like changing dynamics or incomplete observations, but their synergistic interaction, which we term the Epistemic Trap: agents cannot estimate their state without knowing system dynamics, nor learn dynamics without accurate state information. Proof-of-concept experiments in simulated locomotion reveal that combining these uncertainties causes failures far worse than either challenge alone, a 77% observed degradation against the 46% additive prediction, demonstrating that compounding failure modes can emerge and, when they do, far exceed what additive reasoning would predict. Conventional approaches typically adopt a passive epistemic stance that cannot resolve this coupled uncertainty. We propose reframing safety as an information problem. We introduce an Adaptive Safety Architecture built around three contributions. First, the Compound Uncertainty Coefficient (

κ

), a mutual-information based metric that quantifies how tightly state and dynamics uncertainties are coupled. Second, information-seeking policies governed by a MaxInfoRL objective that actively probe system dynamics rather than waiting for the environment to reveal itself passively. Third, regime adaptive safety constraints that tighten automatically as epistemic coupling rises. Together, these constitute a paradigm shift from passive robustness to active perception, offering a principled path toward decision making systems that operate under uncertainty, recognize their own ignorance, and act strategically to resolve it.

Abstract

Autonomous Driving Planning

Explore similar work

Breaking the Epistemic Trap: Active Perception Under Compound Uncertainty

UECP: Uncertainty-Enhanced Collaborative Perception

Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies