As autonomous vehicles (AVs) approach Level 4 and Level 5 operational capability [SAE International, 2018], their on- board decision systems must handle not only safety-critical locomotion but also their subsequent moral weight. This paper details the Ethical Decision Head (EDH), a deep re- inforcement learning (RL) framework that encodes ethical reasoning as a differentiable reward signal, enabling a pol- icy gradient agent to learn morally-aligned driving behavior in scenarios whose state representation is aligned with the CARLA simulation environment [Dosovitskiy et al., 2017]. Two normative frameworks are instantiated and evaluated: a Utilitarian framework minimizing total casualties and a Kan- tian framework enforcing course maintenance as a categori- cal imperative. The EDH is trained via Proximal Policy Op- timization (PPO) [Schulman et al., 2017] against a Bradley- Terry reward model [Bradley and Terry, 1952] learned from pairwise human preference annotations over 200 collision- imminent scenarios. Results reveal an asymmetry in the learnability of normative ethical frameworks under human su- pervision. The Kantian condition, which reduces to a con- stant prediction task under the codebook, serves as a pipeline control: it confirms training stability and rules out infrastruc- ture failure as an explanation for the utilitarian result. The Utilitarian agent learned something more unsettling: human raters rewarded self-sacrifice over casualty minimization, and the model learned that preference faithfully. This divergence between what humans prescribe in theory and what they re- ward in practice suggests that RLHF does not learn ethics as philosophers define it, but as humans live it.
Figures & tables
Figure 1: Iterative codebook development pipeline. Scenario authoring, grading, and inter-grader disagreement analysis cycle until sufficient consistency is reached, producing the ground-truth training labels.
Figure 2: Ground-truth Utilitarian label distribution across all 200 scenarios. Swerve-left constitutes 50% of correct actions, making any policy that fails to learn this action incapable of exceeding 50% accuracy.
Figure 3: Full EDH pipeline architecture. The state vector flows through the policy and value networks, with reward composition branching between Full RLHF and Direct Supervision modes before PPO optimization.
Parameter
Symbol
Value
Discount factor
γ
0.99
GAE smoothing
λGAE
0.95
Rollout buffer size
N
2048
PPO epochs per rollout
Kppo
4
PPO clip threshold
ε
0.2
Entropy coefficient
cent
0.03
Table 1: Training hyperparameters for the EDH.
Figure 4: Training curves comparing Full RLHF (A) and Direct Supervision (B) under the Utilitarian framework over 2,000,000 environment timesteps. Panel A plateaus near the 50% accuracy ceiling imposed by swerve-left collapse following reward model warm-up; Panel B reaches a peak of 90.3%, confirming that the environment reward signal is well-formed and the failure mode is localized to the RLHF pipeline.
Figure 5: Reward signal divergence under Full RLHF. Following the divergence point the proxy reward model saturates while ground truth utility continues to decline, confirming reward model overfitting to human preference noise rather than the normative objective.
Figure 6: Predicted action distributions across all 200 scenarios under each training configuration. Under Full RLHF, the Utilitarian agent never predicts swerve-left, the modal correct action, collapsing instead toward maintain course and swerve-right. Direct Supervision recovers a balanced distribution aligned with ground-truth labels.
Figure 7: A representative collision-imminent scenario as rendered in CARLA for illustrative purposes (left) and its corresponding abstract decision tree (right), showing the three action branches and their pedestrian casualty counts under the Utilitarian framework. The CARLA render is provided for environmental context; the dataset itself consists of codebook-derived 40-dimensional state vectors aligned with the CARLA feature space.
Reinforcement learning from human feedback (RLHF) trains AI systems to better match human preferences using comparisons, corrections, and other feedback.
The overlapping disciplines of machine ethics and value alignment are concerned with designing artificial agents that are aligned with human values and that act in ethically acceptable ways. A recent trend in these disciplines is the use of reinforcement learning (RL) to design such agents, sidelining the philosophical literature that used to play a more central role. Against this backdrop, this paper pursues two goals. The first is to draw out ideas from recent work in metanormative theory that can be useful for designing artificial moral and value-aligned agents. The second is to examine the RL architecture through the lens of these ideas. This will give us clearer criteria for when an RL agent's behavior can be classified as moral, as well as a basis for evaluating and comparing different RL-based approaches to machine ethics and value alignment.
Aleks Knoks, Marija Slavkovik
University of Luxembourg, 2, place de l’Université, L-4365, Esch-sur-Alzette, Luxembourg · University of Bergen, Fosswinckelsgt. 6, 5007 Bergen, Norway
Recent advancements in computer vision have accelerated the development of autonomous driving. Despite these advancements, training machines to drive in a way that aligns with human expectations remains a significant challenge. Human factors are still essential, as humans possess a sophisticated cognitive system capable of rapidly interpreting scene information and making accurate decisions. Aligning machine with human intent has been explored with Reinforcement Learning with Human Feedback (RLHF). Conventional RLHF methods rely on collecting human preference data by manually ranking generated outputs, which is time-consuming and indirect. In this work, we propose an electroencephalography (EEG)-guided decision-making framework to incorporate human cognitive insights without behaviour response interruption into reinforcement learning (RL) for autonomous driving. We collected EEG signals from 20 participants in a realistic driving simulator and analyzed event-related potentials (ERP) in response to sudden environmental changes. Our proposed framework employs a neural network to predict the strength of ERP based on the cognitive information from visual scene information. Moreover, we explore the integration of such cognitive information into the reward signal of the RL algorithm. Experimental results show that our framework can improve the collision avoidance ability of the RL algorithm, highlighting the potential of neuro-cognitive feedback in enhancing autonomous driving systems. Our project page is: https://alex95gogo.github.io/Cognitive-Reward/.
This paper presents a comprehensive comparison of four representative reinforcement learning (RL) methods in autonomous vehicle (AV) decision-making, including: Fitted Value Iteration (FVI), Rollout Classification Policy Iteration (RCPI), Deep Q-Network (DQN), and a discrete-action adaptation of Deep Deterministic Policy Gradient (D-DDPG). The four methods are applied to the same AV decision-making problem at an unsignalized intersection and are compared in terms of optimality and safety, training stability, and training and decision cost. RCPI is the closest to a reference solution and has the most stable training process. However, it is the least data-efficient. D-DDPG reaches the second-smallest gap to a reference solution with the highest data efficiency, but has the least training stability. DQN is the most stable model-free method but has the highest collision rate. FVI achieves a lower collision rate than DQN, but is less data-efficient and less stable. Based on these results, we provide suggestions for selecting a method according to whether a model is available, the budget for training, and whether stable training is required.
Dechen Xie, Mushuang Liu
Department of Mechanical Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA 24060, USA