Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control

Zhuoli Zhuang; Yu-Cheng Chang; Yu-Kai Wang; Thomas Do; Chin-Teng Lin

cs.CVMar 26, 2026

Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control

Authors: Zhuoli Zhuang, Yu-Cheng Chang, Yu-Kai Wang, Thomas Do, Chin-Teng Lin

Organizations: University of Technology Sydney

Abstract

Recent advancements in computer vision have accelerated the development of autonomous driving. Despite these advancements, training machines to drive in a way that aligns with human expectations remains a significant challenge. Human factors are still essential, as humans possess a sophisticated cognitive system capable of rapidly interpreting scene information and making accurate decisions. Aligning machine with human intent has been explored with Reinforcement Learning with Human Feedback (RLHF). Conventional RLHF methods rely on collecting human preference data by manually ranking generated outputs, which is time-consuming and indirect. In this work, we propose an electroencephalography (EEG)-guided decision-making framework to incorporate human cognitive insights without behaviour response interruption into reinforcement learning (RL) for autonomous driving. We collected EEG signals from 20 participants in a realistic driving simulator and analyzed event-related potentials (ERP) in response to sudden environmental changes. Our proposed framework employs a neural network to predict the strength of ERP based on the cognitive information from visual scene information. Moreover, we explore the integration of such cognitive information into the reward signal of the RL algorithm. Experimental results show that our framework can improve the collision avoidance ability of the RL algorithm, highlighting the potential of neuro-cognitive feedback in enhancing autonomous driving systems. Our project page is: https://alex95gogo.github.io/Cognitive-Reward/.

Figures & tables

Figure 1: The framework of the human cognitive reward model. First, raw EEG data are preprocessed, and the most prominent feature, which is the ERP, is extracted. Second, an EEG feature prediction model predicts the ERP from the scene images. Lastly, the prediction probability is used as part of the reward of reinforcement learning for autonomous driving tasks.

Dataset	Active ctlr.	Vehicle data	Camera	Hazards	View	#subjects	# Frames	EEG	Gaze
Ours	+	+	$S^{rgb,depth,sem}$	+	VR	20	720K	+	+
Collision Threat [ 38 ]	+	-	-	-	screen	25	-	+	-
Sustained-attention [ 6 ]	+	+	-	+	screen	27	-	+	-
MPDB [ 51 ]	+	-	-	+	screen	35	-	+	+
DrFixD(night) [ 12 ]	-	-	$S^{rgb}$	-	screen	31	67K*	-	+
LBW [ 31 ]	-	-	$S^{rgb,depth}$	-	screen	28	123K*	-	+

Table 1: Comparison between our dataset and other driving datasets. Ours contains the largest number of frames and is collected using a realistic VR driving simulator. Abbreviations: Camera – S: scene-facing camera, RGB: 3-channel image, sem: semantic segmentation mask. Frame counts marked with * are estimates based on video duration and frame rate.

Panel of Figure 2 — Figure 2: Upper: The dataset collection environment. The HTC VIVE Pro Eye VR headset and Logitech G923 Racing Wheel and Pedal give the subject a more realistic driving experience. Lower: the example of scene image of driver’s view, camera’s view, and bird’s eye view.

Figure 3: The policy network determines the vehicle’s policy from a sequence of three segmentation images, starting with a CNN to extract features. These features are then processed through a self-attention layer. The policy network includes two MLP prediction heads: one for estimating the throttle and brake strength and another for predicting TTC, which aids in training regularization.

Figure 4: The average ERP wave from 20 participants. Condition 1: participants actively react to the emergency braking. Condition 2: participants are not required to react actively. The gray region indicates a significant difference between conditions.

Figure 5: Machine attention visualization of the policy network of RL across three time steps in the emergency braking scenario. Our model consistently focuses on the lead vehicle.

	$\mathbf{F_{1}}$	$\mathbf{F_{2}}$	$\mathbf{F_{3}}$	$\mathbf{F_{4}}$	$\mathbf{F_{5}}$	Mean	FPS
Resnet-18	82	89	79	79	76	81	73
Swin-ViT	80	85	75	80	77	79	62
ConvNeXt	82	89	75	81	76	80	74
Ours	80	85	77	86	81	82	204

Table 2: Five-fold cross-validation result and the FPS of the EEG feature prediction model.

	Emergency Braking			Left Turn
	Route Completion $\uparrow$	Driving Score $\uparrow$	Infraction $\uparrow$	Route Completion $\uparrow$	Driving Score $\uparrow$	Infraction $\uparrow$
Vanilla	$23\pm 27$	$16\pm 19$	$0.66\pm 0.07$	$60\pm 16$	$45\pm 19$	$0.68\pm 0.09$
BC	$65\pm 31$	$55\pm 29$	$0.72\pm 0.08$	$48\pm 5$	$29\pm 3$	$0.62\pm 0.04$
PHIL	$59\pm 23$	$44\pm 29$	$0.67\pm 0.14$	$38\pm 32$	$31\pm 30$	$0.68\pm 0.11$
RLHF	$73\pm 32$	$66\pm 39$	$0.80\pm 0.19$	$63\pm 21$	$49\pm 28$	$0.71\pm 0.15$
TD3-lag	$44\pm 33$	$35\pm 34$	$0.72\pm 0.16$	$40\pm 28$	$32\pm 22$	$0.68\pm 0.06$

Table 3: Summary statistics for Route Completion, Driving Scores, Infraction Scores for emergency braking and left-turn scenario.

RL for Autonomous Driving

Reinforcement learning for autonomous driving trains vehicle policies through interaction to plan, control, and coordinate safely in traffic.

See all 120 papers

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving
May 20, 2026Yang Wu, Qiang Meng, Zhaojiang Liu +3End-to-End Autonomous Driving NavSim
SJTU · FDU · NJU
CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Driving
May 27, 2025Zhennan Wang, Jianing Teng, Canqun Xiang +4End-to-End Autonomous Driving Autonomous Driving Perception
HAOMO.AI Technology Co., Ltd

Mar 18, 2026J/K move · Enter open · S save

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Zilin Huang, Zihao Sheng, Zhengyang Wan, Yansong Qu+3

Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada

VLMs for Autonomous Driving LLM-Guided RL Multimodal Reward Modeling RL for Autonomous Driving Safe RL

Open paper AskPDF arXiv

Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings. Recent vision-language models (VLMs) offer promising semantic understanding capabilities; however, their high inference latency and susceptibility to hallucination hinder direct application to real-time vehicle control. To address these limitations, this paper proposes DriveVLM-RL, a neuroscience-inspired framework that integrates VLMs into RL through a dual-pathway architecture for safe and deployable autonomous driving. Inspired by the human brain's habitual and deliberative visual processing, DriveVLM-RL decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment via CLIP-based contrasting language goals, and a Dynamic Pathway for attention-gated multi-frame semantic risk reasoning via a lightweight detection model and large VLM (LVLM). A hierarchical reward synthesis mechanism fuses these signals with vehicle state information, while an asynchronous training pipeline decouples expensive LVLM inference from environment interaction. Critically, all VLM components operate exclusively during offline training and are completely removed at deployment, eliminating inference latency at test time. Extensive experiments in the CARLA simulator demonstrate that DriveVLM-RL significantly outperforms state-of-the-art baselines in collision avoidance and task success, attaining the highest success rate while reducing collision severity from 10.09 to 1.75 km/h relative to the strongest VLM-based baseline. The demo video, code, and model checkpoints are available at: https://zilin-huang.github.io/DriveVLM-RL-website/

Abstract

RL for Autonomous Driving

Explore similar work

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving

CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Driving