Human Centered Non Intrusive Driver State Modeling Using Personalized Physiological Signals in Real World Automated Driving

David Puertas-Ramirez; Raul Fernandez-Matellan; David Martin Gomez; Jesus G. Boticario

cs.HCApr 13, 2026

Human Centered Non Intrusive Driver State Modeling Using Personalized Physiological Signals in Real World Automated Driving

Authors: David Puertas-Ramirez, Raul Fernandez-Matellan, David Martin Gomez, Jesus G. Boticario

Organizations: UNED, Spain · UC3M, Spain

Abstract

In vehicles with partial or conditional driving automation (SAE Levels 2-3), the driver remains responsible for supervising the system and responding to take-over requests. Therefore, reliable driver monitoring is essential for safe human-automation collaboration. However, most existing Driver Monitoring Systems rely on generalized models that ignore individual physiological variability. In this study, we examine the feasibility of personalized driver state modeling using non-intrusive physiological sensing during real-world automated driving. We conducted experiments in an SAE Level 2 vehicle using an Empatica E4 wearable sensor to capture multimodal physiological signals, including electrodermal activity, heart rate, temperature, and motion data. To leverage deep learning architectures designed for images, we transformed the physiological signals into two-dimensional representations and processed them using a multimodal architecture based on pre-trained ResNet50 feature extractors. Experiments across four drivers demonstrate substantial interindividual variability in physiological patterns related to driver awareness. Personalized models achieved an average accuracy of 92.68%, whereas generalized models trained on multiple users dropped to an accuracy of 54%, revealing substantial limitations in cross-user generalization. These results underscore the necessity of adaptive, personalized driver monitoring systems for future automated vehicles and imply that autonomous systems should adapt to each driver's unique physiological profile.

Figures & tables

Figure 1. Snapshot of human centered DMS recording in a Real World Scenario. From left to right: Intel RealSense (Color), OpenPilot (Interior), OpenPilot(Exterior) Composition of three images during an experiment where the user is looking up while holding the steering wheel: (left) color camera showing whole body, (middle) infrared camera showing upper body, (right) camera showing the road in front of the vehicle

Figure 2. Visual representation of parameters during image generation from physiological signals: local/global maxima and minima, saturation values, working window width and step between consecutive images. A sample graph illustrating an example where the global and local maximum and minimum peaks are highlighted along with their respective local values. The temporal context of the working window is also shown, with a highlighted section representing the window size and step. A clipping area is displayed, indicating the boundaries of the signal domain. This visualization illustrates how to analyze signal behavior over time.

Figure 3. Architecture of the proposed ResNet model Seven inputs (acc_x, acc_y, acc_z, bvp, eda, hr and temp) each goes to a different Resnet50 system. Each of them produces a 2048 wide result. This 2048x7 output is fed into a PCA to reduce it to 100 features. A final classifier gives the final results in one of the levels: LL, L, H, HH

Encoding Method	Accuracy
Continuous RP	70.33%
MTF (4 levels)	63.39%
Binary RP	61.90%
GASF	60.11%
MTF (128 levels)	58.90%
GADF	49.70%

Table 1. Impact of Signal-to-Image Encoding Methods on Accuracy (User 1). Continuous Recurrence Plots (RP) provided the most robust feature representation.

	Trained on
Tested on	User 1	User 2	User 3	User 4
User 1	87.52 $\pm$ 0.78%	50.22 $\pm$ 2.29%	58.37 $\pm$ 1.95%	51.85 $\pm$ 3.31%
User 2	46.26 $\pm$ 2.60%	91.79 $\pm$ 0.63%	41.64 $\pm$ 3.58%	40.62 $\pm$ 4.12%
User 3	63.52 $\pm$ 3.47%	50.88 $\pm$ 2.23%	94.88 $\pm$ 1.21%	57.44 $\pm$ 1.54%
User 4	61.39 $\pm$ 4.83%	48.00 $\pm$ 1.78%	56.17 $\pm$ 2.58%	96.52 $\pm$ 1.63%

Table 2. Cross-Subject Accuracy Matrix. Rows represent the user being tested, and columns represent the user data used for training. The diagonal (bold) highlights the personalized performance (mean

\pm

standard deviation).

	Trained on Users
Tested on	{2, 3, 4}	{1, 3, 4}	{1, 2, 4}	{1, 2, 3}
User 1	57.55 $\pm$ 3.22%	82.26 $\pm$ 1.64%	83.57 $\pm$ 1.43%	81.88 $\pm$ 0.87%
User 2	86.26 $\pm$ 1.51%	45.23 $\pm$ 2.91%	85.23 $\pm$ 2.62%	82.56 $\pm$ 4.51%
User 3	91.36 $\pm$ 1.43%	87.04 $\pm$ 3.01%	57.12 $\pm$ 3.03%	85.76 $\pm$ 3.01%
User 4	86.61 $\pm$ 1.80%	85.57 $\pm$ 3.82%	86.09 $\pm$ 1.23%	55.13 $\pm$ 2.99%

Table 3. Combined-User Accuracy Matrix. Columns represent the specific subset of users used for training. The diagonal (bold) highlights the ”Leave-One-Out” performance, where the tested user was excluded from the training set.

Model Approach	Accuracy	Precision	Recall	F1-Score
Personalized	92.68 $\pm$ 3.67%	92.88 $\pm$ 3.49%	92.64 $\pm$ 3.65%	92.67 $\pm$ 3.57%
Combined (General)	54.00 $\pm$ 6.00%	55.28 $\pm$ 3.31%	53.93 $\pm$ 5.40%	52.76 $\pm$ 4.87%
Significance ( $p$ )	$<0.001$	$<0.001$	$<0.001$	$<0.001$

Table 4. Overall Performance Comparison. The Personalized approach significantly outperforms the Combined (Generalized) approach across all metrics (

p<0.001

). Values are reported as Mean

\pm

Standard Deviation. Bold values indicate the best performance in each column.

Figure 4. Class-wise Recall Analysis. The Personalized models (Blue) maintain high sensitivity across all awareness states. In contrast, the Combined models (Orange) show significant degradation. A matrix of four bar charts, one for each study participant (Users 1–4). Each chart compares the recall of a Personalized Model (blue bars) against a Combined Model (orange bars) across four awareness states: Low-Low, Low, High, and High-High. The personalized models maintain consistently high recall above 90\%, while the combined models show high variability and significant performance drops — Figure 4. Class-wise Recall Analysis. The Personalized models (Blue) maintain high sensitivity across all awareness states. In contrast, the Combined models (Orange) show significant degradation. A matrix of four bar charts, one for each study participant (Users 1–4). Each chart compares the recall of a Personalized Model (blue bars) against a Combined Model (orange bars) across four awareness states: Low-Low, Low, High, and High-High. The personalized models maintain consistently high recall above 90%, while the combined models show high variability and significant performance drops

Driver Monitoring

Driver monitoring uses in-cabin cameras and other sensors to detect driver states and support safer, more responsive vehicles.

See all 30 papers

Risk-Aware Selective Multimodal Driver Monitoring with Driver-State World Modeling
Jun 25, 2026Daosheng Qiu, Haozhuang Chi, Hao Su +4Selective Prediction World Models
Hubei University, Wuhan, China · Nanyang Technological University, Singapore · Osaka University, Osaka, Japan
Physiologically Grounded Driver Behavior Classification: SHAP-Driven Elite Feature Selection and Hybrid Gradient Boosting for Multimodal Physiological Signals
May 6, 2026Sahar Askari, Mohammad Mahdi Mirza Ali Mohammadi, Fatemeh Ensafdoust +2Multimodal Sensor Fusion
Department of Computer Science and Engineering Shiraz University Shiraz, Iran · Department of Electrical Engineering Iran University of Science and Technology Tehran, Iran · Department of Computer Engineering Islamic Azad University, Rasht Branch Rasht, Iran +2
Drowsiness-Aware Adaptive Autonomous Braking System based on Deep Reinforcement Learning for Enhanced Road Safety
Apr 15, 2026Hossem Eddine Hafidi, Elisabetta De Giovanni, Teodoro Montanaro +3Autonomous Driving RL for Autonomous Driving
University of Salento, Lecce, Italy. · Istituto Italiano di Tecnologia (IIT), Lecce, Italy. · Basque Center for Applied Mathematics (BCAM), Bilbao, Spain

Jun 25, 2026J/K move · Enter open · S save

Risk-Aware Selective Multimodal Driver Monitoring with Driver-State World Modeling

Daosheng Qiu, Haozhuang Chi, Hao Su, Shu Long+3

Hubei University, Wuhan, China · Nanyang Technological University, Singapore · Osaka University, Osaka, Japan

Selective Prediction World Models Cost-Aware Inference Driver Monitoring Multimodal Sensor Fusion

Open paper AskPDF arXiv

Continuous driver monitoring in automated vehicles requires low-latency inference while avoiding unsafe decisions under uncertain driver states. Large vision-language models provide broad multimodal priors, but their latency and limited reliability in this setting make them unsuitable as always-on in-cabin monitors. We propose a cost-aware selective inference framework for deployable multimodal driver monitoring. The core system is a lightweight RGB-physiological student that combines in-cabin visual observations with window-level HR/EDA signals, and a learned gate that decides when to accept the fast prediction or abstain for safety intervention. Additional controls show that the learned scores contain sample-level information beyond scenario priors, while exact physiological synchronization remains a limitation. To incorporate predictive evidence, we further study a compact driver-state world modeling module that rolls out latent driver-state features and estimates future fast-model errors and counterfactual system-level action costs. On scenario-induced driver-demand recognition, the RGB-physiological student improves over RGB-only and physiology-only baselines, reaching 0.7440 Macro-F1 and 0.9099 balanced accuracy with 11.39M parameters and 3.08ms inference latency. Cost-aware selective inference reduces unsafe false negatives from 17.37% under always-fast inference to approximately 5% across seeds, while maintaining deployment-level latency. While driver-state world modeling offers valuable predictive signals, worst-group evaluations highlight persistent operating-point calibration drift. Ultimately, reliable edge driver monitoring requires advancing not only perception backbones, but also risk-aware selective control and group-robust calibration.

Abstract

Driver Monitoring

Explore similar work

Risk-Aware Selective Multimodal Driver Monitoring with Driver-State World Modeling

Physiologically Grounded Driver Behavior Classification: SHAP-Driven Elite Feature Selection and Hybrid Gradient Boosting for Multimodal Physiological Signals

Drowsiness-Aware Adaptive Autonomous Braking System based on Deep Reinforcement Learning for Enhanced Road Safety