Multi-task learning for advanced driver assistance systems requires modeling the complex interplay between driver internal states and external traffic environments. However, existing methods treat recognition tasks as flat and independent objectives, failing to exploit the cognitive causal structure underlying driving behavior. In this paper, we propose CauPsi, a cognitive science-grounded causal multi-task learning framework that explicitly models the hierarchical dependencies among Traffic Context Recognition (TCR), Vehicle Context Recognition (VCR), Driver Emotion Recognition (DER), and Driver Behavior Recognition (DBR). The proposed framework introduces two key mechanisms. First, a Causal Task Chain propagates upstream task predictions to downstream tasks via learnable prototype embeddings, realizing the cognitive cascade from environmental perception to behavioral regulation in a differentiable manner. Second, Cross-Task Psychological Conditioning (CTPC) estimates a psychological state signal from driver facial expressions and body posture and injects it as a conditioning input to all tasks including environmental recognition, thereby modeling the modulatory effect of driver internal states on cognitive and decision-making processes. Evaluated on the AIDE dataset, CauPsi achieves a mean accuracy of 82.71% with only 5.05M parameters, surpassing prior work by +1.0% overall, with notable improvements on DER (+3.65%) and DBR (+7.53%). Ablation studies validate the independent contribution of each component, and analysis of the psychological state signal confirms that it acquires systematic task-label-dependent patterns in a self-supervised manner without explicit psychological annotations.
Figures & tables
Figure 1: Overall architecture of CauPsi.
Figure 2: Detailed architecture of the Causal Task Chain.
Pattern
Backbone
αacc (%) ↑
βmacc
P(M)
Multi-view Scene
Driver Images
Joints
DER
DBR
TCR
VCR
(%) ↑
↓
2D
VGG16 ( Simonyan and Zisserman, 2014 )
VGG16 ( Simonyan and Zisserman, 2014 )
3DCNN
69.12
64.57
84.77
74.08
73.15
127.48
Res18 ( He et al., 2016 )
Res18 ( He et al., 2016 )
3DCNN
68.78
64.33
89.76
78.59
75.37
107.77
CMT ( Guo et al., 2022 )
CMT ( Guo et al., 2022 )
3DCNN
68.75
68.75
93.75
81.38
78.16
72.33
GLMDriveNet ( Liu et al., 2024 )
GLMDriveNet ( Liu et al., 2024 )
3DCNN
71.38
66.57
90.23
77.19
76.34
78.17
2D + Timing
PP-Res18+TransE
Res18/34+TransE
MLP+TE
70.83
67.32
90.54
79.97
77.17
-
Table 1: Comparison with existing methods.
Model
βmacc
Δ
DER
DBR
TCR
VCR
CauPsi
82.71
—
78.65
76.84
92.11
83.25
− CTPC
81.77
− 0.94
79.63
73.89
91.95
81.60
− CrossView
82.38
− 0.33
79.96
75.69
92.28
81.60
− CausalChain
80.41
− 2.30
78.16
68.14
92.93
82.43
− FaceBody
80.25
− 2.46
78.65
70.11
92.11
80.13
Table 2: Ablation study results. Δ denotes the change in mean accuracy relative to the full model.
Figure 3: Mean values across the 16 dimensions of ψ , stratified by class for each task.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1: Overall Algorithm of CauPsi
Input: Multi-view video {Xv} , face/body video Xface,Xbody
An agentic AI driver-assistance framework that integrates perception, compliance reasoning, vehicle-state interpretation, and safety arbitration into a modular and auditable pipeline. The architecture, referred to as DriveMCP, incorporates a sensor-like perception stack alongside DriveLM as the vision-language front end to generate a graph-structured scene understanding (Graph Visual Question Answering) and language-grounded driving information. Key compliance elements in world_state, including posted speed limits and jurisdiction cues, are derived from DriveLM outputs through a structured parsing layer rather than being injected as simulator ground truth. A stateful orchestration layer coordinates specialized experts exposed as Model Context Protocol (MCP) servers: (i) a Rules server that performs retrieval-augmented compliance reasoning over jurisdiction-specific traffic codes and sign conventions, (ii) a Weather server that estimates traction risk and contextual speed advisories, and (iii) an MCP-CAN server that surfaces Controller Area Network (CAN)/On-Board Diagnostics (OBD) telemetry and diagnostic context for health-aware risk shaping. These outputs are fused to generate a structured decision that prompts a recommended course of action. The outcome is then further filtered by a Responsibility-Sensitive Safety (RSS)-inspired guardrail that arbitrates speak versus act decisions under bounded online adaptation. In CARLA simulation across multilingual, cross-border, and dynamic speed-limit scenarios, DriveMCP reduces traffic infractions and overspeed relative to the VLM-Direct, VLM-Direct+RAG, and VLM-Tools-NoArbiter baselines, while improving hazard response time and maintaining sub-second advisory latency.
Farzad Nadiri, Mehdi Cina, Ahmad B. Rad
Department of Mechatronics Engineering, Simon Fraser University, Burnaby, BC, Canada
Safe L2/L3 driving automation requires anticipating human-in-the-loop reactions during shared-control transitions. While most driving world models forecast the external environment, in-cabin intelligence remains strictly recognition-oriented and lacks multi-step rollout capabilities for driver dynamics. We introduce Driver-WM, a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context. This formulation unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition. Operating in a compact latent space constructed from frozen vision-language features, Driver-WM adopts a dual-stream architecture to separately encode external traffic and internal driver states. These streams are directionally coupled via a gated causal injection mechanism, which uses a learned vector gate to modulate external contextual perturbations while strictly enforcing temporal causality. Experiments on AIDE show robust long-horizon forecasting on reactive high-motion clips, improved driver/traffic semantic alignment, and controlled interventions that expose the external-to-internal mechanism.
Haozhuang Chi, Daosheng Qiu, Hao Su +4
Nanyang Technological University, Singapore · Hubei University, Wuhan, China · Osaka University, Osaka, Japan
End-to-end autonomous driving, which bypasses traditional modular pipelines by directly predicting future trajectories from sensor inputs, has recently achieved substantial progress. However, existing methods often overlook the causal inter-dependencies in ego-vehicle planning, ignoring the reciprocal relations between the ego vehicle and surrounding agents. This causal oversight leads to inconsistent and unreliable trajectory predictions, especially in interaction-critical scenarios where ego decisions and neighboring agent behaviors must be reasoned about jointly. To address this limitation, we propose CaAD, a Causality-aware end-to-end Autonomous Driving framework that captures these dependencies within a shared latent scene representation. First, we propose an ego-centric joint-causal modeling module that builds on the marginal prediction branch, and learns causal dependencies between the ego vehicle and interaction-relevant agents. Second, we employ a causality-aware policy alignment stage implemented with joint-mode embeddings to align the stochastic ego policy with planning-oriented closed-loop feedback computed from surrounding traffic and map context. On the Bench2Drive and NAVSIM benchmarks, CaAD demonstrates strong closed-loop planning performance, achieving a Driving Score of 87.53 and Success Rate of 71.81 on Bench2Drive, and a PDMS of 91.1 on NAVSIM. The project page is available at https://moonseokha.github.io/CaAD/.