Cognitive-Causal Multi-Task Learning with Psychological State Conditioning for Assistive Driving Perception

Keito Inoshita; Nobuhiro Hayashida; Akira Imanishi

cs.LGApr 8, 2026

Cognitive-Causal Multi-Task Learning with Psychological State Conditioning for Assistive Driving Perception

Authors: Keito Inoshita, Nobuhiro Hayashida, Akira Imanishi

Organizations: Kansai University, 3-3-35, Yamatecho, Suita, Osaka, Japan 564-8680 · ISUZU Advanced Engineering Center, Ltd., 8, Tsuchidana, Fujisawa City, Kanagawa, Japan 252-0881

Abstract

Multi-task learning for advanced driver assistance systems requires modeling the complex interplay between driver internal states and external traffic environments. However, existing methods treat recognition tasks as flat and independent objectives, failing to exploit the cognitive causal structure underlying driving behavior. In this paper, we propose CauPsi, a cognitive science-grounded causal multi-task learning framework that explicitly models the hierarchical dependencies among Traffic Context Recognition (TCR), Vehicle Context Recognition (VCR), Driver Emotion Recognition (DER), and Driver Behavior Recognition (DBR). The proposed framework introduces two key mechanisms. First, a Causal Task Chain propagates upstream task predictions to downstream tasks via learnable prototype embeddings, realizing the cognitive cascade from environmental perception to behavioral regulation in a differentiable manner. Second, Cross-Task Psychological Conditioning (CTPC) estimates a psychological state signal from driver facial expressions and body posture and injects it as a conditioning input to all tasks including environmental recognition, thereby modeling the modulatory effect of driver internal states on cognitive and decision-making processes. Evaluated on the AIDE dataset, CauPsi achieves a mean accuracy of 82.71% with only 5.05M parameters, surpassing prior work by +1.0% overall, with notable improvements on DER (+3.65%) and DBR (+7.53%). Ablation studies validate the independent contribution of each component, and analysis of the psychological state signal confirms that it acquires systematic task-label-dependent patterns in a self-supervised manner without explicit psychological annotations.

Figures & tables

Figure 1: Overall architecture of CauPsi.

Figure 2: Detailed architecture of the Causal Task Chain.

Pattern	Backbone			$\alpha_{\mathrm{acc}}$ (%) $\uparrow$				$\beta_{\mathrm{macc}}$	P(M)
Pattern	Multi-view Scene	Driver Images	Joints	DER	DBR	TCR	VCR	(%) $\uparrow$	$\downarrow$
2D	VGG16 ( Simonyan and Zisserman, 2014 )	VGG16 ( Simonyan and Zisserman, 2014 )	3DCNN	69.12	64.57	84.77	74.08	73.15	127.48
	Res18 ( He et al., 2016 )	Res18 ( He et al., 2016 )	3DCNN	68.78	64.33	89.76	78.59	75.37	107.77
	CMT ( Guo et al., 2022 )	CMT ( Guo et al., 2022 )	3DCNN	68.75	68.75	93.75	81.38	78.16	72.33
	GLMDriveNet ( Liu et al., 2024 )	GLMDriveNet ( Liu et al., 2024 )	3DCNN	71.38	66.57	90.23	77.19	76.34	78.17
2D + Timing	PP-Res18+TransE	Res18/34+TransE	MLP+TE	70.83	67.32	90.54	79.97	77.17	-

Table 1: Comparison with existing methods.

Model	$\beta_{\mathrm{macc}}$	$\Delta$	DER	DBR	TCR	VCR
CauPsi	82.71	—	78.65	76.84	92.11	83.25
$-$ CTPC	81.77	$-$ 0.94	79.63	73.89	91.95	81.60
$-$ CrossView	82.38	$-$ 0.33	79.96	75.69	92.28	81.60
$-$ CausalChain	80.41	$-$ 2.30	78.16	68.14	92.93	82.43
$-$ FaceBody	80.25	$-$ 2.46	78.65	70.11	92.11	80.13

Table 2: Ablation study results.

\Delta

denotes the change in mean accuracy relative to the full model.

Figure 3: Mean values across the 16 dimensions of $\boldsymbol{\psi}$ , stratified by class for each task.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm 1: Overall Algorithm of CauPsi
Input: Multi-view video $\{\mathbf{X}_{v}\}$ , face/body video $\mathbf{X}_{\mathrm{face}},\mathbf{X}_{\mathrm{body}}$
Output: Four-task predictions $\hat{\mathbf{y}}_{1},\hat{\mathbf{y}}_{2},\hat{\mathbf{y}}_{3},\hat{\mathbf{y}}_{4}$
// Feature Extraction (Section 3.2 )
1: For each view $v$ : $\bar{\mathbf{h}}_{v}\leftarrow\frac{1}{T}\sum_{t}\mathbf{W}_{v}^{\mathrm{gap}}\cdot\mathrm{GAP}(\phi_{v}(\mathbf{X}_{v}^{(t)}))$
2: $\mathbf{f}_{\mathrm{face}},\mathbf{f}_{\mathrm{body}}\leftarrow$ face/body encoders $+$ projection
3: $\bar{\mathbf{h}}_{\mathrm{scene}}\leftarrow\sum_{i}\alpha_{i}\bar{\mathbf{h}}_{\mathrm{sc},i}$ (scene integration)

Appendix

Table 6

Figure 4: Normalized confusion matrices for all four tasks.

Task	Class	Prec.	Rec.	F1	N
TCR	TrafficJam	0.766	0.766	0.766	77
	Waiting	0.900	0.947	0.923	133
	Smooth	0.959	0.942	0.951	399
VCR	Parking	0.899	0.922	0.910	154
	Turning	0.661	0.732	0.695	56
	Backward	0.769	0.769	0.769	39

Appendix

Table 3: Per-class performance.

Explore similar work

DriveMCP: An Agentic AI framework for Advanced Driver Assistance System
Sep 15, 2026Farzad Nadiri, Mehdi Cina, Ahmad B. RadVLMs for Autonomous Driving Intelligent Transportation Systems
Department of Mechatronics Engineering, Simon Fraser University, Burnaby, BC, Canada
Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout
May 6, 2026Haozhuang Chi, Daosheng Qiu, Hao Su +4World Model Learning Causal World Models
Nanyang Technological University, Singapore · Hubei University, Wuhan, China · Osaka University, Osaka, Japan
Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling
May 13, 2026Seokha Moon, Minseung Lee, Joon Seo +2Causal Representation Learning End-to-End Autonomous Driving
Korea University · Kakao Mobility

Sep 15, 2026J/K move · Enter open · S save

DriveMCP: An Agentic AI framework for Advanced Driver Assistance System

Farzad Nadiri, Mehdi Cina, Ahmad B. Rad

Department of Mechatronics Engineering, Simon Fraser University, Burnaby, BC, Canada

VLMs for Autonomous Driving Intelligent Transportation Systems Autonomous Driving Runtime Enforcement for AI Agents

Open paper AskPDF arXiv

An agentic AI driver-assistance framework that integrates perception, compliance reasoning, vehicle-state interpretation, and safety arbitration into a modular and auditable pipeline. The architecture, referred to as DriveMCP, incorporates a sensor-like perception stack alongside DriveLM as the vision-language front end to generate a graph-structured scene understanding (Graph Visual Question Answering) and language-grounded driving information. Key compliance elements in world_state, including posted speed limits and jurisdiction cues, are derived from DriveLM outputs through a structured parsing layer rather than being injected as simulator ground truth. A stateful orchestration layer coordinates specialized experts exposed as Model Context Protocol (MCP) servers: (i) a Rules server that performs retrieval-augmented compliance reasoning over jurisdiction-specific traffic codes and sign conventions, (ii) a Weather server that estimates traction risk and contextual speed advisories, and (iii) an MCP-CAN server that surfaces Controller Area Network (CAN)/On-Board Diagnostics (OBD) telemetry and diagnostic context for health-aware risk shaping. These outputs are fused to generate a structured decision that prompts a recommended course of action. The outcome is then further filtered by a Responsibility-Sensitive Safety (RSS)-inspired guardrail that arbitrates speak versus act decisions under bounded online adaptation. In CARLA simulation across multilingual, cross-border, and dynamic speed-limit scenarios, DriveMCP reduces traffic infractions and overspeed relative to the VLM-Direct, VLM-Direct+RAG, and VLM-Tools-NoArbiter baselines, while improving hazard response time and maintaining sub-second advisory latency.

Abstract

Explore similar work

DriveMCP: An Agentic AI framework for Advanced Driver Assistance System

Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout

Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling