Pedestrian Crossing Intent Prediction via Psychological Features and Transformer Fusion

Sima Ashayer; Hoang H. Nguyen; Yu Liang; Mina Sartipi

cs.CVMar 20, 2026

Pedestrian Crossing Intent Prediction via Psychological Features and Transformer Fusion

Authors: Sima Ashayer, Hoang H. Nguyen, Yu Liang, Mina Sartipi

Organizations: The University of Tennessee at Chattanooga, Chattanooga, TN, USA

Abstract

Pedestrian intention prediction needs to be accurate for autonomous vehicles to navigate safely in urban environments. We present a lightweight, socially informed architecture for pedestrian intention prediction. It fuses four behavioral streams (attention, position, situation, and interaction) using highway encoders, a compact 4-token Transformer, and global self-attention pooling. To quantify uncertainty, we incorporate two complementary heads: a variational bottleneck whose KL divergence captures epistemic uncertainty, and a Mahalanobis distance detector that identifies distributional shift. Together, these components yield calibrated probabilities and actionable risk scores without compromising efficiency. On the PSI 1.0 benchmark, our model outperforms recent vision language models by achieving 0.9 F1, 0.94 AUC-ROC, and 0.78 MCC by using only structured, interpretable features. On the more diverse PSI 2.0 dataset, where, to the best of our knowledge, no prior results exist, we establish a strong initial baseline of 0.78 F1 and 0.79 AUC-ROC. Selective prediction based on Mahalanobis scores increases test accuracy by up to 0.4 percentage points at 80% coverage. Qualitative attention heatmaps further show how the model shifts its cross-stream focus under ambiguity. The proposed approach is modality-agnostic, easy to integrate with vision language pipelines, and suitable for risk-aware intent prediction on resource-constrained platforms.

Figures & tables

Fig. 1 : Overview of socially informed, multi-stream architecture for pedestrian crossing intention prediction: Four input streams—attention, positional, situational, and interaction features—are encoded by highway encoders and augmented with stream embeddings. A Transformer model with temporal and cross-stream dependencies and self-attention yields a unified representation. A residual MLP predicts crossing probability, while KL and Mahalanobis modules provide uncertainty and anomaly scores.

Method	Accuracy	F1	AUC-ROC	MCC
SF-GRU	0.788	0.719	0.752	0.452
SingleRNN	0.782	0.714	0.734	0.440
MultiRNN	0.658	0.611	0.666	0.229
PCPA	0.682	0.584	0.611	0.176
ARN	0.688	0.618	0.671	0.237
PSI	0.759	0.681	-	0.374

TABLE I : Performance comparison on the PSI 1.0 test set (baseline numbers are reported from [ 1 ] ).

Ablation Setting	Accuracy	F1	AUC-ROC	MCC
Full Model (Ours)	0.88	0.90	0.94	0.77
w/o Attention stream	0.86	0.88	0.93	0.73
w/o Positional stream	0.85	0.86	0.94	0.73
w/o Situational stream	0.73	0.75	0.79	0.45
w/o Interaction stream	0.89	0.90	0.94	0.78

TABLE II : Ablation study of behavioral streams and training components on PSI 1.0.

Method	Accuracy	F1	AUC-ROC	MCC
Ours	0.73 ± 0.01	0.78 ± 0.01	0.79 ± 0.01	0.44 ± 0.02

TABLE III : PSI 2.0 test performance (mean ± std over five seeds; threshold chosen on validation and kept fixed for test).

Dataset	Class	Precision	Recall	F1
PSI 1.0	Non-Crossing (0)	0.805	0.943	0.868
PSI 1.0	Crossing (1)	0.955	0.840	0.893
PSI 2.0	Non-Crossing (0)	0.698	0.614	0.648
PSI 2.0	Crossing (1)	0.757	0.812	0.781

TABLE IV : Per-class precision, recall, and F1 (mean over five seeds).

Dataset / Setting	Brier $\downarrow$	ECE $\downarrow$	NLL $\downarrow$
PSI 1.0 (raw)	$0.093\pm 0.004$	$0.360\pm 0.033$	$0.320\pm 0.020$
PSI 1.0 ( $\tau$ -scaled)	$0.093\pm 0.004$	$0.367\pm 0.031$	$0.333\pm 0.028$
PSI 2.0 (raw)	$0.214\pm 0.012$	$0.311\pm 0.026$	$0.753\pm 0.079$
PSI 2.0 ( $\tau$ -scaled)	$0.188\pm 0.005$	$0.189\pm 0.031$	$0.562\pm 0.011$

TABLE V : Calibration on test sets (mean

\pm

std over 5 seeds)

Dataset	AUROC		AUPRC
Dataset	KL $\uparrow$	Mahalanobis $\uparrow$	KL $\uparrow$	Mahalanobis $\uparrow$
PSI 1.0	$0.59\pm 0.08$	$\mathbf{0.77\pm 0.03}$	$0.17\pm 0.04$	$\mathbf{0.33\pm 0.09}$
PSI 2.0	$0.53\pm 0.06$	$\mathbf{0.66\pm 0.02}$	$0.33\pm 0.04$	$\mathbf{0.43\pm 0.03}$

TABLE VI : Error-detection performance (mean

\pm

std over 5 seeds). Higher is better.

Dataset	100% (all)	90% kept	80% kept
PSI 1.0	$0.888\pm 0.007$	$\mathbf{0.914\pm 0.007}$	$\mathbf{0.935\pm 0.006}$
PSI 2.0	$0.708\pm 0.027$	$\mathbf{0.731\pm 0.031}$	$\mathbf{0.752\pm 0.026}$

TABLE VII : Accuracy after abstaining on the riskiest samples (Mahalanobis class-conditional ). Mean

\pm

std over 5 seeds.

Fig. 2 : Cross-stream attention heat-maps and uncertainty scores for two PSI 2.0 samples. Each row shows average multi-head attention from one stream token (attn/pos/sit/inter) to all streams. Left: low model risk (KL = 0.00, Mah = 10.71). Right: high model risk (KL = 0.04, Mah = 2329.78). Cell values are mean attention weights across the four behavioral streams.

Panel of Figure 10 — Fig. 3 : Four composite qualitative examples showing input frames with a visualization of the behavioural-stream attention weights and uncertainty estimates (KL and Mahalanobis).

Pedestrian Intention Prediction

Pedestrian intention prediction estimates whether and how people will move, especially whether they will cross, from video and surrounding traffic cues.

See all 14 papers

Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models
Jun 8, 2026Danya Li, Xiang Su, Yan Feng +1Pedestrian Intention Prediction VLM Adaptation
Department of Technology, Management, and Economics, Technical University of Denmark, Denmark · Department of Computer Science and Department of Agricultural Sciences, University of Helsinki, Finland · Department of Transport and Planning, Delft University of Technology, Netherlands
TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs
Sep 9, 2026François G. Landry, Moulay A. AkhloufiPedestrian Intention Prediction Multimodal Fusion
Perception, Robotics and Intelligent Machines Research Group (PRIME) Department of Computer Science, Universit´e de Moncton, Canada
Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction
Jul 14, 2026Md Mahfuzur Rahman, Pengzhan Zhou, A F M Abdun Noor +5Pedestrian Intention Prediction Autonomous Driving
College of Computer Science, Chongqing University, Chongqing, China · Department of Software Engineering, Daffodil International University, Bangladesh · Faculty of Information Science & Technology, Multimedia University, Malaysia

Jun 8, 2026J/K move · Enter open · S save

Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

Danya Li, Xiang Su, Yan Feng, Rico Krueger

Department of Technology, Management, and Economics, Technical University of Denmark, Denmark · Department of Computer Science and Department of Agricultural Sciences, University of Helsinki, Finland · Department of Transport and Planning, Delft University of Technology, Netherlands

Pedestrian Intention Prediction VLM Adaptation Autonomous Driving Perception Egocentric Video Understanding

Open paper AskPDF arXiv

Egocentric vision offers a first-person view of human perception and decision making, yet its potential for traffic-safety prediction remains underexplored. In this work, we study the decoding of pedestrian crossing intentions from short egocentric video clips. We approach this by formulating the task as a closed-ended visual question answering (VQA) problem and leveraging vision language models (VLMs) to predict the pedestrians' intent. We first benchmark three families of state-of-the-art VLMs in a zero-shot setting, finding that they achieve moderate gains over random guessing but exhibit limited higher-level traffic reasoning. Motivated by these findings, we further adapt VLMs to the target task using parameter-efficient fine-tuning. Our results show that the fine-tuned models substantially outperform their zero-shot counterparts and achieve a 9% accuracy improvement over a specialized transformer-based baseline. Finally, we demonstrate that incorporating additional contextual cues, including ego motion, vehicle motion, and eye gaze, further improves predictive performance. In particular, the fine-tuned Qwen3-VL-2B model guided by eye gaze and ego motion achieves a 14.5% accuracy improvement over the transformer baseline, establishing a new state of the art for egocentric pedestrian intent decoding.

Abstract

Pedestrian Intention Prediction

Explore similar work

Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs

Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction