DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoning

Weimin Liu; Qingkun Li; Jiyuan Qiu; Wenjun Wang; Joshua H. Meng

cs.CVMar 30, 2026

DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoning

Authors: Weimin Liu, Qingkun Li, Jiyuan Qiu, Wenjun Wang, Joshua H. Meng

Organizations: School of vehicle and Mobility, Tsinghua University, Beijing, China · Beijing Key Laboratory of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences, Beijing, China · Remote Sensing and Earth Observation Laboratory, University of Copenhagen, Copenhagen K, Denmark · California PATH, University of California, Berkeley, CA, USA

Abstract

Drivers' visual attention provides critical cues for anticipating latent hazards and directly shapes decision-making and control maneuvers, where its absence can compromise traffic safety. To emulate drivers' perception patterns and advance visual attention prediction for intelligent vehicles, we propose DiffAttn, a diffusion-based framework that formulates this task as a conditional diffusion-denoising process, enabling more accurate modeling of drivers' attention. To capture both local and global scene features, we adopt Swin Transformer as encoder and design a decoder that combines a Feature Fusion Pyramid for cross-layer interaction with dense, multi-scale conditional diffusion to jointly enhance denoising learning and model fine-grained local and global scene contexts. Additionally, a large language model (LLM) layer is incorporated to enhance top-down semantic reasoning and improve sensitivity to safety-critical cues. Extensive experiments on four public datasets demonstrate that DiffAttn achieves state-of-the-art (SoTA) performance, surpassing most video-based, top-down-feature-driven, and LLM-enhanced baselines. Our framework further supports interpretable driver-centric scene understanding and has the potential to improve in-cabin human-machine interaction, risk perception, and drivers' state measurement in intelligent vehicles.

Figures & tables

Fig. 1 : Overview of the proposed LLM-enhanced, conditional diffusion-based drivers’ visual attention modeling method DiffAttn .

Fig. 2 : DiffAttn architecture overview. For saliency encoder, we adopt SwinT-Base [ 20 ] pretrained on ImageNet. The decoder is designed with an LLM-enhanced feature fusion pyramid (FFP), which bridges the encoder outputs, and a multi-scale dense-connected conditional diffusion module, where feature maps produced by FFP are densely connected and serve as conditioning signals for noise learning in the diffusion process. The noise predictors generate saliency maps at multiple scales, which are all supervised with groundtruth saliency maps. Saliency map generated at $s=0$ during testing.

Fig. 3 : Network architecture of LLM-based semantic enhancement.

Fig. 4 : Network architecture of multi-scale conditional diffusion.

Dataset			TrafficGaze					DADA-2000					BDD-A			DrFixD-rainy
Method	Input	#. $\downarrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{NSS}\uparrow$	$\textit{AUC-J}\uparrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{NSS}\uparrow$	$\textit{AUC-J}\uparrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{NSS}\uparrow$	$\textit{AUC-J}\uparrow$
MLNet [ 27 ]	I+S	15M	0.87	0.87	0.45	5.69	0.90	11.78	0.04	0.07	0.30	0.59	1.20	0.61	0.43	3.69	0.79	0.63	3.90	0.93
ACLNet [ 28 ]	V+S	-	0.27	0.91	0.77	5.73	0.95	1.95	0.46	0.30	3.23	0.93	1.12	0.61	0.46	-	-	-	-	-
CPFE [ 29 ]	I+S	-	0.26	0.89	0.77	4.33	0.94	2.18	0.33	0.21	2.43	0.91	1.65	0.42	0.30	-	-	-	-	-
TASED-Net [ 30 ]	V+S	21M	1.43	0.94	0.79	5.73	0.97	1.78	0.46	0.31	3.20	0.92	1.24	0.55	0.42	0.85	0.84	0.59	4.21	0.95
CDNN [ 12 ]	I+S	$<$ 1M	0.29	0.95	0.78	5.83	0.97	1.83	0.43	0.31	2.93	0.94	1.14	0.62	0.45	0.52	0.82	0.63	4.11	0.95

TABLE I : Test performance on four datasets (“I” indicates single image input. “V” implies video sequence. “S” indicates single modal of RGB image. “M” indicates multiple modals of input such as semantic segmentation. “#” indicates number of computation parameters. Results best in bold , second best underlined . “-” implies metrics not reported. “N/A” indicates no implementation due to data requirement.)

Fig. 5 : Qualitative results on TrafficGaze: (a) Surrounding vehicle driving in right lane; (b) Changing to right lane with a truck ahead; (c) Straight driving with a traffic sign ahead. Qualitative results on DADA-2000: (d) Motorcycle crossing; (e) Two trucks ahead; (f) Nearby truck changing lane ahead; (g) Pedestrian crossing; (h) Turning right with collision risk involving a taxi; (i) Pedestrian running in front of ego-vehicle. Qualitative results on BDD-A: (j) Vehicle crossing ahead; (k) Straight driving with a traffic light ahead; (l) Driving past parked cars; (m) Lane change with a braking vehicle ahead; (n) Approaching a STOP line. Qualitative results on DrFixD-rainy: (o) Entering a main road with congested traffic; (p) Nearby left vehicle changing into the ego lane; (q) Driving through a green light; (r) Driving on a rural road with a bicyclist on the right; (s) Pedestrians standing at the roadside.

Dataset		BDD-A			DADA-2000
Method	#. $\downarrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{NSS}\uparrow$	$\textit{AUC-J}\uparrow$
w/o $\mathcal{F}_{\text{LLM}}$	91M	1.126	0.632	0.506	1.639	0.502	0.377	3.472	0.949
LLaMA-F	92M	1.087	0.636	0.502	1.582	0.510	0.360	3.486	0.949
LLaMA-T	153M	1.114	0.632	0.501	1.606	0.507	0.370	3.493	0.949
DeepSeek-F	92M	1.097	0.634	0.497	1.660	0.502	0.381	3.474	0.949
DeepSeek-T	139M	1.113	0.629	0.494	1.684	0.495	0.378	3.427	0.948

TABLE II : Ablation study on LLM-based semantic enhancement (“F” and “T” denote frozen and trainable parameters, respectively).

Fig. 6 : Qualitative comparison of saliency prediction results: with LLM enhancement (second row), without LLM enhancement (third row), and CLIP-based method SalM 2 [ 21 ] (last row).

Fig. 7 : Visualization of the denoising process ( $\tau$ : current step).

Dataset	DADA-2000 $(T_{e}=16)$					BDD-A $(T_{e}=15)$
Method	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$	$\textit{NSS}\uparrow$	$\textit{AUC-J}\uparrow$	$\textit{KLD}\downarrow$	$\textit{CC}\uparrow$	$\textit{SIM}\uparrow$
$\boldsymbol{\tau=T_{e}}$	1.582	0.510	0.360	3.486	0.949	1.087	0.636	0.502
(1) Train model with different denoising steps
$T_{e}=10$	1.595	0.507	0.361	3.465	0.949	1.100	0.634	0.502
$T_{e}=5$	1.606	0.506	0.374	3.488	0.950	1.090	0.635	0.500
$T_{e}=2$	1.586	0.509	0.366	3.486	0.950	1.096	0.634	0.500

TABLE III : Ablation study on denoising steps

T_{e}

Fig. 8 : Visualization of denoising results with different $T_{e}$ .

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 62 papers

From Scene to Object: Text-Guided Dual-Gaze Prediction
Apr 22, 2026Zehong Ke, Yanbo Jiang, Jinhao Li +5Gaze Estimation Autonomous Driving Perception
School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China · Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
Inference-Time Attention Steering for Vision-Language-Action Driving Models
Aug 17, 2026Darshan Nagendra Prasad, Lars Ullrich, Knut GraichenVision-Language-Action Models VLMs for Autonomous Driving
FAU Erlangen-Nürnberg, Germany
Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decoupling
Jan 13, 2026Takamichi Miyata, Sumiko Miyata, Andrew MorrisVision-Language Models Autonomous Driving Perception
Chiba Institute of Technology · Chiba, Japan · Institute of Science Tokyo +3

Apr 22, 2026J/K move · Enter open · S save

From Scene to Object: Text-Guided Dual-Gaze Prediction

Zehong Ke, Yanbo Jiang, Jinhao Li, Zhiyuan Liu+4

School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China · Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Gaze Estimation Autonomous Driving Perception Vision-Language Grounding

Open paper AskPDF arXiv

Interpretable driver attention prediction is crucial for human-like autonomous driving. However, existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations, inherently failing to support text-grounded cognitive modeling. Consequently, while Vision-Language Models (VLMs) hold great potential for semantic reasoning, this critical data limitations leads to severe text-vision decoupling and visual-bias hallucinations. To break this bottleneck and achieve precise object-level attention prediction, this paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture. First, we construct G-W3DA, a object-level driver attention dataset. By integrating a multimodal large language model with the Segment Anything Model 3 (SAM3), we decouple macroscopic heatmaps into object-level masks under rigorous cross-validation, fundamentally eliminating annotation hallucinations. Building upon this high-quality data foundation, we propose the DualGaze-VLM architecture. This architecture extracts the hidden states of semantic queries and dynamically modulates visual features via a Condition-Aware SE-Gate, achieving intent-driven precise spatial anchoring. Extensive experiments on the W3DA benchmark demonstrate that DualGaze-VLM consistently surpasses existing state-of-the-art (SOTA) models in spatial alignment metrics, notably achieving up to a 17.8% improvement in Similarity (SIM) under safety-critical scenarios. Furthermore, a visual Turing test reveals that the attention heatmaps generated by DualGaze-VLM are perceived as authentic by 88.22% of human evaluators, proving its capability to generate rational cognitive priors.

Abstract

Visual Attention

Explore similar work

From Scene to Object: Text-Guided Dual-Gaze Prediction

Inference-Time Attention Steering for Vision-Language-Action Driving Models

Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decoupling