NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

Ashkan Yousefi Zadeh; Zishuo Zhu; Xiaomeng Li; Andry Rakotonirainy; Sebastien Glaser; Ronald Schroeter; Patricia Delhomme; Zahra Mehraban

cs.CVAug 14, 2026

NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

Authors: Ashkan Yousefi Zadeh, Zishuo Zhu, Xiaomeng Li, Andry Rakotonirainy, Sebastien Glaser, Ronald Schroeter, Patricia Delhomme, Zahra Mehraban

Organizations: Queensland University of Technology (QUT), Faculty of Health, School of Psychology and Counselling, Australia · ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3), Australia · Université Gustave Eiffel, Laboratory of Applied Psychology and Ergonomics, France

Abstract

Automated vehicles must explain their decisions in ways that passengers can understand, monitor, and trust. Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs, rather than elicited from the driver performing the action. We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads. Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations. NARRATE provides action labels, scenario-context labels spanning six high-level and 32 fine-grained categories, and span-level Situational Awareness (SA) annotations over driver explanations for Perception, Comprehension and Projection. Four benchmark tasks (SA, scenario-context, driver-action classification, and explanation generation) show that this structure is learnable from driver language, while fine-grained context recognition and explanation generation remain challenging. NARRATE paves a path towards more human-centred and domain-aware explanation models for automated driving.

Figures & tables

Figure 1: NARRATE data collection protocol: (A) instrumented vehicle and in-cabin setup; (B) real-time explanation elicitation and event tagging during the drive; (C) post-drive video-cued interview using the synchronised multi-view replay dashboard.

Figure 2: Representative events from NARRATE. Each panel shows an annotated driving event with its driver-action label, high-level and fine-grained scenario-context tags, three front-centre camera frames sampled around the tagged event time, and the corresponding driver explanations. The middle frame marks the tagged event, while the neighbouring frames provide pre- and post-event visual context. In-vehicle explanations were recorded during driving, whereas post-drive explanations were recalled during the video-cued debrief.

Dataset	Cam.	Sensors	Source	Lang.	SA	Act.	Ctx.	Size	Hrs	Drv
Perception & behaviour
nuScenes [ 18 ]	6	V,L,Ra,G,I	None	–	✗	✗	✗	1,000 scn.	5.5	n/r
BDD100K [ 19 ]	1	V,G,I	None	–	✗	✗	✗	100K videos	$\sim$ 1,100 der.	crowd
HDD [ 20 ]	3	V,L,I,G,C	Labels only	Lbl	✗	✓	✓ ∗	137 sess.	104	n/r
Language & explanation
Talk2Car [ 21 ]	6	V,L,Ra,G	Human commands	Cmd. †	✗	✓	✗	11,959 cmds.	n/a	n/a

Table 1: Comparison with related datasets. Columns: Cam. = number of cameras; Lang. = language format; SA = Situational Awareness; Act. = action labels; Ctx. = context labels; Size = headline annotated count (unit as reported); Hrs = hours of driving/video; Drv = distinct human drivers. Sensors: V = video; L = LiDAR; I = IMU; G = GPS; C = CAN; Ra = radar; D = dynamics; Sim. = simulator; var. = variable; mul. = multiple. Language: FT = free text; QA = question–answer; Cmd. = commands; Cat. = categories; Lbl = labels; CF = counterfactual; CoC = chain of causation. Scale: n/r = not reported; n/a = not applicable (built on another dataset or simulated); crowd = crowdsourced (no fixed driver count); der. = derived by us; seg. = segments; scn. = scenes; sess. = sessions.

\ddagger

Reasoning structure, not Endsley L1/L2/L3.

*

Causal/scenario labels only.

\dagger

Instruction or advice, not driver explanation. a Alpamayo hours n/r for the 700K CoC set; the paper’s 80,000 h is the fleet pre-training pool, not annotated reasoning hours. b CoVLA also stated as 6M frames / 431K sampled trajectory points; 83.3 h

=

30 s

\times

10,000. c HAD: 45,629 advice annotations (25,549 action / 20,080 attention) over the clips. d NARRATE Hrs given as annotated / recorded: 8.5 h annotated event footage (2,050

\times

15 s) and

\sim

23 h total driving recorded (35

\times

40 min).

Component	Train	Val	Test	Total
Participants	24	5	6	35
Events	1,402	272	376	2,050
IV explanations	875	171	227	1,273
Post explanations	750	155	205	1,110
Dual (both)	223	54	56	333
IV words (mean $\pm$ SD)	$17.0{\pm}11.4$	$11.8{\pm}9.5$	$12.9{\pm}7.9$	$15.6{\pm}10.8$

Table 2: NARRATE dataset splits. IV = in-vehicle; Post = post-drive.

Panel of Figure 3 — Figure 3: Dataset statistics and distributions. Panels summarise the 14.5 km route, action and context labels, train/validation/test splits, reliability-sample representativeness, explanation length, and content-word differences between in-vehicle and post-drive explanations.

	In-vehicle				Post-drive
Model	L1	L2	L3	MF1	L1	L2	L3	MF1
Majority	0.906	0.964	0.773	0.881	0.904	0.946	0.801	0.884
TF-IDF+LR	0.908	0.964	0.811	0.894	0.916	0.949	0.857	0.907
DistilBERT	0.906	0.964	0.846	0.905 ±.004	0.915	0.946	0.882	0.914 ±.007
BERT-base	0.917	0.963	0.854	0.911 ±.006	0.915	0.949	0.878	0.914 ±.005
RoBERTa-base	0.915	0.964	0.863	0.914 ±.007	0.924	0.949	0.861	0.911 ±.008

Table 4: T1: Situational Awareness classification on the participant-disjoint test set. MF1 = Macro-F1;

\uparrow

higher is better. Trainable models report mean

\pm

std over 5 seeds.

	6-class IV		6-class Post		32-class Primary
Model	MF1	WF1	MF1	WF1	MF1	WF1
Majority	0.000	$-$	0.000	$-$	$-$	$-$
TF-IDF+LR	0.342	$-$	0.377	$-$	$-$	$-$
DistilBERT	0.315 ±.025	0.562 ±.040	0.309 ±.048	0.516 ±.076	$-$	$-$
BERT-base	0.289 ±.045	0.520 ±.072	0.317 ±.049	0.534 ±.079	0.038 ±.023	0.086 ±.061
RoBERTa-base	0.407 ±.023	0.697 ±.023	0.418 ±.004	0.703 ±.008	0.035 ±.011	0.100 ±.048

Table 5: T2: Context classification on the participant-disjoint test set. MF1 = Macro-F1; WF1 = Weighted-F1;

\uparrow

higher is better. The 32-class MF1 is computed over labels with at least five test instances. Trainable models report mean

\pm

std over 5 seeds.

Model	Modality	Acc	MF1
Majority	$-$	0.476	0.065
TF-IDF+LR	Text	0.681	0.228
CLIP+LR	Video	0.468	0.219
Kinematics MLP	Motion	0.586 ±.007	0.241 ±.007
CLIP+Kin MLP	Video+Motion	0.629 ±.015	0.306 ±.016
DistilBERT	Text	0.708 ±.018	0.246 ±.024

Table 6: T3: Driver-action classification on the participant-disjoint test set (

n{=}376

). Acc = Accuracy; MF1 = Macro-F1;

\uparrow

higher is better. Trainable models report mean

\pm

std over 5 seeds.

Model	Type	B4	RL	MET	BS
Random	Retrieval	0.010 ±.001	0.115 ±.005	0.091 ±.004	0.872 ±.001
Action	Retrieval	0.012 ±.004	0.129 ±.028	0.115 ±.047	0.872 ±.003
Action+Context	Retrieval	0.019 ±.007	0.158 ±.016	0.133 ±.021	0.879 ±.005
CLIP-NN	Retrieval	0.012	0.132	0.108	0.875
CLIP+Kin-NN	Retrieval	0.013	0.142	0.110	0.876
FLAN-T5-base	Fine-tuned	0.034 ±.001	0.215 ±.010	0.144 ±.019	0.892 ±.001

Table 7: T4: Explanation generation on the participant-disjoint test set (

n{=}376

). B4 = BLEU-4; RL = ROUGE-L; MET = METEOR; BS = BERTScore-F1;

\uparrow

higher is better. Fine-tuned models report mean

\pm

std over 5 seeds.

Autonomous Driving Datasets

Autonomous driving datasets collect and organize driving-scene data to train and evaluate perception, prediction, planning, and safety systems.

See all 141 papers

An End-to-End Decision-Aware Multi-Scale Attention-Based Model for Explainable Autonomous Driving
Apr 30, 2026Maryam Sadat Hosseini Azad, Shahriar Baradaran Shokouhi, Amir Abbas Hamidi Imani +2Explainable Artificial Intelligence Autonomous Driving
Department of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran · Department of Computing Science, University of Alberta, Edmonton, AB T6G, Canada
DRIV-EX: Counterfactual Explanations for Driving LLMs
Feb 28, 2026Amaia Cardiel, Eloi Zablocki, Elias Ramzi +1Counterfactual Explanations Counterfactual Reasoning in Language Models
Aptikal, Université Grenoble Alpes, Grenoble, France · Valeo.ai, Paris, France
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
Sep 7, 2026Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker +2Spatiotemporal Reasoning Autonomous Driving Datasets
NVIDIA · ETH Zürich · Work done during NVIDIA Internship

Apr 30, 2026J/K move · Enter open · S save

An End-to-End Decision-Aware Multi-Scale Attention-Based Model for Explainable Autonomous Driving

Maryam Sadat Hosseini Azad, Shahriar Baradaran Shokouhi, Amir Abbas Hamidi Imani, Shahin Atakishiyev+1

Department of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran · Department of Computing Science, University of Alberta, Edmonton, AB T6G, Canada

Explainable Artificial Intelligence Autonomous Driving Explainability Evaluation Multi-Scale Attention

Open paper AskPDF arXiv

The application of computer vision is gradually increasing across various domains. They employ deep learning models with a black-box nature. Without the ability to explain the behavior of neural networks, especially their decision-making processes, it is not possible to recognize their efficiency, predict system failures, or effectively implement them in real-world applications. Due to the inevitable use of deep learning in fully automated driving systems, many methods have been proposed to explain their behavior; however, they suffer from flawed reasoning and unreliable metrics, which have prevented a comprehensive understanding of complex models in autonomous vehicles and hindered the development of truly reliable systems. In this study, we propose a multi-scale attention-based model in which driving decisions are fed into the reasoning component to provide case-specific explanations for each decision simultaneously. For quantitative evaluation of our model's performance, we employ the F1-score metric, and also proposed a new metric called the Joint F1 score to demonstrate the accurate and reliable performance of the model in terms of Explainable Artificial Intelligence (XAI). In addition to the BDD-OIA dataset, the nu-AR dataset is utilized to further validate the generalization capability and robustness of the proposed network. The results demonstrate the superiority of our reasoning network over the classic and state-of-the-art models.

Abstract

Autonomous Driving Datasets

Explore similar work

An End-to-End Decision-Aware Multi-Scale Attention-Based Model for Explainable Autonomous Driving

DRIV-EX: Counterfactual Explanations for Driving LLMs

CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving