BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Yuhang Wang; Yiyao Xu; Chaoyun Yang; Lingyao Li; Jingran Sun; Hao Zhou

cs.HCApr 8, 2026

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Authors: Yuhang Wang, Yiyao Xu, Chaoyun Yang, Lingyao Li, Jingran Sun, Hao Zhou

Organizations: University of South Florida Tampa, Florida, USA · Tongji University Shanghai, China · University of Arizona Tucson, Arizona, USA

Abstract

Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.

Figures & tables

Figure 1. Overview of BATON . Column (a) shows the data-collection setup with synchronized front-view and driver cameras; (b) the synchronized multimodal streams (road and cabin video, CAN signals, route context, lead-vehicle detections); (c) dataset scale and diversity (781 routes, 173 drivers, 204.9 hours); (d) the benchmark tasks. A composite figure illustrating the \sysname~dataset: vehicle-mounted sensors for data collection, synchronized multimodal time-series signals with annotated control transition events, geographic distribution of driving routes, and benchmark tasks including action understanding, driving automation system (DAS) handover, and takeover prediction.

Dataset	Setting	Modalities	Scale	Focus	Gap
Drive&Act ( Martin et al., 2019 )	Controlled	Cabin RGB/IR/depth	12 h	Cabin actions	No road view; no control transition
DAD ( Kopuklu et al., 2021 )	Simulator	Cabin IR/depth	31 subjects	Driver anomaly	Simulator only; cabin only
AIDE ( Yang et al., 2023 )	Real-world	Road + cabin video	2,898 clips	Holistic Perception	No control loop; not transition-centered
manD ( Dargahi Nobari and Bertram, 2024 )	Simulator	Cabin + physiol. + vehicle	50 participants	Driver Status	Simulator only; not real-world driving
TD2D ( Hwang et al., 2025 )	Simulator	Cabin + takeover signals	500 cases; 50 drivers	Takeover only	Simulator only; one-sided transition
Lee et al. ( Lee et al., 2025 )	Real-world	CAN + smartphone IMU	4 drivers	Activation only	Small scale; no cabin/road video

Table 1. Comparison with representative datasets and recent studies.

Figure 2. Data-collection setup. A comma 3X device ( comma.ai, 2023 ) records synchronized front-view and in-cabin video; CAN signals are decoded into vehicle-state measurements, and GPS provides route-level context.

Figure 3. Overview of BATON : global route distribution (top), driving time per driver (bottom left), and composition statistics (bottom right).

Figure 4. Multimodal context around control transitions in BATON : (a) aligned cabin views, road views, and map context for takeover and handover events; (b) synchronized vehicle-dynamics, driver-input, radar, driver-monitoring, and planner signals around a single takeover event ( $[-20\,\mathrm{s},+10\,\mathrm{s}]$ ).

Modality	Source	Rate	Coverage	Key parameters	Role	Data origin
Front-view video	Road camera	20 fps	777/781	lanes, curves, traffic, lead vehicle	outside-scene context	raw video
In-cabin video	Cabin camera	20 fps	781/781	head pose, gaze, motion	driver readiness	raw video
Vehicle dynamics	CAN & control	100 Hz	781/781	speed, steering, pedals, DA mode	control-loop state	CAN logs
IMU motion	Device IMU	100 Hz	781/781	acceleration, rotation	motion dynamics	inertial signals
Radar interaction	Forward radar	20 Hz	781/781	relative distance, relative speed	lead interaction	radar tracks
Driver monitoring	DMS outputs	20 Hz	781/781	awareness, distraction, eye state	driver state	openpilot ( comma.ai, 2018 )

Table 2. Modalities in BATON and their roles in bidirectional control-transition analysis.

Figure 5. Distributions in the BATON benchmark: (a) the seven Task-1 action classes; (b, c) positive and negative windows for Task 2 and Task 3.

Item	Setting
Scope	Bidirectional driver–automation transitions
Tasks	T1 action recognition (auxiliary); T2 handover; T3-D takeover detection; T3-A takeover anticipation
Input / Horizon	5 s window; 1 / 3 / 5 s horizon (main: 3 s)
Stride	0.5 s
Splits	Cross-driver (main), cross-vehicle, random
Inputs	Leak-safe (4 ADAS-control variables withheld); T3-A additionally withholds 6 driver-override channels

Table 3. BATON benchmark protocol.

Input protocol	T2	T3
Full input (upper-bound reference)	0.360 / 0.292	0.520 / 0.428
Leak-safe (official; T3-D)	0.222 / 0.091	0.478 / 0.385
Anticipation-safe (T3-A)	–	0.056 / 0.026

Table 4. Protocol ladder (single XGBoost, cross-driver,

h{=}3

s, sample / event AUPRC). Base rates: T2 0.148 / 0.037; T3 0.120 leak-safe, 0.035 anticipation-safe.

Method	Input	T2: Handover	T3-D: Takeover detection	T3-A: Takeover anticipation
Base rate	–	.148 / .037	.120 / –	.035 / –
GRU	struct	.202 $\pm$ .009 / .072 $\pm$ .006	.280 $\pm$ .049 / .119 $\pm$ .029	–
Cross-Modal Transformer	struct	.242 $\pm$ .011 / .103 $\pm$ .005	.316 $\pm$ .028 / .135 $\pm$ .031	–
RG-HBT-Q	struct+video	.254 $\pm$ .020 / .111 $\pm$ .020	.398 $\pm$ .007 / .230 $\pm$ .006	.058 $\pm$ .006 / .026 $\pm$ .002
DI-RG-HBT-Q	struct+video	.219 $\pm$ .011 / .116 $\pm$ .017	.366 $\pm$ .025 / .215 $\pm$ .036	–
TCN	struct	.190 $\pm$ .008 / .058 $\pm$ .008	.332 $\pm$ .064 / .167 $\pm$ .046	–

Table 5. Main transition results (cross-driver,

h{=}3

s, sample / event AUPRC, 3-seed mean

\pm

std; T2 and T3-D leak-safe, T3-A anticipation-safe with base rate 0.035). s : single seed.

Model	T1 $F1_{M}$	T2	T3-D
Gemini-2.5-Flash	0.362	0.184	0.202
GPT-4o	0.303	0.165	0.196
Qwen3-VL-8B	0.280	0.166	0.177
Qwen3-VL-4B	0.212	0.173	0.167
MiniCPM-V-4.5	0.170	0.174	0.182
LLaVA-OneVision-1.5-8B	0.168	0.195	0.149

Table 6. Zero-shot VLM baselines (cross-driver,

h{=}3

s, sample-level, best configuration per task, identical frozen sample draw). T1 uses rule-free camera-only inputs; the Full configuration contains rule-defining vehicle-state text and appears in Appendix E as a leakage-inflated diagnostic.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Full corpus	Benchmark subset
Routes / route bundles	781	565
Driving hours	204.9	162.1
Drivers	173	150
Vehicle models (manufacturers)	108 (22)	99
Control transitions	–	3,593
handover / takeover	–	1,800 / 1,793

Appendix

Table 7. Full living corpus vs. the frozen benchmark subset used for all reported results.

Configuration	Routes	Hours	Handovers	Takeovers
openpilot lateral + stock ACC	409	128.0	1,728	1,726
Stock assistance only	31	11.7	69	63
openpilot lateral + longitudinal	0	–	–	–
Human-only routes ( $<$ 10 s assisted)	125	22.4	3	4

Appendix

Table 8. Benchmark routes by assistance configuration (classified from CAN engagement flags over assisted driving time).

Engagement state	Hours	Share
Neither active (manual)	73.3	45.2%
Longitudinal only (ACC-style)	11.0	6.8%
Lateral only	15.4	9.5%
Both active	62.4	38.5%
Raw OR-transition subtype	Activations	Deactivations
Lateral flag only	1,112	1,003

Appendix

Table 9. Lateral

\times

longitudinal engagement structure of the benchmark subset. Subtype counts are raw flag changes before the debounce/merge filters of Appendix D .

Tier	Access	Contents	Reproduces
GitHub benchmark package	public	sample CSVs (all tasks/horizons incl. T3-A), split files, labels, event lists, generation/eval/baseline code	protocol, metrics, splits
HF BATON -Sample	public	43 full routes (video + all sensor CSVs)	pipeline inspection
Benchmark-ready tier	public	50 Hz tensors, video/pose features for 565 benchmark routes	all reported results
HF BATON full corpus	strict screening	all 781 routes: video + per-route sensor CSVs, in full	everything, incl. new tasks

Appendix

Table 10. Release tiers and what each tier alone reproduces.

	Task 1		Task 2		Task 3
Model	Acc	$F1_{M}$	AUROC	AUPRC	AUROC	AUPRC
LR	.865	.838	.812	.609	.783	.350
XGBoost (5 s)	.936	.920	.828	.631	.877	.653
GRU (5 s)	.926	.910	.815	.590	.843	.429
TCN	.925	.911	.770	.554	.838	.472
XGBoost (last frame)	.790	.700	.782	.449	.870	.608

Appendix

Table 11. Leakage-inflated full-input reference: single-seed model comparison and temporal ablation (cross-driver,

h{=}3

s) with the four ADAS-control variables included.

Action	Rule
Stopped	vEgo below 0.5 m/s for at least 2 s
Lane-change	laneChangeState $>$ 0, or a turn signal is on and steering angle exceeds 5 ∘
Turning	steering angle above 10 ∘ for at least 1 s
Braking	aEgo below $-0.35$ m/s 2 , or brake pressed
Accelerating	aEgo above 0.37 m/s 2 for at least 1 s
Car-following	lead vehicle active, gap below 60 m, absolute acceleration below 1 m/s 2

Appendix

Table 12. Task-1 action-labeling rules.

Signal category (examples)	Task 1	Tasks 2/3	Can it trivially reveal the target?
Engagement flags ( cc_latActive , cruiseState_enabled )	not used by rules	removed	Yes for Tasks 2/3: they define the label, hence removed.
Automation control state / command ( cs_longControlState , actuators_accel )	not used by rules	removed	No (0% within-window change; single-feature AUPRC at base rate); removed conservatively.
Automation warnings / take-over requests	not exposed	not exposed	N/A: no warning or request flag is exposed as input.
Planner outputs ( model_desiredCurvature , model_desiredAcceleration )	not used by rules	kept	No (0% within-window change; single-feature AUPRC at base rate).
Driver input (brake, accelerator, steering, blinkers)	partly rule signals (ablated)	kept	Not the label; strongest retained feature 0.247 vs. 0.332 for a removed flag.
Ego motion & lane state ( vEgo , aEgo , steering angle, lane state, lead distance)	define the action label (ablated)	kept	For Task 1, yes by construction; quantified (rule-free 0.577) and human-validated ( $\kappa=0.953$ ).

Appendix

Table 13. Per-category field audit for the leak-safe protocol.

Model	Input	T1 $F1_{M}$	T2	T3	Model	Input	T1 $F1_{M}$	T2	T3
Qwen2-VL-2B	Front	.131	.147	.131	Qwen3-VL-4B-Th.	Front	.241	.176 ∗	.160
Qwen2-VL-2B	Cabin	.084	.147	.146	Qwen3-VL-4B-Th.	Cabin	.131 ∗	.177 ∗	.175 ∗
Qwen2-VL-2B	F+C	.079	.147	.133	Qwen3-VL-4B-Th.	F+C	.240	.161 ∗	.152
Qwen2-VL-2B	Full	.058	.147	.151	Qwen3-VL-4B-Th.	Full	.459	.201 ∗	.157 ∗
Qwen2.5-VL-3B	Front	.083	.147	.122	Qwen3-VL-8B	Front	.280	.162	.177
Qwen2.5-VL-3B	Cabin	.058	.149	.138	Qwen3-VL-8B	Cabin	.138	.159	.123

Appendix

Table 14. Complete zero-shot VLM results (cross-driver,

h{=}3

s, sample-level, one frozen sample draw).

*

: over 15% parse failures. T1 Full cells are leakage-inflated diagnostics.

Model	Handover	Takeover
XGBoost (controller signals)	0.246 $\pm$ 0.007	0.441 $\pm$ 0.005
V-JEPA2 video only	0.241 $\pm$ 0.007	0.307 $\pm$ 0.010
Multimodal fusion	0.264 $\pm$ 0.008	0.440 $\pm$ 0.004

Appendix

Table 15. Ten-seed leak-safe comparison without driver-monitoring inputs (sample-level AUPRC, mean

\pm

std).

Method	T2: Handover	T3-D: Takeover det.
GRU	.337 $\pm$ .011 / .162 $\pm$ .007	.436 $\pm$ .016 / .308 $\pm$ .014
Cross-Modal Transformer	.302 $\pm$ .019 / .128 $\pm$ .013	.333 $\pm$ .025 / .173 $\pm$ .018
XGBoost (stats)	.360 $\pm$ .006 / .190 $\pm$ .008	.443 $\pm$ .001 / .365 $\pm$ .003
XGBoost (stats+pose)	.434 $\pm$ .006 / .257 $\pm$ .007	.452 $\pm$ .002 / .362 $\pm$ .005

Appendix

Table 16. Cross-vehicle split (vehicle-model fingerprints held out; leak-safe,

h{=}3

s, sample / event AUPRC, 3-seed mean

\pm

std).

Task	Method	Temporal	Cross-driver	Cross-vehicle
T2	XGB stats	.454 $\pm$ .002	.236 $\pm$ .003	.360 $\pm$ .006
T2	XGB stats+pose	.465 $\pm$ .004	.258 $\pm$ .003	.434 $\pm$ .006
T2	GRU	.351 $\pm$ .020	.202 $\pm$ .009	.337 $\pm$ .011
T3-D	XGB stats	.532 $\pm$ .001	.479 $\pm$ .007	.443 $\pm$ .001
T3-D	XGB stats+pose	.567 $\pm$ .001	.514 $\pm$ .000	.452 $\pm$ .002
T3-D	GRU	.414 $\pm$ .028	.280 $\pm$ .049	.436 $\pm$ .016

Appendix

Table 17. Within-driver temporal split vs. the identity-holdout splits (leak-safe,

h{=}3

s, sample AUPRC, 3-seed mean

\pm

std).

Features	Task 2	Task 3
stats	.236 $\pm$ .003 / .103	.479 $\pm$ .007 / .380
stats + pose	.258 $\pm$ .003 / .114	.514 $\pm$ .000 / .413
stats + WM (ViT-L/256)	.306 $\pm$ .005 / .148	.452 $\pm$ .001 / .352
stats + WM (ViT-g/384)	.335 $\pm$ .003 / .171	.460 $\pm$ .002 / .360
WM only (ViT-L/256)	.265 $\pm$ .008 / .130	.280 $\pm$ .006 / .166
WM only (ViT-g/384)	.286 $\pm$ .000 / .165	.365 $\pm$ .002 / .242

Appendix

Table 18. Feature-block ablation for the XGBoost readout (leak-safe, cross-driver,

h{=}3

s; sample / event AUPRC; 3-seed mean

\pm

std, single seed where marked).

Pre-override buffer	Sample AUPRC	Event AUPRC	Events
0.5 s	.067 $\pm$ .002	.031 $\pm$ .001	96
1.0 s (official)	.056 $\pm$ .001	.026 $\pm$ .001	92
1.5 s	.043 $\pm$ .001	.023 $\pm$ .001	87
2.0 s	.030 $\pm$ .001	.019 $\pm$ .000	76
Bootstrap 95% CI	stats	stats+pose
T3-D event AUPRC	.379 [.315, .441]	.411 [.343, .474]

Appendix

Table 19. T3-A robustness: buffer sensitivity (3-seed) and event-level bootstrap 95% CIs (seed 42, 2000 resamples).

Attempt	T3-A AUPRC
Long-window stats (XGB), 30 s	.064 $\pm$ .002
Long-window stats (XGB), 60 s	.068 $\pm$ .002
LCAT, video streams only, 60 s	.039 $\pm$ .003
LCAT, pose stream only, 60 s	.033 $\pm$ .002
LCAT, all streams, 30 s	.047 $\pm$ .003
LCAT, all streams, 60 s	.061 $\pm$ .004

Appendix

Table 20. Additional T3-A attempts (cross-driver,

h{=}3

s, sample AUPRC, 3-seed mean

\pm

std; base rate 0.035; best standard baseline 0.070).

Features (XGBoost readout)	AUPRC
stats	.056 $\pm$ .001 / .026
stats + pose	.070 $\pm$ .001 / .040
stats + WM (ViT-L/256)	.043 $\pm$ .001 / .019
stats + WM (ViT-g/384)	.049 $\pm$ .001 / .024

Appendix

Table 21. Task-3 takeover under the anticipation-safe protocol (driver-override channels withheld; sample / event AUPRC; base rate 0.035).

Input	$F1_{M}$	LC F1	Acc
Full structured (with rule signals)	0.889	0.833	0.903
Rule-free structured	0.577	0.344	0.632
Front-camera video only	0.592	0.106	0.693
IMU only	0.352	0.101	0.391
Driver-monitoring only	0.219	0.016	0.271
Chance (seven classes)	0.14	–	–

Appendix

Table 22. Task-1 ablation (GRU, cross-driver): performance without the rule-defining signals and from modalities the rules never use.

Class (label rate)	Structured (rule-free)	Front video
Stopped (19.0%)	.919 $\pm$ .011	.908 $\pm$ .004
Lane-change (1.3%)	.461 $\pm$ .055	.065 $\pm$ .013
Turning (16.6%)	.718 $\pm$ .027	.736 $\pm$ .038
Braking (21.8%)	.758 $\pm$ .021	.593 $\pm$ .027
Accelerating (11.3%)	.599 $\pm$ .018	.732 $\pm$ .012
Car-following (40.3%)	.978 $\pm$ .004	.908 $\pm$ .002

Appendix

Table 23. Task-1 multi-label baselines (GRU, cross-driver, rule-free inputs, per-class and macro average precision; 3-seed mean

\pm

std).

Protocol	Model	[email protected]	R@1	R@2	R@1, lead $\geq$ 1s	lead $\geq$ 2s	lead $\geq$ 3s	Median lead (s)
T2	XGB stats	.032	.037	.046	.032	.018	.005	2.0
T2	XGB stats+WM(g384)	.041	.073	.115	.069	.041	.005	2.0
T3-D	XGB stats	.267	.281	.295	.277	.240	.124	3.0
T3-D	XGB stats+pose	.267	.277	.323	.272	.217	.115	3.0
T3-A	XGB stats	.000	.000	.000	.000	.000	.000	–
T3-A	XGB stats+pose	.011	.022	.022	.022	.011	.000	2.25

Appendix

Table 24. Operating-point evaluation (cross-driver,

h{=}3

s, seed 42). R@

f

: event recall at

f

false alarms per driving hour; lead columns are at 1 FA/h.

Autonomous Driving Benchmarks

Autonomous driving benchmarks evaluate how well vehicles perceive, predict, plan, and act safely across varied roads and conditions.

See all 140 papers

ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement
Mar 7, 2026Yuhang Wang, Yiyao Xu, Jingran Sun +1Autonomous Driving Datasets Autonomous Driving
Department of Civil and Environmental Engineering, University of South Florida, Tampa, FL 33620, USA
NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
Aug 14, 2026Ashkan Yousefi Zadeh, Zishuo Zhu, Xiaomeng Li +5Autonomous Driving Datasets Autonomous Driving
Queensland University of Technology (QUT), Faculty of Health, School of Psychology and Counselling, Australia · ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3), Australia · Université Gustave Eiffel, Laboratory of Applied Psychology and Ergonomics, France
DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting
Sep 8, 2026Yuhang Wang, Chuheng Wei, Jingxin Yang +2Autonomous Driving Datasets Autonomous Driving Benchmarks
University of South Florida · Purdue University · Stanford University +1

Mar 7, 2026J/K move · Enter open · S save

ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

Yuhang Wang, Yiyao Xu, Jingran Sun, Hao Zhou

Department of Civil and Environmental Engineering, University of South Florida, Tampa, FL 33620, USA

Autonomous Driving Datasets Autonomous Driving Autonomous Driving Safety Evaluation

Open paper AskPDF arXiv

Takeovers remain a key safety vulnerability in production ADAS, yet existing public resources rarely provide takeover-centered, real-world data. We present ADAS-TO, the first large-scale naturalistic dataset dedicated to ADAS-to-manual transitions, containing 15,659 takeover-centered 20s clips from 327 drivers across 22 vehicle brands. Each clip synchronizes front-view video with CAN logs. Takeovers are defined as ADAS ON

\rightarrow

OFF transitions, with the primary trigger labeled as brake, steer, gas, mixed, or system disengagement. We further separate planned driver-initiated terminations (Ego) from forced takeovers (Non-ego) using a rule-based partition. While most events occur within conservative kinematic margins, we identify a long tail of 285 safety-critical cases. For these events, we combine kinematic screening with vision--language (VLM) annotation to attribute hazards and relate them to intervention dynamics. The resulting cross-modal analysis shows distinct kinematic signatures across traffic dynamics, infrastructure degradation, and adverse environments, and finds that in 59.3% of critical cases, actionable visual cues emerge at least 3s before takeover, supporting the potential for semantics-aware early warning beyond late-stage kinematic triggers. The dataset is publicly released at huggingface.co/datasets/HenryYHW/ADAS-TO-Sample.

Abstract

Autonomous Driving Benchmarks

Explore similar work

ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting