ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

Yuhang Wang; Yiyao Xu; Jingran Sun; Hao Zhou

cs.HCMar 7, 2026

ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

Authors: Yuhang Wang, Yiyao Xu, Jingran Sun, Hao Zhou

Organizations: Department of Civil and Environmental Engineering, University of South Florida, Tampa, FL 33620, USA

Abstract

Takeovers remain a key safety vulnerability in production ADAS, yet existing public resources rarely provide takeover-centered, real-world data. We present ADAS-TO, the first large-scale naturalistic dataset dedicated to ADAS-to-manual transitions, containing 15,659 takeover-centered 20s clips from 327 drivers across 22 vehicle brands. Each clip synchronizes front-view video with CAN logs. Takeovers are defined as ADAS ON $\rightarrow$ OFF transitions, with the primary trigger labeled as brake, steer, gas, mixed, or system disengagement. We further separate planned driver-initiated terminations (Ego) from forced takeovers (Non-ego) using a rule-based partition. While most events occur within conservative kinematic margins, we identify a long tail of 285 safety-critical cases. For these events, we combine kinematic screening with vision--language (VLM) annotation to attribute hazards and relate them to intervention dynamics. The resulting cross-modal analysis shows distinct kinematic signatures across traffic dynamics, infrastructure degradation, and adverse environments, and finds that in 59.3% of critical cases, actionable visual cues emerge at least 3s before takeover, supporting the potential for semantics-aware early warning beyond late-stage kinematic triggers. The dataset is publicly released at huggingface.co/datasets/HenryYHW/ADAS-TO-Sample.

Figures & tables

Fig. 1: Role of OP in data collection. (a) Device setup: Comma set up and OP performance (b) Passive logging under OEM ADAS: OP acts as an observer and captures vehicle dynamics and driver inputs around takeovers via CAN. (c) OP engaged (control): the system records its states and outputs from OP’s driving model.

Fig. 2: Clip structure. Each 20 s clip is centered on the takeover event ( $t=0$ ).

Fig. 3: Dataset overview. (a) Top-10 vehicle brands by clip count (out of 22 total brands). (b) Distribution of vehicle speed at the takeover moment (dashed: mean; dash-dotted: median). (c) Distribution of following distance to lead vehicle (when detected, 49.8% of clips). (d) Primary takeover action distribution; the first action applied to take over the vehicle.

Fig. 4: Geographic distribution of the 2,244 routes with GPS coordinates across 323 unique drivers with 4 don’t want to share the location. North America accounts for 84.2% of routes, followed by Europe (4.5%) and Asia (3.2%).

CSV File	Key Signals
carState	Speed ( vEgo ), acceleration, steering angle/
	torque, brake/gas/steering pressed, cruise state
controlsState	ADAS enabled/active, curvature, desired curvature,
	set cruise speed, alert texts
carControl	Lateral/longitudinal active, actuator commands
	(accel, torque, curvature)

TABLE I: Signal topics recorded per clip. Each clip contains eight time-synchronized CSV files.

Fig. 5: Clips per driver. The distribution follows a long tail: the top-10 drivers (orange) contribute 40.4% of all clips.

Statistic	Value
Total takeover clips	15,659
Unique car models	163
Unique brands	22
Unique drivers ( dongle_id )	327
Total video duration	87.0 h
Clip duration	20 s ( $\pm 10$ s)

TABLE II: Summary statistics of the ADAS-TO dataset.

Fig. 6: Rule-based classification flowchart. Each takeover clip enters a seven-level priority cascade from top to bottom. At each decision node, the clip is tested against a specific detector condition; if matched, it is labeled Ego (green) and exits the cascade. Clips not captured by any Ego rule are assigned the default Non-ego label.

Fig. 7: Expert validation of the Ego/Non-ego intent partition on a 500-clip audit set. Left: confusion matrix for the rule-based classifier. Right: per-class precision/recall/F1, with the dashed line indicating overall accuracy (84.0%).

Fig. 8: Takeover action sequence and modality distribution. (LEFT) The action onset distribution within the $[-0.2,+0.5]$ s window illustrates the precise timing of control inputs relative to the ADAS disengagement ( $t=0$ ). (RIGHT) The primary action modality highlights a clear longitudinal preference, with braking and accelerating take up nearly half of the data.

Fig. 9: Example of a lateral takeover response. The subplots depict (a) the vehicle’s deviation to lane center, alongside the driver interventions via (b) steering angle and (c) steering torque.

Fig. 10: Longitudinal takeover context for a representative event. The subplots detail (a) the speed profiles of the ego and lead vehicles, (b) the relative distance to the lead vehicle, and (c) the driver’s pedal inputs within the $[-5,+5]$ s window around the takeover instant ( $t=0$ ).

Fig. 11: Distribution of TTC and THW at the moment of takeover initiation. A critical long-tail of high-risk events is distinctly visible.

Fig. 12: Exemplar of the VLM-based semantic annotation pipeline. The model integrates three pre-takeover frames ( $T-5$ s, $T-3$ s, $T-1$ s) with synchronized sensor data. In this scenario, the VLM achieves a consensus confidence score of 0.9, correctly identifying early visual cues (brake lights and lead vehicle’s low speed) that precede the low-TTC emergency.

Fig. 13: Semantic clustering of VLM-identified risk factors. The 285 critical takeover events are categorized into three macro-archetypes.

Fig. 14: Post-takeover kinematic metrics across semantic archetypes.

Fig. 15: Temporal advantage of VLM-based hazard detection. Left: Distribution of the earliest actionable visual cues. Right: Predictability ( $\geq 3$ s warning) stratified by hazard archetypes.

Fig. 16: Comparison of VLM-based early warning and traditional systems. The VLM detects the red light and brake lights at $T-5$ s, establishing hazard awareness well before kinematic thresholds trigger near $T-2$ s.

Autonomous Driving Datasets

Autonomous driving datasets collect and organize driving-scene data to train and evaluate perception, prediction, planning, and safety systems.

See all 141 papers

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving
Apr 8, 2026Yuhang Wang, Yiyao Xu, Chaoyun Yang +3Autonomous Driving Benchmarks Autonomous Driving Datasets
University of South Florida Tampa, Florida, USA · Tongji University Shanghai, China · University of Arizona Tucson, Arizona, USA
Beyond the Beep: Scalable Collision Anticipation and Real-Time Explainability with BADAS-2.0
Apr 7, 2026Roni Goldshmidt, Hamish Scott, Lorenzo Niccolini +1Traffic Accident Anticipation Long-Tail Autonomous Driving
Nexar AI
Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language
Sep 3, 2026Julian Truetsch, Felix Hauser, Christoph Stiller +1Autonomous Driving Datasets Autonomous Driving Benchmarks
FZI Research Center for Information Technology, 76131 Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe, Germany

Apr 8, 2026J/K move · Enter open · S save

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Yuhang Wang, Yiyao Xu, Chaoyun Yang, Lingyao Li+2

University of South Florida Tampa, Florida, USA · Tongji University Shanghai, China · University of Arizona Tucson, Arizona, USA

Autonomous Driving Benchmarks Autonomous Driving Datasets

Open paper AskPDF arXiv

Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.

Abstract

Autonomous Driving Datasets

Explore similar work

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Beyond the Beep: Scalable Collision Anticipation and Real-Time Explainability with BADAS-2.0

Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language