BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction

Feeza Khan Khanzada; Sudarshan Sridhar; Jaerock Kwon

cs.ROSep 23, 2026

BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction

Authors: Feeza Khan Khanzada, Sudarshan Sridhar, Jaerock Kwon

Organizations: Department of Electrical and Computer Engineering, University of Michigan-Dearborn, 4901 Evergreen Rd, Dearborn, MI 48128, United States

Abstract

Most autonomous-driving datasets record only the action executed by a behavior policy and the single future that followed, providing limited supervision for comparing alternative ego decisions. We introduce BranchDrive, a branch-structured CARLA dataset and benchmark that pairs one canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures spanning acceleration, braking, and left- and right-steering policies at three magnitudes. Each intervention lasts 2.5 s and is followed by expert recovery. Following control-compliance, modality-completeness, replay-fidelity, and action-leakage audits, the frozen benchmark contains 606 independent branch groups and 7,878 associated trajectories. We evaluate prediction of six continuous short-horizon outcomes and a ten-step ego trajectory using action-only, history-only, structured, visual, multimodal, and privileged bird's-eye-view models. On the held-out test split, the structured history-and-action model achieves a macro normalized mean absolute error of 0.5036 and an average displacement error of 2.2042 m, significantly outperforming both restricted baselines. In full-information offline evaluation, its outcome-derived selector increases balanced policy value from 0.5364 to 0.5704 and reduces normalized regret from 0.2674 to 0.1495 relative to the frozen action prior. However, a validation-calibrated minimum-separation guard rejects every intervention, showing that conservative execution remains unresolved. BranchDrive therefore supports action-conditioned short-horizon prediction and fixed-bank offline decision evaluation, but does not establish exact causal effects, binary safety prediction, or closed-loop safety improvement.

Figures & tables

Fig. 1: BranchDrive collection and sample construction. A privileged expert first executes the nominal route and supplies the canonical history, original control trace, and branch-trigger evidence. The matched route is then replayed under the same configuration for each of twelve intervention policies. The intervention is active for 50 simulator ticks and is followed by online-expert recovery. The resulting branch group contains one shared history, thirteen physically executed futures, action-specific continuous targets, and replay-quality metadata.

Policy family	Parameter	Steering command	Throttle command	Brake command
Nominal expert	—	$\delta_{t}^{E}$	$\alpha_{t}^{E}$	$b_{t}^{E}$
Accelerate straight	$\lambda\in\{0.30,0.50,0.80\}$	$\delta_{t}=0$	$\alpha_{t}=\max(\alpha_{t}^{E},\lambda)$	$b_{t}=0$
Brake	$\lambda\in\{0.30,0.50,1.00\}$	$\delta_{t}=\delta_{t}^{E}$	$\alpha_{t}=0$	$b_{t}=\max(b_{t}^{E},\lambda)$
Steer left	$\lambda\in\{0.05,0.10,0.15\}$	$\delta_{t}=-\lambda$	$\alpha_{t}=\alpha_{t}^{E,0}$	$b_{t}=b_{t}^{E,0}$
Steer right	$\lambda\in\{0.05,0.10,0.15\}$	$\delta_{t}=+\lambda$	$\alpha_{t}=\alpha_{t}^{E,0}$	$b_{t}=b_{t}^{E,0}$

TABLE I: Fixed BranchDrive policy bank. Superscript

E

denotes online-expert control evaluated on the current replay state. Superscript

E,0

denotes a control copied from the original nominal expert rollout. Every intervention is active for 50 simulator ticks and is followed by online-expert recovery.

Stream	Rate	Track	Main contents
Six surround RGB views	4 Hz	Sensor	Front, front-left, front-right, rear, rear-left, rear-right
Top-down RGB	4 Hz	Debug	Collection visualization; excluded from sensor-oriented inputs
Semantic BEV	4 Hz	Privileged	Road, lanes, traffic controls, actors, drivable area, and route
Ego and route state	4 Hz	Mixed	Pose, motion, navigation, route-relative and target-point information
Actor and risk evidence	4 Hz	Privileged	Actor distance, relative motion, projected TTC, lane and event evidence
Expert and applied controls	20 Hz	Audit	Online expert, stored nominal expert, and applied steer/throttle/brake

TABLE II: Principal recorded data streams.

Item	Frozen protocol
Statistical unit	One complete branch group containing 13 candidate futures
Population	606 groups; 431/88/87 train/validation/test
Canonical history	16 state frames at 4 Hz and 80 control ticks at 20 Hz
Candidate bank	Nominal expert plus 12 interventions in four families and three magnitudes per family
Action encoding	29-D family, signed magnitude, duration, control semantics, and recovery
Prediction horizon	Ten saved frames over the 2.5-s intervention interval

TABLE III: Frozen BranchDrive benchmark protocol.

Stage or population	Train	Validation	Test	Total groups	Traj.
Discovered branch-group identifiers	–	–	–	2,467	–
Exact thirteen-action banks	–	–	–	1,751	22,763
All-controls-valid banks	–	–	–	649	8,437
Integrity-complete banks	444	95	95	634	8,242
Core replay-valid	431	88	87	606	7,878
Extended replay-valid	427	88	85	600	7,800

TABLE IV: Dataset qualification and frozen benchmark population. “Traj.” denotes associated nominal and intervention trajectories; it is not an independent-sample count. The primary experiments use the core replay-valid population.

Panel of Figure 6 — Fig. 2: Principal experimental results. (a) The dominant dataset reduction arises from enforcing exact intervention-control semantics. (b) Real-data 13-way leakage classifiers remain at chance; deliberately injected positive controls achieve at least 96.2% accuracy. (c) History-plus-action models provide the best aggregate scalar prediction. Error bars show standard deviation across three seeds. (d) O-M3 improves full-information balanced-utility value over the nominal/action-prior baseline. M6 and O-M6 use privileged semantic BEV.

Model	Input	Params. (M)	Macro NMAE	ADE (m)	FDE (m)	Heading ( ∘ )	Speed (m/s)
M0	Mean	0	0.6976	5.8227	10.2097	15.0468	4.9206
M1	Action	0.202	0.6085	4.8983	7.9022	8.9867	4.2353
M2	State/control history	0.398	0.5826	4.2383	8.8550	14.7979	3.7647
M3	State/control + action	0.553	0.5036	2.2042	4.3739	5.9238	2.3022
M4	RGB + action	1.717	0.5495	3.5473	6.1814	8.4443	3.1534
M5	RGB + state/control + action	2.333	0.5008	2.5346	4.6759	6.9289	2.4524

TABLE V: Action-conditioned prediction on the frozen test split. Learned-model values are means over seeds 11, 23, and 37; M0 is deterministic. Lower is better. † M6 uses privileged semantic BEV.

Selector	Value	$\Delta$ Nom.	NReg.	Pair	NDCG	Top-1
Random	0.4669	$-0.0694$	0.4659	0.5000	0.9309	0.0813
Nominal / ActionPrior	0.5364	0.0000	0.2674	0.5263	0.9475	0.1264
AlwaysBrake	0.4255	$-0.1109$	0.6148	0.4820	0.9222	0.0115
O-M3	0.5704	$+0.0341$	0.1495	0.7340	0.9772	0.3946
O-M5	0.5594	$+0.0230$	0.1615	0.7435	0.9741	0.3602
O-M6 †	0.5698	$+0.0335$	0.1441	0.7432	0.9764	0.4674

TABLE VI: Full-information offline policy evaluation on 87 test groups under the balanced utility. Higher is better except NRegret. † Privileged BEV.

Explore similar work

DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions
May 29, 2026Weicheng Zheng, Yixin Huang, Qiao Sun +2VLMs for Autonomous Driving Autonomous Driving
1Shanghai Qi Zhi Institute · 3Tongji University · 2IIIS Tsinghua University
RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts
Sep 29, 2026Hongbin Lin, Chaoda Zheng, Yiming Yang +11End-to-End Autonomous Driving Action-Conditioned World Models
The Chinese University of Hong Kong, Shenzhen · XPeng Motors · Xidian University
DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting
Sep 8, 2026Yuhang Wang, Chuheng Wei, Jingxin Yang +2Autonomous Driving Datasets Autonomous Driving Benchmarks
University of South Florida · Purdue University · Stanford University +1

May 29, 2026J/K move · Enter open · S save

DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions

Weicheng Zheng, Yixin Huang, Qiao Sun, Derun Li+1

1Shanghai Qi Zhi Institute · 3Tongji University · 2IIIS Tsinghua University

VLMs for Autonomous Driving Autonomous Driving Vision-Language-Action Models Autonomous Driving Planning RL for Autonomous Driving

Open paper AskPDF arXiv

Driving Vision-Language-Action Models (Driving VLAs) aim to use language to improve end-to-end planning, but the language-action gap limits this promise. We propose DriveMA, a Driving VLA framework built on verifiable meta-actions, which summarize future ego motion into compact language-domain intentions and can be constructed from expert trajectories with a trajectory-grounded annotation pipeline and can be verified against generated trajectories through rule-based projection. DriveMA exploits this verifiability with action-centric supervised training and a data-efficient turn-level credit assignment reinforcement learning framework, explicitly aligning high-level decisions with low-level trajectory planning through dense rewards and precise credit assignment. DriveMA sets a new state of the art on the Waymo Open Dataset Vision-based E2E Driving, achieving a Rater Feedback Score of 8.060 with a 2B model and further improving it to 8.079 with a 4B model; it also obtains competitive closed-loop planning performance on NAVSIM. These results show that even a simple meta-action interface can achieve state-of-the-art planning when made verifiable and optimized for language-action alignment. Code, data, and models are available at https://tsinghua-mars-lab.github.io/DriveMA.

Abstract

Explore similar work

DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions

RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts

DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting