GatedVLA with Camera-Based Emergency Braking: AlpaSim E2E Closed Loop Challenge Technical Report

Kaneyoshi Hiratsuka; Naoki Takada; Makoto Kawano

cs.CVNov 1, 2026

GatedVLA with Camera-Based Emergency Braking: AlpaSim E2E Closed Loop Challenge Technical Report

Authors: Kaneyoshi Hiratsuka, Naoki Takada, Makoto Kawano

Abstract

This report presents GatedVLA, our entry for the Physical AI AV (PAI-AV) track. Our learned vision-language-action (VLA) policy pairs the driving-pretrained Qwen-Drive-1.0-4B vision-language model with a layer-wise flow-matching action expert that outputs a 2.5 s horizon of acceleration and curvature controls, augmented with a soft visual mask applied to the expert's cross-attention. Compared to an identically trained baseline, the masked policy achieves comparable overall open-loop error while reducing error on left turns. In local closed-loop evaluation across the same 150 scenes (two rollouts per scene), the visual mask increases the mean scene score from 0.479 to 0.517 (paired difference +0.038, 95% bootstrap CI [-0.018, +0.093], wins/losses 37/26, sign test p = 0.21). Incorporating a rule-based automatic emergency braking (AEB) module at deployment further increases the mean score to 0.599 (+0.082, CI [+0.023, +0.143]) and reduces the mean number of scenes with at-fault collisions from 34.0 to 11.0.

Figures & tables

Figure 1: System overview. (a) GatedVLA policy, trained end-to-end: the visual mask $m_{p}$ biases the expert’s cross-attention to the VLM’s image tokens. (b) Pretrained detector and rule-based AEB, integrated only at deployment: they re-time the planned waypoints along their path.

	ADE	FDE	Miss rate
All	0.337 / 0.341	1.029 / 1.023	0.125 / 0.125
Left	0.405 / 0.376	1.213 / 1.100	0.200 / 0.150
Right	0.367 / 0.360	1.136 / 1.089	0.087 / 0.100
Straight	0.239 / 0.287	0.737 / 0.879	0.087 / 0.125

Table 1: Open-loop prediction error at step 38,400 (baseline / mask). ADE and FDE in meters over 2.5 s; miss: any waypoint error

>2

Configuration	Host	Score	At-fault	Offroad	Corridor exit	Rear-ended	Drive (s)
Baseline, 1st	Blackwell	0.476	42	25	8	12	0.133
Baseline, 2nd	Ada	0.481	41	19	15	9	0.066
Mask, 1st	Blackwell	0.499	35	22	12	15	0.110
Mask, 2nd	Ada	0.534	33	23	8	15	0.146
Mask + AEB 10 Hz, 1st	Blackwell	0.607	12	27	11	25	0.201
Mask + AEB 10 Hz, 2nd	Ada	0.591	10	31	13	23	0.085

Table 2: Local closed-loop results across the same 150 scenes. Failure counts are numbers of affected scenes; rear-end collisions by other agents are not at-fault and merely halt progress accrual. Mean Drive latency includes driver video recording.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Epochs / steps	12 / 38,400 (3,200 steps per epoch)
Hardware	1 Aurora node, 12 XPU tiles, one rank per tile (batch size 4 per tile)
Precision	bf16; gradient checkpointing enabled; max gradient norm 1
Optimizer	AdamW, betas $(0.9,0.95)$ , $\epsilon=10^{-8}$ , weight decay 0
Learning rates	VLM $10^{-6}$ ; expert $5\times 10^{-5}$ ; gate $10^{-4}$ (mask run)
Schedule	200 warmup steps, cosine decay to $10^{-8}$ ; VLM frozen in epoch 1, then warmed up and decayed over the rest

Appendix

Table 3: Training configuration (identical for both runs unless otherwise stated). Source: W&B configurations of ccb412c9 and 03bdfc50 ; the model bundles deployed in the container images carry identical values.

Item	Value
Replanning	Every 5th 100 ms interval (0.5 s); plan cached in the local frame and re-sliced at each tick
Temporal ensembling; action samples	Off (1 plan; 1 sample) across all three submissions
Linear-attention kernels	auto : flash-linear-attention Triton kernels, falling back to the reference implementation
Expert denoising	Replayed from a captured CUDA graph
Target point	25 m along the route; route resampled at 0.1 m and extended 2 m past the farthest target
Controller gains	Official defaults (no --controller-gains override)

Appendix

Table 4: Deployment configuration. Replanning, CUDA graph, and detector settings correspond to default parameters or pre-configured overrides baked into the submitted container images.

Diagnostic	Value
Mean keep, history frame	0.200
Mean keep, current frame	0.200
Spearman rank correlation between masks of two samples (1 = identical fixed pattern)	0.055
Mask variance explained by the mean row pattern	0.024
Mask variance explained by the mean column pattern	0.003
Spearman rank correlation between a sample’s mask and its DiT attention	0.827

Appendix

Table 5: Mask diagnostics across the validation set (current frame) at step 38,400.

	Mask run		Baseline
Signal	First	Last	First	Last
Flow-matching loss (Path A)	0.9586	0.1790	0.9424	0.1796
Loss, acceleration	–	–	0.9591	0.2635
Loss, curvature	–	–	0.9256	0.0957
Path B (distillation, DiT)	0.0521	0.0071	–	–
Path C (distillation, gate)	0.0674	0.0122	–	–

Appendix

Table 6: Training signals: mean of the first 50 and last 200 logged points (logged every 10 steps).

Figure 2: Evolution of the visual mask on a left-turn validation sample across training: steps 0, 1,000, 5,000, 20,000, and 38,400 (top to bottom). Panels from left to right: current front-camera frame; retention weight $m$ ; standardized gate score $z$ ; DiT cross-attention map on the frame; and bird’s-eye view showing past trajectory, ground truth, prediction, and target point (lateral axis positive to the right).

Figure 3: Mask-run validation samples at step 38,400: a left turn, a right turn, and straight driving (top to bottom), with panels arranged as in Fig. 2 .

Figure 4: Smoothed training losses (EMA factor 0.05). The baseline flow-matching loss closely tracks Path A of the masked run, while Path C with the learned mask remains consistently below its shuffled-mask and unmasked controls.

Figure 5: Mask diagnostics across training. The rank correlation between the mask and DiT cross-attention decreases from near 1 to approximately 0.83, cross-attention mass on text tokens increases early in training, and mean row/column spatial patterns account for minimal mask variance.

	Run	ADE	FDE	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	Lat.	Lon.	Miss
All	Baseline	0.337	1.029	0.025	0.117	0.300	0.574	0.943	0.478	0.802	0.125
	Mask	0.341	1.023	0.028	0.126	0.308	0.578	0.939	0.457	0.796	0.125
Left	Baseline	0.405	1.213	0.034	0.151	0.365	0.686	1.115	0.768	0.790	0.200
	Mask	0.376	1.100	0.034	0.146	0.347	0.637	1.014	0.667	0.712	0.150
Right	Baseline	0.367	1.136	0.025	0.117	0.320	0.633	1.041	0.516	0.931	0.087
	Mask	0.360	1.089	0.031	0.136	0.321	0.606	0.997	0.526	0.867	0.100

Appendix

Table 7: Open-loop metrics at step 38,400 on the 240 fixed validation samples (in meters; miss rate as a fraction). L2 denotes position error at the specified horizon (in seconds); Lat./Lon. denote absolute final lateral and longitudinal errors.

	All			Left			Right			Straight
Inference path	0.341	1.023	0.125	0.376	1.100	0.150	0.360	1.089	0.100	0.287	0.879	0.125
No mask	0.339	1.016	0.129	0.376	1.096	0.138	0.358	1.086	0.125	0.282	0.866	0.125
Shuffled mask	0.344	1.031	0.133	0.382	1.115	0.150	0.363	1.099	0.125	0.287	0.879	0.125
Mirrored target in gate	0.348	1.049	0.129	0.392	1.154	0.175	0.363	1.114	0.087	0.288	0.881	0.125
No target in DiT, no mask	0.367	1.135	0.167	0.395	1.217	0.188	0.403	1.261	0.163	0.303	0.926	0.150
No target in DiT, mask	0.359	1.094	0.142	0.383	1.150	0.163	0.391	1.210	0.125	0.302	0.921	0.138

Appendix

Table 8: Diagnostic controls for the masked policy at step 38,400, sharing the sampled noise of the main prediction. Each three-column group reports ADE, FDE, and miss rate.

	Baseline	Mask	No mask	Shuffled	Mirrored	No target	No img tokens	Black	Base. black
Corr. of final lateral offset	0.986	0.987	0.987	0.987	0.986	0.984	0.967	0.959	0.969
Std of predicted final lateral (m)	4.451	4.482	4.488	4.497	4.384	4.475	3.981	3.668	3.993
Left: direction accuracy	0.987	1.000	1.000	1.000	0.987	1.000	1.000	1.000	0.987
Right: direction accuracy	1.000	1.000	1.000	1.000	1.000	0.987	0.973	1.000	0.973
Left: median pred./GT lateral	0.993	1.002	1.000	0.993	0.961	0.984	0.919	0.897	0.931
Right: median pred./GT lateral	1.031	1.020	1.018	1.025	0.973	0.986	0.823	0.726	0.838

Appendix

Table 9: Turn response metrics at step 38,400. Direction accuracy and lateral displacement ratios are computed over left/right turn samples with

|y_{\mathrm{GT}}|>0.5

Figure 6: Open-loop validation metrics evaluated every 250 training steps. Vertical axes are clipped for readability, placing initial early-training evaluations off-scale.

Figure 7: L2 position error across the 2.5 s prediction horizon at step 38,400.

Figure 8: Mask-run diagnostic controls across training (validation FDE). Control definitions are provided in Appendix C .

Figure 9: Turn response metrics across training.

Figure 10: Cosine similarity among projected VLM tokens entering the DiT at three depth blocks. In the deepest block, token representations in the masked run exhibit lower pairwise similarity than in the baseline.

Figure 11: Baseline diagnostics: DiT cross-attention statistics (top), and correlations between detached probes (attention pooling and an MLP) on projected VLM features and future motion across three DiT blocks (bottom).

Figure 12: Smoothed optimization signals across training. Throughput is substantially higher during the first epoch while the VLM backbone is frozen.

Shrink	Half-width	Trigger	Gap	Samples	Hits, $\geq 1$ s	Hits, $\geq 0.5$ s	Clean: rollouts / frames
0.0	1.6	3.0	2.5	4	36	38	38 / 769
0.3	1.3	4.0	1.5	6	24	28	15 / 92
0.3	1.3	3.0	2.5	6	29	32	19 / 164
0.3	1.3	4.0	2.5	6	29	32	17 / 127
0.15	1.3	3.0	2.5	6	32	37	22 / 227
0.15	1.45	3.0	2.5	6	34	38	24 / 268

Appendix

Table 10: Offline AEB hyperparameter sweep on replayed detections from the masked policy’s 150 rollouts (comprising 41 front collisions and 62 incident-free rollouts). A braking event is counted only when the speed cap shortens the planned path at a 1 s horizon by more than 0.3 m. The selected deployment configuration is highlighted in bold. Evaluation is open-loop (ego motion does not react to simulated braking).

Figure 13: Per-scene rollout outcomes across the same 150 evaluation scenes.

	Baseline, 1st	Baseline, 2nd	Mask, 1st	Mask, 2nd	AEB 10 Hz, 1st	AEB 10 Hz, 2nd	AEB 5 Hz
Offroad	25 / 0.167	19 / 0.127	22 / 0.147	23 / 0.153	27 / 0.180	31 / 0.207	27 / 0.180
At-fault collision	42 / 0.280	41 / 0.273	35 / 0.233	33 / 0.220	12 / 0.080	10 / 0.067	13 / 0.087
Corridor exit	8 / 0.053	15 / 0.100	12 / 0.080	8 / 0.053	11 / 0.073	13 / 0.087	10 / 0.067
Rear-ended (partial)	12 / 0.022	9 / 0.015	15 / 0.038	15 / 0.037	25 / 0.054	23 / 0.046	28 / 0.067
Short progress	4 / 0.002	4 / 0.003	1 / 0.002	3 / 0.002	5 / 0.005	1 / 0.002	6 / 0.009
Full score	59 / 0.000	62 / 0.000	65 / 0.000	68 / 0.000	70 / 0.000	72 / 0.000	66 / 0.000

Appendix

Table 11: Score-loss decomposition across the same 150 scenes in every column, formatted as: affected scenes / contribution to

(1-\text{mean score})

Comparison	$n$	Ref.	Run	Diff.	95% CI	W/L	$p$
Mask (2) vs. baseline (2)	150	0.479	0.517	+0.038	[-0.018, +0.093]	37/26	0.21
Mask 1st vs. baseline 1st	150	0.476	0.499	+0.023	[-0.049, +0.093]	28/20	0.31
Mask 2nd vs. baseline 2nd	150	0.481	0.534	+0.053	[-0.022, +0.126]	31/19	0.12
Mask 2nd vs. mask 1st	150	0.499	0.534	+0.035	[-0.017, +0.089]	22/16	0.42
Baseline 2nd vs. baseline 1st	150	0.476	0.481	+0.005	[-0.065, +0.076]	25/20	0.55
AEB 10 Hz (2) vs. mask (2)	150	0.517	0.599	+0.082	[+0.023, +0.143]	41/28	0.15

Appendix

Table 12: Paired per-scene comparisons across the same 150 scenes. CI denotes the 95% bootstrap confidence interval (10,000 resamples);

p

denotes the two-sided sign test

p

-value (ignoring ties, uncorrected for multiple comparisons); (2) indicates the per-scene mean across two rollouts.

Without $\backslash$ with	Offroad	Coll.	Exit	Rear	Short	Full
Offroad	19		1			2
At-fault collision	2	9	2	11	2	9
Corridor exit	3		5	1		3
Rear-ended (partial)	1		1	11	2
Short progress					1
Full score	2	3	2	2		56

Appendix

Table 13: Scene outcome transition matrix when adding 10 Hz AEB (first rollouts, same host, 150 scenes): rows denote outcomes without AEB, and columns denote outcomes with AEB.

	Hits	Rate	Accel.	No slowing	Stopped lead	$v_{\mathrm{hit}}$
Baseline (2)	90/301	29.9%	42%	68%	39%	11.6
Mask (2)	77/300	25.7%	30%	64%	38%	10.9
Mask + AEB 10 Hz (2)	28/300	9.3%	32%	50%	61%	9.1
Mask + AEB 5 Hz	12/150	8.0%	17%	42%	67%	2.4
hist_ft	46/147	31.3%	46%	67%	39%	13.2
sig02	50/150	33.3%	32%	62%	36%	12.9

Appendix

Table 14: Pre-impact kinematics for first-incident front collisions across the same 150 scenes: overall front-collision rate; fraction with a speed increase

>2

m/s over the preceding 3 s; fraction whose final planned trajectory failed to command deceleration (planned speed 2 s ahead

\geq v-0.3

m/s); fraction where the nearest lead actor was traveling below 1 m/s (approximate); and median ego impact speed (m/s).

Figure 14: Ego speed profiles across the 11 scenes that incurred an at-fault collision without AEB but achieved a full score with 10 Hz AEB enabled (first rollouts). Dashed vertical lines mark the time of collision without AEB.

Figure 15: Side-by-side comparison 0.3 s prior to collision without AEB versus the same elapsed rollout time with AEB enabled, for 2 of the 11 rescued scenes. Each tile shows the map view (left) and the camera view (right).

Scene	Class	Type	$t$	$v_{-3}$	$v$	$a_{\mathrm{min}}$	$v_{o}$	Lon.	Lat.	Head.	Brake
0caa8f1a	Close range	front	9.4	0.1	1.2	-1.5	0.5	4.4	-1.4	-10	4.5
d641893f	Close range	front	15.1	0.9	4.2	0.8	12.0	-1.1	3.1	-179	none
05855d57	Corridor	front	11.5	30.3	32.0	-0.7	26.8	4.3	-0.0	2	0.3
213dfdac	Corridor	front	3.2	11.3	8.7	-1.5	0.1	2.2	-1.9	7	none
24a50fcc	Corridor	front	11.3	9.8	9.7	-0.7	7.1	1.9	-1.9	4	none
8b25a6ac	Corridor	side	10.6	9.9	9.9	-0.1	0.0	2.7	-2.1	-1	none

Appendix

Table 15: Breakdown of the 12 remaining at-fault collisions under 10 Hz AEB (first rollout). Columns report: elapsed time

t

from rollout start (s); ego speeds 3 s prior (

v_{-3}

) and at impact (

v

) in m/s; minimum acceleration

a_{\mathrm{min}}

over the last 3 s (m/s 2 ); kinematics of the nearest traffic agent at impact (speed

v_{o}

, longitudinal and lateral center offsets in the ego frame, and relative heading in degrees; approximate, as the nearest agent may not be the struck actor); and lead time (s) prior to impact at which the AEB trigger condition was first met in offline detector replay. Failure categories reflect manual inspection.

Figure 16: Views 1 s prior to impact for all 12 remaining at-fault collisions under 10 Hz AEB (first rollout); Map view (left) and camera view (right); Table 15 classifies them.

Thr. (m)	Fail.	Clean	Fail.	Clean
	Ego		Plan at 2 s
0.5	27/38	50/70	29/38	49/70
0.75	21/38	37/70	26/38	39/70
1.0	19/38	30/70	23/38	33/70
1.5	15/38	17/70	17/38	20/70
2.0	13/38	15/70	15/38	16/70

Appendix

Table 16: Offline evaluation of a hypothetical route-distance guard (not deployed): number of lateral failure rollouts flagged at least 1 s prior to failure versus number of full-score clean rollouts falsely flagged at any point.

Failure	Before	$n$	Median	p25	p75	$>1$ m	$>2$ m
Offroad	$-3$ s	18	0.40	0.18	2.56	5	5
	$-2$ s	23	0.49	0.31	1.36	6	4
	$-1$ s	23	0.70	0.33	1.91	11	5
	$-0$ s	23	0.67	0.40	1.99	8	6
Corridor exit	$-3$ s	5	0.81	0.70	0.87	1	1
	$-2$ s	7	0.79	0.34	2.77	3	3

Appendix

Table 17: Lateral distance (m) from the ego vehicle to the served map route prior to lateral failures under 10 Hz AEB. Scenes lacking a covering route segment at the queried timestamp are omitted. For reference, across all frames of the 70 full-score rollouts, the lateral distance distribution has a median of 0.29 m, 90th percentile of 1.14 m, and 99th percentile of 2.99 m.

Figure 17: Views 0.8 s prior to the event under 10 Hz AEB (first rollout), in scene-ID order: the first 3 of the 27 off-road failures (top) and the first 3 of the 11 lateral corridor exits (bottom). Map view (left) and camera view (right) of each scene.

Comparison	$n$	Ref.	Run	Diff.	95% CI	W/L	$p$
Baseline + TE4 vs. baseline (2)	150	0.479	0.504	+0.026	[-0.027, +0.079]	33/22	0.18
Mask + TE4 vs. mask (2)	150	0.517	0.495	-0.021	[-0.082, +0.041]	25/29	0.68
Mask + 8 samples vs. mask (2)	150	0.517	0.512	-0.005	[-0.051, +0.040]	24/21	0.77
hist_ft vs. mask (2)	150	0.517	0.495	-0.021	[-0.082, +0.037]	29/26	0.79
sig02, 1st vs. mask (2)	150	0.517	0.505	-0.012	[-0.077, +0.055]	27/28	1.00
sig02, 2nd vs. mask (2)	150	0.517	0.497	-0.020	[-0.084, +0.043]	29/29	1.00

Appendix

Table 18: Closed-loop evaluation of unsubmitted variants (without AEB), paired as in Table 12 (comparing a single rollout of each variant against the two-rollout mean of the reference model).

	Mean Drive	Planning call	Other calls, mean	Other calls, p50	Calls with detector
AEB 10 Hz	0.111	0.249	0.077	0.026	143/143
AEB 5 Hz	0.104	0.230	0.072	0.015	82/143

Appendix

Table 19: Standalone runtime of the pre-configured submission container images launched without arguments in the challenge sandbox: 2 replicas with 1 active session each on a single shared GPU over an approximately 27 s rollout. All latencies are reported in seconds.

	Sessions	Ticks	Mean	p50	p90	p99	Per tick	Wall
No AEB	6	1071	0.294	0.001	1.457	1.512	0.322	64
AEB 10 Hz	6	1071	0.317	0.169	0.890	1.125	0.346	71
AEB 5 Hz	6	1071	0.306	0.113	1.167	1.365	0.333	67

Appendix

Table 20: Throughput saturation benchmark: logged rollout observations replayed continuously into 3 driver replicas (within the submission sandbox, video recording disabled) on a single shared NVIDIA RTX 6000 Ada GPU, with 2 concurrent sessions per replica and zero simulator rendering delay between consecutive RPC calls. Because eliminating simulator think-time exaggerates GPU contention, these values should be used only for relative comparisons across rows. All times are reported in seconds.

Closed-Loop Control of VLA Models

See all 31 papers

VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
Jun 10, 2026Jin Yao, Dhruva Dixith Kurra, Tom Lampo +3VLMs for Autonomous Driving End-to-End Autonomous Driving
Uber AV Labs · University of Virginia
Inference-Time Attention Steering for Vision-Language-Action Driving Models
Aug 17, 2026Darshan Nagendra Prasad, Lars Ullrich, Knut GraichenVision-Language-Action Models VLMs for Autonomous Driving
FAU Erlangen-Nürnberg, Germany
FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory
Sep 16, 2026Kemal Oksuz, Alexandru Buburuzan, Yuhan Yao +1Memory-Augmented VLMs VLMs for Autonomous Driving
Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom

Jun 10, 2026J/K move · Enter open · S save

VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

Jin Yao, Dhruva Dixith Kurra, Tom Lampo, Zezhou Cheng+2

Uber AV Labs · University of Virginia

VLMs for Autonomous Driving End-to-End Autonomous Driving Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50,m average) and 3-second collision rate (0.18%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.

Abstract

Closed-Loop Control of VLA Models

Explore similar work

VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

Inference-Time Attention Steering for Vision-Language-Action Driving Models

FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory