LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings

Yongshuo Liu; Xu Gao; Morui Zhu; Yongqi Zhu; Qi Chen; Deyuan Qu; Song Fu; Qing Yang

cs.ROSep 6, 2026

LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings

Authors: Yongshuo Liu, Xu Gao, Morui Zhu, Yongqi Zhu, Qi Chen, Deyuan Qu, Song Fu, Qing Yang

Organizations: Department of Computer Science and Engineering, University of North Texas, Denton, TX, USA · Toyota Motor North America, InfoTech Labs, USA

Abstract

We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings. LANTERN separates warning onset, hazard onset, warning termination, and post-hazard recovery, and evaluates each physical event under matched warning and no-warning executions so that the warning's contribution is measured in isolation rather than confounded with onboard vision. The benchmark spans six safety-critical scenario families and provides 3,272 sequences with 236,309 frames for training, together with 120 matched route pairs for closed-loop evaluation. Each hazard route is evaluated under the warning and no-warning conditions, while its no-hazard control penalizes unconditional braking. We further introduce the Cooperative Unified Score (CUS), a safety-gated metric that jointly rewards route progress, anticipation, clearance, and recovery. Fine-tuning a representative VLM driving model raises CUS from 34.6 without warnings to 75.5 with them, demonstrating both the value of cooperative warnings and the discriminative power of the paired protocol. All resources will be made publicly available.

Figures & tables

Figure 1: Driving inputs and outputs under the warning and no-warning conditions. Both conditions receive visual observations, ego state, and navigation state; the warning condition additionally receives the cooperative warning during the warning interval.

Benchmark	Benchmark Setting					Cooperative Evaluation
	Domain	Loop	Task	Hazard scenarios	Training data	Cooperative input	Information timing	Main metrics
Bench2Drive ( Jia et al. 2024 )	CARLA	Closed	E2E driving	✓	✓	–	–	Driving score, success
SafeBench ( Xu et al. 2022a )	CARLA	Closed	Safety testing	✓	$\times$	–	–	Safety score
Bench2Drive-VL ( Jia et al. 2026 )	CARLA	Closed	VLM driving	✓	✓	–	–	Question answering, score
V2X-VLM ( You et al. 2024 )	Real world	Open	Coop. planning	$\times$	✓	Images and text	Frame-aligned	$L_{2}$ error, collision rate
MDrive ( Coscoy et al. 2026 )	CARLA/ Real2Sim	Closed	Multi-agent	✓	$\times$	Perception and negotiation	Online	Driving score, route completion, coop. gain

Table 1: Comparison of representative driving benchmarks across benchmark settings and cooperative evaluation protocols. Checkmarks indicate explicit support, while crosses indicate absence.

Scenario family	Cooperative warning template
Pedestrian Emergence	The pedestrian at position <FRONT_CAR_POS> is about to enter the ego lane.
Vehicle Cut-In	The vehicle at position <FRONT_CAR_POS> will cut into the ego lane soon.
Obstacle Reveal	The static object at position <FRONT_CAR_POS> is blocking the ego lane.
Left-Turn Wrong-Way	A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego left-turn path. Yield before turning left.
Right-Turn Wrong-Way	A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego right-turn path. Yield before turning right.
Right-Turn Yield	The vehicle at position <FRONT_CAR_POS> will cross the ego right-turn path. Yield before turning right.

Table 2: Cooperative warning templates for the six scenario families. Before inference, <FRONT_CAR_POS> is replaced by the hazard or conflict location

(x,y)

in the current ego frame, measured in meters with

x

forward and

y

right.

Figure 2: Temporal organization of a LANTERN scenario. The warning begins before physical-event onset and terminates when the hazard is first directly observed through the ego vehicle’s onboard sensors.

Figure 3: Aligned observation formats and sensor annotations in LANTERN. The six-camera format provides surround coverage, while the front-view format provides a single $110^{\circ}$ camera interface. Both support RGB, depth, point clouds, 3D bounding boxes, LiDAR BEV, and semantic segmentation.

Figure 4: Open-loop dataset composition. Bars report Route A hazard sequences and Route B control sequences at the route and frame levels for each scenario family.

Model	PE	VC	OR	LWW	RWW	RTY	Overall
Coop-SimLingo ( warning )	88.9	86.5	95.9	50.6	65.0	66.2	75.5
Coop-SimLingo ( no-warning )	8.0	59.8	55.9	4.2	17.2	62.5	34.6
warning gain (paired)	+80.9	+26.7	+40.0	+46.4	+47.8	+3.7	+40.9
SimLingo ( warning )	59.0	45.4	56.3	13.8	18.2	65.6	43.1
SimLingo ( no-warning )	47.0	42.0	42.1	0.0	4.4	57.5	32.2
VAD	46.2	45.4	47.9	3.5	0.0	19.6	27.1

Table 3: CUS (

\uparrow

) by scenario family. Shading highlights the paired warning / no-warning comparison and its gain; Overall is the macro-average, and bold marks the best score.

	Continuous components and gate pass (%, $\uparrow$ )							Hard-gate failure rate (%, $\downarrow$ )
Model	A-Prog.	TTC	Clear.	Ant.	Rec.	B-Prog.	Gate Pass	Coll.-A	RD-A	LC-A	Rule-A	Block-A	False Brake-B
Coop-SimLingo ( warning )	95.4	70.3	97.4	86.7	82.4	93.7	82.5	0.0	0.0	0.8	9.2	5.8	1.7
SimLingo ( warning )	85.1	48.3	63.1	58.3	81.3	90.8	50.0	28.3	1.7	3.3	7.5	24.2	0.0
Coop-SimLingo ( no-warning )	89.2	34.8	44.4	53.5	62.7	93.7	43.3	41.7	0.0	1.7	8.3	19.2	1.7
SimLingo ( no-warning )	83.7	38.4	46.4	54.6	60.1	90.8	37.5	46.7	0.8	2.5	12.5	33.3	0.0
VAD	68.1	44.6	53.0	49.1	62.3	75.3	35.0	28.3	0.8	9.2	6.7	42.5	0.0

Table 4: CUS decomposition over 120 matched Route A/B pairs. Continuous components and Gate Pass are higher-is-better, hard-gate failure rates are lower-is-better, and column-best values are bold.

Scenario	ADE $\downarrow$	FDE $\downarrow$	DAC $\uparrow$	Warning Response $\uparrow$
Pedestrian Emergence	2.88	5.76	100.0	93.0
Vehicle Cut-In	2.29	4.65	100.0	88.0
Obstacle Reveal	2.54	4.89	100.0	96.0
Left-Turn Wrong-Way	3.03	6.62	99.0	99.0
Right-Turn Wrong-Way	2.87	6.41	100.0	98.0
Right-Turn Yield	2.81	6.17	95.0	95.0

Table 5: Zero-shot open-loop diagnostics on nuScenes using 100 frames per scenario family. Average Displacement Error (ADE) and Final Displacement Error (FDE) are reported in meters; Drivable-Area Compliance (DAC) and Warning Response are percentages.

Figure 5: Paired nuScenes examples for Coop-SimLingo. Cyan and magenta show no-warning and warning predictions from the same frame; each panel reproduces the complete warning input below the image.

Variant	CUS $\uparrow$	A-Prog. $\uparrow$	Event Exp. $\uparrow$	Ant. $\uparrow$	Rec. $\uparrow$	Coll.-A $\downarrow$	Block-A $\downarrow$
Full temporal protocol	75.5	95.4	99.2	86.7	82.4	0.0	5.8
Late warning onset	54.9	95.2	100.0	0.0	73.8	14.9	7.6
Warning beyond termination	63.3	79.6	100.0	84.5	64.1	0.0	29.7
No anti-deadlock rule	51.2	79.0	71.0	87.0	74.2	9.4	35.7
Noisy warning position	75.5	98.4	100.0	87.4	82.1	3.4	3.3
Held-out warning paraphrase	57.7	85.9	100.0	78.6	75.8	25.7	5.9

Table 6: Sensitivity to warning timing, event activation, localization, and wording. Each variant changes one component of the full protocol; all entries use a 0–100 scale.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Sequence-level distribution of the open-loop release and its environmental extension. (a) Town coverage for all 3,272 sequences. (b) Weather profiles and (c) time periods for the 859-sequence extension.

Scenario	Physical event	Warn lead	Event lead	Actor parameters	Clearance and Route B control
Pedestrian Emergence	A pedestrian emerges from behind a large roadside occluder and crosses the ego lane.	18 m	9 m	3 m/s; 22-m crossing; 30-m recovery	Walker and occluder are removed after the crossing. Route B retains only the occluder.
Vehicle Cut-In	A vehicle 21 m ahead in an adjacent same-direction lane merges into the ego lane.	10/17 m	4 m	80–85% reference speed; $\geq$ 6.5 m/s; 2-s lane change	The actor is handed to Traffic Manager after the merge. Route B keeps it in the adjacent lane.
Obstacle Reveal	A lead van changes lane, exposing a stationary object in the ego lane.	12 m	6 m	Lead at 12–20 km/h; object 12–16 m ahead; 8-s stop limit	The object is removed after the ego response or stop limit. Route B retains the lead vehicle but creates no object.
Left-Turn Wrong-Way	A wrong-way vehicle follows the reverse ego route through the left-turn conflict.	10 m	4 m	7–9 m/s; 25-m spawn offset; 15-m clearance target	Actor is removed after passing the conflict and traveling at least 5 m. Route B sends it in the legal direction.
Right-Turn Wrong-Way	A wrong-way vehicle follows the reverse ego route through the right-turn conflict.	10 m	4 m	7–9 m/s; 20-m spawn offset; 20-m clearance target	Actor is removed after clearing the turn path. Route B sends it in the legal direction.
Right-Turn Yield	Cross traffic from the left traverses the ego vehicle’s right-turn path.	10 m	10 m	8–12 m/s; source interval 10–50 m, extended to 60 m on designated routes; one evaluated passage	Flow ends after the first actor clears the ego route (or a 12-s safety limit). Route B keeps a stationary context actor.

Appendix

Table 7: Temporal and behavioral specification of the six LANTERN scenario families.

Panel of Figure 7 — Figure 7: Matched predictions for the unseen cyclist event at simulation step 170: no warning (left), free-form warning (center), and held-out paraphrase (right). At this point, the ego speed is 10.5, 4.7, and 6.5 m/s, respectively.

Panel of Figure 8 — Figure 8: Matched event-on predictions for the emergency crossing (top) and reversing vehicle (bottom). Left panels are from no-warning rollouts; right panels are from rollouts that received the exact warning, which may already have terminated by the displayed event-on frame. The warning-conditioned rollouts exhibit substantially reduced predicted progress.

Wording	Paraphrase CUS	Matched canonical	Retention	Coll.
Low ambiguity ( $n=39$ )	83.1	77.9	106.7%	0.0%
Medium ambiguity ( $n=39$ )	51.1	69.2	73.9%	28.2%
High ambiguity, no position ( $n=38$ )	37.5	69.3	54.1%	50.0%

Appendix

Table 11: Warning-wording ablation by ambiguity. Retention is paraphrase CUS divided by canonical-warning CUS on the same valid routes.

Panel of Figure 9 — Figure 9: Representative high-ambiguity, position-free warnings. The Obstacle Reveal route completes successfully (left), whereas the Pedestrian Emergence route collides after an insufficient response (right).

Figure 10: Distance-conditioned responses on three held-out S1 control frames. Columns show no-warning , a nearby warning, and the same warning at 100 m. The nearby warning produces a stopping trajectory, whereas the distant warning preserves ordinary progress.

Figure 11: Representative response to a warning about a cyclist traveling away on a physically separated path. The predicted trajectory continues along the route without sustained stopping.

Autonomous Driving Benchmarks

Autonomous driving benchmarks evaluate how well vehicles perceive, predict, plan, and act safely across varied roads and conditions.

See all 141 papers

CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models
Aug 7, 2026Hsu-kuang Chiu, Stephen F. SmithAutonomous Driving Benchmarks End-to-End Autonomous Driving
Carnegie Mellon University, Robotics Institute
CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning
Apr 9, 2026Rui Gan, Junyi Ma, Pei Li +4Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
University of Wisconsin–Madison · University of Wyoming · Columbia University
CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning
Sep 22, 2026Kang Yang, Shuai Liu, Hang Li +3Autonomous Driving VLMs for Autonomous Driving
Renmin University of China · Hebei Key Laboratory of Real-virtual Integrated Autonomous Systems (RIAS)

Aug 7, 2026J/K move · Enter open · S save

CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

Hsu-kuang Chiu, Stephen F. Smith

Carnegie Mellon University, Robotics Institute

Autonomous Driving Benchmarks End-to-End Autonomous Driving Autonomous Driving Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently achieved impressive performance for end-to-end autonomous driving, yet existing approaches are primarily designed for an individual single autonomous driving agent with limited support for cooperative perception, reasoning, and planning. We present Cooperative Multi-agent Unified Driving with Reasoning (CMU-Drive), a closed-loop end-to-end benchmark for evaluating cooperative autonomous driving with multiple connected autonomous vehicles (CAVs) operating in safety-critical driving scenarios with background traffic participants. We further propose Vehicle-to-Vehicle Vision-Language-Action (V2V-VLA), a cooperative VLA model that integrates cooperative driving into a single forward pass by jointly generating driving actions, future waypoints, language reasoning, and communication policies. Experiments on CMU-Drive establish the first benchmark and baseline for cooperative VLA driving and provide a foundation for future research on multi-agent, closed-loop, end-to-end cooperative autonomous driving. Our code, benchmark, and model checkpoint will be publicly released to facilitate open-source research.

Abstract

Autonomous Driving Benchmarks

Explore similar work

CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning

CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning