Closed-Loop Evaluation of Bird's-Eye-View Maps from Cross-View Transformers as Inputs to Behavior-Cloning Policies

Felipe Carlos dos Santos; Eric Antonelo; Gustavo Claudio Karl Couto

cs.ROSep 5, 2026

Closed-Loop Evaluation of Bird's-Eye-View Maps from Cross-View Transformers as Inputs to Behavior-Cloning Policies

Authors: Felipe Carlos dos Santos, Eric Antonelo, Gustavo Claudio Karl Couto

Organizations: Federal University of Santa Catarina (UFSC), Florianópolis, Brazil Department of Automation and Systems Engineering

Abstract

In autonomous driving, Bird's-Eye View (BEV) representations provide a structured, top-down abstraction of the vehicle's surroundings and have become a key input modality for Behavioral Cloning (BC) policies. While ground-truth BEV maps are readily available in simulation, real-world deployment requires replacing them with camera-predicted counterparts - a substitution that introduces perceptual errors whose downstream impact on closed-loop driving performance is not well understood. In this work, we investigate the use of Cross-View Transformer (CVT)-predicted BEV maps as direct policy inputs for a BC agent in the CARLA simulator. We propose a six-channel BEV representation covering road surface, planned route, lane boundaries, vehicles, pedestrians, and traffic lights, and introduce a Kernel Density Estimation (KDE) weighting scheme that rebalances the segmentation loss towards underrepresented driving maneuvers such as curves and intersections. Closed-loop evaluation across two CARLA towns shows that the KDE-weighted model is the only predicted-BEV agent to complete a full episode without infractions, despite not achieving the highest aggregate IoU. This discrepancy reveals that global segmentation metrics are poor proxies for driving performance: what determines navigation success is prediction quality at geometrically critical locations, and the route channel emerges as the primary bottleneck for reliable agent navigation under predicted BEV inputs.

Figures & tables

Channel	CVT 6ch.	CVT 6ch. No Noise	CVT 6ch. KDE
Roads	0.8135	0.5086	0.8614
Trajectory	0.5938	0.1918	0.6455
Lane	0.3618	0.0362	0.4170
Vehicles	0.2460	0.0696	0.2401
Pedestrians	0.0000	0.0000	0.0000
Traffic Light	0.0532	0.0031	0.0499

Table 1: Mean per-channel IoU — City 1 (Town01, validation set).

Channel	CVT 6ch.	CVT 6ch. No Noise	CVT 6ch. KDE	CVT 3ch. †	UNet †
Roads	0.7889	0.6810	0.8160	0.9144	0.6978
Trajectory	0.5569	0.4082	0.5755	0.7808	0.5915
Lane	0.3098	0.2485	0.3086	0.3158	0.3959
Vehicles	0.2419	0.1221	0.2252	–	–
Pedestrians	0.0001	0.0000	0.0003	–	–
Traffic Light	0.0240	0.0051	0.0302	–	–

Table 2: Mean per-channel IoU — City 2 (Town02, test set).

Figure 1: Trajectories generated by each evaluated model overlaid on the CARLA top-down maps for (a) Town01 and (b) Town02. Colors denote different BEV generation approaches, as indicated in the legend. Black filled markers represent episode starting locations, while white filled markers indicate successfully completed episodes. Colored arrows mark the vehicle position and heading at the time of an infraction that caused episode termination. The numerical labels identify the corresponding episode start and end points.

Panel of Figure 2 — Figure 2: BEV predictions at the northwest curve of Town01 near the start of episode three, where the road transitions from an east–west straight to a southward turn. predictions at the same timestep. Only the CVT 6ch. and CVT 6ch. KDE agents complete this curve.

	Town01		Town02
BEV Generator	Route (m)	Infrac.	Route (m)	Infrac.
Ground-truth BEV	598	0/5	604	0/5
UNet	115	5/5	22	5/5
CVT 3ch.	159	5/5	27	5/5
CVT 6ch.	151	5/5	52	5/5
CVT 6ch. KDE	261	4/5	106	5/5

Table 3: Route completion (meters) and number of infractions out of 5 episodes for Town01 and Town02 (route length

{\approx}

1,241 m per town).

Panel of Figure 3 — Figure 3: BEV predictions at the cross road in the middle of Town02 near the finish point of episode one. predictions at the same timestep. All agents fail in this intersection.

	Town01				Town02
BEV Generator	Road	Traj.	Lane	Mean	Road	Traj.	Lane	Mean
CVT 3ch.	0.933	0.808	0.551	0.764	0.871	0.727	0.438	0.679
CVT 6ch.	0.935	0.788	0.332	0.685	0.903	0.680	0.321	0.635
CVT 6ch. KDE	0.943	0.789	0.406	0.713	0.902	0.667	0.340	0.636

Table 4: Mean IoU per channel on the spawn evaluation for Town01 and Town02. Channels: Road (ch. 0), Route (ch. 1), Lane boundary (ch. 2).

Panel of Figure 4 — Figure 4: BEV predictions at the cross road in the east of Town02 when CVT 6ch. KDE and CVT 6ch. fail to make a left turn. Predictions at the same timestep.

Bird's-Eye-View Segmentation

See all 11 papers

Variational Inference for Bird's Eye View Segmentation in Autonomous Driving
Jul 16, 2026Jingyue Shi, Huaicheng Li, Junhui Zhao +1Multi-View Learning Bird's-Eye-View Representation
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China · School of Information Science and Engineering, Southeast University, Nanjing 210096, China
CycleBEV: Regularizing View Transformation Networks via View Cycle Consistency for Bird's-Eye-View Semantic Segmentation
Feb 27, 2026Jeongbin Hong, Dooseop Choi, Taeg-Hyun An +2Bird's-Eye-View Segmentation Cycle Consistency
Electronics and Telecommunications Research Institute (ETRI) · University of Science and Technology (UST)
Investigating Behavior Cloning from Few Demonstrations for Autonomous Driving based on Bird's-Eye View in Simulated Cities
Oct 14, 2026Eric Aislan Antonelo, Gustavo Claudio Karl Couto, Christian Möller +1Instance Reweighting Autonomous Driving
Automation and Systems Engineering Department, Federal University of Santa Catarina, Florianópolis, Brazil · Faculty of Science and Engineering, Information Technology Åbo Akademi University, Finland

Jul 16, 2026J/K move · Enter open · S save

Variational Inference for Bird's Eye View Segmentation in Autonomous Driving

Jingyue Shi, Huaicheng Li, Junhui Zhao, Yanxiang Jiang

School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China · School of Information Science and Engineering, Southeast University, Nanjing 210096, China

Multi-View Learning Bird's-Eye-View Representation Bird's-Eye-View Segmentation Variational Inference Semantic Segmentation

Open paper AskPDF arXiv

The bird's eye view (BEV) has emerged as a pivotal approach for environmental perception in autonomous driving, providing a unified spatial representation for vehicles. Nevertheless, despite BEV's significance in addressing the challenges inherent to autonomous driving, effectively fusing data from multiple camera sensors and operating in complex external driving environments remains a considerable challenge. To mitigate this issue, we recast the BEV segmentation problem within a variational inference framework. In this paper, we propose a novel transformer-based variational flow transformation network for BEV segmentation, denoted as TVB. Our architecture implicitly learns the mapping from multiple camera views to a unified canonical BEV map during training by exploiting posterior BEV supervision. TVB employs a conditional variational auto encoder (CVAE) as its backbone and produces multiple BEV map candidates. To augment the realism of the generated BEV maps, we integrate normalizing flows into the map generation process, enabling the construction of more complex and expressive probability distributions. Furthermore, we design a BEV-attention fusion (BAF) module that harnesses attention mechanisms to adaptively integrate the multiple candidate BEV maps. Experimental results, evaluated on both the nuScenes and OPV2Vdatasets, demonstrate that our proposed method achieves superior performance in multi-camera view BEV segmentation and lane environment perception.

Abstract

Bird's-Eye-View Segmentation

Explore similar work

Variational Inference for Bird's Eye View Segmentation in Autonomous Driving

CycleBEV: Regularizing View Transformation Networks via View Cycle Consistency for Bird's-Eye-View Semantic Segmentation

Investigating Behavior Cloning from Few Demonstrations for Autonomous Driving based on Bird's-Eye View in Simulated Cities