A robust awareness of how dynamic scenes evolve is essential for Autonomous Driving systems, as they must accurately detect, track, and predict the behaviour of surrounding obstacles. Traditional perception pipelines that rely on modular architectures tend to suffer from cumulative errors and latency. Instance Prediction models provide a unified solution, performing Bird's-Eye-View segmentation and motion estimation across current and future frames using information directly obtained from different sensors. However, a key challenge in these models lies in the effective processing of the dense spatial and temporal information inherent in dynamic driving environments. This level of complexity demands architectures capable of capturing fine-grained motion patterns and long-range dependencies without compromising real-time performance. We introduce BEVPredFormer, a novel camera-only architecture for BEV instance prediction that uses attention-based temporal processing to improve temporal and spatial comprehension of the scene and relies on an attention-based 3D projection of the camera information. BEVPredFormer employs a recurrent-free design that incorporates gated transformer layers, divided spatio-temporal attention mechanisms, and multi-scale head tasks. Additionally, we incorporate a difference-guided feature extraction module that enhances temporal representations. Extensive ablation studies validate the effectiveness of each architectural component. When evaluated on the nuScenes dataset, BEVPredFormer was on par or surpassed State-Of-The-Art methods, highlighting its potential for robust and efficient Autonomous Driving perception.
Figures & tables
Figure 1: BEVPredFormer proposed architecture
Figure 2: Difference module architecture present in the temporal block of BEVPredFormer
Figure 3: Temporal attention architecture . a) Overall diagram of spatio-temporal processing. b) PredFormer attention types
Architecture
Code
Encoder
View Transf.
Long range
Short range
VPQ ↑
IoU ↑
VPQ ↑
IoU ↑
PowerBEV Li et al. (2023a) a
✓
EfficientNet
Lift-Splat
32.2
38.9
55.5
62.5
Fiery Hu et al. (2021)
✓
EfficientNet
Lift-Splat
29.9
36.7
50.2
59.4
StrechBEV Akan and Güney (2022)
✓
EfficientNet
Lift-Splat
29.0
37.1
46.0
55.5
BEVerse Zhang et al. (2022) a
✓
Swin
Lift-Splat
33.3
38.7
52.2
60.3
ST-P3 Hu et al. (2022)
✓
EfficientNet
Lift-Splat
32.0
38.9
-
-
Table 1: Comparative model performance on nuScenes validation set
Figure 4: BEVPredFormer results with short range. Different highlighted areas represent the same object in the camera and BEV
Figure 5: BEVPredFormer results with long range. Different highlighted areas represent the same object in the camera and BEV
Block type
Long range
Short range
VPQ ↑
IoU ↑
VPQ ↑
IoU ↑
-
32.5
40.5
54.8
63.4
Binary TS
32.9
40.7
54.9
63.8
Triplet TST a
33.0
40.6
54.4
63.8
Triplet TST
33.3
40.9
54.9
63.9
Quad. TSST
33.3
40.8
55.0
64.0
Table 2: Temporal module block type comparative performance 0 0 footnotetext: All experiments with patch size 4 and 2 blocks.
N block
VPQ ↑
IoU ↑
Flops (G)
Par (M)
1
32.7
40.5
248
6.7
2
33.3
40.9
295
9.9
4
33.1
40.9
390
16.2
6
33.2
41.1
485
22.5
Table 3: Long range performance comparison of the number of temporal processing blocks 0 0 footnotetext: All experiments use Triplet TST block with patch size 4.
Resolution
VPQ ↑
IoU ↑
Latency (ms) ↓
224x480
31.0
38.8
170
448x800
33.3
40.9
220
640x1600
33.3
40.9
360
Table 4: Input image size performance impact 0 0 footnotetext: Results for long-range evaluation using Triplet TST temporal model.
Bird's-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception, effectively mitigating the error propagation inherent in traditional modular pipelines. However, current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow, effectively treating scene agents as generic moving obstacles. This absence of explicit semantic awareness imposes limitations on the capacity of the model to solve ambiguities in complex scenarios, particularly those where object-specific behavior is essential for accurate forecasting (e.g. overtaking, intersections). In this paper, we introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop. The proposed teacher-student pipeline employs Vision-Language Foundation Models to generate dense, semantic-enhanced BEV maps from multi-camera images. These maps serve as auxiliary supervision during training, guiding the network to learn spatio-temporal representations that are not only geometrically consistent but also semantically discriminative. To the best of our knowledge, this represents the first attempt to unify semantic guidance with the temporal task of future instance prediction. The experimental results demonstrate that TGRIP surpasses existing state-of-the-art models in nuScenes, validating the hypothesis that semantic enrichment is a fundamental element for robust, end-to-end motion prediction. Code is available on https://github.com/miguelag99/TGRIP.
Miguel Antunes-García, Santiago Montiel-Marín, Fabio Sánchez-García +3
Electronics Department, University of Alcalá (UAH), Alcalá de Henares, Spain
End-to-end driving requires planning-relevant bird's-eye-view (BEV) representations, but existing pretraining approaches often rely on task annotations or dense scene reconstruction. We introduce PAVER, Planning-Aligned BEV Encoder Pretraining. From a single LiDAR sweep, PAVER constructs sparse risk and unknown targets describing occupied and unobserved evidence along rule-based ego motions. A 10K-parameter head predicts these targets from masked BEV features conditioned on the action state, directing supervision toward geometric constraints on candidate motions. Pretraining requires no driving-task annotations or dense reconstruction. Only the BEV encoder is transferred, preserving the downstream architecture and camera-only inference. On nuScenes, PAVER reduces VAD-Tiny's average collision rate from 0.51% to 0.19%, while improving planning L2, motion prediction, detection, and mapping. The selected VAD-Tiny and VAD-Base schedules use about 36% less estimated total training time than scratch training, including pretraining. On Bench2Drive Town05 Long, PAVER improves UniAD-Tiny's closed-loop Driving Score from 48.45 to 58.79. The project page is available at https://archiiive99.github.io/PAVER.
The bird's eye view (BEV) has emerged as a pivotal approach for environmental perception in autonomous driving, providing a unified spatial representation for vehicles. Nevertheless, despite BEV's significance in addressing the challenges inherent to autonomous driving, effectively fusing data from multiple camera sensors and operating in complex external driving environments remains a considerable challenge. To mitigate this issue, we recast the BEV segmentation problem within a variational inference framework. In this paper, we propose a novel transformer-based variational flow transformation network for BEV segmentation, denoted as TVB. Our architecture implicitly learns the mapping from multiple camera views to a unified canonical BEV map during training by exploiting posterior BEV supervision. TVB employs a conditional variational auto encoder (CVAE) as its backbone and produces multiple BEV map candidates. To augment the realism of the generated BEV maps, we integrate normalizing flows into the map generation process, enabling the construction of more complex and expressive probability distributions. Furthermore, we design a BEV-attention fusion (BAF) module that harnesses attention mechanisms to adaptively integrate the multiple candidate BEV maps. Experimental results, evaluated on both the nuScenes and OPV2Vdatasets, demonstrate that our proposed method achieves superior performance in multi-camera view BEV segmentation and lane environment perception.
Jingyue Shi, Huaicheng Li, Junhui Zhao +1
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China · School of Information Science and Engineering, Southeast University, Nanjing 210096, China