Humans combine prediction and perception to observe the world. When faced with rapidly moving birds or insects, we can only perceive them clearly by predicting their next position and focusing our gaze there. Inspired by this, this paper proposes the Prediction-As-Perception (PAP) framework, integrating a prediction-perception architecture into 3D object perception tasks to enhance the model's perceptual accuracy. The PAP framework consists of two main modules: prediction and perception, primarily utilizing continuous frame information as input. Firstly, the prediction module forecasts the potential future positions of ego vehicles and surrounding traffic participants based on the perception results of the current frame. These predicted positions are then passed as queries to the perception module of the subsequent frame. The perceived results are iteratively fed back into the prediction module. We evaluated the PAP structure using the end-to-end model UniAD on the nuScenes dataset. The results demonstrate that the PAP structure improves UniAD's target tracking accuracy by 10% and increases the inference speed by 15%. This indicates that such a biomimetic design significantly enhances the efficiency and accuracy of perception models while reducing computational resource consumption.
Figures & tables
Figure 1: Fig. 1. An Overall architecture of PAP. The PAP framework consists of perception and prediction moules. The input to the perception module is the current frame’s image and queries, which include both randomly generated queries for the current frame and queries generated from the output results of the prediction module from the previous frame. The input to the pre-diction module is the output results of the perception module, and the output is the possible future positions of traffic participants in the current frame. These position coordinates, once converted into queries, are stored in the queries bank for future frame calls.
Figure 2: Fig. 2. Perception module of PAP. The perception module primarily uses queries to detect objects. Let’s assume the perception module has a structure similar to DETR3D[13]. The queries input to the perception module consist of two parts: one part generated from the prediction results of the previous frame, and another part randomly generated for the current frame. When the positions of the queries generated from the previous frame’s prediction results are projected onto the map, it is evident that compared to the randomly generated queries, the former are closer to the locations of traffic participants in the current environment.
Figure 3: Fig. 3. PAP with UniAD[9]. In UniAD, the interaction between modules is inherently based on queries. Therefore, we only need to take the queries output by the Motion Former module in UniAD, embed them to match the dimensions of the Track Queries, and then feed them together with the Track Queries into Track Former for detection.
AMOTA
AMOTP
Recall
IDS
Training time
FPS
UniAD
0.359
1.32
0.467
906
91h
14
UniAD+PAP
0.395
1.22
0.493
826
78h
16
Table 1: Table 1. Performance comparison between UniAD and UniAD+PAP on the nuScenes validation dataset. UniAD+PAP outperforms UniAD in all performance metrics, and it is also more efficient in terms of training and computation.
AMOTA
AMOTP
Recall
IDS
Bicycle
0.372
1.297
0.453
15
Bus
0.465
1.225
0.535
8
Car
0.613
0.744
0.667
405
Motor
0.438
1.253
0.5
24
Pedest
0.411
1.192
0.487
342
Trailer
0.33
1.551
0.201
4
Table 2: Table 2. Performance metrics of UniAD + PAP for each category.
Real-time 3D object detection is a critical component for the safe operation of autonomous driving systems and robotics. While LiDAR point clouds provide accurate spatial information, processing them efficiently remains a significant challenge. Traditional methods rely on complex 3D convolutions or anchor-based paradigms that struggle to balance detection accuracy with inference speed. In this paper, we propose PillarDETR, a novel end-to-end 3D object detection architecture that combines the efficiency of pillar-based LiDAR encoding with the representational power of modern 2D vision models. Specifically, PillarDETR replaces standard convolutional backbones with a Cross Stage Partial (CSP) network derived from YOLOv8, enabling richer feature extraction from pseudoimages. Furthermore, we discard conventional anchor-based or center-based detection heads in favor of a Real-Time Detection Transformer (RT-DETR) decoder. This hybrid design allows the network to capture global context and directly predict 3D bounding boxes without relying on non-maximum suppression (NMS). Extensive experiments on the KITTI and nuScenes benchmarks demonstrate that PillarDETR achieves a compelling trade-off between mean Average Precision (mAP) and inference latency. Our ablation studies confirm that integrating the YOLOv8 backbone and RT-DETR head yields substantial improvements over the PointPillars baseline, establishing PillarDETR as a highly effective solution for real-time 3D perception.
Smit Kadvani, Shriya Gumber, Kriti Faujdar +1
Independent Researcher · University of Southern California · Illinois Institute of Technology
A robust awareness of how dynamic scenes evolve is essential for Autonomous Driving systems, as they must accurately detect, track, and predict the behaviour of surrounding obstacles. Traditional perception pipelines that rely on modular architectures tend to suffer from cumulative errors and latency. Instance Prediction models provide a unified solution, performing Bird's-Eye-View segmentation and motion estimation across current and future frames using information directly obtained from different sensors. However, a key challenge in these models lies in the effective processing of the dense spatial and temporal information inherent in dynamic driving environments. This level of complexity demands architectures capable of capturing fine-grained motion patterns and long-range dependencies without compromising real-time performance. We introduce BEVPredFormer, a novel camera-only architecture for BEV instance prediction that uses attention-based temporal processing to improve temporal and spatial comprehension of the scene and relies on an attention-based 3D projection of the camera information. BEVPredFormer employs a recurrent-free design that incorporates gated transformer layers, divided spatio-temporal attention mechanisms, and multi-scale head tasks. Additionally, we incorporate a difference-guided feature extraction module that enhances temporal representations. Extensive ablation studies validate the effectiveness of each architectural component. When evaluated on the nuScenes dataset, BEVPredFormer was on par or surpassed State-Of-The-Art methods, highlighting its potential for robust and efficient Autonomous Driving perception.
Miguel Antunes-García, Santiago Montiel-Marín, Fabio Sánchez-García +3
RobeSafe Research Group, Department of Electronics, University of Alcalá, Alcalá de Henares, Madrid, Spain.
Three-dimensional feature extraction and multi-task perception are fundamental components of modern autonomous driving systems. Although large image encoders, high-resolution inputs, and long temporal contexts can substantially improve representation quality and overall performance, jointly leveraging these strategies remains challenging due to prohibitive computational costs during both training and inference. Furthermore, different perception tasks often require distinct feature representations, making it difficult for a unified architecture to achieve end-to-end multi-task performance comparable to specialized single-task systems. To address these challenges, we propose HENet++, a unified framework for multi-task 3D perception and end-to-end autonomous driving that employs a hybrid image encoding strategy, using a large encoder for short-term frames and a lightweight encoder for long-term temporal context to strike a favorable balance between accuracy and efficiency. The framework jointly extracts dense background features and sparse foreground features, enabling task-specific representations that reduce cumulative errors and provide richer information for downstream prediction and planning modules. HENet++ is compatible with diverse 3D feature extraction pipelines and supports multi-modal inputs, including camera and radar data. Extensive experiments demonstrate state-of-the-art performance on the nuScenes multi-task 3D perception benchmark, achieving the lowest collision rate on the nuScenes planning benchmark and higher PDMS on the NAVSIM benchmark.
Zhongyu Xia, Zhiwei Lin, Yongtao Wang +1
Wangxuan Institute of Computer Technology, Peking University, Beijing, China. · University of California, Merced, USA.