HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving

Zhongyu Xia; Zhiwei Lin; Yongtao Wang; Ming-Hsuan Yang

cs.CVNov 10, 2025

HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving

Authors: Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Ming-Hsuan Yang

Organizations: Wangxuan Institute of Computer Technology, Peking University, Beijing, China. · University of California, Merced, USA.

Abstract

Three-dimensional feature extraction and multi-task perception are fundamental components of modern autonomous driving systems. Although large image encoders, high-resolution inputs, and long temporal contexts can substantially improve representation quality and overall performance, jointly leveraging these strategies remains challenging due to prohibitive computational costs during both training and inference. Furthermore, different perception tasks often require distinct feature representations, making it difficult for a unified architecture to achieve end-to-end multi-task performance comparable to specialized single-task systems. To address these challenges, we propose HENet++, a unified framework for multi-task 3D perception and end-to-end autonomous driving that employs a hybrid image encoding strategy, using a large encoder for short-term frames and a lightweight encoder for long-term temporal context to strike a favorable balance between accuracy and efficiency. The framework jointly extracts dense background features and sparse foreground features, enabling task-specific representations that reduce cumulative errors and provide richer information for downstream prediction and planning modules. HENet++ is compatible with diverse 3D feature extraction pipelines and supports multi-modal inputs, including camera and radar data. Extensive experiments demonstrate state-of-the-art performance on the nuScenes multi-task 3D perception benchmark, achieving the lowest collision rate on the nuScenes planning benchmark and higher PDMS on the NAVSIM benchmark.

Figures & tables

Figure 1 : HENet++ reduces training cost by enabling the joint use of high-resolution images and long temporal sequences through hybrid encoding. By integrating hybrid encoding, joint sparse and dense feature extraction, and pretraining via model merging, HENet++ achieves state-of-the-art multi-task performance while attaining the lowest collision rate in end-to-end driving on the nuScenes benchmark.

Figure 2 : Overall architecture of HENet++. The proposed framework performs hybrid encoding to jointly learn sparse foreground representations and dense background voxel features, enabling end-to-end multi-task prediction. A model-merging-based pre-training strategy is further introduced to improve multi-task performance.

Figure 3 : Design of Hybrid Image Encoding Network for dense feature extraction. Hybrid Image Encoding Network uses image encoders of varying complexity to encode long-sequence frames and short-term images, respectively. Temporal Integration Module fuses multi-frame features from the various encoders.

Figure 4 : Hybrid image encoding network for sparse feature extraction. The architecture extracts object-centric representations through hybrid encoding of multi-view image features.

Figure 5 : Initialization of HENet++ via model merging.

Figure 6 : End-to-end autonomous driving architecture based on HENet++. An attention-based trajectory planner uses instance features (including the ego vehicle) as queries and combines instance and panoramic dense features as key-value pairs to enable iterative prediction of future states and planning. The hybrid dense-sparse design of HENet++ further allows straightforward extension to multi-modal inputs.

Methods	Backbone	Frames	Time/E	Detection		BEV Segmentation				Occupancy
Methods	Backbone	Frames	Time/E	NDS $\uparrow$	mAP $\uparrow$	$\textbf{mIoU}\uparrow$	$\textbf{IoU}_{\textbf{v}}\uparrow$	$\textbf{mIoU}_{\textbf{a}}\uparrow$	$\textbf{mIoU}_{\textbf{d}}\uparrow$	$\textbf{IoU}\uparrow$	$\textbf{mIoU}\uparrow$
VPN	ResNet50	1	-	33.4	25.7	43.8	37.3	76.0	18.0	-	-
LSS	ResNet50	1	-	41.0	34.4	45.0	42.8	73.9	18.3	-	11.4
BEVFormer-S	ResNet101	1	-	45.3	38.0	47.3	44.4	77.6	19.8	-	-
BEVFormer	ResNet101	5	213min	52.0	41.2	49.4	46.7	77.5	23.9	-	30.5
OccNet	ResNet101	5	230min	52.0	41.2	24.6	12.9	47.2	13.8	41.1	27.0

Table 1 : Comparison of end-to-end multi-task learning on the nuScences val set . Time/E represent the training time per epoch with FP32 on 8

\times

A800 GPUs.

\textbf{mIoU}_{\textbf{v}}

\textbf{mIoU}_{\textbf{a}}

, and

\textbf{mIoU}_{\textbf{d}}

represent the mIoU for vehicles, drivable area, and lane & road divider, respectively.

Method	Input	Backbone	UniAD Metrics									VAD/STP3 Metrics
			L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$				CCR $\downarrow$	L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$
			1s	2s	3s	Avg.	1s	2s	3s	Avg.	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
ST-P3	C	EfficientNet-b4	1.72	3.26	4.86	3.28	0.44	1.08	3.01	1.51	8.37	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
OccNet	C	ResNet101-DCN	1.29	2.13	2.99	2.14	0.21	0.59	1.37	0.72	-	-	-	-	-	-	-	-	-
UniAD	C	ResNet101	0.48	0.96	1.65	1.03	0.05	0.17	0.71	0.31	1.59	0.45	0.70	1.04	0.73	0.62	0.58	0.63	0.61
Ego-MLP	C	ResNet50	0.15	0.32	0.59	0.35	0.00	0.27	0.85	0.37	2.93	-	-	-	-	-	-	-	-

Table 2 : Comparison of End-to-end Autonomous Driving results on nuScenes val set. L2 refers to the L2 error between the predicted trajectory and the ground-truth trajectory of human driving.

Method	Mod.	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
UniAD	C	97.8	91.9	92.9	100.0	78.8	83.4
PARA-Drive	C	97.9	92.4	93.0	99.8	79.3	84.0
iPad	C	98.6	98.3	94.9	100.0	88.0	91.7
HENet++	C	98.9	98.7	94.2	100.0	88.3	92.6

Table 3 : Open-loop results with closed-loop metrics on NAVSIM.

Figure 7 : The baseline trajectory (red) has a lower L2 error with respect to the ground truth (black), but results in a collision. In contrast, HENet++ (green) produces a higher L2 error but successfully avoids collision.

Methods	Backbone	NDS $\uparrow$	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
BEVDet	ResNet50	37.9	29.8	0.725	0.279	0.589	0.860	0.245
BEVDet4D $\dagger$	ResNet50	45.7	32.2	0.703	0.278	0.495	0.354	0.206
PETRv2 $\dagger$	ResNet50	45.6	34.9	0.700	0.275	0.580	0.437	0.187
BEVStereo $\dagger$	ResNet50	50.0	37.2	0.598	0.270	0.438	0.367	0.190
SOLOFusion $\ddagger$	ResNet50	53.4	42.7	0.567	0.274	0.511	0.252	0.181
Sparse4Dv2 $\ddagger$	ResNet50	53.9	43.9	0.598	0.270	0.475	0.282	0.179

Table 4 : Comparison of 3D object detection results on the nuScences val set .

*

indicates the result benefits from the perspective pre-training.

\dagger

indicates using one temporal frame.

\ddagger

denotes integrating two or more temporal frames. The best and second best results are marked in red and blue .

Methods	Backbone	NDS $\uparrow$	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
BEVDet4D	Swin-B	56.9	45.1	0.511	0.241	0.386	0.301	0.121
PolarFormer	V2-99	57.2	49.3	0.556	0.256	0.364	0.439	0.127
PETRv2	V2-99	58.2	49.0	0.561	0.243	0.361	0.343	0.120
HoP-BEVFormer	V2-99	60.3	51.7	0.501	0.245	0.346	0.362	0.105
BEVDepth	ConvNeXt-B	60.9	52.0	0.445	0.243	0.352	0.347	0.127
BEVStereo	V2-99	61.0	52.5	0.431	0.246	0.358	0.357	0.138

Table 5 : Comparison of 3D object detection results on nuScences test set. The best and second best results are marked in red and blue . ‡uses test-time augmentation.

Methods	Backbone	$\textbf{mIoU}\uparrow$	$\textbf{mIoU}_{\textbf{veh}}\uparrow$	$\textbf{mIoU}_{\textbf{area}}\uparrow$	$\textbf{mIoU}_{\textbf{div}}\uparrow$
VPN	ResNet50	42.7	31.8	76.9	19.4
LSS	ResNet50	46.5	41.7	77.7	20.0
BEVFormer	ResNet101-DCN	50.2	44.8	80.1	25.7
FIERY	ResNet-101	-	38.2	-	-
M2BEV	ResNeXt-101	-	-	77.2	40.5
PETRv2	V2-99	60.3	46.3	85.6	49.0

Table 6 : Comparison of BEV semantic segmentation results on nuScences val set. The best and second best results are marked in red and blue . For both the HENet and HENet++ frameworks, the single-task BEV segmentation models are identical.

Method	Backbone	Visible Mask	RayIoU	mIoU $\uparrow$	others	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive. suf.	other flat	sidewalk	terrain	manmade	vegetation
BEVFormer	R-50	✔	32.4	23.7	5.0	38.8	10.0	34.4	41.1	13.2	16.5	18.2	17.8	18.7	27.7	49.0	27.7	29.1	25.4	15.4	14.5
SparseOcc	R-50	✔	36.1	30.9	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
OPUS	R-50	✔	38.4	33.2	10.7	39.8	21.3	39.8	45.3	23.4	21.8	17.8	19.3	27.5	33.2	71.6	37.1	45.1	43.6	33.8	33.2
TPVFormer	R-50	✔	-	34.2	7.7	44.0	17.7	40.9	47.0	15.1	20.5	24.7	24.7	24.3	29.3	79.3	40.7	48.5	49.4	32.6	29.8
ODG	R-50	✔	39.2	35.5	13.7	39.0	23.0	46.8	49.3	25.8	23.6	20.7	18.5	30.0	35.6	76.8	39.3	45.0	46.8	37.5	32.2
OccFormer	R-50	✔	-	37.4	9.2	45.8	18.2	42.8	50.3	24.0	20.8	22.9	21.0	31.9	38.1	80.1	38.2	50.8	54.3	46.4	40.2

Table 7 : Comparison of Occupancy results on the nuScenes-Occ3D benchmark. We present the RayIoU and the mean IoU across categories, as well as the IoUs for different classes. The best and second best results are marked in red and blue .

	Model	Backbone	Input	Frames	NDS	mAP	FPS	GPU memory	Time
A	BEVDepth4D	R18	640 $\times$ 1152	2 frames	48.6	34.8	14.3	31.2G / BS=64	14h
B	BEVDepth4D	R18	256 $\times$ 704	9 frames	48.6	34.9	21.7	22.9G / BS=64	14h
C	BEVDepth4D	R18	640 $\times$ 1152	9 frames	51.2	39.2	14.1	43.5G / BS=64	44h
A+B	Model Ensemble	-	-	-	48.9	35.2	7.93	-	-
A+B	Dense Hybrid Encoding	R18 & R18	640 $\times$ 1152 & 256 $\times$ 704	2 + 7 frames	52.1	39.8	8.91	41.3G / BS=64	13h
D	BEVDepth4D	R50	256 $\times$ 704	9 frames	53.8	40.9	19.1	14.1G / BS=16	16h

Table 8 : Ablation of Hybrid Image Encoding Network on Detection Task. ‘Time’ denotes the overall training time. ‘BS’ indicates the batch size. For A

\sim

F, FPS is measured on RTX3090 GPU with FP32. For H

\sim

I, FPS is measured on A100-SMX4 GPU with FP32 and FP16 mixed precision. For A

\sim

F, Training cost (GPU memory and Time) is estimated on 8

\times

Tesla A800 GPUs with FP32. For H

\sim

I, Training cost is estimated on 8

\times

A100-SMX4 GPUs with FP32 and FP16 mixed precision.

	Model	Backbone	Input	Frames	NDS	$\textbf{mIoU}_{bev}$	$\textbf{mIoU}_{occ}$
A	BEVDepth4D	R50	256 $\times$ 704	9 frames	53.0	51.5	-
B	BEVStereo	V2-99	640 $\times$ 1152	2 frames	58.0	55.8	-
C	BEVStereo	V2-99	896 $\times$ 1600	2 frames	58.9	56.7	-
A+B	Dense Hybrid Encoding w/o AFFM	V2-99 & R50	640 $\times$ 1152 & 256 $\times$ 704	2 + 7 frames	59.0	56.9	-
A+B	Dense Hybrid Encoding	V2-99 & R50	640 $\times$ 1152 & 256 $\times$ 704	2 + 7 frames	59.9	58.0	-
D	HENet++ w/o Hybrid Encoding	R50	256 $\times$ 704	9 frames	54.1	51.9	39.4

Table 9 : Ablation study of the Hybrid Image Encoding Network in the multi-task setting.

\textbf{mIoU}_{bev}

and

\textbf{mIoU}_{occ}

denote the mIoU for BEV semantic segmentation and semantic occupancy, respectively.

Figure 8 : Visualization results of Hybrid Encoding and baseline methods. From left to right: multi-view image inputs, BEVDepth4D, BEVStereo, Hybrid Encoding results, and ground truth. Hybrid Encoding improves occluded object estimation by leveraging long-term temporal information and yields more accurate predictions using high-resolution features.

Load from	NDS	$\textbf{mIoU}_{bev}$	$\textbf{mIoU}_{occ}$
Detection	53.6 $\pm$ 0.2	54.3 $\pm$ 0.5	36.2 $\pm$ 0.3
Segmentation	52.8 $\pm$ 0.4	55.0 $\pm$ 0.3	34.9 $\pm$ 0.6
Occupancy	53.2 $\pm$ 0.2	54.9 $\pm$ 0.5	37.3 $\pm$ 0.2
Model Merge	53.8 $\pm$ 0.3	55.3 $\pm$ 0.3	37.6 $\pm$ 0.2

Table 10 : Ablation of Pretrain Method of HENet++. This experiment is conducted on a triple-task R50 small model, repeated five times with different random seeds, and reports the mean and standard deviation. Compared to using single-task detection or occupancy weights, loading the merged model for pre-training improves multi-task performance.

$\mathcal{L}_{plan}$	$\mathcal{L}_{pred}$	$\mathcal{L}_{col}$	UniAD Metrics		VAD/STP3 Metrics
$\mathcal{L}_{plan}$	$\mathcal{L}_{pred}$	$\mathcal{L}_{col}$	mean L2	mean Col.	mean L2	mean Col.
✓			1.69	0.52	0.71	0.17
✓	✓		1.36	0.20	0.58	0.10
✓	✓	✓	1.29	0.13	0.55	0.05

Table 11 : Ablation on the Loss for End-to-End Autonomous Driving.

Figure 9 : Ablation study on the hyperparameter $\alpha$ for model merging.

Explore similar work

Jul 27, 2026cs.CV

MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks. However, such a single feature map hardly carries sufficient information to simultaneously meet the requirements of various perception tasks, leading to a very limited perception performance. To mitigate this limitation, this paper proposes MATS, a novel multi-modality multi-task learning approach with modality-adaptive BEV fusion and task-specific Mixture-of-Experts (MoE) for 3D perception. Specifically, a simple modality-adaptive BEV fusion module is designed to adaptively recalibrate the BEV features by modeling the global cross-modality dependencies, generating diverse BEV feature maps for various perception tasks. For joint multi-task learning, this paper proposes a task-specific MoE module to decouple the tasks and enable the network to automatically choose the appropriate BEV feature candidates for each specific task. To validate the effectiveness of the proposed approach, we conduct extensive experiments on the large-scale benchmark nuScenes. With the camera- and LiDAR-modality input data, the proposed approach outperforms the state-of-the-art (SOTA) by a significant margin. Furthermore, the experimental results on the single tasks show that the proposed approach significantly outperforms the baselines. The code and trained models will be available upon publication.

Junchen Huo, Wanming Hao, Song Wang +3

School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China · Tianping College of Suzhou University of Science and Technology, Suzhou 215009, China · department of computer science, Toronto Metropolitan University, Toronto M5B2K3, Canada

Mixture of Experts Camera-LiDAR Fusion Autonomous Driving Perception Multi-Task Learning Bird's-Eye-View Representation

Ask PDF

Aug 13, 2026cs.CV

Geometry-Grounded Unified 3D Perception for Autonomous Driving

Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.

Longfei Xu, Xiaohui Wang, Zehao Huang +4

School of Computer Science and Engineering Beihang University Beijing, China · School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · School of Artificial Intelligence Beihang University Beijing, China

Depth Estimation Autonomous Driving Perception 3D Occupancy Prediction 3D Object Detection 3D Representation Learning 3D Scene Understanding

Ask PDF

Jun 2, 2026cs.CV

Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

We present a novel compact deep multi-task learning model to handle various autonomous driving perception tasks in one forward pass. The model performs multiple views of semantic segmentation, depth estimation, light detection and ranging (LiDAR) segmentation, and bird's eye view projection simultaneously without being supported by other models. We also provide an adaptive loss weighting algorithm to tackle the imbalanced learning issue that occurred due to plenty of given tasks. Through data pre-processing and intermediate sensor fusion techniques, the model can process and combine multiple input modalities retrieved from RGB cameras, dynamic vision sensors (DVS), and LiDAR placed at several positions on the ego vehicle. Therefore, a better understanding of a dynamically changing environment can be achieved. Based on the ablation study, the model variant trained with our proposed method achieves a better performance. Furthermore, a comparative study is also conducted to clarify its performance and effectiveness against the combination of some recent models. As a result, our model maintains better performance even with much fewer parameters. Hence, the model can inference faster with less GPU memory utilization. Moreover, the result tends to be consistent in 3 different CARLA simulation datasets and 1 real-world nuScenes-lidarseg dataset. To support future research, we share codes and other files publicly at https://github.com/oskarnatan/compact-perception.

Oskar Natan, Jun Miura

Department of Computer Science and Engineering, Toyohashi University of Technology, Aichi 441-8580, Japan · Department of Computer Science and Electronics, Gadjah Mada University, Yogyakarta 55281, Indonesia

Depth Estimation Autonomous Driving Perception Multi-Task Learning 3D Semantic Segmentation Adaptive Loss Weighting Multimodal Sensor Fusion

Ask PDF