Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Rui Ding; Meng Yang; Nanning Zheng

cs.CVMar 8, 2026

Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Authors: Rui Ding, Meng Yang, Nanning Zheng

Organizations: Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, P.R.China.

Abstract

Monocular 3D object detection is a promising yet ill-posed task for autonomous vehicles due to the lack of accurate depth information. Cross-modality knowledge distillation could effectively transfer depth information from LiDAR to image-based network. However, modality gap between image and LiDAR seriously limits its accuracy. In this paper, we systematically investigate the negative transfer problem induced by modality gap in cross-modality distillation for the first time, including not only the architecture inconsistency issue but more importantly the feature overfitting issue. We propose a selective learning approach named MonoSTL to overcome these issues, which encourages positive transfer of depth information from LiDAR while alleviates the negative transfer on image-based network. On the one hand, we utilize similar architectures to ensure spatial alignment of features between image-based and LiDAR-based networks. On the other hand, we develop two novel distillation modules, namely Depth-Aware Selective Feature Distillation (DASFD) and Depth-Aware Selective Relation Distillation (DASRD), which selectively learn positive features and relationships of objects by integrating depth uncertainty into feature and relation distillations, respectively. Our approach can be seamlessly integrated into various CNN-based and DETR-based models, where we take three recent models on KITTI and a recent model on NuScenes for validation. Extensive experiments show that our approach considerably improves the accuracy of the base models and thereby achieves the best accuracy compared with all recently released SOTA models.

Figures & tables

Fig. 1: Negative transfer problem in cross-modality distillation. (a) A visual example of positive/negative transfer. (b) The accuracy of monocular 3D detection is seriously decreased when fully transferring the features to student from teacher with a similar architecture based on MonoDLE* 1 1 1 MonoDLE* refers to the base model in Monodistill [ 19 ] , which is developed on the basis of MonoDLE [ 9 ] . . Our approach uses selective transfer learning to alleviate the negative transfer problem and considerably improves the accuracy of four base models on (c) KITTI and (d) NuScenes datasets.

Fig. 2: Overview of our MonoSTL framework . The framework comprises three components including the teacher network, the student network, and three distillation modules. First, we train the teacher network with GT depth map from LiDAR. It adopts similar architectures to the student network. Second, three distillation modules are used to selectively transfer features from the teacher network to the student network including our DASRD and DASFD modules as well as the general response distillation module. Finally, only the student network is retained to predict 3D objects from single images in the inference stage.

Method	$AP_{3D}$ (Car test)			$AP_{3D}$ (Car val)			$AP_{BEV}$ (Car val)
Method	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
MonoDLE * [ 9 ]	17.23	12.26	10.29	19.29	15.13	12.78	26.47	20.24	18.29
MonoDLE* + Ours	24.04	16.50	13.85	25.59	19.23	16.60	34.20	25.82	22.47
Improvement	+6.81	+4.24	+3.56	+6.3	+4.1	+3.82	+7.73	+5.58	+4.18
MonoCon [ 10 ]	22.50	16.46	13.95	26.33	19.01	15.98	34.65	25.39	21.93
MonoCon + Ours	23.47	17.07	14.20	27.13	20.57	17.40	36.31	26.60	22.87

TABLE I: Comparison to the three base models on KITTI . Our approach consistently outperforms the three base models across all difficulty levels in both the training and validation sets.

Method	Reference	$AP_{3D}$ (Car test)			$AP_{BEV}$ (Car test)			$AP_{3D}$ (Car val)
Method	Reference	Mod.	Easy	Hard	Mod.	Easy	Hard	Mod.	Easy	Hard
CMAN [ 53 ]	TITS2022	11.87	17.77	9.16	17.04	25.89	12.88	16.75	24.91	12.46
MonoDLE [ 9 ]	CVPR2021	12.26	17.23	10.29	18.89	24.79	16.00	13.66	17.45	11.68
MonoRUn [ 54 ]	CVPR2021	12.30	19.65	10.58	17.34	27.94	15.24	14.65	20.02	12.61
MonoRCNN [ 55 ]	ICCV2021	12.65	18.36	10.03	18.11	25.48	14.10	14.87	19.07	12.59
DDMP-3D [ 56 ]	CVPR2021	12.78	19.71	9.80	17.89	28.08	13.44	-	-	-

TABLE II: Comparison to recent SOTA models. The best results are in bold . Our approach consistently achieves the best accuracy on both the KITTI validation and test splits.

Method	$AP_{3D}$ (Car test)			$AP_{3D}$ ( Car val)
Method	Easy	Mod.	Hard	Easy	Mod.	Hard
MonoDLE*	17.23	12.26	10.29	19.29	15.13	12.78
MonoDistill	22.97	16.03	13.60	24.31	18.47	15.76
MonoDLE*+ Ours	24.04	16.50	13.85	25.59	19.23	16.60
vs Monodistill	+1.07	+0.47	+0.25	+1.28	+0.76	+0.84

TABLE III: Comparison to Monodistill with the same base model on KITTI. The results show that our approach more effectively alleviates feature overfitting issue in cross-modality distillation. The gain of our approach is mainly attributed to the developed DASFD and DASRD modules.

Method	$AP_{3D}$ (Pedestrian)			$AP_{3D}$ (Cyclist)
Method	Easy	Mod.	Hard	Easy	Mod.	Hard
MonoEF	4.27	2.79	2.21	1.80	0.92	0.71
D4LCN	4.55	3.42	2.83	2.45	1.67	1.36
M3D-RPN	4.92	3.48	2.94	0.94	0.65	0.47
DDMP-3D	4.93	3.55	3.01	4.18	2.50	2.32
AutoShape	5.46	3.74	3.03	5.99	3.06	2.70

TABLE IV: Comparison for pedestrian and cyclist categories on the KITTI test set with IOU thresholds of 0.5. Our approach achieves the best accuracy across all difficulty levels on pedestrian and competitive accuracy on cyclist.

#	General		Ours			$AP_{3D}$ (Car val)
#	FD	RD	DASFD	DASRD	ED	Easy	Mod.	Hard
a.						19.29	15.13	12.78
b.					$\checkmark$	21.63	17.24	14.71
c.	$\checkmark$	$\checkmark$			$\checkmark$	24.31	18.47	15.76
d.	$\checkmark$			$\checkmark$	$\checkmark$	24.78	18.57	16.50
e.		$\checkmark$	$\checkmark$		$\checkmark$	25.37	19.06	16.33

TABLE V: Ablation studies of distillation modules on KITTI validation. FD, RD, and ED refer to general feature, relation, and response distillation. DASFD and DASRD refer to our distillation and relation distillation.

Method	DASFD	DASRD	MAP↑	NDS↑
FCOS3D*			0.321	0.395
FCOS3D*+DASFD	✓		0.331	0.400
FCOS3D*+DASRD		✓	0.328	0.396
FCOS3D*+ Ours	✓	✓	0.335	0.400

TABLE VI: Ablation studies of distillation modules on NuScenes validation. FCOS3D* refers to the results of its open-source model. Notably, all methods don’t utilize test-time augmentation (TTA) and model ensemble.

Method	$AP_{3D}$ (Car val)
Method	Easy	Mod.	Hard
w/ depth error	24.27	18.36	15.59
w/ depth uncertainty	25.59	19.23	16.60

TABLE VII: Ablation study on depth uncertainty. Employing depth uncertainty consistently achieves better results across all difficult levels compared with depth error.

	Scheme	$AP_{3D}$ (Car val)
	Scheme	Easy	Mod.	Hard
separately	Student	25.59	19.23	16.60
separately	Teacher	24.65	19.14	16.45
fusion	sum	24.04	18.12	15.47
fusion	multiply	24.37	18.42	16.41

TABLE VIII: Ablation studies on weight schemes of depth uncertainty. Both the two fusion strategies yield even worse results than a separate weight scheme.

Method	$\lambda_{3}$	$AP_{3D}$ (Car val)
Method	$\lambda_{3}$	Easy	Mod.	Hard
MonoDLE*	0.1	21.68	17.04	14.55
	1	25.59	19.23	16.60
	10	22.48	16.65	14.31
MonoDETR	0.1	32.24	22.06	18.50
	1	29.59	21.58	18.06

TABLE IX: Ablation Studies of the loss weight

\lambda_{3}

. The loss weights

\lambda_{1}

and

\lambda_{2}

have been set to be 10 and 1 following Monodistil [ 19 ] .

Method	network	$AP_{3D}$ (Car val)
Method	network	Easy	Mod.	Hard
Monocon	Teacher	68.13	49.72	41.48
	Base model	26.33	19.01	15.98
	Ours	27.13	20.57	17.40
MonoDETR	Teacher	40.07	27.68	22.49
	Base model	28.84	20.61	16.38

TABLE X: Effect of the teacher network. Our approach is not highly influenced by the accuracy of the teacher network.

Fig. 3: Visual results in BEV view . The first row in each example displays 3D boxes detected by our approach on images. The second row compares our approach with the base model MonoDLE* in BEV view. The third row compares our approach with the recent Monodistill using the same base model in BEV view. The improvement of our approach is mainly attributed to the developed DASFD and DASRD modules.

Fig. 4: Visual results in lateral view . The first and second col in each example displays 3D boxes detected by our approach and Monodistill, respectively. Our approach performs better than Monodistill in cases of false positives and false negatives.

Fig. 5: Failure examples. A few objects are missed or detected inaccurately by our approach. The failure case is mainly caused by the inaccurate depth estimation of object centers.

Fig. 6: t-SNE visualization of network features on both train and validation sets. Nearby objects, i.e. objects with close GT labels, are distinguished more easily in our approach than the base model and monodistill, such as blue (20-21m) and pink (21-22m). It indicates that the feature overfitting issue in our approach is well alleviated than others.

Monocular 3D Object Detection

Monocular 3D object detection estimates objects’ positions, dimensions, and orientations in 3D from one camera, without depth sensors.

See all 22 papers

RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection
Mar 8, 2026Rui Ding, Zhaonian Kuang, Zongwei Zhou +3Multi-View 3D Object Detection 3D Object Detection
Xi’an Jiaotong University, Xi’an, China · Horizon Robotics, Beijing, China · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1
LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection
Dec 5, 2025Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7Efficient Object Detection Real-Time Object Detection
DeepScenario · ETH Zurich · TU Munich +5
Towards robust multimodal 3D object detection via visual foundation models
Sep 20, 2026Ziying Song, Lin Liu, Hongyu Pan +4Multimodal Robustness Vision Foundation Models
School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · Horizon Robotics, Beijing 100000, China · University of Macau, Macao 999078, China +1

Mar 8, 2026J/K move · Enter open · S save

RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection

Rui Ding, Zhaonian Kuang, Zongwei Zhou, Meng Yang+2

Xi’an Jiaotong University, Xi’an, China · Horizon Robotics, Beijing, China · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China+1

Multi-View 3D Object Detection 3D Object Detection Cross-Modal Knowledge Distillation Depth Estimation

Open paper AskPDF arXiv

Multi-view 3D detection with bird's eye view (BEV) is crucial for autonomous driving and robotics, but its robustness in real-world is limited as it struggles to predict accurate depth values. A mainstream solution, cross-modal distillation, transfers depth information from LiDAR to camera models but also unintentionally transfers depth-irrelevant information (e.g. LiDAR density). To mitigate this issue, we propose RayD3D, which transfers crucial depth knowledge along the ray: a line projecting from the camera to true location of an object. It is based on the fundamental imaging principle that predicted location of this object can only vary along this ray, which is finally determined by predicted depth value. Therefore, distilling along the ray enables more effective depth information transfer. More specifically, we design two ray-based distillation modules. Ray-based Contrastive Distillation (RCD) incorporates contrastive learning into distillation by sampling along the ray to learn how LiDAR accurately locates objects. Ray-based Weighted Distillation (RWD) adaptively adjusts distillation weight based on the ray to minimize the interference of depth-irrelevant information in LiDAR. For validation, we widely apply RayD3D into three representative types of BEV-based models, including BEVDet, BEVDepth4D, and BEVFormer. Our method is trained on clean NuScenes, and tested on both clean NuScenes and RoboBEV with a variety types of data corruptions. Our method significantly improves the robustness of all the three base models in all scenarios without increasing inference costs, and achieves the best when compared to recently released multi-view and distillation models.

Abstract

Monocular 3D Object Detection

Explore similar work

RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection

LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection

Towards robust multimodal 3D object detection via visual foundation models