AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelines
Organizations: Zhejiang University · University of Macau · Institute of Collaborative Innovation
Abstract
The contemporary paradigm of scaling data annotation, crucial for developing machine learning solutions, is to hire ordinary human annotators and instruct them with expert-crafted guidelines to label data. This paradigm is laborious, tedious, and costly, motivating us to study an open problem, auto-annotation with expert-crafted guidelines (dubbed AutoExpert). We develop benchmarks by redesigning the evaluation protocol and re-annotating data with nuScenes and PandaSet, two 3D detection datasets for autonomous driving research that provide expert-crafted annotation guidelines. Their guidelines define 18 and 25 object classes, respectively, using nuanced language descriptions and a few visual examples. Following the guidelines that require using 3D cuboids to label LiDAR data, AutoExpert requires algorithms to learn on few-shot labeled images and texts to perform the task of 3D detection on LiDAR data. Apparently, the challenges of AutoExpert lie in the data-modality and task discrepancy. Nevertheless, public foundation models (FMs) serve as promising tools to tackle these challenges. To address AutoExpert, we adopt a conceptually simple pipeline consisting of three components: (1) 2D object detection and segmentation in RGB images, (2) lifting 2D detections into 3D using known sensor poses, and (3) 3D cuboids generation for the 2D detections. Within this pipeline, we enhance and evaluate a variety of methods such as open-vocabulary detectors, few-shot detectors, and self-supervised learned detectors. We also develop novel techniques, leading to refined components that boost 3D detection mAP from 12.1 to 25.4 on the AutoExpert-nuScenes benchmark.
Figures & tables
| Method | pub. | mAP 3D | NDS | ATE | ASE | AOE | AVE | AAE |
| Oyster [ 81 ] | CVPR’23 | 6.3 | 10.7 | 0.755 | 0.715 | 1.451 | 1.201 | 0.771 |
| w/ frustum | ours | 12.9 | 15.6 | 0.723 | 0.651 | 1.563 | 1.008 | 0.711 |
| LISO [ 8 ] | ECCV’24 | 8.9 | 13.1 | 0.725 | 0.679 | 1.411 | 1.152 | 0.733 |
| w/ frustum | ours | 15.7 | 19.8 | 0.679 | 0.583 | 1.429 | 0.903 | 0.644 |
| UNION [ 28 ] | NeurIPS’24 | 9.7 | 13.8 | 0.726 | 0.667 | 1.412 | 1.112 | 0.713 |
| w/ frustum | ours | 16.8 | 20.6 | 0.677 | 0.579 | 1.397 | 0.901 | 0.622 |
| name | mAP 2D | mAP 3D | NDS | |
| Detic | o-name | 16.5 | 12.1 | 16.6 |
| GD | o-name | 16.9 | 16.1 | 21.3 |
| ft-GD | o-name | 20.0 | 16.6 | 21.2 |
| GD | r-name | 18.2 | 15.7 | 22.1 |
| ft-GD | r-name | 20.8 | 18.2 | 23.1 |
| Class | 10+C | 6+C | 2+C | C | C+2 | C+6 | C+10 | 1+C+1 | 3+C+3 | 5+C+5 |
| bus | 24.3 | 25.8 | 28.1 | 30.8 | 28.3 | 27.2 | 26.4 | 29.4 | 26.7 | 25.3 |
| bicycle | 22.5 | 25.1 | 28.6 | 30.1 | 32.4 | 29.0 | 26.7 | 30.8 | 29.4 | 28.4 |
| emergency - vehicle | 12.1 | 13.1 | 12.2 | 12.8 | 11.9 | 12.4 | 12.3 | 12.2 | 12.2 | 12.1 |
| adult | 34.4 | 43.7 | 56.6 | 59.3 | 60.2 | 46.8 | 36.1 | 61.2 | 56.5 | 49.3 |
| child | 4.4 | 4.9 | 4.5 | 3.4 | 2.8 | 2.6 | 1.9 | 3.5 | 2.9 | 2.7 |
| construction - worker | 13.6 | 16.3 | 22.9 | 25.6 | 28.6 | 24.3 | 20.5 | 27.9 | 25.1 | 22.3 |
| v-MHT | SA. | track | mAP 3D | NDS | |
| 18.2 | 23.1 | ||||
| ✓ | 21.9 | 25.2 | |||
| ✓ | ✓ | 22.8 | 25.9 | ||
| ✓ | ✓ | ✓ | 23.6 | 26.4 | |
| ✓ | ✓ | ✓ | ✓ | 25.4 | 27.2 |
| original class name | refined class name by GPT-4o | refined class name by Qwen |
| car | car | car |
| truck | truck | truck |
| trailer | trailer, container | trailer, container |
| bus | bus | bus |
| construction - vehicle | construction-vehicle | construction-vehicle |
| bicycle | bicycle bike | bicycle |
| GD r-name by GPT-4o | GD r-name by Qwen | ft-GD r-name by GPT-4o | ft-GD r-name by Qwen | |
| mAP 2D | 18.2 | 18.0 | 20.8 | 20.7 |
| mAP 3D | 15.7 | 15.6 | 18.2 | 18.1 |
| NDS | 22.1 | 21.9 | 23.1 | 23.0 |
| class | Detic (o-name) | GD (o-name) | ft-GD (o-name) | GD (r-name) | ft-GD (r-name) | |||||
| mAP 2D | mAP 3D | mAP 2D | mAP 3D | mAP 2D | mAP 3D | mAP 2D | mAP 3D | mAP 2D | mAP 3D | |
| car | 58.3 | 31.9 | 52.6 | 25.1 | 56.4 | 29.1 | 51.3 | 26.1 | 54.2 | 25.6 |
| truck | 37.2 | 17.6 | 34.3 | 14.2 | 34.4 | 15.0 | 36.3 | 14.1 | 37.5 | 12.6 |
| trailer | 4.2 | 0.8 | 8.5 | 2.0 | 7.0 | 1.3 | 8.1 | 1.8 | 8.5 | 1.7 |
| bus | 59.0 | 6.4 | 59.0 | 5.7 | 60.2 | 8.3 | 59.7 | 5.3 | 59.8 | 6.4 |
| construction - vehicle | 11.1 | 14.7 | 9.9 | 9.7 | 4.5 | 12.0 | 10.2 | 8.9 | 11.0 | 9.5 |
| Method | pub. | mAP 3D | NDS | mATE | mASE | mAOE | mAVE | mAAE |
| CenterPoint [ 75 ] | CVPR’21 | 3.6 | 19.0 | 0.971 | 0.517 | 0.794 | 0.546 | 0.447 |
| auto3D | ours | 25.4 | 27.2 | 0.552 | 0.534 | 0.992 | 0.927 | 0.536 |
| LiDAR Configuration | ||
| parameter/feature | nuScenes [ 10 ] | AV2 [ 70 ] |
| LiDAR Model | Single HDL-32E | Dual VLP-32C |
| Number of channels | 32 | 32 2 |
| Measurement Range | Max 100m, Effective 70m | Max 200m, Effective 100m |
| Vertical FOV | -30.67 ∘ to + 10.67 ∘ | -25 ∘ to + 15 ∘ |
| Horizontal FOV | 360 ∘ | 360 ∘ |
| mAP 3D | NDS | mATE | mASE | mAOE | mAVE | mAAE | |
| MHT | 19.2 | 23.8 | 0.570 | 0.538 | 1.130 | 0.920 | 0.555 |
| v-MHT | 21.9 | 25.2 | 0.560 | 0.535 | 0.994 | 0.933 | 0.549 |
| Rotation Step Size (in radian) | |||||
| Rotation Step | |||||
| mAP 3D | 22.0 | 22.0 | 21.9 | 21.9 | 21.4 |
| NDS | 25.5 | 25.3 | 25.2 | 25.2 | 24.6 |
| Translation Step Size (in meter) | |||||
| Translation Step | 0.3m | 0.5m | 0.8m | 1.0m | 1.5m |
| mAP 3D | 22.1 | 21.9 | 21.3 | 21.0 | 19.1 |
| Method | 2D Det. (sec.) | Segmt. (sec.) | VLM Inf. (sec.) | 3D Gen. (sec.) | Total Time (sec.) |
| CM3D [ 23 ] | 0.08 | 0.01 | N/A | 0.65 | 0.74 |
| MHT (w/o VLM) | 0.08 | 0.01 | N/A | 0.86 | 0.95 |
| v-MHT (Ours) | 0.08 | 0.01 | 0.40 | 0.15 | 0.64 |
| Method | mAR 3D /Occlusion Level | mAP 3D /Distance (m) | ||||||
| 60-100% | 40-60% | 20-40% | 0-20% | 0-10 | 10-20 | 20-30 | 0-50 | |
| CM3D [ 23 ] | 33.5% | 49.4% | 58.1% | 59.9% | 26.2 | 23.9 | 13.9 | 18.2 |
| v-MHT (ours) | 34.1% | 51.5% | 59.6% | 61.9% | 30.2 | 26.2 | 15.6 | 21.9 |
| (+0.6%) | (+2.1%) | (+1.5%) | (+2.0%) | (+4.0) | (+2.3) | (+1.7) | (+3.7) | |
| auto3D | 36.5% | 54.0% | 60.8% | 63.1% | 30.4 | 29.8 | 19.8 | 25.4 |
| (+3.0%) | (+4.6%) | (+2.7%) | (+3.2%) | (+4.2) | (+5.9) | (+5.9) | (+7.2) | |
| Method | car | truck | trailer | bus | CV | bicycle | motorcycle | EV |
| SAM3D [ 77 ] | 6.2 | 5.2 | 0.2 | 2.1 | 0.5 | 3.3 | 4.1 | 0.1 |
| Oyster [ 81 ] | 13.1 | 4.1 | 0.4 | 2.7 | 3.1 | 10.0 | 17.1 | 0.8 |
| w/ frustum | 20.1 | 9.0 | 1.2 | 4.6 | 6.8 | 21.1 | 35.7 | 1.9 |
| LISO [ 8 ] | 19.0 | 7.0 | 0.8 | 6.4 | 4.3 | 13.7 | 23.1 | 2.5 |
| w/ frustum | 25.0 | 12.8 | 1.4 | 11.7 | 7.9 | 25.1 | 42.3 | 4.5 |
| UNION [ 28 ] | 18.8 | 7.2 | 0.6 | 6.6 | 4.8 | 14.2 | 22.8 | 2.4 |
| Method | adult | child | PO | CW | stroller | PM | PP | debris | TC | barrier |
| SAM3D [ 77 ] | 0.9 | 0.1 | 0.2 | 0.2 | 1.5 | 0.9 | 0.1 | 0.1 | 1.6 | 1.5 |
| Oyster [ 81 ] | 19.8 | 1.2 | 0.9 | 8.6 | 6.9 | 3.7 | 0.1 | 0.1 | 16.9 | 3.9 |
| w/ frustum | 42.0 | 2.5 | 1.6 | 18.4 | 15.3 | 6.7 | 0.1 | 0.1 | 37.1 | 8.1 |
| LISO [ 8 ] | 26.7 | 1.6 | 1.0 | 11.7 | 9.8 | 4.0 | 0.1 | 0.1 | 23.6 | 5.2 |
| w/ frustum | 47.2 | 2.9 | 1.8 | 21.4 | 17.9 | 7.3 | 0.1 | 0.1 | 43.3 | 9.6 |
| UNION [ 28 ] | 25.9 | 1.7 | 1.0 | 12.0 | 9.6 | 3.8 | 0.1 | 0.1 | 38.1 | 4.9 |
| center | size | orientation | score | mAP 3D | NDS | mATE | mASE | mAOE | mAVE | mAAE |
| 25.4 | 27.2 | 0.552 | 0.534 | 1.133 | 0.927 | 0.536 | ||||
| 24.5 | 24.7 | 0.601 | 0.589 | 1.142 | 0.976 | 0.585 | ||||
| 25.4 | 28.7 | 0.552 | 0.386 | 1.113 | 0.927 | 0.536 | ||||
| 25.4 | 27.7 | 0.552 | 0.492 | 0.990 | 0.927 | 0.536 | ||||
| 25.4 | 28.0 | 0.552 | 0.460 | 1.111 | 0.927 | 0.536 | ||||
| 23.8 | 23.8 | 0.613 | 0.600 | 1.241 | 0.982 | 0.614 |
| Class | 10+C | 6+C | 2+C | C | C+2 | C+6 | C+10 | 1+C+1 | 3+C+3 | 5+C+5 |
| car | 33.6 | 36.4 | 39.9 | 41.6 | 40.8 | 38.2 | 35.5 | 40.7 | 38.6 | 37.4 |
| truck | 20.7 | 21.6 | 22.7 | 22.7 | 22.8 | 22.3 | 21.4 | 22.9 | 22.7 | 22.2 |
| trailer | 1.7 | 1.7 | 1.8 | 1.7 | 1.7 | 1.7 | 1.7 | 1.8 | 1.7 | 1.7 |
| bus | 24.3 | 25.8 | 28.1 | 30.8 | 28.3 | 27.2 | 26.4 | 29.4 | 26.7 | 25.3 |
| construction - vehicle | 13.5 | 14.3 | 14.6 | 14.9 | 14.9 | 14.4 | 14.3 | 14.8 | 14.8 | 14.6 |
| bicycle | 22.5 | 25.1 | 28.6 | 30.1 | 32.4 | 29.0 | 26.7 | 30.8 | 29.4 | 28.4 |
| mAP 3D | NDS | mATE | mASE | mAOE | mAVE | mAAE | |
| baseline w/o SA | 21.9 | 25.2 | 0.560 | 0.535 | 0.994 | 0.933 | 0.549 |
| per-instance SA | 22.1 | 25.4 | 0.559 | 0.537 | 0.994 | 0.928 | 0.544 |
| per-class SA | 22.8 | 25.9 | 0.556 | 0.536 | 0.993 | 0.919 | 0.538 |
| Class Name | GT Count |
| car | 2,464 |
| pedestrian | 890 |
| pylons | 263 |
| temporary_construction_barriers | 179 |
| pedestrian_with_object | 122 |
| cones | 111 |
| original class name | refined class name by GPT-4o |
| car | car |
| pedestrian | pedestrian |
| pylons | pylons |
| temporary_construction_barriers | plastic_pedestrian_barricade |
| pedestrian_with_object | pedestrian_with_open_umbrella |
| cones | tall_safety_cone |
| mAP 2D | mAP 3D | NDS | |
| GD o-name | 17.3 | 10.1 | 17.7 |
| ft-GD o-name | 25.9 | 17.9 | 26.1 |
| GD r-name | 20.5 | 13.6 | 20.9 |
| ft-GD r-name | 27.8 | 18.4 | 27.6 |
| Method | pub. | mAP 3D | NDS | mATE | mASE | mAOE |
| OpenBox [ 27 ] | NeurIPS’25 | 10.9 | 15.0 | 0.856 | 0.491 | 1.185 |
| w/ ft-GD | ours | 15.9 | 18.9 | 0.820 | 0.461 | 1.076 |
| AnnofreeOD [ 60 ] | ICCV’25 | 11.5 | 15.4 | 0.855 | 0.486 | 1.190 |
| w/ ft-GD | ours | 16.1 | 19.3 | 0.811 | 0.452 | 1.015 |
| CM3D [ 23 ] | CoRL’24 | 12.3 | 16.0 | 0.831 | 0.522 | 1.182 |
| w/ ft-GD | ours | 16.6 | 19.7 | 0.803 | 0.455 | 1.104 |
| Class Name | mAP 3D | NDS | ATE | ASE | AOE |
| Animals-Other | 36.4 | 48.5 | 0.513 | 1.084 | 1.358 |
| Bicycle | 37.4 | 31.3 | 0.539 | 0.296 | 0.743 |
| Bus | 17.1 | 3.6 | 2.011 | 0.257 | 0.552 |
| Car | 40.8 | 35.6 | 0.846 | 0.058 | 0.610 |
| Cones | 41.4 | 34.3 | 0.345 | 0.163 | 0.900 |
| Const_Signs | 16.6 | 7.4 | 0.583 | 0.661 | 0.803 |