VRU detection in low-light conditions is a key component in ensuring VRU safety during the most collision prone time of the day. Currently, advances in low-light detection are limited due to the lack of large scale datasets and automated labeling methods. Most autolabeling methods use RGB detections as their labels source, which cannot be replicated in low-light conditions. To fill this gap in this paper we present an automated pipeline for low-light VRU labeling. We use LWIR-RGB image pairs and source our labels from LWIR as VRUs differentiate from the background due to their body temperature. Unlike our previous work that relied on a complex hardware setup to align the two modalities, we perform this experiment using a LWIR-RGB stereo setup, lowering hardware complexity and cost, making the pipeline easier to reproduce. We evaluate method end-to-end on a dataset collected in the capital region of Finland. Models trained on our generated labels beat both off-the-shelf baselines and models fine tuned on a hand labeled subset on F1, mAP@50 and LAMR metrics when tested on a manually labeled dataset, and the result holds for two different LWIR detectors. This shows that low-light VRU autolabeling can be made both accessible and effective without specialized alignment hardware.
Figures & tables
Figure 1: The proposed LWIR-RGB autolabeling pipeline.
Figure 2: LWIR-RGB stereo camera setup on the research vehicle.
Figure 3: LWIR and RGB chessboard images captured during calibration, with the chessboard being illuminated with a flood light. Top row corresponds to original images, and bottom row shows images undistorted with the calibrated parameters.
Figure 4: Visualizations of corresponding RGB and LWIR images, with the rectified images overlaid on top of each other to show the quality of the calibration.
Figure 5: Training data samples collected from various locations during data collection.
Base Model
Training Data
P
R
mAP@50
mAP@50-95
RF-DETR
FLIR-ADAS
0.682
0.5653
0.801
0.452
YOLOv11l
FLIR-ADAS
0.827
0.750
0.827
0.501
SAM3
None
0.560
0.315
0.339
0.110
Table I : Evaluation of the LWIR detection models on the FLIR-ADAS dataset. Bolded values indicate the best result in each column.
LWIR-YOLO labels
LWIR-DETR labels
ht
Frames
YOLO
DETR
Frames
YOLO
DETR
0.10
6164
0.208
0.161
5163
0.165
0.161
0.08
9489
0.226
0.187
8123
0.195
0.190
0.06
16455
0.252
0.210
13914
0.238
0.207
0.04
27930
0.243
0.225
24617
0.250
0.228
0.02
51483
0.282
0.219
53684
0.252
0.215
Table II : RGB detector model accuracies reported as mAP 50 on the test sequence, with different height thresholds used in the autolabel filtering. Confidence threshold selected as ct=0.25 . Best result per column in bold, second best underlined. Number of frames corresponds to the amount of image samples remaining in the training data after filtering.
Training set
Test
Labels
ct
ht
Images
Inst.
P
R
F1
mAP@50
mAP@50-95
LAMR ↓
Base Model YOLOv11l
COCO (zero-shot)
–
–
–
–
0.596
0.154
0.244
0.199
0.118
0.815
Manual (val seq.)
–
–
441
363
0.342
0.162
0.220
0.188
0.097
0.832
LWIR-YOLO
0.50
0.04
29513
75161
0.584
0.228
0.328
0.231
0.080
0.785
LWIR-YOLO
0.50
0.02
47914
174370
0.625
0.217
0.323
0.239
0.085
0.778
Table III : Low-light RGB detection on the test sequence (1440 images, 1173 annotated instances). Training-set size is reported per row, varying based on the filtering parameters. Green marks results better than the 441-frame manual reference within the same base model, red worse; color is applied to F1, mAP and LAMR only, since P and R are not informative as standalone values. P, R, F1, and LAMR are computed with an IoU-threshold of 0.5. LAMR is log-average miss rate over FPPI [10−2,100] , lower is better.
Training set
Test
Labels
ct
ht
Images
Inst.
P
R
F1
mAP@50
mAP@50-95
LAMR ↓
YOLOv11l
COCO (zero-shot)
–
–
–
–
0.596
0.154
0.244
0.199
0.118
0.815
Manual (val seq.)
–
–
441
363
0.342
0.162
0.220
0.188
0.097
0.832
LWIR-YOLO
0.25
0.02
51483
216466
0.613
0.208
0.311
0.282
0.124
0.750
LWIR-DETR
0.25
0.02
53684
212098
0.522
0.213
0.303
0.252
0.089
0.769
Table IV : Low-light RGB detection on the test sequence (1440 images, 1173 annotated instances). All autolabeled models use the same filtration setting, ct=0.25 and ht=0.02 . Resulting training set size is reported per row. Green marks results better than the 441-frame manual reference with the same base model, red worse; color is applied to F1, mAP and LAMR only, since P and R are not informative as standalone values. P, R, F1, and LAMR are computed with an IoU-threshold of 0.5. LAMR is log-average miss rate over FPPI [10−2,100] , lower is better.
Figure 6: Precision–Recall and log-average miss rate (MR) curves for YOLOv11l (top) and RF-DETR-Base (bottom).
Figure 7: Qualitative detection results on test set for YOLOv11l across four scenes (columns) for ground truth and selected models (rows). The LWIR-YOLO and LWIR-DETR labels were generated with ct=0.25 and ht=0.02 .
Pedestrian detection in RGB images is a key task in pedestrian safety, as the most common sensor in autonomous vehicles and advanced driver assistance systems is the RGB camera. Low-light pedestrian detection lacks large public datasets and autolabelling pipelines. This research proposes a solution in the form of an automated infrared-RGB pipeline. The pipeline consists of 1) Infrared detection, where a fine-tuned model for infrared pedestrian detection is used 2) Label transfer process from the infrared detections to their RGB counterparts 3) Training object detection models using the generated labels for low-light RGB pedestrian detection. The research was performed using the KAIST dataset. For evaluation, three object detection models, DETR, YOLO, and RCNN, were trained on generated and ground truth labels. When compared on previously unseen images, the results showed that the models trained on generated labels out-performed the ones trained on ground-truth in 5 out of 6 cases for the mAP@50 and LAMR metrics, and outperformed ground-truth on mAP@50-95 in all cases. Acquired results indicate that the proposed auto-labelling pipeline could be used for scalable annotation of low-light datasets for pedestrian detection. The source code for this research is available on GitHub: https://github.com/BouzoulasDimitrios/IR-RGB-autoamed-low-light-pedestrian-labelling
Dimitrios Bouzoulas, Eerik Alamikkotervo, Risto Ojala
Energy and Mechanical Engineering, Aalto University, Espoo, Finland
Reliable 3D perception of vulnerable road users (VRUs) such as cyclists and pedestrians is essential for their safety in urban traffic and a core requirement for autonomous driving (AD). Alongside advances in vehicle-based perception, research increasingly equips bicycles with sensors to study traffic from a perspective native to VRUs. Such platforms still rely on LiDAR detectors originally trained on vehicle data, yet annotated 3D data from a cyclist's perspective is scarce. How well these detectors generalise to this setting has not been evaluated. We present a 3D object detection benchmark of 1,027 annotated LiDAR keyframes (over 18,000 3D bounding boxes) from the FUSE-Bike platform in urban Munich. We evaluate four nuScenes-pre-trained detectors against 1,854 human-verified ground-truth (GT) boxes both in their original form and after finetuning on training labels produced by a VRU-dedicated auto-labelling pipeline that requires no manual annotation. The zero-shot domain gap is concentrated on the VRU classes. Finetuning recovers most of it, improving mean average precision (mAP) by up to 23.4 points with the largest gains on pedestrians and cyclists, and the adapted detectors even surpass the quality of the auto-labels they were trained on. The benchmark provides a reproducible baseline for VRU-centric 3D detection and shows that auto-labels are a viable substitute for manual annotation when adapting vehicle-trained detectors to a cyclist platform.
Mario Finkbeiner, Max A. Buettner, Kanak Mazumder +1
Intelligent Vehicles Lab (IVL), Munich University of Applied Sciences, Munich, Germany
Detecting small unmanned aerial vehicles from RGB-infrared remote-sensing pairs remains challenging due to tiny target scale, cluttered backgrounds, and spatial misalignment between heterogeneous sensors. Existing bimodal detectors often align or fuse features without assessing the reliability of local cross-sensor correspondence, allowing mismatch artifacts to propagate into the detection head. To address this issue, we propose LER-YOLO, a reliability-aware sparse mixture-of-experts framework for misaligned RGB-infrared UAV detection. LER-YOLO first introduces an Uncertainty-Aware Target Alignment module that resamples visible features toward the infrared reference and estimates a spatial reliability map. This reliability prior is then used by a Reliability-Guided Sparse MoE Fusion module to adaptively select k experts from RGB-dominant, infrared-dominant, and interactive fusion experts, enabling trustworthy cross-modal interaction while suppressing unreliable fusion. Experiments on the public MBU benchmark under a YOLOv5s-family protocol show that LER-YOLO achieves 89.7+/-0.2% AP50 over three independent seeds, with a best result of 89.9%. Extensive ablations, parameter-matched comparisons, synthetic-shift evaluations, and complexity analysis demonstrate that the gains mainly come from reliability-guided expert routing rather than increased model capacity.
Liming Hou, Yueping Peng, Hexiang Hao +6
Engineering University of PAP, Xi’an 710086, China · Unit Command Department, Officers College of PAP, Chengdu 610213, China