AURORA-KITTI: Any-Weather Depth Completion and Denoising
Authors: Yiting Wang, Tim Brödermann, Hamed Haghighi, Haonan Zhao, Christos Sakaridis, Kurt Debattista, Valentina Donzella
Organizations: Warwick Manufacturing Group, University of Warwick, Coventry, CV4 7AL, United Kingdom · School of Engineering and Materials Science, Queen Mary University of London, London, E1 4NS, United Kingdom · Computer Vision Laboratory, ETH Zürich, Zurich, 8057, Switzerland · Photogrammetry and Remote Sensing, ETH Zürich, Zurich, 8093, Switzerland
Robust depth completion is fundamental to real-world 3D scene understanding, yet existing RGB--LiDAR fusion methods degrade significantly under adverse weather, where both camera images and LiDAR measurements suffer from weather-induced corruption. In this paper, we introduce AURORA-KITTI, the first large-scale, physics-consistent synthetic RGB-LiDAR benchmark for any-weather depth completion and denoising. We further formulate Depth Completion and Denoising~(DCD) as a unified task that reconstructs clean dense metric depth from weather-corrupted RGB images and sparse LiDAR inputs while suppressing weather-induced noise. AURORA-KITTI contains over 82K weather-consistent RGB--LiDAR pairs with metric depth supervision, spanning diverse weather types, three severity levels, day/night scenes, paired clean references, lens occlusion conditions, and textual descriptions.Moreover, we provide DDCD as a strong and efficient distillation-based reference baseline. DDCD is trained on AURORA-KITTI using a frozen teacher applied to paired clean RGB images only during training; at inference, it uses weather-corrupted RGB and sparse LiDAR. Experiments on AURORA-KITTI and the real-world DENSE benchmark show that existing depth completion and foundation depth models remain vulnerable under severe weather, while DDCD improves over weather-aware retraining. Controlled experiments further show that physics-consistent adverse-weather data is the dominant driver of robustness, with clean-prior distillation providing complementary gains. Comparisons with off-the-shelf zero-shot methods use different training protocols; same-data retraining and component ablations separately quantify the contributions of weather-aware data and DDCD. Github: https://github.com/Warwick-Jocelyn/AURORA-KITTI.
Depth completion from sparse LiDAR measurements and corresponding RGB images is a prerequisite for accurate 3D perception in robotic systems. Existing methods achieve high accuracy on standard benchmarks but rely on heavy backbone architectures that preclude real-time deployment on embedded hardware. We present EfficientPENet, a two-branch depth completion network that replaces the conventional ResNet encoder with a modernized ConvNeXt backbone, introduces sparsity-invariant convolutions for the depth stream, and refines predictions through a Convolutional Spatial Propagation Network (CSPN). The RGB branch leverages ImageNet-pretrained ConvNeXt blocks with Layer Normalization, 7x7 depthwise convolutions, and stochastic depth regularization. Features from both branches are merged via late fusion and decoded through a multi-scale deep supervision strategy. We further introduce a position-aware test-time augmentation scheme that corrects coordinate tensors during horizontal flipping, yielding consistent error reduction at inference. On the KITTI depth completion benchmark, EfficientPENet achieves an RMSE of 631.94 mm with 36.24M parameters and a latency of 20.51 ms, operating at 48.76 FPS. This represents a 3.7 times reduction in parameters and a 23 times speedup relative to BP-Net, while maintaining competitive accuracy. These results establish EfficientPENet as a practical solution for real-time depth completion on resource-constrained edge platforms such as the NVIDIA Jetson.
Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi +4
Canizaro Livingston Gulf States Center for Environmental Informatics, the University of New Orleans, New Orleans, USA · US Army Corps of Engineers, Engineer Research and Development Center, Vicksburg, Mississippi, USA
Robust 3D object detection under adverse weather remains a critical hurdle for autonomous driving. Despite progress with LiDAR-4D radar fusion, most methods are constrained by a closed-world assumption, implicitly requiring training and test weather to align in both type and severity. This premise fails in practice: the open-ended nature of weather, and even variations within a single type like rain, cause dramatically different LiDAR degradation patterns, leading to significant performance drops in unseen conditions. To address this, we present Dual-Critic Guided Diffusion Alignment (DCDA), a weather-agnostic framework that learns to recover degraded LiDAR features toward a clean manifold. Rather than modeling specific weather types, DCDA employs a 4D radar-conditioned diffusion process to progressively refine features, guided by two complementary critics. (i) A detection-guided critic, anchored by a pre-trained clean-weather model, ensures that the refined features retain object-level discriminability and localization accuracy. (ii) A weather adversarial critic enforces holistic distributional consistency with clean-weather representations. By aligning features through semantic and distributional constraints rather than explicit weather modeling, DCDA generalizes effectively to unseen weather types and severities without requiring paired data or weather labels. We further introduce a structured open-weather benchmark with held-out type-severity combinations and extensive experiments verify DCDA's advantages.
Shuyao Li, Chuanxing Geng, Heyang Sun +2
Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
We introduce Marigold-SSD, a single-step, late-fusion depth completion framework that leverages strong diffusion priors while eliminating the costly test-time optimization typically associated with diffusion-based methods. By shifting computational burden from inference to finetuning, our approach enables efficient and robust 3D perception under real-world latency constraints. Marigold-SSD achieves significantly faster inference with a training cost of only 4.5 GPU days. We evaluate our method across four indoor and two outdoor benchmarks, demonstrating strong cross-domain generalization and zero-shot performance compared to existing depth completion approaches. Our approach significantly narrows the efficiency gap between diffusion-based and discriminative models. Finally, we challenge common evaluation protocols by analyzing performance under varying input sparsity levels. Page: https://dtu-pas.github.io/marigold-ssd/
Jakub Gregorek, Paraskevas Pegios, Nando Metzger +3
DTU - Technical University of Denmark · Pioneer Centre for AI · ETH Zürich