Real-Time Diffusion Relighting as a Neural Sensor: Closing the Low-Light Domain Gap for Robotic Perception
Organizations: University of Maryland, College Park · Dolby Laboratories
Abstract
Low-light conditions severely degrade robotic perception: object detectors trained on daylight imagery are prone to failure under dark and adverse weather conditions that fall outside of their training distribution. Furthermore, from an efficiency standpoint, existing Low-Light Image Enhancement (LLIE) methods are designed for offline processing and are therefore impractical to integrate into live control loops. To address these issues, we propose to embed a Generative Neural Sensor within the control loop that treats illumination as a generative prior rather than a signal processing problem, enabling task-relevant semantic recovery. We further propose an Real-Time Perceptual Architecture that decouples high-latency semantic recovery from low-latency robotic control. Rather than blocking the control loop, our system runs a large-scale image-to-image diffusion model in a background thread, and is activated only when a drop in detection confidence signals a perceptual uncertainty event, significantly reducing computational overhead. Our method sets a new state-of-the-art performance on the ExDark low-light benchmark with 87.1% mAP, and under adverse weather conditions, significantly boosts detection rates over competing methods. On a navigation task in a closed-loop autonomous driving simulator (CARLA), we achieve a 100% lap completion rate, establishing a simulation-based proof-of-concept for integrating high-latency generative models with standard robotic control autonomy stacks.
Figures & tables
| Methods | Bicycle | Boat | Bottle | Bus | Car | Cat | Chair | Cup | Dog | Motor | People | Table | Mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RetinexNet [ 36 ] | 0.790 | 0.741 | 0.743 | 0.908 | 0.820 | 0.665 | 0.651 | 0.750 | 0.721 | 0.703 | 0.784 | 0.556 | 0.736 |
| KinD [ 39 ] | 0.800 | 0.721 | 0.788 | 0.919 | 0.822 | 0.718 | 0.672 | 0.771 | 0.775 | 0.736 | 0.803 | 0.555 | 0.757 |
| Zero-DCE [ 6 ] | 0.806 | 0.750 | 0.762 | 0.914 | 0.837 | 0.681 | 0.677 | 0.769 | 0.788 | 0.728 | 0.801 | 0.535 | 0.754 |
| Retinexformer [ 40 ] | 0.809 | 0.769 | 0.753 | 0.914 | 0.814 | 0.688 | 0.689 | 0.763 | 0.766 | 0.769 | 0.805 | 0.543 | 0.757 |
| MambaIR [ 41 ] | 0.803 | 0.763 | 0.752 | 0.903 | 0.830 | 0.687 | 0.684 | 0.761 | 0.721 | 0.738 | 0.813 | 0.556 | 0.751 |
| MambaSFLNet [ 3 ] | 0.821 | 0.775 | 0.773 | 0.891 | 0.824 | 0.665 | 0.696 | 0.820 | 0.779 | 0.783 | 0.809 | 0.558 | 0.766 |
| Cond. | Met. | Baseline | CLAHE [ 35 ] | MambaSFLNet [ 3 ] | Ours |
|---|---|---|---|---|---|
| Fog | F1 | 0.681 | 0.781 | 0.579 | 0.855 |
| mAP | 0.544 | 0.635 | 0.452 | 0.807 | |
| Rain | F1 | 0.904 | 0.916 | 0.872 | 0.871 |
| mAP | 0.816 | 0.816 | 0.813 | 0.812 | |
| Ovrcst | F1 | 0.942 | 0.938 | 0.933 | 0.866 |
| mAP | 0.907 | 0.906 | 0.906 | 0.812 |
| Controller | Collisions (%) | CTE 1 (m) | Avg Latency |
|---|---|---|---|
| MPC (Relight) | 62.0 ms | ||
| MPC_DLC (Relight) | 62.0 ms |
| Method | Collisions | Rate (%) | Latency |
|---|---|---|---|
| Daytime (Oracle) | 1.7 ms | ||
| Sunset (No Fix) | 1.7 ms | ||
| Sunset + CLAHE | 2.0 ms | ||
| Sunset + Gen. (Ours) | 62.0 ms | ||
| Dark (No Fix) | 1.7 ms | ||
| Dark + CLAHE | 2.0 ms |
| Mode | Collisions | Activated (%) | Latency | GPU Util. |
|---|---|---|---|---|
| Continuous | 62.0 ms | |||
| Event-Driven | 0.4% | 2.0 ms | 0.4% |
Robotic Perception
Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.
See all 337 papers