eSharp: Diffusion-based Sharp Monocular Metric Event-to-Depth Estimation
Organizations: Rice University · Indian Institute of Science
Abstract
Depth perception is fundamental for autonomous navigation, industrial inspection, and mixed-reality systems. It often operates under strict constraints on power and form factor. In such cases, monocular depth estimation (MDE) is often used. Even as recent advances continue to improve monocular depth quality, conventional frame-based MDE systems remain challenged by fast motion and high-dynamic-range scenes. This work introduces eSharp, an MDE framework that addresses these problems using an event camera. eSharp repurposes pretrained diffusion priors for event-only MDE and produces accurate metric depth with unparalleled sharpness on real data. Real-world event datasets provide only sparse LiDAR depth maps for ground-truth supervision, making dense prediction challenging. We address this limitation through cross-modal distillation, using LiDAR-calibrated dense pseudo-labels generated from a spatially co-registered frame camera to train eSharp. Experiments show that eSharp outperforms prior event-to-depth baselines in both metric-depth accuracy and spatial fidelity, establishing diffusion models as a promising foundation for event-driven 3D perception.
Figures & tables
| Error Metrics | Accuracy Metrics | ||||||||
| Dataset | Method | Abs Rel | Sq Rel | RMSE | RMSE log | SILog | |||
| DSEC | E2Depth [ 15 ] | 0.344 | 0.253 | 13.467 | 0.376 | 0.098 | 0.447 | 0.755 | 0.915 |
| E2Depth ∗ [ 15 ] | 0.253 | 0.130 | 10.119 | 0.315 | 0.107 | 0.574 | 0.861 | 0.956 | |
| EReFormer [ 28 ] | 0.387 | 0.401 | 13.954 | 0.395 | 0.124 | 0.486 | 0.776 | 0.892 | |
| EReFormer ∗ [ 28 ] | 0.286 | 0.208 | 11.369 | 0.325 | 0.109 | 0.569 | 0.839 | 0.944 | |
| DepthAnyEvent [ 1 ] | 0.277 | 0.170 | 11.117 | 0.292 | 0.051 | 0.585 | 0.860 | 0.955 | |
| Error Metrics | Accuracy Metrics | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Training Strategy | Abs Rel | Sq Rel | RMSE | RMSE log | SILog | |||
| DSEC | latent optimization | 0.342 | 0.189 | 6.749 | 0.325 | 0.039 | 0.392 | 0.859 | 0.972 |
| [latent + metric] optimization | 0.239 | 0.146 | 6.591 | 0.285 | 0.074 | 0.644 | 0.890 | 0.960 | |
| latent [latent + metric] optimization | 0.149 | 0.055 | 5.268 | 0.203 | 0.037 | 0.794 | 0.949 | 0.987 | |
| MVSEC | latent optimization | 0.449 | 1.167 | 7.899 | 0.447 | 0.166 | 0.486 | 0.742 | 0.869 |
| [latent + metric] optimization | 0.503 | 2.035 | 9.766 | 0.478 | 0.191 | 0.508 | 0.743 | 0.858 | |
| Warping method | Image-to-depth method | MAE | RMSE | AbsRel | |||
|---|---|---|---|---|---|---|---|
| Camera-matrix homography | DA2/LiDAR | 3.033 | 5.683 | 0.120 | 0.870 | 0.963 | 0.986 |
| 10 m plane | DA2/LiDAR | 3.057 | 5.721 | 0.120 | 0.868 | 0.963 | 0.986 |
| DA3-LiDAR warp | DA3/LiDAR | 2.941 | 5.642 | 0.135 | 0.861 | 0.956 | 0.982 |
| DA3-metric warp | DA3-metric estimation | 7.669 | 9.639 | 0.359 | 0.244 | 0.917 | 0.977 |
| DA2-LiDAR warp | DA2/LiDAR | 2.983 | 5.572 | 0.117 | 0.873 | 0.966 | 0.987 |
| Method | Abs Rel | Sq Rel | RMSE | |||
|---|---|---|---|---|---|---|
| Raw prediction | 0.306 | 0.225 | 9.641 | 0.463 | 0.727 | 0.853 |
| Depth affine LS | 0.407 | 0.784 | 7.196 | 0.452 | 0.762 | 0.877 |
| Inverse-depth affine LS | 0.302 | 0.328 | 8.517 | 0.498 | 0.777 | 0.891 |
| Log-depth affine LS | 0.315 | 0.458 | 7.521 | 0.529 | 0.779 | 0.902 |
| Quantile matching | 0.268 | 0.591 | 6.499 | 0.652 | 0.859 | 0.935 |
| Method | Stage | Trainable modules | Trainable params. | Frozen params. |
|---|---|---|---|---|
| eSharp | latent optimization | U-Net | 865.9M | 424.0M |
| [latent + metric] optimization | U-Net + calibration head | 865.9M + 1.9K | 424.0M | |
| eSharp-fast | latent optimization | U-Net | 867.6M | 424.0M |
| [latent + metric] optimization | U-Net + calibration head | 867.6M + 1.9K | 424.0M |
| Error Metrics | Accuracy Metrics | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Training Strategy | Abs Rel | Sq Rel | RMSE | RMSE log | SILog | |||
| DSEC | latent optimization | 0.176 | 0.058 | 5.792 | 0.232 | 0.045 | 0.720 | 0.956 | 0.988 |
| [latent + metric] optimization | 0.271 | 0.108 | 6.858 | 0.371 | 0.068 | 0.391 | 0.749 | 0.927 | |
| latent [latent + metric] optimization | 0.146 | 0.055 | 5.046 | 0.196 | 0.034 | 0.810 | 0.953 | 0.987 | |
| MVSEC | latent optimization | 0.399 | 0.824 | 7.693 | 0.457 | 0.191 | 0.392 | 0.714 | 0.872 |
| [latent + metric] optimization | 0.480 | 1.073 | 7.519 | 0.470 | 0.185 | 0.409 | 0.698 | 0.848 | |
| Dataset | Resolution | Method | Latency ( ) | FPS | VRAM (GB) |
|---|---|---|---|---|---|
| DSEC | Depth AnyEvent [ 1 ] | 0.012 | 83 | 0.29 | |
| eSharp | 0.47 | 2.1 | 5.6 | ||
| eSharp-fast | 0.073 | 14 | 2.3 | ||
| MVSEC | Depth AnyEvent [ 1 ] | 0.0097 | 100 | 0.15 | |
| eSharp | 0.36 | 2.8 | 5.2 | ||
| eSharp-fast | 0.049 | 20.4 | 2.0 |
| Error Metrics | Accuracy Metrics | |||||||||
| Method | Abs Rel | Sq Rel | RMSE | RMSE log | SILog | |||||
| eSharp | 0.0 | 1.0 | 0.262 | 0.132 | 5.973 | 0.266 | 0.035 | 0.598 | 0.912 | 0.976 |
| 0.3 | 0.7 | 0.161 | 0.066 | 5.341 | 0.212 | 0.039 | 0.782 | 0.944 | 0.984 | |
| 0.5 | 0.5 | 0.166 | 0.072 | 5.390 | 0.214 | 0.038 | 0.777 | 0.940 | 0.983 | |
| 0.7 | 0.3 | 0.149 | 0.055 | 5.268 | 0.203 | 0.037 | 0.794 | 0.949 | 0.987 | |
| eSharp-fast | 0.0 | 1.0 | 0.146 | 0.055 | 5.031 | 0.196 | 0.034 | 0.809 | 0.953 | 0.987 |
| Figure | Dataset and sample identifier |
|---|---|
| Fig. 1 | dsec_interlaken_00_b_rgb_000950 |
| dsec_interlaken_01_a_rgb_000090 | |
| dsec_zurich_city_09_d_0000000094 | |
| Fig. 2 | dsec_interlaken_00_f_0000000020 |
| Fig. 3 | dsec_pipeline_interlaken_00_f_0000000720 |
| Fig. 4 | dsec_zurich_city_05_b_0000000043 |
Depth Estimation
Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.
See all 207 papers