LangDepth: Vision-Language Scene Priors for Semantic Binning and Cross-Domain Monocular Depth Estimation
Organizations: School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China
Abstract
Monocular depth estimation has advanced through large-scale pretraining and strong visual backbones, yet metric depth remains sensitive to scene-level scale. Indoor rooms and outdoor driving scenes occupy very different depth ranges, but most depth decoders infer this global structure implicitly from visual features. We ask whether scene-level semantic structure can instead act as an upstream prior that shapes the metric output space before dense prediction. We present LangDepth (SemanticBins), a monocular depth architecture that couples strong visual features with a scene-conditioned output space. A frozen CLIP encoder and a learnable bank of scene- and geometry-oriented concept keys convert image features into a compact scene descriptor, without requiring per-image text or an external captioner. A Scene Prior Module (SPM) maps this descriptor to a scene-conditioned metric depth interval, providing a global range anchor before dense decoding. In parallel, a DINOv2-L backbone with a lightweight FPN extracts dense visual features, and a Per-Pixel Assignment Network (PPAN) assigns pixels to the scene-prescribed bin structure. A confidence gate fuses the dense visual estimate with the SPM prior. We train LangDepth jointly on NYU Depth V2 and KITTI under a modest supervised depth budget, without web-scale depth supervision or pseudo-labelled depth data. A single model achieves delta_1 = 0.960 on NYU, 0.960 on KITTI, 0.887 on DIODE indoor, 0.946 on iBims-1, and 0.861 on SUN RGB-D. Beyond aggregate accuracy, the SPM predicts compact indoor ranges and much wider outdoor KITTI ranges, the gate becomes SPM-dominant on outdoor scenes, and perturbing the routed scene context causes large drops on KITTI and iBims-1. These results support scene-conditioned output spaces as a practical and interpretable mechanism for cross-domain monocular depth estimation.
Figures & tables
| NYU Depth V2 | KITTI Eigen | DIODE indoor | iBims-1 | SUN RGB-D | ||||||||||||||||
| Method | AbsRel | RMSE | AbsRel | RMSE | AbsRel | RMSE | AbsRel | RMSE | AbsRel | RMSE | ||||||||||
| AdaBins [ 2 ] | 0.103 | 0.364 | 0.903 | 0.984 | 0.058 | 2.360 | 0.964 | 0.995 | 0.186 | 0.768 | 0.743 | 0.916 | 0.125 | 0.655 | 0.835 | 0.968 | 0.159 | 0.476 | 0.771 | 0.944 |
| LocalBins [ 3 ] | 0.087 | 0.362 | 0.936 | 0.990 | — | — | — | — | — | — | — | — | 0.210 | 0.880 | 0.560 | 0.880 | 0.156 | 0.470 | 0.777 | 0.949 |
| DPT [ 18 ] | 0.110 | 0.357 | 0.904 | 0.988 | 0.062 | 2.575 | 0.959 | 0.995 | 0.132 | 0.613 | 0.853 | 0.945 | 0.098 | 0.537 | 0.888 | 0.986 | 0.358 | 1.230 | 0.852 | 0.925 |
| DepthFormer [ 11 ] | 0.103 | 0.390 | 0.902 | 0.983 | 0.220 | 8.771 | 0.608 | 0.852 | 0.161 | 0.695 | 0.784 | 0.937 | 0.114 | 0.596 | 0.862 | 0.972 | 0.143 | 0.431 | 0.790 | 0.964 |
| NeWCRFs [ 24 ] | 0.095 | 0.334 | 0.922 | 0.992 | 0.052 | 2.071 | 0.975 | 0.997 | 0.160 | 0.703 | 0.805 | 0.931 | 0.112 | 0.599 | 0.864 | 0.976 | 0.367 | 1.320 | 0.836 | 0.922 |
| Method | NYU AbsRel | NYU | KITTI AbsRel | KITTI | DIODE | iBims | SUN |
|---|---|---|---|---|---|---|---|
| Ours | 0.0727 | 0.9600 | 0.0540 | 0.9601 | 0.9030 | 0.9572 | 0.8663 |
| w/o gate fusion | 0.0725 | 0.9600 | 0.0539 | 0.9607 | 0.9028 | 0.9556 | 0.8664 |
| w/o concept routing | 0.0726 | 0.9600 | 0.0562 | 0.9554 | 0.8966 | 0.9543 | 0.8653 |
| w/o learned SPM | 0.0729 | 0.9597 | 0.0521 | 0.9653 | 0.8997 | 0.9565 | 0.8650 |
| SPM range only | 0.0729 | 0.9597 | 0.0543 | 0.9598 | 0.8991 | 0.9573 | 0.8661 |
| Dataset | (m) | (m) | Width (m) |
|---|---|---|---|
| NYU | |||
| DIODE | |||
| iBims | |||
| SUN | |||
| KITTI |
| Dataset | mean( ) | median( ) | std( ) | ||
|---|---|---|---|---|---|
| NYU | 0.539 | 0.532 | 0.058 | 8.8% | 0.6% |
| KITTI | 0.359 | 0.318 | 0.129 | 90.1% | 1.5% |
| DIODE | 0.514 | 0.517 | 0.039 | 23.3% | 0.01% |
| iBims | 0.528 | 0.527 | 0.037 | 14.7% | 0.001% |
| SUN | 0.510 | 0.504 | 0.091 | 46.6% | 1.3% |
| Perturbation | DIODE | iBims | SUN | KITTI | ||||
|---|---|---|---|---|---|---|---|---|
| Original | 0.9030 | — | 0.9572 | — | 0.8663 | — | 0.9601 | — |
| Zero context | 0.7333 | 0.5900 | 0.8933 | 0.4685 | ||||
| Wrong domain | 0.8711 | 0.8452 | 0.8518 | 0.8747 | ||||
| Shuffled | 0.8940 | 0.9448 | 0.8640 | 0.8833 | ||||
| Batch mean | 0.8957 | 0.9497 | 0.8667 | 0.8614 | ||||
| Gaussian | 0.8965 | 0.9508 | 0.8664 | 0.8611 |
Depth Estimation
Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.
See all 207 papers