Depth Any Seen: Which Surfaces and How Far?

Xiaohao Xu; Xiaonan Huang

cs.CVSep 25, 2026

Depth Any Seen: Which Surfaces and How Far?

Authors: Xiaohao Xu, Xiaonan Huang

Organizations: Robotics Department, University of Michigan, Ann Arbor

Abstract

When several surfaces are visible along a ray, recovering visible 3D structure from one image requires jointly estimating their presence and metric depth. Depth Any Seen represents these surfaces as image-conditioned multi-Bernoulli depth sets, whose components each contribute one depth or remain absent. Its auxiliary-free Exact Multi-Bernoulli objective (ExactMB) learns depth and presence by marginalizing one-to-one assignments to complete, distinct targets. Our analysis shows that matching expected count can leave component-surface assignment unresolved. We extend real and synthetic layered-depth benchmarks to evaluate depth accuracy, recovered support, and overprediction. Compared to depth stacking, ExactMB reduces overprediction by a relative 88.2% on LD-Real and 80.5% on MD-3K while retaining most ordinal accuracy, with comparable conditional metric-depth error on LD-Syn. Further ablation studies show that ordered assignment improves depth-accurate recall and precision over marginalization, whereas the count-regularized configuration achieves higher deeper-rank precision than ordered assignment at lower recall. Our code will be publicly released.

Figures & tables

Figure 1: Recovering visible 3D geometry requires two answers: which surfaces are present, and how far away are they? (a) A single RGB image: purple circles and amber diamonds mark opaque and through-glass rays. (b) Depth Anything V1/V2 ( Yang et al., 2024a ; Yang et al., 2024b ) compress layered visibility into one depth per ray. (c) SeeGroup’s released evaluation decoder ( Wen and Deng, 2026 ) retains up to four ranked relative depths per ray. (d) Depth Any Seen predicts ray-adaptive metric depth sets with explicit absence for unused components. Gray denotes no retained depth.

Figure 2: One likelihood learns depth and absence. (a) Our default feed-forward model uses recurrent decoding to explicitly predict depth, scale, and existence probability. Selection and sorting form the output set. (b) ExactMB sums over one-to-one assignments to explain observed returns and score unused components as absent. Ordered assignment retains optional emission. (c) Annotated visibility determines the target return count. Scale describes localization conditional on presence.

			LD-Real		MD-3K			LD-Syn
	Emission	Assignment	Ordinal ACC $\uparrow$	OverPred $\downarrow$	Recall $\uparrow$	OverPred $\downarrow$	Pattern ACC $\uparrow$	AbsRel $\downarrow$	Count ACC $\uparrow$	MAE $\downarrow$
(i)	All	Ordered	$67.4_{\pm 0.7}$	$98.4_{\pm 0.3}$	$100.0_{\pm 0.0}$	$98.5_{\pm 0.2}$	$0.03_{\pm 0.04}$	$18.3_{\pm 0.1}$	$3.7_{\pm 0.2}$	$2.244_{\pm 0.031}$
(ii)	Count reg.	Ordered	$58.9_{\pm 0.9}$	$7.4_{\pm 0.6}$	$79.1_{\pm 1.0}$	$8.6_{\pm 0.5}$	$65.9_{\pm 1.2}$	$17.2_{\pm 0.1}$	$95.0_{\pm 0.0}$	$0.067_{\pm 0.001}$
(iii)	Presence	MAP	$60.7_{\pm 1.2}$	$12.1_{\pm 1.1}$	$81.7_{\pm 0.7}$	$20.9_{\pm 1.4}$	$47.7_{\pm 2.7}$	$18.7_{\pm 0.3}$	$88.4_{\pm 0.2}$	$0.183_{\pm 0.002}$
(iv)	Presence	Marginalized	$61.7_{\pm 1.8}$	$11.6_{\pm 1.3}$	$81.5_{\pm 2.9}$	$19.2_{\pm 3.8}$	$46.3_{\pm 4.0}$	$19.2_{\pm 0.2}$	$88.2_{\pm 0.5}$	$0.194_{\pm 0.007}$
(v)	Presence	Ordered	$63.2_{\pm 1.9}$	$10.1_{\pm 1.2}$	$82.6_{\pm 3.9}$	$21.7_{\pm 1.2}$	$52.4_{\pm 3.1}$	$17.6_{\pm 0.2}$	$89.7_{\pm 0.1}$	$0.154_{\pm 0.002}$

Table 1: Primary cardinality and assignment ablations. Separately trained variants (means

\pm

sample SD). All denotes depth stacking and Marginalized denotes ExactMB. Count reg.: explicit count regularization. Ordered: compacted annotation-channel assignment. Rows (iii–v) differ only in recorded assignment per seed. Section 4 defines ACC/OverPred. AbsRel (%) averages conditional errors over four ranks. MAE is in layers. Markers match Figure 3 (a).

		LD-Real		MD-3K			LD-Syn
Released model	Scale	Ordinal ACC $\uparrow$	OverPred $\downarrow$	Recall $\uparrow$	OverPred $\downarrow$	Pattern ACC $\uparrow$	GT-cal. AbsRel ${}^{\dagger}\!\downarrow$	Count ACC $\uparrow$	MAE $\downarrow$
SeeGroup ( Wen and Deng, 2026 )	Relative	74.2	100.0	100.0	100.0	0.0	15.8	2.4	2.756
WT r69l ( Zhang et al., 2026 )	Relative	58.2	100.0	100.0	100.0	0.0	24.5	2.4	2.789
LaRI scenes ( Li et al., 2026 )	Relative	52.0	100.0	100.0	100.0	0.0	29.5	2.4	2.789
LaRI objects ( Li et al., 2026 )	Relative	12.9	50.2	79.1	80.5	38.1	37.7	33.1	0.750
WT r75b ( Zhang et al., 2026 )	Metric	17.0	100.0	100.0	100.0	0.0	46.1	2.4	2.789

Table 2: Released-model transfer references. Single estimates from unretrained models for different tasks. Scale denotes native output.

\dagger

AbsRel (%) averages four conditional rank errors after per-image/layer GT least-squares calibration and clipping; primary errors are unaligned.

Figure 3: Count and presence metrics favor different modeling choices. (a) Primary-study LD-Syn Count ACC by true count. (b) Separate-recipe paired effects. Positive values indicate improvement. ExactMB denotes marginalization. pp denotes percentage points. Means $\pm$ sample SD.

Figure 4: Evaluating depth quality requires both localization and recovered support. The same five LD-Syn ablations, means $\pm$ sample SD. (a) AbsRel on jointly valid target–prediction pixels. (b) Target-support recall. (c) Depth-accurate recall. (d) Depth-accurate precision. Section 4 defines these metrics. Decoded ranks 1–4 are paired with supplied target channels L1/L3/L5/L7. Principal outputs are unaligned and unclipped. Full rank-wise scores and denominators are in Appendix C.2 .

Figure 5: Correct ordering does not guarantee correct visible-surface membership. All 3,161 MD-3K pairs; principal operating point. Joint ACC requires correct presence and ordering against the annotations. Bars/whiskers: four-seed means/sample SD; circles: per-seed Joint ACC.

Figure 6: Visible depth layers with spatially varying support. Selected auxiliary-regularized ExactMB predictions. Columns: RGB, count, front-to-back depth ranks 1–4. (a–h) Real predictions. (i) Synthetic GT/prediction on a shared metric scale, without alignment. Gray: absent or invalid depth. The supplementary video demo provides additional qualitative cases and clearer views.

Figure 7: Layered 3D reconstruction for relighting. (a) Textured and depth-colored unprojections (purple near, yellow far, per-scene scales). (b) Inserted-light re-rendering and detail crops.

Figure 8: Local route screening with reconstructed surfaces. (a) Monocular input. (b) Layered reconstructions. (c) MicroDuck routes ( Pollen Robotics, 2026 ) clear or intersect selected first-layer facets (green/red). (d) Rejected (red) and feasible (green) rollouts.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Reference choice	Operational meaning
Representation	$K=4$ unordered Bernoulli–Laplace components	Capacity bounds the emitted set; all four decoder passes execute.
Encoder / decoder	DINOv2 ViT-L ( Oquab et al., 2024 ) ; metric Depth Anything V2 initialization; shared recurrent DPT ( Ranftl et al., 2021 )	One encoder, shared depth/scale/existence heads, absolute rather than cumulative depths.
Parameterization	$d=20\operatorname{softplus}(u)$ ; $q=\sigma(e)$ ; $\beta=\operatorname{clip}_{[.1,10]}(.1+\operatorname{softplus}(v))$	Camera- $Z$ meters. The factor $20$ is not a depth clamp.
Recurrence	Detached existence gate; learned removal strength initialized $.05$ ; update cap $.25$	Bound ( S17 ) is relative to each feature token’s norm.
Targets / mask	Compacted provided L1/L3/L5/L7 channels; element mask from positive valid depth	No numeric target sorting or deduplication; all-false masks contribute $M=0$ in the historical all-ray mean.
Complete criterion	ExactMB only; outer weight $1$ , temperature $1$ , existence weight $1$ ; no alignment	All geometric auxiliary weights are zero. Padded enumeration includes the $(K-M)!$ correction.

Appendix

Table S1: Auxiliary-free ExactMB configuration. The configuration connects each implementation choice to its operational role. Variant-specific changes are stated with the corresponding comparisons. Section B.3 describes the available historical configuration records and their limits.

Benchmark	Available annotation	Reported measures	Not established by this view alone
LD-Real 300 real images	Sparse valid/invalid tuples, requested ranks, and near-to-far relations	ACC on valid tuples; OverPred on invalid tuples; same/mixed-rank breakdowns	Dense metric accuracy, physical surface count on every ray, or a literal transparent/opaque material partition.
MD-3K 3,161 real pairs	Transparent mask at query points; foreground/background order; TT/TO/OO and Same/Reverse strata	Recall, OverPred, pattern ACC, ordinal ACC, and Joint ACC on their declared point/pair populations	A census of all physical interfaces or dense real metric depth.
LD-Syn 500 synthetic images	Four dense metric channels and validity masks, in camera- $Z$ meters	Count ACC, Count MAE, under/over-count, and conditional same-rank AbsRel with support	Unconditional real-image metric generalization or calibration of existence probabilities.

Appendix

Table S2: Complementary annotations for visible-layer evaluation. Each benchmark defines absence through its annotations. Together, they assess ordering, presence, and metric depth.

Context	Selection and geometry	Population
Principal terminal suite	Native selector; Bernoulli cutoff $.01$ ; $.02$ m floor and last-retained gap	Full splits; first four accepted ranks on MD/Syn; no alignment
Primary cutoff sweep	Same 12 Bernoulli-slot checkpoints; six cutoffs; fixed geometry	Evaluates operating points without treating them as replicates or test-selected optima
Training / Syn histories	Raw expected mass or $q>.5$ activation; no principal geometric filter	Minibatch exposure / fixed 64-image cohort; image-conditional validation means
Fixed-ray geometry	$q>.01$ , depth $>.02$ m, gap $>.02$ m; injective tolerance	4,513 fixed supplied-distinct-target rays from 20 images; one auxiliary-regularized run
Qualitative displays	$q>.5$ with depth sorting; pinhole geometry	Selected examples and checkpoints; illustrative geometry and graphics
Released models	Dataset-specific masks, order and scale below	Single released estimates; native-task comparisons remain separate

Appendix

Table S3: Decoding rules and populations across evaluation settings. Each setting specifies its own thresholds and aggregation rules. Diagnostic settings leave the principal decoder unchanged.

Benchmark	Outcome	Signed change Count reg. $-$ fixed
LD-Real	Ordinal ACC $\uparrow$	$-8.5_{\mathord{\pm}1.3}$
LD-Real	Tuple-level OverPred $\downarrow$	$-91.0_{\mathord{\pm}0.8}$
MD-3K	Pattern ACC $\uparrow$	$+65.8_{\mathord{\pm}1.2}$
MD-3K	Transparent-point Recall $\uparrow$	$-20.9_{\mathord{\pm}1.0}$
MD-3K	Opaque-point OverPred $\downarrow$	$-89.9_{\mathord{\pm}0.5}$
LD-Syn	Occupied-micro count ACC $\uparrow$	$+91.32_{\mathord{\pm}0.18}$

Appendix

Table S4: The count-regularized configuration reduces false emission at the cost of valid returns. Paired count-regularized minus fixed-cardinality differences, reported as mean

\pm

sample SD over seeds

7/42/61/123

, with native selectors and common geometric filters. Rate differences are percentage points. Count MAE is in layers. Every paired seed shares the displayed direction, including losses in LD valid-query accuracy, transparent-point recall, and four-return count accuracy.

Panel of Table 15 — Table S5: Depth error, support, and depth-and-presence recovery for the primary five systems. Mean $\pm$ sample SD over four training seeds. All entries except RMS (meters) are percentages. AbsRel, RMS, and $\delta_{1}$ are conditional on both same-rank values being valid. Support and depth-accurate recall use the fixed GT denominators shown below. Depth-accurate precision uses each system’s predicted support. Different conditional supports preclude interpreting AbsRel alone as complete recovery.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 198 papers

SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping
May 27, 2026Hongyu Wen, Jia DengDepth Estimation
Department of Computer Science, Princeton University
MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources
Jan 29, 2026Baorui Ma, Jiahui Yang, Donglin Di +5Vision Foundation Models Metric Depth Estimation
Li Auto Inc
One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models
Jun 28, 2026Xiaohao Xu, Feng Xue, Xiang Li +53D Foundation Models Depth Estimation
University of Michigan, Ann Arbor, MI, USA · Carnegie Mellon University, Pittsburgh, PA, USA · New York University, New York, NY, USA +1

May 27, 2026J/K move · Enter open · S save

SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping

Hongyu Wen, Jia Deng

Department of Computer Science, Princeton University

Depth Estimation

Open paper AskPDF arXiv

Transparent objects are common in daily life, and it is important to understand their multilayer depth, including the transparent surface and the objects behind it. Existing methods for multilayer depth typically extend single-layer prediction. They define layers by the front-to-back ordering of 3D points and predict the layers sequentially. However, as layered geometry can admit multiple valid groupings of 3D points into layers, a predefined grouping strategy is inherently restrictive. In this work, we propose SeeGroup, a multi-layer depth estimation method that avoids imposing a predefined grouping and allows the model itself to adaptively assign surfaces to depth maps. We formulate per-pixel multi-layer depth as a point process, treating depth layers as unordered events along each camera ray. This induces a permutation-invariant likelihood over the observed depth layers, yielding a loss that naturally supports arbitrary layer groupings. Experiments demonstrate that our method significantly advances the state of the art of multi-layer depth estimation, improving quadruplet relative depth accuracy on LayeredDepth benchmark from 61.34% to 70.09%. Code is available at https://github.com/princeton-vl/SeeGroup.

Abstract

Depth Estimation

Explore similar work

SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping

MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models