PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration

Xiujian Liang; Jiacheng Liu; Mingyang Sun; Qichen He; Anda Cheng; Cewu Lu; Jianhua Sun

cs.ROOct 21, 2026

PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration

Authors: Xiujian Liang, Jiacheng Liu, Mingyang Sun, Qichen He, Anda Cheng, Cewu Lu, Jianhua Sun

Organizations: Fudan University · Shanghai Inovation Institute · Zhejiang University · Shanghai JiaoTong University ·

Abstract

Real-world physical sensing exhibits complex, heterogeneous noise patterns that deviate significantly from idealized simulation, posing a fundamental bottleneck for sim-to-real transfer. Existing sensor modelings typically treat depth noise as a monolithic black-box process, overlooking the distinct physical mechanisms that govern different error modalities. In this work, we introduce a physics-grounded paradigm that disentangles monolithic noise into two complementary modalities: sensing invalidation and measurement inaccuracy, enabling a tailored treatment of noise sources based on their physical origins. Building on this insight, we propose PRISM, a tripartite framework that distills 3D Visual Foundation Model features as rich spatial-semantic priors for physics-based reasoning. To address the inherent sparsity and class imbalance of invalidation regions, we develop Hierarchical Positive-Prioritized Supervision, integrating multi-scale positive-weighted objectives with a positive-preserving dynamic hard mining strategy to enforce precise artifact delineation. Extensive benchmarks demonstrate that PRISM achieves state-of-the-art fidelity in noisy depth synthesis. Furthermore, downstream robotic experiments show that PRISM facilitates a 92.1% average success rate in the real world, marking a significant improvement over monolithic baselines. Our project page is available at: \resizebox{\linewidth}{!}{https://shawnleung-sii.github.io/prism-icml2026/.}

Figures & tables

Figure 1: The Anatomy of Depth Perception and Modeling. (a) The Reality Gap: Unlike pristine simulation, real-world physical sensing exhibits a bimodal noise distribution: black voids and gray residuals. (b) Paradigm Shift: While prior methods conflate noises into a monolithic modeling, our PRISM framework disentangles noise into distinct modalities based on physics-grounded structural priors, enabling targeted synthesis for each noise source.

Figure 2: The overview of PRISM framework. The PRISM orchestrates three synergistic modules: the SPR extracts material-aware priors, the BND predicts sensing invalidation, and the NRG synthesizes measurement residuals. The diagram highlights the cross-stream modulation, where the Semantic Stream (from VFM) implicitly guides the Geometric Stream to enable physics-grounded noise synthesis.

Figure 3: Overview of the PRISM-enabled Sim-to-Real Pipeline. (a) Simulation Data Collection: Large-scale expert demonstrations are collected in a simulator. (b) Offline Depth Enhancement: PRISM acts as a plug-in enhancer, transforming ideal simulation depth into real-world-like observations with physically grounded artifacts. (c) Depth-Aware Policy Learning: A modified robotic policy, equipped with a dedicated depth encoder, is trained on the enhanced dataset to learn robustness against sensor corruption. (d) Zero-Shot Real-World Transfer: The trained policy is directly deployed on physical robots, demonstrating stable manipulation without online domain adaptation.

Figure 4: Qualitative evaluation of physics-grounded depth synthesis. (a) Disentangled Generation: PRISM decomposes noise into Sensing Invalidation (binary masks) and Measurement Inaccuracy (continuous residuals) to compose final depth. (b) Baseline Comparison: Across diverse scenes involving specular and transparent objects, PRISM precisely localizes artifacts to physically unstable regions.

Method	Overall Metrics				Sensing Invalidation			Meas. Inaccuracy
Method	MAE $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$	$\delta<1.25\uparrow$	IoU $\uparrow$	F1 $\uparrow$	Recall $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	Acc- $\delta\uparrow$
GazeboDR	0.203	0.258	0.167	0.724	-	-	-	0.178	0.224	68.3
DepthSynth	0.115	0.143	0.092	0.884	0.651	0.789	0.734	0.094	0.121	86.1
ActiveStereo	0.094	0.118	0.075	0.916	0.806	0.893	0.858	0.078	0.101	89.4
DCL-Depth	0.098	0.125	0.078	0.908	0.785	0.879	0.845	0.082	0.106	88.6
Stable-Sim2Real	0.087	0.106	0.069	0.931	0.876	0.934	0.912	0.070	0.091	91.7

Table 1: Quantitative comparison of depth synthesis fidelity on ByteCameraDepth (In-Domain of Realsense D435). We evaluate three aspects: Overall Metrics (global reconstruction quality), Sensing Invalidation (detection of sensor failures), and Measurement Inaccuracy (precision in valid regions). Note : Baselines marked with ‘-’ do not support separate invalidation masks from continuous noise.

Figure 5: Hardware Requirement of Downstream Evaluation. The real-world setup consists of two platforms: Franka Research 3 for specular and deformable manipulation tasks, and Realman RM75 for long-horizon articulation tasks.

Method	Overall Fidelity		Sensing Inv.
Method	MAE $\downarrow$	$\delta<1.25\uparrow$	IoU $\uparrow$	F1 $\uparrow$
GazeboDR	0.248	0.678	-	-
DepthSynth	0.138	0.832	0.542	0.703
ActiveStereo	0.112	0.878	0.685	0.813
DCL-Depth	0.118	0.865	0.658	0.794
Stable-Sim2Real	0.102	0.894	0.756	0.861

Table 2: Cross-camera synthesis generalization. Models trained on ByteCameraDepth are directly tested on NYU-Depth-v2 ( Zero-shot in Kinect Camera ) to assess robustness.

Task	Explicit Analytical Modeling			Implicit Data-Driven Modeling
Task	GazeboDR	DepthSynth	ActiveStereo	DCL-Depth	Stable-S2R	PRISM
Can-to-Plate	18.7 $\pm$ 0.6	20.0 $\pm$ 0.0	20.0 $\pm$ 0.0	20.0 $\pm$ 0.0	20.0 $\pm$ 0.0	20.0 $\pm$ 0.0
Banana-to-Plate	18.0 $\pm$ 1.0	19.3 $\pm$ 0.6	19.7 $\pm$ 0.6	19.7 $\pm$ 0.6	20.0 $\pm$ 0.0	20.0 $\pm$ 0.0
Stack Cubes	12.7 $\pm$ 0.6	14.3 $\pm$ 0.6	16.7 $\pm$ 0.6	16.3 $\pm$ 1.5	18.7 $\pm$ 0.6	19.3 $\pm$ 0.6
Bucket-to-Plate	11.7 $\pm$ 0.6	13.7 $\pm$ 1.2	15.7 $\pm$ 0.6	15.3 $\pm$ 1.2	17.7 $\pm$ 0.6	17.7 $\pm$ 0.6
Toyball-to-Drawer	4.7 $\pm$ 0.6	7.7 $\pm$ 0.6	11.7 $\pm$ 0.6	11.3 $\pm$ 1.2	16.3 $\pm$ 0.6	17.3 $\pm$ 1.2

Table 3: Multi-task manipulation success rates with RISE policy. We evaluate 6 tasks with diverse properties: Specular ( ), Deformable ( ), Articulated ( ), and Long-Horizon ( ). Results are reported as mean

\pm

std across 3 random seeds (

/20

trials).

Baselines	DP3	PPT	RISE	Average
Sim + Clean	35.3	38.3	43.1	38.9
Sim + GazeboDR	49.2	53.3	57.8	53.4
Sim + DepthSynth	60.6	66.4	67.8	64.9
Sim + ActiveStereo	75.6	80.6	78.3	78.2
Sim + DCL-Depth	74.2	78.6	77.8	76.9
Sim + Stable-Sim2Real	86.7	89.7	90.8	89.1

Table 4: Cross-policy Sim2Real generalization. We report the average success rate (%) across all 6 tasks using three different policy architectures. PRISM consistently outperforms baselines.

Figure 6: Components Contribution. Assessing the impact of removing SPR and BND on Invalidation IoU and Robot SR.

Figure 7: Semantics Efficacy. Comparing generic vs. geometric priors. 3D VFMs show superior material awareness.

Figure 8: Supervision Strategy. Progressive improvement by adding Positive-Weighting, Multi-Scale, and PP-DHM.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Input Dim	Output Dim	Description
VFM Backbone	$[B,3,H,W]$	$[B,N,1024]$	MoGe2-Large, frozen
(MoGe2-L)			$N=\frac{H}{14}\times\frac{W}{14}$
Global Avg Pool	$[B,N,1024]$	$[B,1024]$	Spatial aggregation
Context MLP	$[B,1024]$	$[B,256]$	Linear(1024 $\to$ 512)
			+ SiLU + Linear(512 $\to$ 256)
Scale Attn $\ell=1$	$[B,256]$	$[B,64]$	Linear + Tanh

Appendix

Table 5: SPR Architecture Specification. The VFM backbone is frozen during training.

Stage	Input	Output	Resolution	Operations
Pre-Stem	$[B,4,H,W]$	$[B,32,H,W]$	$H\times W$	Conv3 $\times$ 3 + BN + ReLU
Stem	$[B,32,H,W]$	$[B,64,\frac{H}{2},\frac{W}{2}]$	$\frac{H}{2}$	Conv3 $\times$ 3(s=2) + BN + ReLU
Stage 1	$[B,64,\frac{H}{2},\frac{W}{2}]$	$[B,64,\frac{H}{2},\frac{W}{2}]$	$\frac{H}{2}$	ConvBlock $\times 2$
Stage 2	$[B,64,\frac{H}{2},\frac{W}{2}]$	$[B,128,\frac{H}{4},\frac{W}{4}]$	$\frac{H}{4}$	ConvBlock(s=2) + ConvBlock
Stage 3	$[B,128,\frac{H}{4},\frac{W}{4}]$	$[B,256,\frac{H}{8},\frac{W}{8}]$	$\frac{H}{8}$	ConvBlock(s=2) + ConvBlock
Stage 4	$[B,256,\frac{H}{8},\frac{W}{8}]$	$[B,512,\frac{H}{16},\frac{W}{16}]$	$\frac{H}{16}$	ConvBlock(s=2) + ConvBlock

Appendix

Table 6: BND Encoder Architecture. ConvBlock consists of Conv3

\times

3 + BatchNorm + ReLU.

Stage	Input	Skip	Output	Resolution	Modulation
Dec4	$[B,512,\frac{H}{16}]$	$[B,256,\frac{H}{8}]$	$[B,512,\frac{H}{8}]$	$\frac{H}{8}$	$\mathbf{F}\odot(1+\mathbf{S}_{4})$
Dec3	$[B,512,\frac{H}{8}]$	$[B,128,\frac{H}{4}]$	$[B,256,\frac{H}{4}]$	$\frac{H}{4}$	$\mathbf{F}\odot(1+\mathbf{S}_{3})$
Dec2	$[B,256,\frac{H}{4}]$	$[B,64,\frac{H}{2}]$	$[B,128,\frac{H}{2}]$	$\frac{H}{2}$	$\mathbf{F}\odot(1+\mathbf{S}_{2})$
Dec1	$[B,128,\frac{H}{2}]$	$[B,32,H]$	$[B,128,H]$	$H$	Skip only
Head	$[B,128,H,W]$	—	$[B,1,H,W]$	$H$	Conv + Sigmoid

Appendix

Table 7: BND Decoder Architecture with Semantic-Prompted Modulation.

Module	Input	Output	Trainable	Description
VFM (MoGe2-L)	$[B,3,H,W]$	$[B,N,1024]$	✗	Semantic extraction
VFM Projector	$[B,1024]$	$[B,320]$	✓	MLP for conditioning
VAE Encoder	$[B,3,H,W]$	$[B,4,\frac{H}{8},\frac{W}{8}]$	✗	Latent compression
VAE Decoder	$[B,4,\frac{H}{8},\frac{W}{8}]$	$[B,3,H,W]$	✓	Fine-tuned decoding
ControlNet	$[B,3,H,W]$	Multi-scale	✓	Spatial conditioning
U-Net	$[B,4,\frac{H}{8},\frac{W}{8}]$	$[B,4,\frac{H}{8},\frac{W}{8}]$	✓ (Stage 2)	Denoising backbone

Appendix

Table 8: NRG Architecture Specification. “Trainable” indicates whether parameters are updated during training.

Parameter	Value	Parameter	Value
Base Model	Stable Diffusion 2.1	DDIM $\eta$	0.0 (deterministic)
DDPM Timesteps	1000 (training)	Guidance Scale	9.0
DDIM Steps	50 (inference)	Residual Scale	2.0
Depth Range	$[0.05\text{m},5.0\text{m}]$	Latent Channels	4

Appendix

Table 9: Diffusion Configuration for NRG.

Module	Parameters	Components
SPR	$\sim$ 0.8M	Context MLP + Scale Attention heads
BND	$\sim$ 21M	CNN Encoder + Decoder + Prediction Heads
NRG	$\sim$ 361M	ControlNet + VFM Projector + VAE Decoder (fine-tuned)
Total	$\sim$ 382M

Appendix

Table 10: Trainable Parameter Summary for PRISM.

Category	Parameter	Value
Hardware	GPU	8 $\times$ NVIDIA H200
	Framework	PyTorch 2.1 + CUDA 12.1
	Distributed	DDP (DistributedDataParallel)
	Precision	FP16 (AMP enabled)
Common	Image Resolution	$512\times 512$
	Optimizer	AdamW ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999)

Appendix

Table 11: PRISM Training Configuration. All experiments use mixed-precision (FP16) training with DDP.

Module	Loss Component	Weight	Description
BND (H-PPS)	Positive-Weighted BCE	$\lambda=1.0$	Main supervision
	Positive Weight	$w_{\text{pos}}=3.0$	Recall enhancement
	Dice Loss	$\lambda_{\text{dice}}=0.3$	Shape regularization
	Multi-Scale Auxiliary	$\lambda_{\text{aux}}=0.5$	Deep supervision (4 scales)
	OHEM Start Ratio	$\gamma_{\text{start}}=0.25$	25% hardest negatives
	OHEM End Ratio	$\gamma_{\text{end}}=0.10$	10% at final epoch

Appendix

Table 12: Loss Function Hyperparameters for H-PPS and NRG.

Split	Samples	Scene Instances	Usage
Training	45,000	39	Model training
Validation	5,000	39	Hyperparameter tuning
Test	5,000	39	Final evaluation
Total	55,000	39

Appendix

Table 13: ByteCameraDepth Dataset Statistics.

Property	Value
Sensor Type	Legacy Structured-Light Sensor
Test Split	654 images (Standard Eigen Split)
Depth Range	0.1m – 10.0m
Resolution	$640\times 480$ (Center Cropped)

Appendix

Table 14: NYU-Depth-v2 Dataset Configuration for Zero-Shot Evaluation.

Figure 9: Hardware Requirement of Downstream Evaluation. The real-world setup consists of two platforms: Franka Research 3 for precision tasks, and Realman RM75 for long-horizon articulation tasks. We utilize RealSense D435 cameras for RGB-D perception.

Method	Paradigm	Overall Metrics		Invalidation (Safety)		Accuracy (Precision)		Downstream
Method	Config	MAE $\downarrow$	$\delta<1.25\uparrow$	IoU $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	$\delta<1.05\uparrow$	Robot SR $\uparrow$
NRG Only	Baseline	0.118	87.3%	0.412	0.583	0.089	71.5%	72.5%
w/o SPR	Physics-Blind	0.095	91.8%	0.823	0.891	0.068	82.4%	83.7%
w/o BND	Monolithic	0.098	90.6%	0.756	0.852	0.071	80.8%	81.5%
PRISM (Full)	Ours	0.076	96.4%	0.952	0.975	0.052	91.3%	92.5%

Appendix

Table 15: Component Ablation Study. We evaluate the contribution of Semantic-Physics Reasoner (SPR) and Bimodal Noise Disentangler (BND) against the pure Noise Residual Generator (NRG) baseline. Inv-IoU and Inv-F1 measure the detection of sensor holes, while Acc-MAE measures the precision in valid regions. Robot SR denotes the success rate in Sim2Real manipulation.

Backbone	Type	Depth Fidelity		Physics Awareness		Efficiency
Backbone	Type	MAE $\downarrow$	$\delta<1.25\uparrow$	Inv-IoU $\uparrow$	Robot SR $\uparrow$	Params	Inference
ResNet-50 (Scratch)	CNN	0.142	82.6%	0.685	60.0%	25M	12ms
ResNet-50 (ImageNet)	CNN	0.125	86.1%	0.738	67.0%	25M	12ms
DINOv2-Base	Generic ViT	0.108	89.7%	0.812	73.5%	86M	28ms
DINOv2-Large	Generic ViT	0.095	92.3%	0.856	78.0%	304M	45ms
DepthAnything-V2-L	3D VFM	0.082	95.1%	0.925	88.0%	335M	48ms

Appendix

Table 16: VFM Backbone Comparison in SPR. We compare CNNs, Generic ViTs, and 3D VFMs. Robot SR refers to the average success rate across all 6 manipulation tasks. Inference time is measured on a single H100 GPU.

Config	Mechanism Added	Invalidation Detection			Fine-Grained
Config	Mechanism Added	IoU $\uparrow$	F1 $\uparrow$	Recall $\uparrow$	Boundary-IoU $\uparrow$
Baseline	BCE Only	0.548	0.642	0.485	0.312
+ Weight	Positive-Weighted ( $w_{pos}=3.0$ )	0.685	0.756	0.712	0.425
+ Dice	Shape Constraint	0.768	0.845	0.792	0.538
+ Aux	Multi-Scale Supervision	0.856	0.912	0.875	0.652
+ Mining	PP-DHM (Static $\gamma=0.25$ )	0.912	0.948	0.945	0.735

Appendix

Table 17: Progressive Ablation of H-PPS. We add components sequentially to the standard BCE baseline. Boundary-IoU evaluates the precision of artifact edges (within 5px).

Impact of Positive Weight $w_{pos}$				Impact of Mining Ratio $\gamma_{start}$
$w_{pos}$	Recall $\uparrow$	Precision $\uparrow$	F1 $\uparrow$	$\gamma_{start}$	Inv-IoU $\uparrow$	B-IoU $\uparrow$
1.0	0.485	0.962	0.642	0.10	0.905	0.752
2.0	0.824	0.951	0.883	0.25	0.952	0.823
3.0	0.991	0.959	0.975	0.50	0.931	0.789
4.0	0.993	0.885	0.936	0.75	0.884	0.741
5.0	0.995	0.812	0.894	1.00	0.856	0.705

Appendix

Table 18: Sensitivity Analysis. We analyze the trade-offs for Positive Weight

w_{pos}

(Left) and Initial Mining Ratio

\gamma_{start}

(Right). The optimal configurations (

w_{pos}=3.0

and

\gamma_{start}=0.25

) represent the best balance.

Figure 10: Qualitative synthesis on the Kitchen scene .

Figure 11: Qualitative synthesis results on the Living Room scene .

Figure 12: Qualitative synthesis results on the Market scene .

Figure 13: Qualitative synthesis results on the Office scene .

Figure 14: Qualitative synthesis results on the Lounge scene .

Figure 15: Real-world rollout for T1 (Can-to-Plate) .

Figure 16: Real-world rollout for T2 (Banana-to-Plate) .

Figure 17: Real-world rollout for T3 (Stack Cubes) .

Figure 18: Real-world rollout for T4 (Bucket-to-Plate) .

Figure 19: Real-world rollout for T5 (Toyball-to-Drawer) .

Figure 20: Real-world rollout for T6 (Bread-to-Microwave) .

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation
Mar 17, 2026Xinhao Cai, Gensheng Pei, Zeren Sun +3Monocular Depth Estimation Diffusion Models
Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Department of Electrical and Computer Engineering, Sungkyunkwan University +2
Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption
Oct 17, 2026Zhiyuan Yang, Jiaju He, Ruiteng Zhao +3Robotic Manipulation Noise Robustness
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore · National University of Singapore (NUS), Singapore
MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources
Jan 29, 2026Baorui Ma, Jiahui Yang, Donglin Di +5Vision Foundation Models Metric Depth Estimation
Li Auto Inc

Mar 17, 2026J/K move · Enter open · S save

Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

Xinhao Cai, Gensheng Pei, Zeren Sun, Yazhou Yao+2

Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Department of Electrical and Computer Engineering, Sungkyunkwan University+2

Monocular Depth Estimation Diffusion Models Diffusion Model Guidance

Open paper AskPDF arXiv

In this paper, we propose Iris, a deterministic framework for Monocular Depth Estimation (MDE) that integrates real-world priors into the diffusion model. Conventional feed-forward methods rely on massive training data, yet still miss details. Previous diffusion-based methods leverage rich generative priors yet struggle with synthetic-to-real domain transfer. Iris, in contrast, preserves fine details, generalizes strongly from synthetic to real scenes, and remains efficient with limited training data. To this end, we introduce a two-stage Priors-to-Geometry Deterministic (PGD) schedule: the prior stage uses Spectral-Gated Distillation (SGD) to transfer low-frequency real priors while leaving high-frequency details unconstrained, and the geometry stage applies Spectral-Gated Consistency (SGC) to enforce high-frequency fidelity while refining with synthetic ground truth. The two stages share weights and are executed with a high-to-low timestep schedule. Extensive experimental results confirm that Iris achieves significant improvements in MDE performance with strong in-the-wild generalization.

Abstract

Sim-to-Real Transfer

Explore similar work

Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption

MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources