PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation

Chuanxin Song; Peixi Peng

cs.CVNov 5, 2026

PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation

Authors: Chuanxin Song, Peixi Peng

Organizations: Peking University · Pengcheng Laboratory

Abstract

Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.

Figures & tables

Figure 1: Illustration of the unique spatiotemporal challenges within ERP. For each timestamp, the side insets display the distortion-free up and down cubemap views. Spatially , ERP severely stretches the polar regions, allocating excessive pixels to represent sparse information. Temporally , ERP’s geometric distortion causes smooth object motion to manifest as discontinuous, non-linear dynamics in the polar regions.

Figure 2: Overview of the PanoCARLA construction pipeline. In Stage I , human-in-the-loop waypoint adjustment ensures smooth and realistic 6-DoF trajectories. In Stage II , an unobstructed recording setup enables drone-view roaming with dynamic altitude variations, supported by six-view collision detection to eliminate environment-clipping artifacts. Finally , the collected data are processed modality-specifically, yielding seamlessly stitched RGB panoramas and geometrically consistent depth annotations.

Figure 3: Overview of the proposed PVDepth . Building upon a pre-trained video diffusion backbone, PVDepth incorporates two key spatiotemporal adaptations to bridge the geometric gap between perspective and panoramic domains: (1) Progressive Sphere-aware Noise Initialization , which smoothly transitions the input noise from a planar to a spherical distribution to adapt to non-uniform ERP information density; and (2) Cube-rectified Temporal Modeling , which introduces an auxiliary distortion-free Cubemap branch to capture linear motion modeling, rectifying ERP-induced non-linear dynamics in the primary ERP branch.

Dataset	Scene	Depth	Syn./Real	Video	#Frames
Structured3D Zheng et al. (2020)	Indoor	$\checkmark$	Syn.	$\times$	196K
Matterport3D Chang et al. (2017)	Indoor	$\checkmark$	Real	$\times$	10K
Stanford2D3D Armeni et al. (2017)	Indoor	$\checkmark$	Real	$\times$	1K
Omni360-Scene Ge et al. (2025)	Outdoor	$\checkmark$	Syn.	$\times$	61K
PanoCARLA	Outdoor	$\checkmark$	Syn.	$\checkmark$	126.7K

Table 1: Comparison with representative panoramic RGB-D datasets.

Panorama image depth-based
Method	fps02_len50		fps10_len90		fps20_len110
Method	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
DA-2 ${}_{\text{ZS}}$ Li et al. (2025a)	1.367 (0.781)	0.573 (0.742)	1.125 (0.759)	0.617 (0.759)	1.021 (0.731)	0.656 (0.761)
UniK3D ${}_{\text{ZS}}$ Piccinelli et al. (2025)	0.644 (0.486)	0.610 (0.724)	0.580 (0.455)	0.638 (0.741)	0.551 (0.442)	0.666 (0.742)
PanoDA ${}_{\text{ZS}}$ Cao et al. (2025)	0.767 (0.527)	0.262 (0.344)	0.721 (0.526)	0.281 (0.342)	0.642 (0.524)	0.298 (0.341)
DepthAnywhere ${}_{\text{ZS}}$ Wang & Liu (2024)	0.487 (0.260)	0.405 (0.788)	0.493 (0.220)	0.444 (0.806)	0.368 (0.220)	0.543 (0.807)

Table 2: Quantitative comparison. The best and second best performances are bolded and underlined . ZS and FT denote Zero-Shot and Fine-Tuned settings. (Cube-based) refers to the strategy of inferring on decomposed cubemap faces independently and stitching them back. Gray values denote frame-level performance of image-based methods.

Figure 4: Qualitative comparisons on in-the-wild panoramic videos. To visualize the temporal consistency, we show the temporal profiles in red boxes, obtained by slicing the depth values along the time axis at the vertical red line positions.

Method	Components		Metrics
Method	PSNI	CRTM	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
Baseline	-	-	0.248 (0.232)	0.757 (0.754)
w/ PSNI	✓	-	0.228 (0.215)	0.782 (0.784)
w/ CRTM	-	✓	0.208 (0.213)	0.779 (0.777)
PVDepth (Full)	✓	✓	0.199	0.798

Table 3: Effectiveness of Proposed Components. Values in parentheses denote results trained for 40k steps to match the total computational budget of the full model.

Noise Strategy			CRTM	Metrics
Spherical	Anneal	Res. Normal	CRTM	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
-	-	-	-	0.248	0.757
✓	-	-	-	0.235	0.758
✓	-	✓	-	0.231	0.770
✓	✓	-	-	0.228	0.782
✓	✓	✓	-	0.229	0.785

Table 4: Ablation studies on Noise Initialization Strategy. Spherical : Sphere-projected noise; Anneal : Progressive transition strategy; Res. Normal : Residual normal Gaussian noise.

Method Variant	Fusion	KV	Metrics
Method Variant	Strategy	Source	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
Equi-only (Baseline)	-	Equi	0.248	0.757
Naive Average	Avg	Equi + Equi	0.267	0.704
Naive Add	Add	Equi + Equi	0.356	0.649
CRTM (Self-guided)	Proj. & Add	Equi + Equi	0.214	0.772
CRTM (Cube-guided)	Proj. & Add	Equi + Cube	0.208	0.779

Table 5: Ablation on CRTM Design. KV Source: Source for Attention keys/values in the temporal layers.

Strategy	Noise	Training	Spatial	Metrics
Strategy	Init	Stages	Update	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
Baseline (40k step)	Normal	1	-	0.232	0.754
One-Stage Joint	PSNI	1	-	0.221	0.763
Two-Stage Unfrozen	PSNI	2	Train	0.215	0.769
Two-Stage Frozen	Normal	2	Freeze	0.204	0.775
Two-Stage Frozen (Ours)	PSNI	2	Freeze	0.199	0.798

Table 6: Ablation studies on Training Strategy. We compare One-Stage Joint (simultaneously optimizing spatial and temporal modules) against a Two-Stage Protocol (

1_{th}

: Spatial Adaptation via PSNI;

2_{th}

: Temporal Modeling via CRTM). Spatial Update : Whether spatial weights are updated during

2_{th}

stage.

Resolution	DepthCrafter ${}_{\text{ZS}}$	DepthCrafter ${}_{\text{FT}}$	PVDepth
$384\times 192$	0.679 (0.473)	0.493 (0.709)	0.455 (0.738)
$640\times 320$	0.351 (0.571)	0.354 (0.761)	0.275 (0.801)
$768\times 384$	0.310 (0.570)	0.263 (0.769)	0.212 (0.806)
$896\times 448$	0.292 (0.577)	0.254 (0.763)	0.207 (0.802)
$1024\times 512$	0.290 (0.567)	0.232 (0.754)	0.199 (0.798)

Table 7: Sensitivity to inference resolution. Each fine-tuned model is trained at

640\times 320

and evaluated at different inference resolutions. Each entry reports AbsRel

\downarrow

, with

\delta_{1}\uparrow

shown in parentheses. Both metrics are averaged over the three evaluation settings. ZS and FT denote Zero-Shot and Fine-Tuned models.

Metric	UniK3D	ViPE	Matrix-3D	DepthCrafter	PVDepth
Latency (s/frame) $\downarrow$	0.05	0.5–3	20	0.355	0.482
GPU Memory (G/clip) $\downarrow$	3.84	16.8–34.4	9.22	30.6	30.7
FLOPs (PFLOPs/clip) $\downarrow$	0.317	0.758–2.258	2.518	1.986	2.553
AbsRel $\downarrow$	0.592	0.670	0.494	0.232	0.199
$\delta_{1}\uparrow$	0.638	0.721	0.513	0.754	0.798

Table 8: Inference Efficiency. Latency is the average inference time per frame at

1024\times 512

resolution. GPU memory and FLOPs are measured for 110-frame clips at the same resolution. AbsRel and

\delta_{1}

are averaged over the three evaluation settings. ViPE shows a range because the convergence behavior of SLAM optimization varies across sequences.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Comparison of ERPs with and without vehicle self-occlusion. For each ERP, the left inset shows its up and down cubemap views. In the self-occlusion case (a), the vehicle body blocks a fixed portion of the down view, introducing persistent visual noise.

Panel of Figure 6 — Figure 6: Examples of recorded trajectory visualizations in Town01 (a) and Town03 (b). The colored loops represent smooth driving paths generated by our pipeline.

Figure 7: Illustration of the conversion from Cubemap to Equirectangular Projection (ERP).

Statistic	T01	T02	T03	T04	T05	T06	T07	T10	Total
# Clips	18	14	22	13	19	9	14	8	117
# Frames	15512	6626	17822	23531	25436	15788	14899	7082	126696
Distance (km)	5.82	2.53	6.46	10.49	9.63	7.45	5.71	2.52	50.61

Appendix

Table 9: Statistics of different towns in the PanoCARLA dataset. T denotes Town index.

Metric	ViPE Huang et al. (2025)	DepthCrafter Hu et al. (2025)	DA-2 Li et al. (2025a)	PVDepth
SSIM $\uparrow$	0.4107	0.3290	0.3147	0.4194

Appendix

Table 10: Reprojection-based geometric consistency evaluation on the GenEx Real-World dataset using SSIM. Higher values indicate better geometric consistency.

Figure 8: Reprojection-based geometric consistency under varying temporal offsets. We report SSIM scores of different methods under varying temporal offsets between the source and target frames. While ViPE achieves strong performance at small temporal offsets due to test-time optimization, its performance degrades more rapidly as the temporal offset increases. In contrast, PVDepth exhibits more stable performance across larger temporal offsets, indicating stronger long-range temporal consistency.

Figure 9: Qualitative visualization of reprojection-based geometric consistency. The top row shows the target panoramic frame. The second and third rows show synthesized target views obtained by forward-projecting point clouds from an anchor frame at a temporal offset of 40 frames using different methods. Gray regions indicate invalid areas that are masked out during evaluation. Compared to other methods, DA-2 exhibits the largest discrepancy from the target frame due to poor depth–pose alignment, while DepthCrafter suffers from noticeable geometric distortions caused by the domain gap between perspective training data and panoramic imagery. In contrast, PVDepth produces more geometrically consistent reprojections that better align with the target frame.

Pose Source	ViPE Huang et al. (2025)	DepthCrafter Hu et al. (2025)	DA-2 Li et al. (2025a)	PVDepth
MapAnything-based	0.3460	0.3023	0.2572	0.3767
ViPE-based	0.4107	0.3290	0.3147	0.4194

Appendix

Table 11: Reprojection-based geometric consistency on the GenEx Real-World dataset under different pose sources, measured by SSIM. Higher values indicate better geometric consistency.

Figure 10: Qualitative visualization of the cube-based strategy after re-projecting cubemap predictions back to the ERP format. The first row shows the input RGB panoramic frames. The second and third rows present the corresponding depth predictions obtained by DepthCrafter in the zero-shot and fine-tuned settings, respectively. Although depth predictions are internally consistent within each cubemap region, significant scale discrepancies can be observed across different cubemap regions after re-projection, leading to visible seams in the ERP panorama. Notably, such cross-region scale inconsistencies persist even after fine-tuning.

Figure 11: Frame-wise qualitative comparisons on in-the-wild panoramic video depth estimation. Columns show individual frames sampled at different temporal positions (Frame 0, 25, 50, 75, and 100) from the same video sequence, while rows correspond to the input RGB panorama and depth predictions from different methods.

Figure 12: Qualitative visualizations of PVDepth on real-world indoor panoramic videos.

Figure 13: Failure cases of PVDepth on reflective water surfaces . While PVDepth produces plausible depth estimates on calm water regions (top row), it struggles in scenarios with strong reflections (bottom row), where reflected structures on the water surface are incorrectly interpreted as physical geometry.

Figure 14: Additional frame-wise qualitative visualization results of PVDepth on in-the-wild panoramic videos.

Figure 15: Additional visualizations of the PanoCARLA dataset. We present sample RGB-D pairs from the collected sequences. The top rows display the stitched RGB panoramas across different environments, while the bottom rows show the corresponding depth visualizations. For better visualization of fine-grained details, each frame is visualized individually rather than using a sequence-level visualization.

Video Depth Estimation

See all 22 papers

Depth Anything in $360^\circ$ : Towards Scale Invariance in the Wild
Dec 28, 2025Hualie Jiang, Ziyang Song, Zhiqiang Lou +2Depth Estimation Monocular Depth Estimation
Insta360 Research
RePer-360: Releasing Perspective Priors for 360 $^\circ$ Depth Estimation via Self-Modulation
Mar 6, 2026Cheng Guan, Chunyu Lin, Zhijie Shen +2Domain Adaptation Depth Estimation
Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China. · Visual Intelligence +X International Cooperation Joint Laboratory of MOE, Beijing 100044, China.
DVD: Deterministic Video Depth Estimation with Generative Priors
Mar 12, 2026Hongfei Zhang, Harold Haodong Chen, Chenfei Liao +12Video Depth Estimation Depth Estimation
HKUST(GZ) · HKUST · UCSD +5

Dec 28, 2025J/K move · Enter open · S save

Depth Anything in $360^\circ$ : Towards Scale Invariance in the Wild

Hualie Jiang, Ziyang Song, Zhiqiang Lou, Rui Xu+1

Insta360 Research

Depth Estimation Monocular Depth Estimation Vision Foundation Model Adaptation Panoramic Depth Estimation

Open paper AskPDF arXiv

Panoramic depth estimation captures the complete 360

^\circ

scene geometry, being essential for robotics and AR/VR applications. While perspective depth models have achieved remarkable zero-shot generalization via large-scale training, panoramic methods lag behind, especially for open-world scenes, due to data scarcity. To bridge this gap, we introduce DA360, a panoramic-adapted version of Depth Anything V2. Our key insight is that the base DAV2 model, trained on perspective images to predict affine-invariant disparity, already exhibits good zero-shot performance on panoramas. Building on this, we design a lightweight adaptation framework that (i) learns a per-image shift from the ViT class token with scale-invariant supervision, transforming affine-invariant disparity into scale-invariant disparity that directly yields well-formed 3D point clouds, and (ii) integrates circular padding into the DPT decoder to eliminate seam artifacts, ensuring spatial coherence. Fine-tuned on a combination of synthetic indoor and outdoor panoramic data, DA360 is evaluated on standard real-world indoor benchmarks and our newly curated outdoor dataset, Metropolis. Results show that DA360 not only outperforms the original DAV2 by over 50% and 12% relative error reduction indoors and outdoors, but also surpasses prior specialized methods like PanDA by about 25--35% across all tests, establishing state-of-the-art zero-shot panoramic depth estimation.

Abstract

Video Depth Estimation

Explore similar work

Depth Anything in 360∘360^\circ360∘: Towards Scale Invariance in the Wild

RePer-360: Releasing Perspective Priors for 360∘^\circ∘ Depth Estimation via Self-Modulation

DVD: Deterministic Video Depth Estimation with Generative Priors

Depth Anything in $360^\circ$ : Towards Scale Invariance in the Wild

RePer-360: Releasing Perspective Priors for 360 $^\circ$ Depth Estimation via Self-Modulation