STATIC: Surface-normal-guided Two-branch Alignment and Temporal Integration for Consistent Video Depth Estimation

Sunghun Yang; Minhyeok Lee; Suhwan Cho; Jungho Lee; Sangyoun Lee

cs.CVDec 2, 2024

STATIC: Surface-normal-guided Two-branch Alignment and Temporal Integration for Consistent Video Depth Estimation

Authors: Sunghun Yang, Minhyeok Lee, Suhwan Cho, Jungho Lee, Sangyoun Lee

Organizations: Yonsei University

Abstract

Video depth estimation is essential for applications such as autonomous driving, AR/VR, and robotics. Recent transformer-based single-image monocular depth estimation models perform well on a single image but struggle with depth consistency across video frames. Traditional methods aim to improve temporal consistency using multi-frame temporal modules or explicit priors like optical flow and camera parameters. However, these approaches face issues such as high memory use, reduced performance with dynamic or irregular motion, and a heavy reliance on the accuracy of external motion estimators. We propose STATIC, a novel model that independently learns temporal consistency in geometrically stable and unstable regions using only two adjacent frames. By leveraging intrinsic geometric cues rather than external motion priors, STATIC generates a depth-derived geometry-aware difference mask from surface normals to identify regions with inter-frame geometric variation by measuring directional variance. For geometrically stable regions, the Masked Stable (MS) module enhances temporal consistency by focusing on structurally stable regions. For geometrically unstable regions, the Surface Normal Similarity (SNS) module models inter-frame correspondences and ensures temporal consistency by measuring feature similarities between frames. A final refinement integrates the independently learned stable and unstable features, enabling STATIC to improve consistent depth prediction across consecutive frames. Experiments on KITTI, NYUv2, and Sintel demonstrate strong depth accuracy and temporal consistency, as well as stable performance across the tested frame-rate and camera-motion conditions.

Figures & tables

Figure 1: Results of the (a) Depth Anything V2 [ 42 ] , (b) GEDepth [ 43 ] and proposed (c) STATIC from sequential frames. Temporal profiles are generated from the marked slice regions to visualize frame-to-frame depth continuity. (c) shows improved ground continuity over the single-frame approach (a) and more stable depth across frames than the compared video depth estimation method (b). For visual comparison, each method is rescaled.

Figure 2: Overall architecture of the proposed STATIC model. The model primarily consists of an encoder, depth embedder, and SNS/MS modules, with a surface normal decoder and head as submodules.

Figure 3: Generation of the proposed difference mask. The pixel-wise directional variance between two surface-normal maps is thresholded to produce the coarse mask $M^{d}$ . The refinement module combines $M^{d}$ with the surface-normal feature $Z_{t}^{n}$ to generate the final mask $M^{l}$ used by the SNS and MS modules.

Figure 4: The structure of the SNS module. First, a similarity map $S$ is generated using two features. Then, by multiplying this map with the depth features of other frame, a process similar to warping is performed.

Figure 5: The structure of the MS module. First, an attention mechanism is applied using masked features that retain only geometrically stable regions. Next, a refinement process is conducted to integrate the aligned feature with the depth feature.

Method	Frame	C	O	M	Abs Rel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$
PEM [ 17 ]	SF	-	-	-	0.068	0.221	2.127	0.958	0.993	0.9983
CATNet [ 32 ]	SF	-	-	-	0.052	0.160	2.153	0.972	0.997	0.9990
SSRDepth [ 49 ]	SF	-	-	-	0.053	0.157	2.125	0.973	0.997	0.9990
AdaBins [ 3 ]	SF	-	-	-	0.058	0.190	2.360	0.964	0.995	0.9993
BinsFormer [ 19 ]	SF	-	-	-	0.058	0.190	2.336	0.964	0.996	0.9994
DepthFormer [ 1 ]	SF	-	-	-	0.053	0.187	2.285	0.970	0.996	0.9994

Table 1: Performance comparison between various methods on the KITTI Eigen dataset. The best results are in bold. "MF" indicates multi-frame methods, "SF" indicates single-frame methods, "O" represents optical flow, "C" represents camera parameter, and "M" represents memory. Baseline results are cited from the corresponding published papers.

Type	Method	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$
SF	SSRDepth [ 49 ]	0.933	0.086
	MiDas-v2.1-Large [ 28 ]	0.910	0.095
	DPT-Large [ 27 ]	0.928	0.084
	WSVD [ 34 ]	0.768	0.164
	ST-CLSTM [ 47 ]	0.833	0.131
MF	DeepV2D [ 33 ]	0.924	0.082

Table 2: Performance comparison between various methods on the NYU v2 dataset. Baseline results are cited from the corresponding published papers. † : trained with large-scale pretraining under a different supervision setting.

Method	Type	$\delta_{1}\uparrow$	$Rel\downarrow$	$OPW\downarrow$	$\textit{T}(ms)$
DPT [ 27 ]	Image	0.597	0.339	0.612	67.7
MiDas [ 28 ]	Image	0.485	0.410	0.843	53.0
CVD [ 23 ]	Flow	0.518	0.406	0.497	-
WSVD [ 34 ]	Flow	0.501	0.439	0.577	296.6
Robust-CVD [ 14 ]	Camera	0.521	0.422	0.475	-
GEDepth † [ 43 ]	Camera	0.522	0.430	0.473	185

Table 3: Performance comparison of various methods on the Sintel dataset. The best results are highlighted in bold , and the second best results are underlined . Note that † indicates retrained model and T denotes inference time. Baseline results are cited from the corresponding published papers.

Figure 6: Temporal consistency comparison on the KITTI Eigen dataset. The temporal profiles are generated from the marked slice regions in the RGB frames to visualize frame-to-frame depth continuity. The red boxes indicate zoomed regions where temporal discontinuities are more clearly observed.

Type	Method	rTC $\uparrow$	aTC $\downarrow$
	NeWCRFs [ 46 ]	0.914	0.116
SF	iDisc [ 26 ]	0.923	0.108
	GEDepth [ 43 ]	0.919	0.133
	Depth Anything V2 [ 42 ]	0.946	0.099
	TC-Depth-FS [ 30 ]	0.901	0.122
MF	Many-Depth-FS [ 38 ]	0.920	0.111

Table 4: Comparison of temporal consistency across various methods on rTC and aTC.

Figure 7: Qualitative comparison of video methods on the MPI-Sintel dataset. The temporal profiles are generated from the marked slice regions in the RGB frames to visualize frame-to-frame depth continuity. The red boxes highlight regions where differences in temporal stability are noticeable.

Figure 8: Qualitative comparison of video methods on the KITTI Eigen dataset.

SNS	MS	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta<1.25$ $\uparrow$
-	-	0.051	2.022	0.977
$\checkmark$	-	0.047	1.951	0.982
-	$\checkmark$	0.045	1.939	0.984
$\checkmark$	$\checkmark$	0.041	1.917	0.988

Table 5: The ablation experiment evaluates the individual effects of the SNS module and the MS module on the KITTI Eigen dataset.

Figure 11: Qualitative comparison under stationary-camera and moving-camera scenarios. Video frames, surface normal, and the corresponding difference mask across consecutive frames are shown, with optical flow for comparison.

Frame Speed	$\delta_{1}\uparrow$	$Rel\downarrow$	$OPW\downarrow$
$\times 1$	0.598	0.330	0.405
$\times 0.5$	0.595	0.334	0.411
$\times 0.33$	0.571	0.359	0.424

Table 6: Performance Evaluation at Different Frame Speeds.

Figure 12: Comparison of Difference Mask Variations with Increased Frame Intervals.

Type	Method	Params (M)	FLOPs (G)	GPU Mem. (MB)	Inference Time (ms)
Image	Depth Anything V2 [ 42 ]	335.316	416.815	1483	38.767
Video	NVDS-MiDaS [ 37 ]	198.581	238.846	1469	26.459
	NVDS-DPT	437.274	425.770	2378	42.831
	GEDepth [ 43 ]	277.135	318.211	1692	32.812
	Video Depth Anything [ 5 ]	384.390	453.725	4333	30.707
	STATIC	280.610	243.384	1626	30.553

Table 7: Computational cost comparison between image-based and video-based depth estimation methods at an input resolution of

240\times 480

Figure 13: Failure cases. Red dashed boxes highlight failure regions in the predicted depth and normal maps.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 198 papers

GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth
May 11, 2026Yuecheng Liu, Junda Cheng, Longliang Liu +4Depth Estimation Monocular Depth Estimation
1Huazhong University of Science & Technology · 2Carizon · 3Optics Valley Laboratory
3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation
Jun 14, 2026Yuanye Liu, Ke Zhang, Junzhe Jiang +3Depth Estimation Monocular Depth Estimation
Fudan University, Shanghai, China · Johns Hopkins University, Baltimore, USA
PTC-Depth: Pose-Refined Monocular Depth Estimation with Temporal Consistency
Apr 2, 2026Leezy Han, Seunggyu Kim, Dongseok Shim +1Depth Estimation Monocular Depth Estimation
Ajou University · Sony Creative AI Lab

May 11, 2026J/K move · Enter open · S save

GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth

Yuecheng Liu, Junda Cheng, Longliang Liu, Wenjing Liao+3

1Huazhong University of Science & Technology · 2Carizon · 3Optics Valley Laboratory

Depth Estimation Monocular Depth Estimation Video Depth Estimation

Open paper AskPDF arXiv

Video depth estimation extends monocular prediction into the temporal domain to ensure coherence. However, existing methods often suffer from spatial blurring in fine-detail regions and temporal inconsistencies. We argue that current approaches, which primarily rely on temporal smoothing via Transformers, struggle to maintain strict 3D geometric consistency-particularly under rotations or drastic view changes. To address this, we propose GemDepth, a framework built on the insight that an explicit awareness of camera motion and global 3D structure is a prerequisite for 3D consistency. Distinctively, GemDepth introduces a Geometry-Embedding Module (GEM) that predicts inter-frame camera poses to generate implicit geometric embeddings. This injection of motion priors equips the network with intrinsic 3D perception and alignment capabilities. Guided by these geometric cues, our Alternating Spatio-Temporal Transformer (ASTT) captures latent point-level correspondences to simultaneously enhance spatial precision for sharp details and enforce rigorous temporal consistency. Furthermore, GemDepth employs a data-efficient training strategy, effectively bridging the gap between high efficiency and robust geometric consistency. As shown in Fig.2, comprehensive evaluations demonstrate that GemDepth achieves state-of-the-art performance across multiple datasets, particularly in complex dynamic scenarios. The code is publicly available at: https://github.com/Yuecheng919/GemDepth.

Abstract

Depth Estimation

Explore similar work

GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth

3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation

PTC-Depth: Pose-Refined Monocular Depth Estimation with Temporal Consistency