SMFormer: Empowering Self-supervised Stereo Matching via Foundation Models and Data Augmentation

Yun Wang; Zhengjie Yang; Jiahao Zheng; Zhanjie Zhang; Dapeng Oliver Wu; Yulan Guo

cs.CVApr 11, 2026

SMFormer: Empowering Self-supervised Stereo Matching via Foundation Models and Data Augmentation

Authors: Yun Wang, Zhengjie Yang, Jiahao Zheng, Zhanjie Zhang, Dapeng Oliver Wu, Yulan Guo

Organizations: Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong, China · Hong Kong Generative AI Research and Development Center, The Hong Kong University of Science and Technology, Hong Kong, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310000, China · School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University, Sun Yat-sen University, Shenzhen, China

Abstract

Recent self-supervised stereo matching methods have made significant progress. They typically rely on the photometric consistency assumption, which presumes corresponding points across views share the same appearance. However, this assumption could be compromised by real-world disturbances, resulting in invalid supervisory signals and a significant accuracy gap compared to supervised methods. To address this issue, we propose SMFormer, a framework integrating more reliable self-supervision guided by the Vision Foundation Model (VFM) and data augmentation. We first incorporate the VFM with the Feature Pyramid Network (FPN), providing a discriminative and robust feature representation against disturbance in various scenarios. We then devise an effective data augmentation mechanism that ensures robustness to various transformations. The data augmentation mechanism explicitly enforces consistency between learned features and those influenced by illumination variations. Additionally, it regularizes the output consistency between disparity predictions of strong augmented samples and those generated from standard samples. Experiments on multiple mainstream benchmarks demonstrate that our SMFormer achieves state-of-the-art (SOTA) performance among self-supervised methods and even competes on par with supervised ones. Remarkably, in the challenging Booster benchmark, SMFormer even outperforms some SOTA supervised methods, such as CFNet.

Figures & tables

Fig. 1: (a), our method, equipped with the Vision Foundation Model (SAM [ 1 ] ), which is a robust pre-trained model, surpassing previous state-of-the-art domain generalized methods. (b), the fine-tuned performance significantly surpasses previous self-supervised methods, even surpassing some supervised ones. (c) Our SMFormer, equipped with the proposed feature extractors and self-supervised losses, demonstrates robust performance in challenging regions such as occluded areas (Class 1), reflective areas (Class 2), textureless areas (Class 3), thin structure areas (Class 4), and areas with illumination changes (low lighting in the right image) compared to solely relying on the photometric consistency loss (Baseline). Best zoomed in.

Fig. 2: Hard cases in Booster [ 9 ] with reflective and texture-less regions. (c), (e) indicates Winner-Take-All (WTA) disparity from the feature correlation (1/4 scale) achieved by the dot products among left and right features before the 3D CNN cost aggregation. WTA disparity enhanced with pre-trained VFM (SAM [ 1 ] ) contains less noise than the original ones without VFM priors. Compared with (d), (f) achieves better disparity maps. The models are trained in a self-supervised manner.

Fig. 3: An overview of SMFormer. During training, the framework adopts a data augmentation branch with an augmented pair (the upper part of subfigure (a)) and a standard branch with a standard pair (the lower part of subfigure (a)). Only the standard branch is available for inference. Both branches share weights in the cost aggregation module except for the feature extractors. For the feature-level stereo contrastive loss, the key encoder for the augmented image pair is a moving average of the query encoder to mitigate negative sample effects. The details regarding the feature extractor are in subfigure (b). Color-dashed boxes are our main contributions.

Fig. 4: The visualization of the reconstructed left and the left image on the MotorcycleE image pair. For the Ground Truth disparity with occlusions, we use the predicted disparity values to fill in the gaps. The error map is calculated from the absolute difference between these two images.

Fig. 7: A pipeline of valid checks. T denotes the horizontal flip operation.

		KIT 2012				KIT 2015
Method	Supervised.	Out-Noc (%)	Out-All (%)	Avg-Noc (px)	Avg-All (px)	D1-bg (%)	D1-fg (%)	D1-All (%)	Time (s)
DispNet [ 61 ]	✓	4.11	4.65	0.9	1.0	4.32	4.41	4.34	0.06
GCNet [ 64 ]	✓	1.77	2.30	0.6	0.7	2.21	6.16	2.87	0.9
MABNet [ 65 ]	✓	2.71	3.31	0.7	0.8	3.04	8.07	3.88	0.11
SGM-Net [ 66 ]	✓	2.29	3.50	0.7	0.9	2.66	8.64	3.66	67
OASM [ 67 ]	✗	6.39	8.60	1.3	2.0	6.89	19.42	8.98	0.73

TABLE I: Comparative results achieved on the KITTI 2012 & 2015 benchmarks. The latest state-of-the-art self-supervised stereo matching method is ChiT but using KITTI Eigen Splits (22600 image pairs), denoted as

\ast

. “-” indicates that results are not available.

Fig. 8: Visualization results achieved by our method and other self-supervised stereo matching methods on KITTI benchmarks. Our SMFormer performs well in reflective and detailed regions, highlighted by orange boxes. Best zoomed in.

Method	KIT 2012	KIT 2015	Middle	ETH3D
Method	Bad 3.0 (All)	Bad 3.0 (All)	Bad 2.0 (Noc)	Bad 1.0 (Noc)
CFNet [ 2 ]	4.7	5.8	15.4	5.8
PCWNet [ 42 ]	4.2	5.6	15.8	${5.2}$
RAFT-Stereo [ 5 ]	5.1	5.7	12.6	3.3
IGEVStereo [ 3 ]	5.7	6.0	7.2	4.1
UCFNet_pretrain [ 68 ]	4.5	5.2	26.0	4.8

TABLE II: Domain generalization evaluation on target training sets. ‡ indicates the use of pre-trained VFM. The best results in bold and the sub-optimal best results in blue ).

Table 10

Table 11

Fig. 9: Visualization results achieved by our method and other supervised methods. Disparity maps are generated from Middlebury & ETH3D benchmarks.

Fig. 10: Visualization results achieved by our method and the baseline method. Note that the baseline method uses the CFNet backbone and employs the vanilla photometric loss and disparity smooth loss as supervisory signals.

VFMs	Model	KIT 2012 (All)		KIT 2015 (All)		MID (Noc)		ETH3D (Noc)
VFMs	Capacity	EPE	Bad 3.0	EPE	Bad 3.0	EPE	Bad 2.0	EPE	Bad 1.0
Baseline ∗ [ 2 ]	✗	1.10	4.73	1.46	5.78	3.21	15.4	0.56	5.80
DUST3R [ 16 ]	Large	0.92	4.67	1.01	5.06	1.37	9.45	0.37	4.67
DINOV2 [ 13 ]	Base	0.93	4.56	1.02	5.14	1.41	9.79	0.37	3.93
DINOV2 [ 13 ]	Large	0.87	4.35	0.96	4.92	1.29	8.61	0.34	3.49
EVA2 [ 15 ]	Base	0.94	4.46	1.04	5.10	1.55	10.4	0.36	3.71

TABLE VII: Ablation study on different VFMs on zero-shot generalization. Note that all model variants are only trained on the synthetic SceneFlow dataset and tested on four real datasets.

\ast

indicates that the officially provided weights are used for evaluation. And VFMs include the proposed MLA module.

Fig. 11: Visual comparison of SMFormer on Middlebury PianoL pair. Top row: the predicted disparity maps with different combinations of the proposed components. Bottom row: the error maps. $\mathcal{L}_{photo}$ is the photometric consistency loss; $\mathcal{L}_{s}$ is the disparity smooth loss. $\mathcal{L}_{flc}$ : The feature-level Stereo Contrastive Loss; $\mathcal{L}_{ild}$ : The image-level Disparity Difference Loss.

Model	Feature Extractor	$\mathcal{L}_{photo}$	$\mathcal{L}_{s}$	$\mathcal{L}_{flc}$	$\mathcal{L}_{ild}$	KIT 2012 (All)		KIT 2015 (All)		Middle (All)		ETH3D (All)
Model	Feature Extractor	$\mathcal{L}_{photo}$	$\mathcal{L}_{s}$	$\mathcal{L}_{flc}$	$\mathcal{L}_{ild}$	EPE	D1	EPE	D1	EPE	Bad 2.0	EPE	Bad 1.0
Baseline (CFNet)	FPN	✓	✓			0.84	4.10	0.96	4.27	2.87	13.8	1.62	10.2
SMFormer	VFM	✓	✓			0.87	4.56	1.01	4.83	1.74	11.3	1.27	9.24
	FPN + VFM	✓	✓			0.77	3.98	0.85	4.05	1.29	9.81	0.49	5.25
	FPN + VFM	✓	✓	✓		0.73	3.59	0.82	3.67	1.10	8.86	0.36	3.55
	FPN + VFM	✓	✓	✓	✓	0.68	3.01	0.76	3.10	0.95	8.14	0.28	2.93

TABLE VIII: Ablation study on the proposed components. We evaluate the model on the KITTI 2012 & 2015, Middlebury, and ETH3D training sets, using all values (All) rather than just the non-occluded regions (Noc). VFM denotes SAM [ 1 ] (ViT-Large). Cyan color in the table represents the results of the baseline, while pink color represents the results of the final model.

Feature Extractor		KIT 2012		KIT 2015		Middle		ETH3D
Backbone	MLA	EPE	D1	EPE	D1	EPE	Bad 2.0	EPE	Bad 1.0
SAM	✗	0.87	4.56	1.01	4.83	1.74	11.3	1.27	9.24
SAM + Adapter [ 18 ]	✗	0.84	4.15	0.93	4.26	1.49	10.4	0.62	6.86
SAM + FPN	✗	0.81	4.04	0.92	4.13	1.37	10.2	0.55	5.79
SAM + FPN	✓	0.77	3.98	0.85	4.05	1.29	9.81	0.49	5.25

TABLE IX: Ablation studies with different components on Feature extractor using the baseline losses (

\mathcal{L}_{photo}

and

\mathcal{L}_{s}

). Following [ 18 ] , we adopt a DPT decoder strategy to act as the adaptor.

ID	Data	KIT 2012		KIT 2015		Middle		ETH3D
ID	Augmentation	EPE	D1	EPE	D1	EPE	Bad 2.0	EPE	Bad 1.0
1	✗	0.77	3.98	0.85	4.05	1.29	9.81	0.49	5.25
2	Vanilla	0.81	4.23	0.90	4.36	1.96	13.1	0.81	9.65
3	Intermediate	0.74	3.75	0.83	3.87	1.11	9.10	0.41	4.58
4	Ours	0.68	3.01	0.76	3.10	0.95	8.14	0.28	2.93

TABLE X: Ablation study on different data augmentation strategies. “Vanilla” and “Intermediate” use the augmented pair as input in a single standard branch and the baseline losses (

\mathcal{L}_{photo}

and

\mathcal{L}_{s}

), where

\mathcal{L}_{photo}

uses augmented (Vanilla) or clean (Intermediate) image pair. In contrast, our method adopts an additional data augmentation branch and the proposed losses (

\mathcal{L}_{flc}

and

\mathcal{L}_{ild}

Fig. 12: Comparison with different pre-trained weights on KITTI 2012, KITTI 2015, Middlebury, and ETH3D datasets. D1 denotes KITTI (Bad 3.0), Middlebury (Bad 2.0), and ETH3D (Bad 1.0) metrics.

VFMs	KIT 2012		KIT 2015		Middle		ETH3D
VFMs	EPE	D1	EPE	D1	EPE	Bad 2.0	EPE	Bad 1.0
DINOV2 [ 13 ]	0.72	3.27	0.79	3.39	1.27	8.96	0.40	4.29
EVA2 [ 15 ]	0.71	3.30	0.81	3.46	1.30	9.21	0.35	4.10
DAM [ 48 ]	0.66	3.02	0.78	3.18	1.01	8.35	0.29	3.24
SAM [ 1 ]	0.68	3.01	0.76	3.10	0.95	8.14	0.28	2.93

TABLE XI: Performance comparison with the proposed SMFormer across multiple VFMs as backbones. We incorporate the ViT-Large architectures of VFMs into our framework while keeping the other proposed components and training strategy unchanged.

Occlusion Rate	KIT 2015		Middle
Occlusion Rate	EPE	D1	EPE	Bad 2.0
$\alpha$ = 0	0.84	3.86	1.19	9.45
$\alpha$ = 0.05	0.78	3.20	1.05	8.70
$\alpha$ = 0.15	0.76	3.10	0.95	8.14
$\alpha$ = 0.25	0.82	3.35	1.02	8.47

TABLE XII: Ablation study of the occlusion rate

\alpha

for the pixel level contrastive sample on the KITTI 2015 and Middlebury training sets.

Temperature	KIT 2015		Threshold	KIT 2015
Temperature	EPE	D1	Threshold	EPE	D1
$\tau$ = 1 (Uniform)	0.80	3.46	$\tau_{warp}$ = 1	0.79	3.35
$\tau$ = 0.1	0.77	3.18	$\tau_{warp}$ = 2	0.76	3.21
$\tau$ = 0.07	0.76	3.10	$\tau_{warp}$ = 3	0.76	3.10
$\tau$ = 0.01	0.79	3.37	$\tau_{warp}$ = 5	0.81	3.51

TABLE XIII: Ablation study of hyperparameters temperature

\tau

and the threshold value

\tau_{warp}

on the KITTI 2015 training sets.

Training Dataset	KIT 2012		KIT 2015
Training Dataset	EPE	D1	EPE	D1
KIT	0.68	3.01	0.76	3.10
KIT + VKITTI2	0.61	2.68	0.70	2.79
KIT + VKITTI2 + KITTI RAW	0.57	2.55	0.66	2.62

TABLE XIV: Performance Gains from Data Capacity. KIT, VKITTI2, and KITTI RAW denote the mixed KITTI 2012& KITTI 2015, Virtual KITTI 2, and KITTI RAW datasets, respectively.

VFM	VFM	Method	Training	Number of	Training
Model	Size	Method	Paradigm	Epochs	Time
SAM	ViT-Large	Baseline	S	200	6.0 h
	ViT-Large	Full	D	200	16.9 h
	ViT-Large	Ours	D	200	8.6 h

TABLE XV: The quantitative training time comparisons. “Full” denotes full-finetuning. ‘S’ denotes the single-branch baseline using vanilla photometric consistency loss, while ‘D’ denotes the dual-branch setting used in our training strategy.

Stereo Matching

Stereo matching estimates depth by finding corresponding points in paired images and measuring their disparity.

See all 57 papers

Aug 30, 2026

PhasorNet: Learning Structure from Frequency for Real-Time Stereo Matching

Accurate stereo matching remains challenging in ill-posed regions such as fine structures, reflective, or transparent objects, where appearance cues are often ambiguous or unreliable. To tackle this, we propose PhasorNet, a lightweight yet powerful framework that boosts geometric discrimination via frequency-domain cues. At its core, the Phase-Augmented Transformer (PAT) injects Fourier-derived phase information into the attention mechanism, yielding photometrically robust, structure-preserving features that prioritize structural consistency in difficult areas. Additionally, we develop a Geometry-Context Fusion Refinement Module (GCFRM) that combines a full-resolution convolutional stream with a lightweight attention-based stream (leveraging WQA and CDGA blocks) to efficiently preserve fine details and object boundaries without excessive overhead. Training is further enhanced by a multi-scale Edge-guided High-Error Region (EHR) loss that adaptively focuses optimization on high-error and edge regions, guiding hierarchical cost volume refinement. With only 5.3M parameters, PhasorNet achieves state-of-the-art performance on the challenging ETH3D benchmark while exhibiting excellent cross-domain generalization on KITTI, delivering an efficient and practical solution for accurate real-time stereo matching.

Md Raqib Khan, Santosh Kumar Vipparthi, Subrahmanyam Murala

CVPR Lab, Trinity College Dublin, The University of Dublin, Dublin, Ireland · CVPR Lab, Indian Institute of Technology Ropar, Rupnagar, Punjab, India

Frequency-Domain Feature Learning Stereo Matching

Ask PDF

Jul 10, 2026

Rethinking Monocular Depth Embedding for Generalized Stereo Matching

Generally, monocular methods capture rich contextual priors but lack geometric precision, whereas stereo methods are geometrically accurate yet struggle in textureless and occluded regions. Several approaches attempt to combine their strengths to enhance the generalization of stereo matching (SM) by aligning monocular depth with stereo information. However, establishing a stable and generalizable alignment is challenging, and unreliable monocular cues can substantially degrade performance. This paper rethinks monocular depth embedding. First, to prevent shortcut learning, we reduce branch coupling instead of expanding network width. Second, we construct soft constraints instead of hard ones from monocular depth to improve tolerance to monocular depth errors. Based on the principles, we integrate monocular information into both feature extraction and GRU iterations. Specifically, the monocular depth map is fused with the RGB image to sharpen depth boundary perception and suppress matching ambiguities. The fused image is then used for feature extraction, allowing the contextual features to encode global geometric information. Furthermore, the monocular depth gradient feature is employed to guide disparity updates, helping to escape local oscillations. Finally, to address the boundary blurring of supervised disparity caused by data augmentation, we propose an edge confidence estimation method and an edge-aware loss function. Our method achieves state-of-the-art (SOTA) performance on multiple standard benchmarks, demonstrating excellent generalization while improving accuracy. The code is available at https://github.com/linliboabc-maker/stereo-matching-digital.

Libo Lin, Shuangli Du, Minghua Zhao +3

Shaanxi Key Laboratory for Network Computing and Security Technology, Xi’an University of Technology, No. 5, Jinhua South Road, Xi’an, 710048, Shaanxi, China · College of Computer Science, Sichuan University, , Chengdu, 610065, Sichuan, China

Stereo Matching

Ask PDF

Nov 19, 2024

Motif Channel Opened in a White-Box: Stereo Matching via Motif Correlation Graph

Real-world applications of stereo matching, such as autonomous driving, place stringent demands on both safety and accuracy. However, learning-based stereo matching methods inherently suffer from the loss of geometric structures in certain feature channels, creating a bottleneck in achieving precise detail matching. Additionally, these methods lack interpretability due to the black-box nature of deep learning. In this paper, we propose MoCha-V2, a novel learning-based paradigm for stereo matching. MoCha-V2 introduces the Motif Correlation Graph (MCG) to capture recurring textures, which are referred to as ``motifs" within feature channels. These motifs reconstruct geometric structures and are learned in a more interpretable way. Subsequently, we integrate features from multiple frequency domains through wavelet inverse transformation. The resulting motif features are utilized to restore geometric structures in the stereo matching process. Experimental results demonstrate the effectiveness of MoCha-V2. MoCha-V2 achieved 1st place on the Middlebury benchmark at the time of its release. Code is available at https://github.com/ZYangChen/MoCha-Stereo.

Ziyang Chen, Yongjun Zhang, Wenting Li +3

College of Computer Science, the State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China · School of Information Engineering, Guizhou University of Commerce, Guiyang 550021, China · School of Software, Northwestern Polytechnical University, Xi’an, 710129, China +2

Representation Learning Stereo Matching

Ask PDF

Abstract

Stereo Matching

Explore similar work

PhasorNet: Learning Structure from Frequency for Real-Time Stereo Matching

Rethinking Monocular Depth Embedding for Generalized Stereo Matching

Motif Channel Opened in a White-Box: Stereo Matching via Motif Correlation Graph