Beyond Row Alignment: Virtual-Camera-Aware Online Stereo Rectification

Yizhao Peng; Rui Gong; Zaiwang Gu; Xudong Jiang; Jun Cheng

cs.CVNov 7, 2026

Beyond Row Alignment: Virtual-Camera-Aware Online Stereo Rectification

Authors: Yizhao Peng, Rui Gong, Zaiwang Gu, Xudong Jiang, Jun Cheng

Organizations: Nanyang Technological University, Singapore · Institute of Advanced Intelligence and Computing, A*STAR, Singapore

Abstract

Stereo matchers estimate depth by comparing the left and right images from a stereo camera pair, but they assume that the two cameras are accurately calibrated and rectified. In real deployments, small physical shifts after calibration, caused by camera motion, mechanical drift, vibration, or installation changes, can break this assumption and degrade depth estimation. Online stereo rectification is therefore needed to correct the image pair during operation and recover a stereo geometry where standard stereo matching can work reliably. Recent online rectification methods mainly optimize row alignment, making the left and right views of the same scene point fall on the same image row. We show that this target, while necessary, is insufficient for deployable stereo depth estimation. A rectifier may make a small set of matched points look well row-aligned, but still distort the full images in ways that hurt stereo matching: useful image regions may be cut off, large blank areas may appear, objects may be stretched or squeezed, or image scale may change unevenly across the view. We propose VirtualRect, an online stereo rectification method designed for reliable real-world deployment. Rather than treating rectification as simply making the left and right image rows line up, this method estimates a rectified virtual stereo camera and combines image-wide supervision of image warp behavior. This makes the rectified images and the camera parameters come from the same geometry, so the images used for matching and the model used for depth computation remain consistent. Experiments on multiple datasets show that, compared with the prior state-of-the-art online rectifier, VirtualRect reduces vertical flow error by 17% on Carla-Flowguided and 7% on Semi-Truck Highway, and lowers depth MAE by 30-41% on Carla-Flowguided across three downstream stereo matchers.

Figures & tables

Figure 1: VirtualRect Online Rectification. The proposed framework consists of a semi-dense matcher, a solver-aware estimator, and a virtual stereo deployment module. The matcher takes the left and right images $\mathbf{I}_{l},\mathbf{I}_{r}$ as input and outputs correspondences. The solver-aware estimator samples and scores for rectification hypotheses, predicts correspondence reliability, and solves for pre-deployment rectifying homographies $\mathbf{H}^{\mathrm{pre}}_{l},\mathbf{H}^{\mathrm{pre}}_{r}$ . The virtual-camera-aware adaptor then predicts a shared rectified camera $\mathbf{K}^{*}$ and deploys the final homographies $\mathbf{H}^{*}_{l},\mathbf{H}^{*}_{r}$ with $\mathbf{K}^{*}$ for stereo matching.

Figure 2: Qualitative downstream stereo disparity comparison by different online rectification methods on Semi-Truck Highway and Carla-Flowguided. Colored panels are disparity maps; the rightmost reference uses offline-calibration rectification.

Dataset	Method	SIFT Offset $\downarrow$	SuperGlue Offset $\downarrow$	Vertical Flow $\downarrow$
		(pixels)	(pixels)	(pixels)
Semi-Truck Highway	SIFT + MAGSAC (2017) [ 11 , 18 ]	21.10	1.15	1.37
	SP+SG + MAGSAC (2019) [ 19 , 20 , 18 ]	14.97	1.20	1.51
	LoFTR + MAGSAC (2021) [ 21 , 18 ]	14.94	0.76	0.86
	EfficientLoFTR + MAGSAC (2024) [ 22 , 18 ]	8.07	0.73	0.82
	DirectionNet (2021) [ 27 ]	8.41	6.10	6.44

Table 1: Stereo rectification accuracy. We follow the keypoint-offset and vertical flow evaluation convention on Semi-Truck Highway and Carla-Flowguided. SIFT and SuperGlue offsets are independent sparse matcher checks. The VirtualRect row is the mean over five seeds. Arrows indicate that lower values are better.

Method	Monster++			DLNR			HITNet			Time
Method	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	Time
SIFT + MAGSAC [ 11 , 18 ]	2.03	5.12	87.05	2.52	6.49	88.24	2.67	9.45	88.21	367
SP+SG + MAGSAC [ 19 , 20 , 18 ]	8.58	30.22	71.67	9.89	31.14	72.69	10.57	38.04	72.76	98
LoFTR + MAGSAC [ 21 , 18 ]	3.60	9.90	86.25	4.13	11.10	87.45	4.22	14.31	87.41	237
EfficientLoFTR + MAGSAC [ 22 , 18 ]	4.86	13.65	84.55	5.50	14.76	85.76	5.61	18.21	85.75	163
FAR [ 49 ]	3.85	9.94	86.71	4.35	11.03	87.94	4.37	13.81	87.61	1672

Table 2: Downstream stereo checks and disparity coverage on Carla-Flowguided. Each rectified input is evaluated by three stereo matchers. For each matcher, Cov is Eq. ( 15 ). MAE is in meters, AbsRel and Cov are percentages, and Time reports the rectifier runtime only in milliseconds per stereo pair.

Method	Carla-Flowguided					Perturbed SceneFlow
Method	RE $\downarrow$	$R_{x}$ $\downarrow$	$R_{y}$ $\downarrow$	$R_{z}$ $\downarrow$	TE $\downarrow$	RE $\downarrow$	$R_{x}$ $\downarrow$	$R_{y}$ $\downarrow$	$R_{z}$ $\downarrow$	TE $\downarrow$
EfficientLoFTR + MAGSAC [ 22 , 18 ]	0.208	0.007	0.205	0.018	2.736	0.263	0.014	0.254	0.035	1.385
FAR [ 49 ]	0.157	0.007	0.153	0.017	2.727	0.203	0.011	0.199	0.021	1.290
RsCeR [ 10 ]	0.081	0.003	0.077	0.006	0.860	0.152	0.008	0.147	0.018	0.678
VirtualRect (Ours)	0.021	0.001	0.016	0.001	0.258	0.115	0.005	0.111	0.014	0.460

Table 3: Pose diagnostics. These report how well the pre-gauge geometry recovers the row-alignment pose. RE is the overall rotation angular error,

R_{x}

R_{y}

, and

R_{z}

are absolute axis-angle components, and TE is the translation-direction angular error; all are in degrees. Lower is better.

Method	SIFT Offset $\downarrow$	SuperGlue Offset $\downarrow$	RAFT Vertical Flow $\downarrow$
FloSR [ 9 ]	14.00	5.24	10.65
RsCeR [ 10 ]	3.76	0.43	0.62
VirtualRect (Ours)	3.10	0.42	0.57

Table 4: Carla-trained, zero-shot on Semi-Truck. Models are trained on Carla-Flowguided and evaluated on Semi-Truck Highway without fine-tuning. SIFT and SuperGlue report keypoint offsets, and RAFT reports vertical flow. All errors are in pixels; lower is better.

Method	Temporal state	VOF-I (px) $\downarrow$	DC-I (m) $\downarrow$
TESO [ 50 ]	Yes	0.07	0.51
VirtualRect (Ours)	No	0.02	0.25

Table 5: Zero-shot on Carla-Drift, a temporally continuous dataset. Results on all 155 sequences (155,000 stereo pairs), with fixed stereo translation. Lower is better.

Supervision	$\mathcal{L}_{\rm row}$	$\mathcal{L}_{H}$	$\mathcal{L}_{\rm risk}$	Vertical Flow $\downarrow$	RE $\downarrow$	TE $\downarrow$
Row alignment	$\checkmark$	–	–	0.61	0.036	0.432
+ behavior supervision	$\checkmark$	$\checkmark$	–	0.48	0.025	0.288
+ inlier calibration	$\checkmark$	$\checkmark$	$\checkmark$	0.43	0.021	0.258

Table 6: Stage-1 supervision ablation on Carla-Flowguided under final deployment. All variants use the same gauge/

\mathbf{K}^{\!*}

deployment. RE and TE are angular errors in degrees. Lower is better.

RsCeR			VirtualRect
Deployment	MAE $\downarrow$	AbsRel (%) $\downarrow$	Deployment	MAE $\downarrow$	AbsRel (%) $\downarrow$
pre-gauge + old $\mathbf{K}$	2.23	5.38	pre-gauge + old $\mathbf{K}$	1.61	3.86
post-gauge + old $\mathbf{K}$	3.21	9.25	post-gauge + old $\mathbf{K}$	3.00	9.03
post-gauge + $\mathbf{K}^{\!*}$	2.14	4.86	post-gauge + $\mathbf{K}^{\!*}$	1.50	3.31

Table 7: Virtual-camera deployment ablation on Carla-Flowguided with DLNR. Pre-gauge is the solver output; post-gauge uses the deployed virtual-camera warp. MAE is in meters and AbsRel is in percent. Lower is better.

Stereo Vision

Stereo vision uses images from two or more viewpoints to estimate depth, recover 3D structure, and support spatial perception.

See all 44 papers

Mar 25, 2026

WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching

We introduce WAFT-Stereo, a simple and effective warping-based method for stereo matching. WAFT-Stereo demonstrates that cost volumes, a common design used in many leading methods, are not necessary for strong performance and can be replaced by warping with improved efficiency. WAFT-Stereo ranks first on ETH3D (BP-0.5), Middlebury (RMSE), and KITTI (all metrics), reducing the zero-shot error by 81% on ETH3D, while being 1.8-6.7x faster than competitive methods. Code and model weights are available at https://github.com/princeton-vl/WAFT-Stereo.

Yihan Wang, Jia Deng

Department of Computer Science, Princeton University

Stereo Matching Stereo Vision Stereo Depth Estimation

Ask PDF

May 14, 2026

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors

Stereo matching on top-bottom equirectangular images provides an effective framework for full-surround perception, as vertically aligned epipolar lines enable the use of advanced perspective stereo architectures that are largely driven by large-scale datasets and monocular priors. However, the performance of such adaptations is severely limited by the scarcity of omnidirectional stereo datasets and the degradation of perspective monocular priors under spherical distortions. To address these challenges, we propose H-OmniStereo, a zero-shot omnidirectional stereo matching framework. First, we construct high-quality synthetic dataset comprising over 2.8 million top-bottom equirectangular stereo pairs to scale up training. Second, we introduce an equirectangular monocular normal estimator, specifically operating in a heading-aligned coordinate system. Beyond providing distortion-robust and cross-view-consistent geometric priors for establishing reliable correspondences in stereo matching, this design boosts training efficiency and accommodates train-test FoV mismatches. Extensive experiments show that our approach achieves higher accuracy than existing methods on out-of-domain datasets and successfully generalizes to real-world consumer camera setups using a single model. The model and dataset will be released at https://github.com/JIANG-CX/H-OmniStereo.

Chenxing Jiang, Zhe Tong, Pusen Gao +5

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China

Omnidirectional Vision Stereo Matching

Ask PDF

Jun 24, 2026

LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching

Existing Vision Foundation Model (VFM)-based iterative stereo pipelines under-exploit three information pathways: multi-scale backbone features are collapsed into single-level correlations, geometric priors remain untapped at initialization, and context propagates only locally. These gaps widen under degraded photometric cues, making underwater scenes a stringent generalization test. To address this, we propose LinStereo, built upon Depth Anything V3, whose core is a Position-Aware Linear Attention (PALA) module that replaces local recurrence with global aggregation at linear cost, propagating reliable estimates from well-matched regions into degraded areas while preserving disparity structure. PALA is made effective by two enabling components: Hierarchical Semantic Cost Volumes (HSCV), which supply scale-aligned correlations from the VFM feature hierarchy, and a Depth Prior Initialization (DPI) that converts monocular depth into a metrically calibrated warm start. LinStereo achieves state-of-the-art-level accuracy on standard benchmarks and strong cross-domain generalization, particularly on underwater scene where severe photometric degradation makes stereo matching particularly challenging, attaining the best overall accuracy with consistent gains 28% lower AbsRel on TartanAir-UW, 26% on SQUID, a real-world underwater dataset).

Yiran Wang, Oliver Turner, Viorela Ila

Australian Centre for Robotics (ACFR), School of Aerospace, Mechanical and Mechatronic Engineering (AMME), Faculty of Engineering, The University of Sydney, Sydney, NSW, Australia

Multi-Scale Feature Fusion Stereo Matching Linear Attention Stereo Depth Estimation

Ask PDF

Abstract

Stereo Vision

Explore similar work

WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors

LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching