Beyond Row Alignment: Virtual-Camera-Aware Online Stereo Rectification

Yizhao Peng; Rui Gong; Zaiwang Gu; Xudong Jiang; Jun Cheng

cs.CVNov 7, 2026

Beyond Row Alignment: Virtual-Camera-Aware Online Stereo Rectification

Authors: Yizhao Peng, Rui Gong, Zaiwang Gu, Xudong Jiang, Jun Cheng

Organizations: Nanyang Technological University, Singapore · Institute of Advanced Intelligence and Computing, A*STAR, Singapore

Abstract

Stereo matchers estimate depth by comparing the left and right images from a stereo camera pair, but they assume that the two cameras are accurately calibrated and rectified. In real deployments, small physical shifts after calibration, caused by camera motion, mechanical drift, vibration, or installation changes, can break this assumption and degrade depth estimation. Online stereo rectification is therefore needed to correct the image pair during operation and recover a stereo geometry where standard stereo matching can work reliably. Recent online rectification methods mainly optimize row alignment, making the left and right views of the same scene point fall on the same image row. We show that this target, while necessary, is insufficient for deployable stereo depth estimation. A rectifier may make a small set of matched points look well row-aligned, but still distort the full images in ways that hurt stereo matching: useful image regions may be cut off, large blank areas may appear, objects may be stretched or squeezed, or image scale may change unevenly across the view. We propose VirtualRect, an online stereo rectification method designed for reliable real-world deployment. Rather than treating rectification as simply making the left and right image rows line up, this method estimates a rectified virtual stereo camera and combines image-wide supervision of image warp behavior. This makes the rectified images and the camera parameters come from the same geometry, so the images used for matching and the model used for depth computation remain consistent. Experiments on multiple datasets show that, compared with the prior state-of-the-art online rectifier, VirtualRect reduces vertical flow error by 17% on Carla-Flowguided and 7% on Semi-Truck Highway, and lowers depth MAE by 30-41% on Carla-Flowguided across three downstream stereo matchers.

Figures & tables

Figure 1: VirtualRect Online Rectification. The proposed framework consists of a semi-dense matcher, a solver-aware estimator, and a virtual stereo deployment module. The matcher takes the left and right images $\mathbf{I}_{l},\mathbf{I}_{r}$ as input and outputs correspondences. The solver-aware estimator samples and scores for rectification hypotheses, predicts correspondence reliability, and solves for pre-deployment rectifying homographies $\mathbf{H}^{\mathrm{pre}}_{l},\mathbf{H}^{\mathrm{pre}}_{r}$ . The virtual-camera-aware adaptor then predicts a shared rectified camera $\mathbf{K}^{*}$ and deploys the final homographies $\mathbf{H}^{*}_{l},\mathbf{H}^{*}_{r}$ with $\mathbf{K}^{*}$ for stereo matching.

Figure 2: Qualitative downstream stereo disparity comparison by different online rectification methods on Semi-Truck Highway and Carla-Flowguided. Colored panels are disparity maps; the rightmost reference uses offline-calibration rectification.

Dataset	Method	SIFT Offset $\downarrow$	SuperGlue Offset $\downarrow$	Vertical Flow $\downarrow$
		(pixels)	(pixels)	(pixels)
Semi-Truck Highway	SIFT + MAGSAC (2017) [ 11 , 18 ]	21.10	1.15	1.37
	SP+SG + MAGSAC (2019) [ 19 , 20 , 18 ]	14.97	1.20	1.51
	LoFTR + MAGSAC (2021) [ 21 , 18 ]	14.94	0.76	0.86
	EfficientLoFTR + MAGSAC (2024) [ 22 , 18 ]	8.07	0.73	0.82
	DirectionNet (2021) [ 27 ]	8.41	6.10	6.44

Table 1: Stereo rectification accuracy. We follow the keypoint-offset and vertical flow evaluation convention on Semi-Truck Highway and Carla-Flowguided. SIFT and SuperGlue offsets are independent sparse matcher checks. The VirtualRect row is the mean over five seeds. Arrows indicate that lower values are better.

Method	Monster++			DLNR			HITNet			Time
Method	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	MAE $\downarrow$	AbsRel $\downarrow$	Cov $\uparrow$	Time
SIFT + MAGSAC [ 11 , 18 ]	2.03	5.12	87.05	2.52	6.49	88.24	2.67	9.45	88.21	367
SP+SG + MAGSAC [ 19 , 20 , 18 ]	8.58	30.22	71.67	9.89	31.14	72.69	10.57	38.04	72.76	98
LoFTR + MAGSAC [ 21 , 18 ]	3.60	9.90	86.25	4.13	11.10	87.45	4.22	14.31	87.41	237
EfficientLoFTR + MAGSAC [ 22 , 18 ]	4.86	13.65	84.55	5.50	14.76	85.76	5.61	18.21	85.75	163
FAR [ 49 ]	3.85	9.94	86.71	4.35	11.03	87.94	4.37	13.81	87.61	1672

Table 2: Downstream stereo checks and disparity coverage on Carla-Flowguided. Each rectified input is evaluated by three stereo matchers. For each matcher, Cov is Eq. ( 15 ). MAE is in meters, AbsRel and Cov are percentages, and Time reports the rectifier runtime only in milliseconds per stereo pair.

Method	Carla-Flowguided					Perturbed SceneFlow
Method	RE $\downarrow$	$R_{x}$ $\downarrow$	$R_{y}$ $\downarrow$	$R_{z}$ $\downarrow$	TE $\downarrow$	RE $\downarrow$	$R_{x}$ $\downarrow$	$R_{y}$ $\downarrow$	$R_{z}$ $\downarrow$	TE $\downarrow$
EfficientLoFTR + MAGSAC [ 22 , 18 ]	0.208	0.007	0.205	0.018	2.736	0.263	0.014	0.254	0.035	1.385
FAR [ 49 ]	0.157	0.007	0.153	0.017	2.727	0.203	0.011	0.199	0.021	1.290
RsCeR [ 10 ]	0.081	0.003	0.077	0.006	0.860	0.152	0.008	0.147	0.018	0.678
VirtualRect (Ours)	0.021	0.001	0.016	0.001	0.258	0.115	0.005	0.111	0.014	0.460

Table 3: Pose diagnostics. These report how well the pre-gauge geometry recovers the row-alignment pose. RE is the overall rotation angular error,

R_{x}

R_{y}

, and

R_{z}

are absolute axis-angle components, and TE is the translation-direction angular error; all are in degrees. Lower is better.

Method	SIFT Offset $\downarrow$	SuperGlue Offset $\downarrow$	RAFT Vertical Flow $\downarrow$
FloSR [ 9 ]	14.00	5.24	10.65
RsCeR [ 10 ]	3.76	0.43	0.62
VirtualRect (Ours)	3.10	0.42	0.57

Table 4: Carla-trained, zero-shot on Semi-Truck. Models are trained on Carla-Flowguided and evaluated on Semi-Truck Highway without fine-tuning. SIFT and SuperGlue report keypoint offsets, and RAFT reports vertical flow. All errors are in pixels; lower is better.

Method	Temporal state	VOF-I (px) $\downarrow$	DC-I (m) $\downarrow$
TESO [ 50 ]	Yes	0.07	0.51
VirtualRect (Ours)	No	0.02	0.25

Table 5: Zero-shot on Carla-Drift, a temporally continuous dataset. Results on all 155 sequences (155,000 stereo pairs), with fixed stereo translation. Lower is better.

Supervision	$\mathcal{L}_{\rm row}$	$\mathcal{L}_{H}$	$\mathcal{L}_{\rm risk}$	Vertical Flow $\downarrow$	RE $\downarrow$	TE $\downarrow$
Row alignment	$\checkmark$	–	–	0.61	0.036	0.432
+ behavior supervision	$\checkmark$	$\checkmark$	–	0.48	0.025	0.288
+ inlier calibration	$\checkmark$	$\checkmark$	$\checkmark$	0.43	0.021	0.258

Table 6: Stage-1 supervision ablation on Carla-Flowguided under final deployment. All variants use the same gauge/

\mathbf{K}^{\!*}

deployment. RE and TE are angular errors in degrees. Lower is better.

RsCeR			VirtualRect
Deployment	MAE $\downarrow$	AbsRel (%) $\downarrow$	Deployment	MAE $\downarrow$	AbsRel (%) $\downarrow$
pre-gauge + old $\mathbf{K}$	2.23	5.38	pre-gauge + old $\mathbf{K}$	1.61	3.86
post-gauge + old $\mathbf{K}$	3.21	9.25	post-gauge + old $\mathbf{K}$	3.00	9.03
post-gauge + $\mathbf{K}^{\!*}$	2.14	4.86	post-gauge + $\mathbf{K}^{\!*}$	1.50	3.31

Table 7: Virtual-camera deployment ablation on Carla-Flowguided with DLNR. Pre-gauge is the solver output; post-gauge uses the deployed virtual-camera warp. MAE is in meters and AbsRel is in percent. Lower is better.

Stereo Vision

Stereo vision uses images from two or more viewpoints to estimate depth, recover 3D structure, and support spatial perception.

See all 44 papers

WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching
Mar 25, 2026Yihan Wang, Jia DengStereo Matching Stereo Vision
Department of Computer Science, Princeton University
H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors
May 14, 2026Chenxing Jiang, Zhe Tong, Pusen Gao +5Omnidirectional Vision Stereo Matching
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China
LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching
Jun 24, 2026Yiran Wang, Oliver Turner, Viorela IlaMulti-Scale Feature Fusion Stereo Matching
Australian Centre for Robotics (ACFR), School of Aerospace, Mechanical and Mechatronic Engineering (AMME), Faculty of Engineering, The University of Sydney, Sydney, NSW, Australia

Mar 25, 2026J/K move · Enter open · S save

WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching

Yihan Wang, Jia Deng

Department of Computer Science, Princeton University

Stereo Matching Stereo Vision Stereo Depth Estimation

Open paper AskPDF arXiv

We introduce WAFT-Stereo, a simple and effective warping-based method for stereo matching. WAFT-Stereo demonstrates that cost volumes, a common design used in many leading methods, are not necessary for strong performance and can be replaced by warping with improved efficiency. WAFT-Stereo ranks first on ETH3D (BP-0.5), Middlebury (RMSE), and KITTI (all metrics), reducing the zero-shot error by 81% on ETH3D, while being 1.8-6.7x faster than competitive methods. Code and model weights are available at https://github.com/princeton-vl/WAFT-Stereo.

Abstract

Stereo Vision

Explore similar work

WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors

LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching