cs.CVNov 6, 2026
SaveLeft-Right-Consistent Iterative Stereo Matching via Cross-Injection
Organizations: Meta Reality Labs
Abstract
State-of-the-art stereo matching models predominantly predict the disparity of only the left (reference) view. However, applications such as per-eye AR/VR rendering, novel view synthesis, or two-view depth fusion require disparity for both views, and need those two maps to agree. The usual workaround of running the network a second time on the horizontally flipped and swapped pair gives two independent predictions that can be inconsistent, especially on transparent surfaces, in textureless regions, and at occlusion boundaries. We introduce cross-injection (CI), a lightweight extension for iterative (refinement-based) models that can be added to any pretrained left-only iterative matcher. It couples the two passes by injecting the explicit, signed left-right inconsistency as an additional input cue at every step of the refinement stage. We empirically show that CI substantially reduces left-right disagreement on every benchmark (more than halving it on Booster for FoundationStereo), while largely preserving sharp object boundaries. Furthermore, using two state-of-the-art stereo matching models, we show that CI also reduces the single-view disparity error, making it a worthwhile extension even if only left disparity is needed.
Figures & tables
Figure 1: Sample predictions with and without our proposed cross-injection (CI) extension. CI can be added to any iterative stereo matcher (here FoundationStereo [ 56 ] ) and substantially improves left-right consistency between two otherwise independent inference passes (left example); it also reduces per-view failure modes (right example). Here, we show two examples from the Booster dataset [ 39 ] where inconsistent predictions arise mainly on reflective and transparent surfaces. Additional visualizations are shown in Figs. 4 , 5 and 6 .
Figure 2: Overview of the proposed cross-injection architecture. Two refinement passes (the left pass and the flipped-swapped right pass), whose feature encoders and GRU share weights, each iteratively update their own disparity. At every step, each pass receives its own masked left-right inconsistency with respect to the other view (Fig. 3 , Eq. 2 ) as an additional input cue, injected through a zero-initialized encoder. Because the encoder emits zero at initialization, augmenting an already-trained refiner with cross-injection reproduces its original (uncoupled) behavior exactly, and the cross-view correction is learned by fine-tuning.
Figure 3: Illustration of the proposed cross-injection mechanism. The detached partner prediction is horizontally flipped and warped into the frame of the own prediction, using the own disparity as the shift field (Eq. 2 ), and subtracted from the own prediction. The resulting signed inconsistency (“consistency vector”) is passed explicitly to the refiner’s GRU block. We do this with a tensor of shape : the two channels are (i) the signed inconsistency with zeros at non-co-visible pixels, and (ii) the binary co-visibility mask derived from warping. This way, the model can distinguish non-visibility from perfect consistency.
Figure 4: FoundationStereo zero-shot predictions on selected ETH3D [ 42 ] samples with and without our cross-injection method. Even though we are not addressing the ambiguity for transparent objects directly, left/right predictions are more consistent in which visible layer to predict (a). The CI model also successfully corrects major (b) and minor (c) disagreements between the views. Background shrubs in (b) lose some degree of detail.
Figure 5: FoundationStereo zero-shot predictions on selected Booster [ 39 ] (half resolution) samples with and without our cross-injection method. CI corrects errors caused by reflections (a), transparency (b+c) and other random failure modes (d).
Figure 6: FoundationStereo zero-shot predictions on selected KITTI-15 [ 35 ] samples with and without our cross-injection method. (a) shows occasional “smearing” effects on thin vertical structures such as traffic signs or street lamps in front of the sky. Interestingly, this failure mode was already present in FoundationStereo, but our method seems to amplify it. (b) shows an example where both models predict equally sharp edges on road signs and high frequency details (vegetation).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| FS | S 2 M 2 -L | S 2 M 2 -XL | |
| Optimizer | AdamW, , wd | ||
| Peak LR | |||
| LR schedule | step-drop | cosine | cosine |
| Optimizer updates | k | k | k |
| Precision | bf16 | fp32 | fp32 |
| GPUs ( bs/GPU) | |||
Table S.1 : Fine-tuning configuration for the models in Tab. 2 . Each column applies to both the cross-injection run and its matched no-CI control, which differ only in the cross-injection refiner and the left-right consistency loss. All runs start from the public left-only checkpoint with the cross-injection pathway zero-initialized and fine-tune the full network on crops. The S 2 M 2 cosine is defined over exactly the number of updates performed, so the learning rate anneals and the final step is the reported one.
| Model | 1 pass | 2 passes | + CI | CI overhead | |
|---|---|---|---|---|---|
| (ms) | (ms) | (ms) | vs. 2 passes | vs. 1 pass | |
| FoundationStereo ( iters) | 447 | 896 | 1111 | ||
| S 2 M 2 -L ( iters) | 141 | 283 | 298 | ||
| S 2 M 2 -XL ( iters) | 240 | 480 | 498 | ||
Table S.2 : What cross-injection costs at inference, and what that is a fraction of. Wall-clock at in fp32 (the frozen monocular branch stays fp16 ), batch 1, on one A100-SXM4-80GB; median of 36 CUDA-event-timed runs per configuration, the configurations interleaved round-robin so that GPU clock drift cannot be mistaken for overhead. 1 pass predicts left disparity only. 2 passes is the standard way to obtain both views (two weight-shared, uncoupled passes) and is the baseline cross-injection should be judged against, because any both-view predictor pays it: the second pass costs what the first one does ( ). Cross-injection is then charged per refinement step, which is why the same mechanism costs five to seven times less, relative to the two-pass baseline, on models that iterate three times instead of .
| Stage (ms) | 1 pass | 2 passes | + CI |
| Feature extraction (both images) | 54.6 | 111.7 | 113.4 |
| stem | 0.311 | 0.610 | 0.613 |
| Cost-volume construction | 22.4 | 48.3 | 49.3 |
| Cost-volume filtering (3D UNet) | 92.5 | 185.0 | 184.7 |
| Initial disparity decode | 0.125 | 0.250 | 0.251 |
| Geometry-encoding pyramid | 1.97 | 3.94 | 3.92 |
Table S.3 : Inference time breakdown. Per-stage GPU time for FoundationStereo at refinement iterations, under the conditions of Tab. S.2 ; CUDA events are recorded inline and resolved after a single synchronization, so no stage boundary stalls the stream. Columns are the three configurations of Tab. S.2 . Refinement accounts for most of a pass, which is why a per-step cost is material for this model. Per iteration and per view, the cue costs 0.33 ms of warp and 1.60 ms of co-visibility mask, against 7.0 ms for one GRU update: the injected cue is a cheap tensor operation, not a second network, and the mask, not the warp, is what it costs. The sum of stages falls – short of the independently measured end-to-end time, which bounds the unattributed time.
| Model | Latency | Middlebury-H | ETH3D | ||
|---|---|---|---|---|---|
| (ms) | BP-2 | LR-BP-1 | BP-2 | LR-BP-1 | |
| FoundationStereo ( ) | 469 | 3.43 | 10.54 | 0.50 | 4.98 |
| FoundationStereo ( ) | 514 | 3.12 | 10.13 | 0.44 | 4.92 |
| FoundationStereo ( ) | 635 | 2.91 | 10.13 | 0.44 | 5.49 |
| FoundationStereo ( ) | 780 | 2.84 | 10.13 | 0.43 | 5.57 |
| FoundationStereo ( ) | 882 | 2.77 | 9.89 | 0.43 | 5.59 |
Table S.4 : Accuracy against latency. Latency is measured at as in Tab. S.2 , accuracy at each benchmark’s native resolution. Cross-injection at is better than the baseline at on every column shown, at a lower latency : the extra per-step cost is more than repaid by needing fewer steps. On the columns shared with Tab. 2 , the rows reproduce it; the CI row differs by at most because it is evaluated in fp32 here.
Figure S.1 : Cross-injection trained from scratch. A matched pair trained end to end for k steps and differing only in the cross-injection refiner and the consistency loss, shown on four benchmarks at all evaluated steps. Color is the benchmark and line style is the arm ( solid with cross-injection, dashed its matched control), so each solid/dashed pair of one color is a controlled comparison. The left panel reports each benchmark’s own disparity metric (legend), the right panel LR-BP-1 throughout. The dotted line marks the learning-rate drop at k and the shaded band the final k. Right: on every benchmark the solid curve lies below its dashed control at every step. Left: cross-injection starts above its control and closes the gap over the run. Vertical axes are logarithmic. These two runs are comparable only to each other.
| Middlebury-H | Booster-H | |||||
|---|---|---|---|---|---|---|
| Region | share % | BP-2 | noise | share % | BP-2 | noise |
| all valid pixels | 100.0 | 100.0 | ||||
| textureless | 22.4 | 72.9 | ||||
| depth edges | 9.7 | 0.9 | ||||
| occluded | 6.7 | 1.9 | ||||
| out of view | 4.0 | 5.6 | ||||
Table S.7 : Single-view error change by region. Change in BP-2 (percentage points) for the full method against its matched control (rows (e) and (a) of Tab. 1 ) as the mean over the same six checkpoints; negative is better. Share is the fraction of valid-ground-truth pixels the region covers, exact over the full evaluation set. Noise is the same comparison between control checkpoints, which contain no cross-injection. Bold marks effects at least twice the noise. The rows overlap and do not sum to the all-valid-pixels row.
| Pixel group | share % | control | CI | noise | |
|---|---|---|---|---|---|
| Middlebury-H | |||||
| consistent ( px) | 96.8 | ||||
| inconsistent | 3.2 | ||||
| Booster-H | |||||
| consistent ( px) | 94.7 | ||||
| inconsistent | 5.3 | ||||
Table S.8 : Where does cross-injection change the error? Non-occluded valid left-view pixels, split by whether the control’s left and right predictions agree with each other ( px), for rows (e) and (a) of Tab. 1 , pooled over the six checkpoint pairs. Values are mean absolute disparity error in pixels, for the control and under cross-injection, and their difference ( CI minus control; negative is better). Noise is the same comparison between control checkpoints, which contain no cross-injection. Because the groups are defined by the control’s predictions, is biased by the grouping; the text tests this.
More in this topic
Stereo Matching
Stereo matching estimates depth by finding corresponding points in paired images and measuring their disparity.
See all 62 papers