EMCStereo: Attention-Enhanced Stereo Matching for Thin-Structure Depth Estimation with a Synthetic Tree-Branch Benchmark
Authors: Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green
Organizations: Centre for Data Science and Artificial Intelligence Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering University of Canterbury, Canterbury, New Zealand
Thin structures such as tree branches are among the hardest cases for stereo matching: a branch is only a few pixels wide, the background is cluttered, and dense ground truth for real branches is nearly impossible to label by hand. We make three contributions. First, EMCStereo integrates three lightweight attention modules into a PSMNet-style cost-volume backbone: Efficient Multi-scale Attention (EMA) on deep semantic features, a Multi-Scale Fusion block (MSFblock) learning spatial pyramid weights instead of concatenating them, and Coordinate Attention (CoordAtt) on final matching features. Because MSFblock collapses four pyramid branches into one, the modules leave the network 2.0% smaller and add only 1.7% inference time overhead. Second, VirtualTree is a synthetic stereo dataset rendered in Unreal Engine 5 with a simulated ZED Mini rig, providing 5,520 pairs with exact disparity for thin branches. Third, an eight-way ablation establishes a run-to-run noise floor of 0.009 px end-point error (EPE). EMCStereo achieves 1.31 px EPE (5.96% D1-all) on the VirtualTree test split, 1.00 px on SceneFlow, and 0.80, 0.73, 0.62, and 3.19 px on KITTI 2012, KITTI 2015, ETH3D, and Middlebury, with depth accuracy delta_1 from 92.6% to 98.7%. Evaluated against the noise floor, the attention stack is accuracy-neutral at a 100-epoch budget, while MSFblock and CoordAtt cost 0.03-0.05 px unless EMA is present.
Figures & tables
Fig. 1: EMCStereo architecture. A weight-shared Base CNN extracts 1/4 -resolution features from both images. Along the main (top) path the deep 128 -channel feature is re-weighted by EMA, the four spatial-pyramid branches are adaptively fused by MSFblock, and the final 32 -channel feature is refined by CoordAtt before the concatenation cost volume is built. The volume is regularized by a stacked-hourglass 3D network and decoded by soft-argmin into three deeply supervised multi-scale predictions. The coloured insets detail the three modules.
Setting
Value
Optimizer
AdamW ( β=0.9/0.999 , wd 10−4 )
Learning rate
5×10−4 , cosine, 500-iter warm-up
Precision / clipping
AMP; gradient clip 1.0 ; seed 0
Max disparity / crop
192 ; 256×512 ; batch 4
Colour augmentation
Jitter 0.4/0.4/0.4/0.16 , asymmetric p=0.2
Epochs (VirtualTree)
300, warm-started from a VirtualTree ckpt
TABLE I: Training configuration and measured cost for EMCStereo. Inference figures are fp32, batch 1, on an RTX 3060 Laptop GPU.
Split
EPE ↓
D1-all ↓
Bad 1.0 ↓
Bad 3.0 ↓
RMSE ↓
δ1↑
(px)
(%)
(%)
(%)
(px)
(%)
Validation
1.315
5.80
15.33
7.33
4.83
96.20
Test (held out)
1.308
5.96
15.16
7.31
4.71
96.11
TABLE II: EMCStereo on the two VirtualTree splits ( 552 pairs each). The validation split selects the checkpoint; the test split takes part in neither training nor selection.
EMA
MSF
Coord
Params
EPE ↓
D1-all ↓
Bad 1.0 ↓
RMSE ↓
(M)
(px)
(%)
(%)
(px)
5.225
1.749
8.01
19.73
5.346
✓
5.226
1.740
8.07
19.99
5.300
✓
5.120
1.777
8.10
19.87
5.401
✓
5.226
1.785
8.15
19.97
5.407
✓
✓
5.120
1.738
8.00
19.79
5.318
TABLE III: Ablation of the three attention modules on the VirtualTree validation split ( 552 pairs); lower is better. The eight upper rows share a fixed 100 -epoch budget from scratch and are mutually comparable; an independent repeat of the top row gave 1.740 px, so differences below 0.009 px EPE ( 0.03 pp D1-all, 0.11 pp Bad- 1.0 , 0.059 px RMSE) sit inside the noise floor. The last row is the deployed model of Table II , whose margin is a budget effect (Section VI-B ), not an architectural one.
Fig. 2: Four VirtualTree validation scenes, two per row; each scene shows the left RGB frame, the ground-truth disparity from the UE5 geometry buffer and the EMCStereo prediction, with the disparity panels sharing the JET map over [0,Dmax] . The top row holds two views with a near branch spanning much of the search range, the bottom row two canopy views dominated by thin twigs. Twigs stay connected and silhouettes stay sharp; the residual speckle sits in the sky and background foliage, which the metrics exclude.
Fig. 3: Real-world stereo pair (physical ZED Mini). Trained only on synthetic data, EMCStereo recovers the trunk, the branches and their depth order; the cluttered indoor background is far outside its training domain. Our map is colour-coded over 0 – 50 px and the reference uses its own normalization, so only structure is comparable.
Fig. 4: Two scenes from the held-out SceneFlow split: ground truth (left) against EMCStereo (right), JET-coloured over [0,Dmax] . Wire racks, chair legs and aircraft edges survive at close to their ground-truth width, and most of the residual error is in the thin occlusion bands the ground truth marks invalid.
Dataset
Pairs
EPE ↓
D1-all ↓
RMSE ↓
AbsRel ↓
δ1↑
(px)
(%)
(px)
(%)
SceneFlow
3,545
1.00
3.59
5.13
0.097
97.67
KITTI 2012
19
0.80
3.25
2.79
0.022
98.74
KITTI 2015
20
0.73
2.20
1.94
0.027
98.63
ETH3D
3
0.62
3.86
1.50
0.078
92.60
Middlebury
5
3.19
12.65
9.43
0.050
96.17
TABLE IV: Cross-dataset evaluation of EMCStereo trained on each benchmark’s public training set (all from scratch; ETH3D as a one-epoch fine-tune of the SceneFlow checkpoint). “Pairs” is the size of the 10% held-out split, which also supplies the early-stopping signal, so the metrics are selection-biased upper bounds rather than leaderboard results.
Method
SceneFlow
KITTI 2015
Params
EPE (px) ↓
D1-all (%) ↓
(M)
Published, official test splits
DispNetC [ 25 ]
1.68
4.34
42
MC-CNN-acrt [ 11 ]
—
3.88
—
GC-Net [ 5 ]
2.51
2.87
2.85
CRL [ 12 ]
1.32
2.67
78.77
TABLE V: Comparison with published stereo networks, ordered by KITTI 2015 D1-all. Accuracy is quoted from the cited work; parameter counts come from the cited work where it reports one, otherwise from a published comparison (DispNetC [ 18 ] , GwcNet and ACVNet [ 15 ] , DeepPruner-Best [ 19 ] ); “—” marks a number none of these sources gives. Our row is not a leaderboard entry and is printed at the precision we measured: † 10% held-out SceneFlow split ( 3,545 pairs); ‡ 20 -pair held-out KITTI 2015 split. Only the parameter column is comparable across the two blocks.
Centre for Data Science and Artificial Intelligence, Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering, University of Canterbury, Canterbury, New Zealand
School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China · Pika Labs · Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China +1
Centre for Data Science and Artificial Intelligence, Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering, University of Canterbury, Canterbury, New Zealand