EMCStereo: Attention-Enhanced Stereo Matching for Thin-Structure Depth Estimation with a Synthetic Tree-Branch Benchmark
Authors: Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green
Organizations: Centre for Data Science and Artificial Intelligence Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering University of Canterbury, Canterbury, New Zealand
Thin structures such as tree branches are among the hardest cases for stereo matching: a branch is only a few pixels wide, the background is cluttered, and dense ground truth for real branches is nearly impossible to label by hand. We make three contributions. First, EMCStereo integrates three lightweight attention modules into a PSMNet-style cost-volume backbone: Efficient Multi-scale Attention (EMA) on deep semantic features, a Multi-Scale Fusion block (MSFblock) learning spatial pyramid weights instead of concatenating them, and Coordinate Attention (CoordAtt) on final matching features. Because MSFblock collapses four pyramid branches into one, the modules leave the network 2.0% smaller and add only 1.7% inference time overhead. Second, VirtualTree is a synthetic stereo dataset rendered in Unreal Engine 5 with a simulated ZED Mini rig, providing 5,520 pairs with exact disparity for thin branches. Third, an eight-way ablation establishes a run-to-run noise floor of 0.009 px end-point error (EPE). EMCStereo achieves 1.31 px EPE (5.96% D1-all) on the VirtualTree test split, 1.00 px on SceneFlow, and 0.80, 0.73, 0.62, and 3.19 px on KITTI 2012, KITTI 2015, ETH3D, and Middlebury, with depth accuracy delta_1 from 92.6% to 98.7%. Evaluated against the noise floor, the attention stack is accuracy-neutral at a 100-epoch budget, while MSFblock and CoordAtt cost 0.03-0.05 px unless EMA is present.
Figures & tables
Fig. 1: EMCStereo architecture. A weight-shared Base CNN extracts 1/4 -resolution features from both images. Along the main (top) path the deep 128 -channel feature is re-weighted by EMA, the four spatial-pyramid branches are adaptively fused by MSFblock, and the final 32 -channel feature is refined by CoordAtt before the concatenation cost volume is built. The volume is regularized by a stacked-hourglass 3D network and decoded by soft-argmin into three deeply supervised multi-scale predictions. The coloured insets detail the three modules.
Setting
Value
Optimizer
AdamW ( β=0.9/0.999 , wd 10−4 )
Learning rate
5×10−4 , cosine, 500-iter warm-up
Precision / clipping
AMP; gradient clip 1.0 ; seed 0
Max disparity / crop
192 ; 256×512 ; batch 4
Colour augmentation
Jitter 0.4/0.4/0.4/0.16 , asymmetric p=0.2
Epochs (VirtualTree)
300, warm-started from a VirtualTree ckpt
TABLE I: Training configuration and measured cost for EMCStereo. Inference figures are fp32, batch 1, on an RTX 3060 Laptop GPU.
Split
EPE ↓
D1-all ↓
Bad 1.0 ↓
Bad 3.0 ↓
RMSE ↓
δ1↑
(px)
(%)
(%)
(%)
(px)
(%)
Validation
1.315
5.80
15.33
7.33
4.83
96.20
Test (held out)
1.308
5.96
15.16
7.31
4.71
96.11
TABLE II: EMCStereo on the two VirtualTree splits ( 552 pairs each). The validation split selects the checkpoint; the test split takes part in neither training nor selection.
EMA
MSF
Coord
Params
EPE ↓
D1-all ↓
Bad 1.0 ↓
RMSE ↓
(M)
(px)
(%)
(%)
(px)
5.225
1.749
8.01
19.73
5.346
✓
5.226
1.740
8.07
19.99
5.300
✓
5.120
1.777
8.10
19.87
5.401
✓
5.226
1.785
8.15
19.97
5.407
✓
✓
5.120
1.738
8.00
19.79
5.318
TABLE III: Ablation of the three attention modules on the VirtualTree validation split ( 552 pairs); lower is better. The eight upper rows share a fixed 100 -epoch budget from scratch and are mutually comparable; an independent repeat of the top row gave 1.740 px, so differences below 0.009 px EPE ( 0.03 pp D1-all, 0.11 pp Bad- 1.0 , 0.059 px RMSE) sit inside the noise floor. The last row is the deployed model of Table II , whose margin is a budget effect (Section VI-B ), not an architectural one.
Fig. 2: Four VirtualTree validation scenes, two per row; each scene shows the left RGB frame, the ground-truth disparity from the UE5 geometry buffer and the EMCStereo prediction, with the disparity panels sharing the JET map over [0,Dmax] . The top row holds two views with a near branch spanning much of the search range, the bottom row two canopy views dominated by thin twigs. Twigs stay connected and silhouettes stay sharp; the residual speckle sits in the sky and background foliage, which the metrics exclude.
Fig. 3: Real-world stereo pair (physical ZED Mini). Trained only on synthetic data, EMCStereo recovers the trunk, the branches and their depth order; the cluttered indoor background is far outside its training domain. Our map is colour-coded over 0 – 50 px and the reference uses its own normalization, so only structure is comparable.
Fig. 4: Two scenes from the held-out SceneFlow split: ground truth (left) against EMCStereo (right), JET-coloured over [0,Dmax] . Wire racks, chair legs and aircraft edges survive at close to their ground-truth width, and most of the residual error is in the thin occlusion bands the ground truth marks invalid.
Dataset
Pairs
EPE ↓
D1-all ↓
RMSE ↓
AbsRel ↓
δ1↑
(px)
(%)
(px)
(%)
SceneFlow
3,545
1.00
3.59
5.13
0.097
97.67
KITTI 2012
19
0.80
3.25
2.79
0.022
98.74
KITTI 2015
20
0.73
2.20
1.94
0.027
98.63
ETH3D
3
0.62
3.86
1.50
0.078
92.60
Middlebury
5
3.19
12.65
9.43
0.050
96.17
TABLE IV: Cross-dataset evaluation of EMCStereo trained on each benchmark’s public training set (all from scratch; ETH3D as a one-epoch fine-tune of the SceneFlow checkpoint). “Pairs” is the size of the 10% held-out split, which also supplies the early-stopping signal, so the metrics are selection-biased upper bounds rather than leaderboard results.
Method
SceneFlow
KITTI 2015
Params
EPE (px) ↓
D1-all (%) ↓
(M)
Published, official test splits
DispNetC [ 25 ]
1.68
4.34
42
MC-CNN-acrt [ 11 ]
—
3.88
—
GC-Net [ 5 ]
2.51
2.87
2.85
CRL [ 12 ]
1.32
2.67
78.77
TABLE V: Comparison with published stereo networks, ordered by KITTI 2015 D1-all. Accuracy is quoted from the cited work; parameter counts come from the cited work where it reports one, otherwise from a published comparison (DispNetC [ 18 ] , GwcNet and ACVNet [ 15 ] , DeepPruner-Best [ 19 ] ); “—” marks a number none of these sources gives. Our row is not a leaderboard entry and is printed at the precision we measured: † 10% held-out SceneFlow split ( 3,545 pairs); ‡ 20 -pair held-out KITTI 2015 split. Only the parameter column is comparable across the two blocks.
Autonomous drone-based tree pruning needs accurate, real-time depth estimation from stereo cameras. Depth is computed from disparity maps using Z=fB/d, so even small disparity errors cause noticeable depth mistakes at working distances. Building on our earlier work that identified DEFOM-Stereo as the best reference disparity generator for vegetation scenes, we present the first study to train and test ten deep stereo matching networks on real tree branch images. We use the Canterbury Tree Branches dataset -- 5,313 stereo pairs from a ZED Mini camera at 1080P and 720P -- with DEFOM-generated disparity maps as training targets. The ten methods cover step-by-step refinement, 3D convolution, edge-aware attention, and lightweight designs. Using perceptual metrics (SSIM, LPIPS, ViTScore) and structural metrics (SIFT/ORB feature matching), we find that BANet-3D produces the best overall quality (SSIM = 0.883, LPIPS = 0.157), while RAFT-Stereo scores highest on scene-level understanding (ViTScore = 0.799). Testing on an NVIDIA Jetson Orin Super (16 GB, independently powered) mounted on our drone shows that AnyNet reaches 6.99 FPS at 1080P -- the only near-real-time option -- while BANet-2D gives the best quality-speed balance at 1.21 FPS. We also compare 720P and 1080P processing times to guide resolution choices for forestry drone systems.
Yida Lin, Bing Xue, Mengjie Zhang +2
Centre for Data Science and Artificial Intelligence, Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering, University of Canterbury, Canterbury, New Zealand
Standard attention mechanisms are not well suited to stereo matching. Global attention scales quadratically and provides no explicit matching constraint, while local attention is efficient but loses long-range correspondences. We propose MatchAttention, an attention mechanism that embeds an explicit matching constraint into attention by treating the relative position between a query and its matched key as a learnable component of attention sampling. Centering a small contiguous sampling window on this learnable relative position enforces the matching constraint and supports long-range correspondence at strictly linear attention complexity. A differentiable contiguous attention sampling (CAS) operator enables sub-pixel accuracy, and cascaded MatchAttention blocks iteratively refine the relative positions through residual connections. We instantiate MatchAttention as a hierarchical coarse-to-fine stereo network with two variants. MatchAttentionXL targets accuracy and MatchAttentionRT targets real-time edge inference. MatchAttentionXL achieves state-of-the-art accuracy on Middlebury V3 and top results across KITTI 2012/2015 and ETH3D. MatchAttentionRT runs at 9.3 ms on RTX 4060 Ti and 79.1 ms on Jetson Orin NX 16 GB at 1024 x 512, making it the first stereo model to deliver real-time edge inference without sacrificing zero-shot generalization. The code is available at https://github.com/TingmanYan/MatchAttention.
Tingman Yan, Tao Liu, Chenghao Li +4
School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China · Pika Labs · Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China +1
Dense ground-truth disparity maps are practically unobtainable in forestry environments, where thin overlapping branches and complex canopy geometry defeat conventional depth sensors -- a critical bottleneck for training supervised stereo matching networks for autonomous UAV-based pruning. We present UE5-Forest, a photorealistic synthetic stereo dataset built entirely in Unreal Engine 5 (UE5). One hundred and fifteen photogrammetry-scanned trees from the Quixel Megascans library are placed in virtual scenes and captured by a simulated stereo rig whose intrinsics -- 63 mm baseline, 2.8 mm focal length, 3.84 mm sensor width -- replicate the ZED Mini camera mounted on our drone. Orbiting each tree at up to 2 m across three elevation bands (horizontal, +45 degrees, -45 degrees) yields 5,520 rectified 1920 x 1080 stereo pairs with pixel-perfect disparity labels. We provide a statistical characterisation of the dataset -- covering disparity distributions, scene diversity, and visual fidelity -- and a qualitative comparison with real-world Canterbury Tree Branches imagery that confirms the photorealistic quality and geometric plausibility of the rendered data. The dataset will be publicly released to provide the community with a ready-to-use benchmark and training resource for stereo-based forestry depth estimation.
Yida Lin, Bing Xue, Mengjie Zhang +2
Centre for Data Science and Artificial Intelligence, Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering, University of Canterbury, Canterbury, New Zealand