Organizations: Department of Biomedical and Electrical Engineering, Marshall University, Huntington, WV 25755, USA. · Department of Computer Science, Marshall University, Huntington, WV 25755, USA. · Department of Civil Engineering, Marshall University, Huntington, WV 25755, USA.
This paper presents a depth-enhanced YOLO-SAM2 framework for detecting ballast insufficiency in railway tracks using RGB-D data. Although YOLOv8 provides reliable localization, the RGB-only model shows limited safety performance, achieving high precision (0.99) but low recall (0.49) due to insufficient ballast, as it tends to over-predict the sufficient class. To improve reliability, we incorporate depth-based geometric analysis enabled by a sleeper-aligned depth-correction pipeline that compensates for RealSense spatial distortion using polynomial modeling, RANSAC, and temporal smoothing. SAM2 segmentation further refines region-of-interest masks, enabling accurate extraction of sleeper and ballast profiles for geometric classification. Experiments on field-collected top-down RGB-D data show that depth-enhanced configurations substantially improve the detection of insufficient ballast. Depending on bounding-box sampling (AABB or RBB) and geometric criteria, recall increases from 0.49 to as high as 0.80, and F1-score improves from 0.66 to over 0.80. These results demonstrate that integrating depth correction with YOLO-SAM2 yields a more robust and reliable approach for automated railway ballast inspection, particularly in visually ambiguous or safety-critical scenarios.
Figures & tables
Fig. 1: RGB frame (left) and corresponding raw depth map (right) captured by the RealSense D435.
Fig. 2: Flowchart of the proposed ballast sufficiency detection algorithm based on RGB–D sensing and rotated bounding box geometry
Fig. 3: Comparison of RGB image (left), raw RealSense depth map (middle), and depth map after tilt and bias correction (right).
Fig. 4: An example of axis align bounding boxes (left) and rotated bounding boxes (right).
Method
Precision
Recall
F1
YOLO Only
0.9896
0.4947
0.6620
CD-YOLO-SAM2-AABB-C1-C2
0.6791
0.7644
0.7192
CD-YOLO-SAM2-RBB-C1-C2
0.7377
0.7068
0.7219
CD-YOLO-SAM2-AABB-C1-CY
0.7450
0.7801
0.7621
CD-YOLO-SAM2-RBB-C1-CY
0.8000
0.7330
0.7650
CD-YOLO-SAM2-AABB-C1-C2-CY
0.8191
0.8063
0.8127
TABLE I: Performance Comparison of Detection Methods
Fault detection identifies abnormal behavior in machines, sensors, infrastructure, software, and autonomous systems before it causes harm or disruption.