OSDaR-AR: Enhancing Railway Perception Datasets via Multi-modal Augmented Reality

Federico Nesti; Gianluca D'Amico; Mauro Marinoni; Giorgio Buttazzo

cs.CVFeb 26, 2026

OSDaR-AR: Enhancing Railway Perception Datasets via Multi-modal Augmented Reality

Authors: Federico Nesti, Gianluca D'Amico, Mauro Marinoni, Giorgio Buttazzo

Organizations: Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, Pisa, Italy · Simulatrix MV srl, Pisa, Italy

Abstract

Although deep learning has significantly advanced the perception capabilities of intelligent transportation systems, railway applications continue to suffer from a scarcity of high-quality, annotated data for safety-critical tasks like obstacle detection. While photorealistic simulators offer a solution, they often struggle with the ``sim-to-real" gap; conversely, simple image-masking techniques lack the spatio-temporal coherence required to obtain augmented single- and multi-frame scenes with the correct appearance and dimensions. This paper introduces a multi-modal augmented reality framework designed to bridge this gap by integrating photorealistic virtual objects into real-world railway sequences from the OSDaR23 dataset. Utilizing Unreal Engine 5 features, our pipeline leverages LiDAR point-clouds and INS/GNSS data to ensure accurate object placement and temporal stability across RGB frames. This paper also proposes a segmentation-based refinement strategy for INS/GNSS data to significantly improve the realism of the augmented sequences, as confirmed by the comparative study presented in the paper. Carefully designed augmented sequences are collected to produce OSDaR-AR, a public dataset designed to support the development of next-generation railway perception systems. The dataset is available at the following page: https://syndra.retis.santannapisa.it/osdarar.html

Figures & tables

Fig. 2 : Preparation and rendering pipeline to obtain OSDaR-AR sequences from the original OSDaR data. The images also show, from top left to bottom left clock-wise, a sample RGB image from sequence 6_station_klein_flottbek_6.2 ( rgb_highres_center camera), the segmented point-cloud, the minimal reconstructed virtual environment, and the final rendered frame (cropped to better show details).

Fig. 3 : A comparison between different localization methods for sequence 6_station_klein_flottbek_6.2 : the INS/GNSS deviates from the rail geometry, the LiDAR Odometry is quite accurate, while the segmentation-based projection refinement of the INS/GNSS improves the original position.

			INS/ GNSS	LiDAR Odometry	Segmentation- refined (ours)
Seq 3.1	C1	$RPE$	$81.3\pm 50.1$	$\mathbf{25.3\pm 17.4}$	$31.1\pm 12.5$
	C1	$J$	$2.71\pm 3.16$	$1.91\pm 2.70$	$\mathbf{1.54\pm 1.62}$
	C2	$RPE$	$89.0\pm 92.0$	$56.8\pm 45.1$	$\mathbf{22.0\pm 27.7}$
	C2	$J$	$4.23\pm 5.59$	$3.11\pm 4.31$	$\mathbf{2.14\pm 2.46}$
Seq 5.1	C1	$RPE$	$63.7\pm 44.7$	$\mathbf{43.9\pm 27.0}$	$60.1\pm 43.5$
	C1	$J$	$3.10\pm 4.07$	$\mathbf{2.17\pm 2.43}$	$3.10\pm 4.03$

TABLE I : Experimental results. For each Camera (C1: rgb_center , C2: rgb_highres_center ), top:

RPE

[px], bottom:

J

[px]. The best result in each row is highlighed in bold .

Explore similar work

Apr 24, 2026

Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain

Automated train operation on existing railway infrastructure requires robust camera-based perception, yet the railway domain lacks public benchmark suites with standardized evaluation protocols that would enable reproducible comparison of approaches. We present RAIL-BENCH, the first perception benchmark suite for the railway domain. It comprises five challenges - rail track detection, object detection, vegetation segmentation, multi-object tracking, and monocular visual odometry - each tailored to the specific characteristics of railway environments. RAIL-BENCH provides curated training and test datasets drawn from diverse real-world scenarios, evaluation metrics, and public scoreboards (https://www.mrt.kit.edu/railbench). For the rail track detection challenge we introduce LineAP, a novel segment-based average precision metric that evaluates the geometric accuracy of polyline predictions independently of instance-level grouping, addressing key limitations of existing line detection metrics.

Annika Bätz, Pavel Klasek, Seo-Young Ham +3

Karlsruhe Institute of Technology, Engler-Bunte-Ring 21, 76131 Karlsruhe, Germany · Deutsches Zentrum für Schienenverkehrsforschung (DZSF), August-Bebel-Straße 10, 01219 Dresden, Germany · DB InfraGO AG, EUREF-Campus 17, 10829 Berlin, Germany

Visual Odometry Object Detection Multi-Object Tracking

Ask PDF

Apr 16, 2026

Integrating Object Detection, LiDAR-Enhanced Depth Estimation, and Segmentation Models for Railway Environments

Obstacle detection in railway environments is crucial for ensuring safety. However, very few studies address the problem using a complete, modular, and flexible system that can both detect objects in the scene and estimate their distance from the vehicle. Most works focus solely on detection, others attempt to identify the track, and only a few estimate obstacle distances. Additionally, evaluating these systems is challenging due to the lack of ground truth data. In this paper, we propose a modular and flexible framework that identifies the rail track, detects potential obstacles, and estimates their distance by integrating three neural networks for object detection, track segmentation, and monocular depth estimation with LiDAR point clouds. To enable a reliable and quantitative evaluation, the proposed framework is assessed using a synthetic dataset (SynDRA), which provides accurate ground truth annotations, allowing for direct performance comparison with existing methods. The proposed system achieves a mean absolute error (MAE) as low as 0.63 meters by integrating monocular depth maps with LiDAR, enabling not only accurate distance estimates but also spatial perception of the scene.

Enrico Francesco Giannico, Federico Nesti, Gianluca D'Amico +5

Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, Pisa, Italy · Progress Rail Signaling S.p.A., Serravalle Pistoiese (PT), Italy

Monocular Depth Estimation Camera-LiDAR Fusion Object Detection

Ask PDF

Aug 5, 2026

A Multi-Sensor Dataset for Monitoring the Operational Environment of Rail Vehicles

Reliable environment monitoring is essential for the safe and efficient operation of automated railway systems, covering all Grades of Automation (GoA), from partially automated (GoA2) to fully automated operation (GoA4). Artificial Intelligence (AI) plays a central role in enabling these systems to detect, classify, and react to potential hazards in real time. The development of such AI-based perception systems requires large volumes of accurately annotated data for training and validation. Within the Digitale Schiene Deutschland (DSD) program, DB InfraGO AG and understandAI GmbH have developed a comprehensive multi- sensor dataset tailored to the needs of railway environment perception. This dataset contains over 7 million high-quality annotations of both railway-specific and general perception objects, captured under varying operational scenarios. The finalized dataset can now be requested at the DB InfraGO AG and serve as a valuable resource for advancing AI-driven environment monitoring in the railway domain.

Claudio Diotallevi, Rodrigo Gudiño, Zaharia Pachalieva +5

understandAI GmbH, An der RaumFabrik 33a, 76227 Karlsruhe, Germany · DB InfraGO AG, EUREF-Campus 17, 10829 Berlin, Germany

Intelligent Transportation Systems

Ask PDF