RGB-Event HyperGraph Prompt for Kilometer Marker Recognition based on Pre-trained Foundation Models

Xiaoyu Xian; Shiao Wang; Xiao Wang; Daxin Tian; Yan Tian

cs.CVFeb 25, 2026

RGB-Event HyperGraph Prompt for Kilometer Marker Recognition based on Pre-trained Foundation Models

Authors: Xiaoyu Xian, Shiao Wang, Xiao Wang, Daxin Tian, Yan Tian

Organizations: School of Transportation Science and Engineering, Beihang University, Beijing 100191, China · CRRC Academy Co., Ltd, Beijing, 100070, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China.

Abstract

Metro trains often operate in highly complex environments, characterized by illumination variations, high-speed motion, and adverse weather conditions. These factors pose significant challenges for visual perception systems, especially those relying solely on conventional RGB cameras. To tackle these difficulties, we explore the integration of event cameras into the perception system, leveraging their advantages in low-light conditions, high-speed scenarios, and low power consumption. Specifically, we focus on Kilometer Marker Recognition (KMR), a critical task for autonomous metro localization under GNSS-denied conditions. In this context, we propose a robust baseline method based on a pre-trained RGB OCR foundation model, enhanced through multi-modal adaptation. Furthermore, we construct the first large-scale RGB-Event dataset, EvMetro5K, containing 5,599 pairs of synchronized RGB-Event samples, split into 4,479 training and 1,120 testing samples. Extensive experiments on EvMetro5K and other widely used benchmarks demonstrate the effectiveness of our approach for KMR. Both the dataset and source code will be released on https://github.com/Event-AHU/EvMetro5K_benchmark

Figures & tables

Fig. 1: The multi-modal imaging device proposed in this paper and typical metro perception scenarios. Specifically, sub-figures (c) and (d) show the imaging results of the metro under low-illumination conditions in high-speed motion/static scenes and overexposed scenes, respectively. In each sub-figure (c, d), the four images arranged clockwise are: the RGB image, the NIR image, the stacked event stream rendered as a red-blue map, and the reconstructed grayscale image from the event stream.

Fig. 2: An overview of our proposed RGB-Event based Hypergraph Prompt for Kilometer Marker Recognition based on foundation models.

Fig. 3: Example samples from the EvMetro5K dataset. Each pair shows the RGB image (left) and the corresponding event-reconstructed grayscale image (right). While the RGB modality often suffers from low light, motion blur, and overexposure conditions, the event modality provides clearer structural information for milestone recognition.

Algorithm	Publish	Backbone	Accuracy	Params	Code
IPAD [ 60 ]	IJCV2025	ViT	69.5	14.1M	URL
Qwen3VL [ 61 ]	arXiv2025	ViT	39.1	235.0B	URL
CAM [ 62 ]	PR2024	ResNet+ViT	81.3	23.3M	URL
CCD [ 24 ]	ICCV 2023	ViT	86.0	32.0M	URL
SIGA [ 25 ]	CVPR 2023	ResNet	81.3	40.4M	URL
CDistNet [ 26 ]	IJCV 2023	ResNet+ViT	91.9	65.5M	URL

TABLE I: The accuracy comparisons with SOTA methods on the EvMetro5K dataset.

Algorithm	Publish	Backbone	Accuracy		Params(M)	Code
Algorithm	Publish	Backbone	WordArt*	IC15*	Params(M)	Code
CCD [ 24 ]	ICCV 2023	ViT	62.1	91.6	52.0	URL
SIGA [ 25 ]	CVPR 2023	ResNet	70.9	73.7	40.4	URL
DiG [ 63 ]	ACM MM 2022	ViT	74.1	78.2	52.0	URL
MGP-STR [ 64 ]	ECCV 2022	ViT	80.5	84.9	148.0	URL
BLIVA [ 65 ]	AAAI 2024	ViT	56.7	51.3	7531.3	URL

TABLE II: The accuracy comparisons with SOTA methods on WordArt* and IC15*.

Table 6

Fig. 4: Ablation studies of the Fusion layers and Permutations.

Fig. 5: Visualization of attention maps of our method on the EvMetro5K dataset.

RGB-Event Sensor Fusion

See all 15 papers

FEMOT: Multi-Object Tracking using Frame and Event Cameras
Jun 12, 2026Shiao Wang, Xiao Wang, Chao Wang +6Event-Based Vision Visual Object Tracking
School of Computer Science and Technology, Anhui University, Hefei, 230601, China. · 2Peng Cheng Laboratory, Shenzhen, China. · 5Harbin Institute of Technology, Shenzhen, China. +2
Event-RGB Adaptive Tracking for Nighttime Highway Perception
Jul 13, 2026Haidong Wang, Hengxing Cai, Wanlei Li +2Autonomous Driving Datasets Intelligent Transportation Systems
School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China · School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China
Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain
Apr 24, 2026Annika Bätz, Pavel Klasek, Seo-Young Ham +3Visual Odometry Object Detection
Karlsruhe Institute of Technology, Engler-Bunte-Ring 21, 76131 Karlsruhe, Germany · Deutsches Zentrum für Schienenverkehrsforschung (DZSF), August-Bebel-Straße 10, 01219 Dresden, Germany · DB InfraGO AG, EUREF-Campus 17, 10829 Berlin, Germany

Jun 12, 2026J/K move · Enter open · S save

FEMOT: Multi-Object Tracking using Frame and Event Cameras

Shiao Wang, Xiao Wang, Chao Wang, Yitao Li+5

School of Computer Science and Technology, Anhui University, Hefei, 230601, China. · 2Peng Cheng Laboratory, Shenzhen, China. · 5Harbin Institute of Technology, Shenzhen, China.+2

Event-Based Vision Visual Object Tracking Multimodal Sensor Fusion Multi-Object Tracking Multimodal Visual Object Tracking

Open paper AskPDF arXiv

Conventional RGB cameras have been widely used in multi-object tracking due to their ability to capture rich appearance and semantic information. However, their performance is often degraded under complex real-world challenges, such as motion blur, low illumination, and overexposure. Bio-inspired event cameras offer high temporal resolution and high dynamic range, providing complementary cues under extreme scenarios. Nevertheless, RGB-event multi-object tracking remains underexplored due to the lack of large-scale and well-annotated datasets. To address this issue, we propose FEMOT, a large-scale RGB-event multi-object tracking dataset that covers diverse real-world scenarios and 14 challenging attributes. With both RGB and event data as well as high-quality annotations, FEMOT provides a reliable platform for systematically evaluating RGB-event multi-object tracking methods. Based on FEMOT, we retrain and evaluate over ten strong trackers, thereby establishing a comprehensive benchmark for future research. Furthermore, we propose FEMOTR, a multimodal tracking framework that decouples RGB and event features and fuses them in the frequency domain, thereby effectively exploiting their complementary characteristics for robust object localization and identity association. Extensive experiments on FEMOT and DSEC-MOT datasets demonstrate the effectiveness of the proposed method. The source code and benchmark dataset have been released on https://github.com/Event-AHU/FEMOT.

Abstract

RGB-Event Sensor Fusion

Explore similar work

FEMOT: Multi-Object Tracking using Frame and Event Cameras

Event-RGB Adaptive Tracking for Nighttime Highway Perception

Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain