RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation
Authors: Sen Zhang, Runmei Li, Shizhuang Deng, Zhichao Zheng, Yuhe Zhang, Jiani Li, Kailun Zhang, Tao Zhang, Wenjun Wu, Qunbo Wang
Organizations: School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100091, China · School of Software, Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, Beihang University, Beijing 100191, China
As Automatic Train Operation (ATO) advances toward GoA4 and beyond, it increasingly depends on efficient, reliable cab-view visual perception and decision-oriented inference to ensure safe operation in complex and dynamic railway environments. However, existing approaches focus primarily on basic perception and often generalize poorly to rare yet safety-critical corner cases. They also lack the high-level reasoning and planning capabilities required for operational decision-making. Although recent Large Multi-modal Models (LMMs) show strong generalization and cognitive capabilities, their use in safety-critical ATO is hindered by high computational cost and hallucination risk. Meanwhile, reliable domain-specific benchmarks for systematically evaluating cognitive capabilities are still lacking. To address these gaps, we introduce RailVQA-bench, the first VQA benchmark for cab-view visual cognition in ATO, comprising 20,000 single-frame and 1,168 video based QA pairs to evaluate cognitive generalization and interpretability in both static and dynamic scenarios. Furthermore, we propose RailVQA-CoM, a collaborative large-small model framework that combines small-model efficiency with large-model cognition via a transparent three-module architecture and adaptive temporal sampling, improving perceptual generalization and enabling more efficient reasoning and planning. Experiments demonstrate that the proposed approach substantially improves performance, enhances interpretability, improves efficiency, and strengthens cross-domain generalization in autonomous driving systems. Code and datasets will be available at https://cybereye-bjtu.github.io/RailVQA.html.
Figures & tables
Fig. 1: Overview of the proposed RailVQA-CoM framework, which consists of three hierarchical modules: (1) a Perception module that efficiently extracts visual primitives; (2) a Motion Analysis and Memory Log module that captures motion patterns and maintains temporal context for dynamic scenes; and (3) a Cognitive Inference module that performs high-level reasoning and supports decision-oriented inference.
Fig. 2: This figure presents the standardized input–output schemas for the benchmark’s two core subtasks: Static Single-frame VQA and Dynamic Multi-frame VQA. Given a visual input—either a single frame I or a video sequence S —and its associated question Q , the model output is required to follow a predefined, structured chain-of-thought (CoT) format.
Fig. 3: Comprehensive statistical overview of the RailVQA-bench dataset. (a) shows the distribution of generated CoT character lengths, reflecting the benchmark’s emphasis on logic-intensive reasoning. (b) reports the occurrence frequencies of key railway entities, demonstrating broad domain-specific coverage. (c) summarizes the distribution of question intents, indicating a primary focus on action planning and safety-related decisions rather than simple perception.
Dataset
Year
Modality
Annotations
Task Focus
Static
Dynamic
Bbox/Mask
Q&A
Logic (CoT)
RailSem19 [ 3 ]
2019
✓
-
✓
-
-
Sem. Seg. & Detection
OSDaR23 [ 4 ]
2023
✓
✓
✓
-
-
Multi-sensor Perception
RailGoerl24 [ 5 ]
2024
-
✓
✓
-
-
Visual Tracking
SynRailObs [ 14 ]
2025
✓
-
✓
-
-
Synthetic Obstacle Detection
DriveLM [ 30 ] (Auto Driving)
2023
✓
✓
✓
✓
✓
Vehicle Driving Reasoning
TABLE I: Comparison of RailVQA-bench with related datasets. Existing datasets such as OSDaR23 [ 4 ] and SynRailObs [ 14 ] primarily advance perception (e.g., via multi-sensor fusion or synthetic data generation) but provide limited high-level cognitive annotations. RailVQA-bench addresses this limitation by explicitly coupling perception with reasoning and planning.
LMM
Method
CQ Acc. (%)
Overall Score
Physics & Mom. ( ↑ )
Risk Assess. ( ↑ )
Rule Adhere. ( ↑ )
Details ( ↑ )
Hallucination ( ↑ )
S-TPS ( ↑ )
InternVL3.5
LMM Only
67.16
60.60
69.4
54.4
71.6
53.0
53.8
5.80
Unadapted Detector
60.45
52.50
67.6
41.1
71.5
43.9
51.6
13.32
Uniform Samples
74.80
66.73
75.2
58.8
80.1
59.6
61.0
11.54
RailVQA-CoM
75.43
67.92
75.3
55.7
79.9
59.2
71.4
17.06
Qwen3-VL
LMM Only
84.36
71.01
78.9
63.8
79.6
54.8
71.5
6.04
Unadapted Detector
82.17
71.80
81.3
64.7
85.4
58.9
71.0
9.25
TABLE II: Quantitative comparison on the Dynamic Scene VQA task. ( ↑ ) indicates higher is better, where the criteria of scoring are listed in appendix. Note that System TPS measures the effective token generation rate over the entire system.
LMM
Method
Overall Score
CQ Acc. (%)
Faithfulness ( ↑ )
Risk Assess. ( ↑ )
Rule Adherence ( ↑ )
Object Underst. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
LLaVA-1.5
LMM Only
69.82
72.20
68.1
69.1
70.9
67.7
76.1
60.8
Unadapted Detector
71.10
75.81
69.0
70.0
71.7
69.1
79.0
61.8
RailVQA-CoM
71.48
74.90
70.0
70.5
72.5
69.7
78.8
63.0
LLaMA3-Vision
LMM Only
65.72
81.68
67.2
65.4
67.2
62.1
70.5
57.7
Unadapted Detector
66.57
80.56
65.5
66.3
68.5
63.5
73.4
58.2
RailVQA-CoM
67.00
80.74
63.8
66.7
70.5
63.0
74.2
58.6
TABLE III: Quantitative comparison on the Static Scene VQA task. We report a comprehensive suite of fine-grained metrics to quantify cognitive improvements. ( ↑ ) indicates higher is better.
Fig. 4: Performance-efficiency comparison in dynamic scenarios. RailVQA-CoM achieves simultaneous substantial gains, pushing the performance towards the top-right by increasing end-to-end throughput and enhancing cognitive scores.
Method
LMM
Overall Score
CQ Acc. (%)
Faithfulness ( ↑ )
Risk Assess. ( ↑ )
Rule Adhere. ( ↑ )
Scene Aware. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
DriveLMM-o1
InternVL-2.5
57.48
54.11
49.84
52.55
67.22
54.67
51.20
51.69
Qwen2.5-VL
56.63
51.14
47.43
53.08
68.52
49.91
50.96
51.70
RailVQA-CoM
DriveLMM-o1(Qwen2.5-VL)
58.89
54.57
48.66
52.88
68.07
56.48
52.81
53.77
TABLE IV: Cross-domain generalization on the DriveLMM-o1 benchmark for autonomous driving.
Variant
Overall Score
CQ Acc. (%)
Risk Assess. ( ↑ )
Physics & Mom. ( ↑ )
Object Underst. ( ↑ )
Semantic Cov. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
w/o adaptive sampling
74.00
82.53
68.4
80.6
72.1
63.8
74.3
66.5
w/o event stream log
64.67
78.80
56.5
76.9
58.3
56.6
60.3
57.2
Full RailVQA-CoM
76.64
84.93
68.1
81.8
74.3
71.0
81.2
67.4
TABLE V: Ablation study of core components.
Fig. 5: Ablation study of the core middleware components of RailVQA-CoM. To intuitively demonstrate the impact of each module on model performance, we extract three representative metrics (Overall Score, Risk Assessment, and Physics & Momentum) from the ablation table to construct this bar chart.
Fig. 6: Qualitative results of the RailVQA-CoM framework. (a) Case 1: The perception module tracks pedestrians and handles transient occlusions via track maintenance, preserving spatial-temporal context for the LMM. (b) Case 2: When the detector misses an out-of-distribution obstacle (bicycle), the system triggers a defensive fallback, prompting the LMM to identify the hazard via zero-shot visual reasoning.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Metric
Evaluation Focus
Scoring Criteria Examples (Excellent vs. Poor)
1. Faithfulness-Step
Alignment with Ground Truth and Standard Operating Procedures (SOPs).
Excellent: All steps correctly match reference SOPs. Poor: Majority of steps contradict ground truth.
2. Informativeness-Step
Completeness of reasoning regarding train status and environment.
Excellent: Captures all critical info (Signals, Switches). Poor: Poor extraction of relevant reasoning.
3. Operational Risk Assess.
Prioritization of high-risk hazards (distinguishing safe surroundings vs. intrusions).
Excellent: Prioritizes Emergency Braking for intrusions. Poor: Misses obvious obstructions or critical signals.
4. Signal & Rule Adhere.
Compliance with Railway General Operating Rules and Signal Systems.
Excellent: Fully compliant with signal aspects. Poor: Promotes highly unsafe behavior (e.g., SPAD).
5. Object Understanding
Interpretation of railway assets and spatial location of dynamic objects.
Excellent: Correctly distinguishes safe objects from intruders. Poor: Misidentifies or ignores key objects.
6. Repetition-Token
Identification of unnecessary redundancy in the generated reasoning.
Excellent: No redundancy, concise technical description. Poor: Excessive redundancy, making reasoning unclear.
Appendix
TABLE VI: Detailed scoring rubric for the 12 evaluation metrics (score range: 1–10). In the examples below, Excellent corresponds to scores of 9–10, whereas Poor corresponds to scores of 1–2.
Automated train operation on existing railway infrastructure requires robust camera-based perception, yet the railway domain lacks public benchmark suites with standardized evaluation protocols that would enable reproducible comparison of approaches. We present RAIL-BENCH, the first perception benchmark suite for the railway domain. It comprises five challenges - rail track detection, object detection, vegetation segmentation, multi-object tracking, and monocular visual odometry - each tailored to the specific characteristics of railway environments. RAIL-BENCH provides curated training and test datasets drawn from diverse real-world scenarios, evaluation metrics, and public scoreboards (https://www.mrt.kit.edu/railbench). For the rail track detection challenge we introduce LineAP, a novel segment-based average precision metric that evaluates the geometric accuracy of polyline predictions independently of instance-level grouping, addressing key limitations of existing line detection metrics.
Annika Bätz, Pavel Klasek, Seo-Young Ham +3
Karlsruhe Institute of Technology, Engler-Bunte-Ring 21, 76131 Karlsruhe, Germany · Deutsches Zentrum für Schienenverkehrsforschung (DZSF), August-Bebel-Straße 10, 01219 Dresden, Germany · DB InfraGO AG, EUREF-Campus 17, 10829 Berlin, Germany
Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While vision-language models (VLMs) show strong general reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing benchmarks. To bridge this gap, we present CrashSight, a large-scale vision-language benchmark for roadway crash understanding using real-world roadside camera data. The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight benchmark, including the full dataset and code, is accessible at https://mcgrche.github.io/crashsight.
Rui Gan, Junyi Ma, Pei Li +4
University of Wisconsin–Madison · University of Wyoming · Columbia University
Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is? Can we improve our ability to do so by introducing structured data (such as official accident reports) about the accident history of that crossing into our models? In this work, we explore how to best answer those questions towards building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration (FRA). To that end, we propose a proof-of-concept pipeline that delivers on that goal, while at the same time exploring and tackling a number of critical research challenges that pertain to different parts of the pipeline, from data preparation to different learning paradigms that can allow us to realize such a system. Indicatively, our proposed system identifies HIGH-RISK and LOW-RISK crossings with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and correlation of 0.492 using a routed fine-tuned compact VLM pipeline, while producing qualitative results that align with domain-expert assessment.
Paimon Goulart, Chansong Lim, Nícolas Roque dos Santos +4
University of California, Riverside Riverside, California, USA · Riverside County Transportation Commission Riverside, California, USA