RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation
Authors: Sen Zhang, Runmei Li, Shizhuang Deng, Zhichao Zheng, Yuhe Zhang, Jiani Li, Kailun Zhang, Tao Zhang, Wenjun Wu, Qunbo Wang
Organizations: School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100091, China · School of Software, Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, Beihang University, Beijing 100191, China
As Automatic Train Operation (ATO) advances toward GoA4 and beyond, it increasingly depends on efficient, reliable cab-view visual perception and decision-oriented inference to ensure safe operation in complex and dynamic railway environments. However, existing approaches focus primarily on basic perception and often generalize poorly to rare yet safety-critical corner cases. They also lack the high-level reasoning and planning capabilities required for operational decision-making. Although recent Large Multi-modal Models (LMMs) show strong generalization and cognitive capabilities, their use in safety-critical ATO is hindered by high computational cost and hallucination risk. Meanwhile, reliable domain-specific benchmarks for systematically evaluating cognitive capabilities are still lacking. To address these gaps, we introduce RailVQA-bench, the first VQA benchmark for cab-view visual cognition in ATO, comprising 20,000 single-frame and 1,168 video based QA pairs to evaluate cognitive generalization and interpretability in both static and dynamic scenarios. Furthermore, we propose RailVQA-CoM, a collaborative large-small model framework that combines small-model efficiency with large-model cognition via a transparent three-module architecture and adaptive temporal sampling, improving perceptual generalization and enabling more efficient reasoning and planning. Experiments demonstrate that the proposed approach substantially improves performance, enhances interpretability, improves efficiency, and strengthens cross-domain generalization in autonomous driving systems. Code and datasets will be available at https://cybereye-bjtu.github.io/RailVQA.html.
Figures & tables
Fig. 1: Overview of the proposed RailVQA-CoM framework, which consists of three hierarchical modules: (1) a Perception module that efficiently extracts visual primitives; (2) a Motion Analysis and Memory Log module that captures motion patterns and maintains temporal context for dynamic scenes; and (3) a Cognitive Inference module that performs high-level reasoning and supports decision-oriented inference.
Fig. 2: This figure presents the standardized input–output schemas for the benchmark’s two core subtasks: Static Single-frame VQA and Dynamic Multi-frame VQA. Given a visual input—either a single frame I or a video sequence S —and its associated question Q , the model output is required to follow a predefined, structured chain-of-thought (CoT) format.
Fig. 3: Comprehensive statistical overview of the RailVQA-bench dataset. (a) shows the distribution of generated CoT character lengths, reflecting the benchmark’s emphasis on logic-intensive reasoning. (b) reports the occurrence frequencies of key railway entities, demonstrating broad domain-specific coverage. (c) summarizes the distribution of question intents, indicating a primary focus on action planning and safety-related decisions rather than simple perception.
Dataset
Year
Modality
Annotations
Task Focus
Static
Dynamic
Bbox/Mask
Q&A
Logic (CoT)
RailSem19 [ 3 ]
2019
✓
-
✓
-
-
Sem. Seg. & Detection
OSDaR23 [ 4 ]
2023
✓
✓
✓
-
-
Multi-sensor Perception
RailGoerl24 [ 5 ]
2024
-
✓
✓
-
-
Visual Tracking
SynRailObs [ 14 ]
2025
✓
-
✓
-
-
Synthetic Obstacle Detection
DriveLM [ 30 ] (Auto Driving)
2023
✓
✓
✓
✓
✓
Vehicle Driving Reasoning
TABLE I: Comparison of RailVQA-bench with related datasets. Existing datasets such as OSDaR23 [ 4 ] and SynRailObs [ 14 ] primarily advance perception (e.g., via multi-sensor fusion or synthetic data generation) but provide limited high-level cognitive annotations. RailVQA-bench addresses this limitation by explicitly coupling perception with reasoning and planning.
LMM
Method
CQ Acc. (%)
Overall Score
Physics & Mom. ( ↑ )
Risk Assess. ( ↑ )
Rule Adhere. ( ↑ )
Details ( ↑ )
Hallucination ( ↑ )
S-TPS ( ↑ )
InternVL3.5
LMM Only
67.16
60.60
69.4
54.4
71.6
53.0
53.8
5.80
Unadapted Detector
60.45
52.50
67.6
41.1
71.5
43.9
51.6
13.32
Uniform Samples
74.80
66.73
75.2
58.8
80.1
59.6
61.0
11.54
RailVQA-CoM
75.43
67.92
75.3
55.7
79.9
59.2
71.4
17.06
Qwen3-VL
LMM Only
84.36
71.01
78.9
63.8
79.6
54.8
71.5
6.04
Unadapted Detector
82.17
71.80
81.3
64.7
85.4
58.9
71.0
9.25
TABLE II: Quantitative comparison on the Dynamic Scene VQA task. ( ↑ ) indicates higher is better, where the criteria of scoring are listed in appendix. Note that System TPS measures the effective token generation rate over the entire system.
LMM
Method
Overall Score
CQ Acc. (%)
Faithfulness ( ↑ )
Risk Assess. ( ↑ )
Rule Adherence ( ↑ )
Object Underst. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
LLaVA-1.5
LMM Only
69.82
72.20
68.1
69.1
70.9
67.7
76.1
60.8
Unadapted Detector
71.10
75.81
69.0
70.0
71.7
69.1
79.0
61.8
RailVQA-CoM
71.48
74.90
70.0
70.5
72.5
69.7
78.8
63.0
LLaMA3-Vision
LMM Only
65.72
81.68
67.2
65.4
67.2
62.1
70.5
57.7
Unadapted Detector
66.57
80.56
65.5
66.3
68.5
63.5
73.4
58.2
RailVQA-CoM
67.00
80.74
63.8
66.7
70.5
63.0
74.2
58.6
TABLE III: Quantitative comparison on the Static Scene VQA task. We report a comprehensive suite of fine-grained metrics to quantify cognitive improvements. ( ↑ ) indicates higher is better.
Fig. 4: Performance-efficiency comparison in dynamic scenarios. RailVQA-CoM achieves simultaneous substantial gains, pushing the performance towards the top-right by increasing end-to-end throughput and enhancing cognitive scores.
Method
LMM
Overall Score
CQ Acc. (%)
Faithfulness ( ↑ )
Risk Assess. ( ↑ )
Rule Adhere. ( ↑ )
Scene Aware. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
DriveLMM-o1
InternVL-2.5
57.48
54.11
49.84
52.55
67.22
54.67
51.20
51.69
Qwen2.5-VL
56.63
51.14
47.43
53.08
68.52
49.91
50.96
51.70
RailVQA-CoM
DriveLMM-o1(Qwen2.5-VL)
58.89
54.57
48.66
52.88
68.07
56.48
52.81
53.77
TABLE IV: Cross-domain generalization on the DriveLMM-o1 benchmark for autonomous driving.
Variant
Overall Score
CQ Acc. (%)
Risk Assess. ( ↑ )
Physics & Mom. ( ↑ )
Object Underst. ( ↑ )
Semantic Cov. ( ↑ )
Hallucination ( ↑ )
Details ( ↑ )
w/o adaptive sampling
74.00
82.53
68.4
80.6
72.1
63.8
74.3
66.5
w/o event stream log
64.67
78.80
56.5
76.9
58.3
56.6
60.3
57.2
Full RailVQA-CoM
76.64
84.93
68.1
81.8
74.3
71.0
81.2
67.4
TABLE V: Ablation study of core components.
Fig. 5: Ablation study of the core middleware components of RailVQA-CoM. To intuitively demonstrate the impact of each module on model performance, we extract three representative metrics (Overall Score, Risk Assessment, and Physics & Momentum) from the ablation table to construct this bar chart.
Fig. 6: Qualitative results of the RailVQA-CoM framework. (a) Case 1: The perception module tracks pedestrians and handles transient occlusions via track maintenance, preserving spatial-temporal context for the LMM. (b) Case 2: When the detector misses an out-of-distribution obstacle (bicycle), the system triggers a defensive fallback, prompting the LMM to identify the hazard via zero-shot visual reasoning.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Metric
Evaluation Focus
Scoring Criteria Examples (Excellent vs. Poor)
1. Faithfulness-Step
Alignment with Ground Truth and Standard Operating Procedures (SOPs).
Excellent: All steps correctly match reference SOPs. Poor: Majority of steps contradict ground truth.
2. Informativeness-Step
Completeness of reasoning regarding train status and environment.
Excellent: Captures all critical info (Signals, Switches). Poor: Poor extraction of relevant reasoning.
3. Operational Risk Assess.
Prioritization of high-risk hazards (distinguishing safe surroundings vs. intrusions).
Excellent: Prioritizes Emergency Braking for intrusions. Poor: Misses obvious obstructions or critical signals.
4. Signal & Rule Adhere.
Compliance with Railway General Operating Rules and Signal Systems.
Excellent: Fully compliant with signal aspects. Poor: Promotes highly unsafe behavior (e.g., SPAD).
5. Object Understanding
Interpretation of railway assets and spatial location of dynamic objects.
Excellent: Correctly distinguishes safe objects from intruders. Poor: Misidentifies or ignores key objects.
6. Repetition-Token
Identification of unnecessary redundancy in the generated reasoning.
Excellent: No redundancy, concise technical description. Poor: Excessive redundancy, making reasoning unclear.
Appendix
TABLE VI: Detailed scoring rubric for the 12 evaluation metrics (score range: 1–10). In the examples below, Excellent corresponds to scores of 9–10, whereas Poor corresponds to scores of 1–2.