ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation

Zihe Wang; Yihuan Wang; Haiyang Yu. Zhiyong Cui; Xiaojian Liao; Chengcheng Wang; Yonglin Tian; Yongxin Tong

cs.AIMar 17, 2026

ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation

Authors: Zihe Wang, Yihuan Wang, Haiyang Yu. Zhiyong Cui, Xiaojian Liao, Chengcheng Wang, Yonglin Tian, Yongxin Tong

Organizations: Beihang University Beijing, China · Shandong Hi-speed Group Co., Ltd Jinan, China · Institute of automation, Chinese Academy of Sciences Beijing, China

Abstract

The current expressway operation relies on rule-based and isolated models, which limits the ability to jointly analyze knowledge across different systems. Meanwhile, Large Language Models (LLMs) are increasingly applied in intelligent transportation, advancing traffic models from algorithmic to cognitive intelligence. However, general LLMs are unable to effectively understand the regulations and causal relationships of events in unconventional scenarios in the expressway field. Therefore, this paper constructs a pre-trained multimodal large language model (MLLM) for expressways, ExpressMind, which serves as the cognitive core for intelligent expressway operations. This paper constructs the industry's first full-stack expressway dataset, encompassing traffic knowledge texts, emergency reasoning chains, and annotated video events to overcome data scarcity. This paper proposes a dual-layer LLM pre-training paradigm based on self-supervised training and unsupervised learning. Additionally, this study introduces a Graph-Augmented RAG framework to dynamically index the expressway knowledge base. To enhance reasoning for expressway incident response strategies, we develop a RL-aligned Chain-of-Thought (RL-CoT) mechanism that enforces consistency between model reasoning and expert problem-solving heuristics for incident handling. Finally, ExpressMind integrates a cross-modal encoder to align the dynamic feature sequences under the visual and textual channels, enabling it to understand traffic scenes in both video and image modalities. Extensive experiments on our newly released multi-modal expressway benchmark demonstrate that ExpressMind comprehensively outperforms existing baselines in event detection, safety response generation, and complex traffic analysis. The code and data are available at: https://wanderhee.github.io/ExpressMind/.

Figures & tables

Figure 2. The Overall Framework of ExpressMind. The architectural diagram of ExpressMind showing three key components: Adaptive Visual Encoding, RL-Enhanced CoT Reasoning, and Domain Knowledge Integration.

Figure 3. Task-oriented Domain Data Profiling.

Figure 4. Schematic of the RL-based Reasoning Enhancement.

Figure 5. Multimodal Encoding Framework.

Expressway Laws & Regulations QA
Models	MCQ [%]			True/False [%]			Fill-in-the-Blank [%]		Short Answer [%]
Models	Acc	Rec	F1	Acc	Rec	F1	Emb	F1	GPT-Score	F1
Qwen-32B	97.9	96.5	97.2	96.1	95.8	96.4	93.7	85.3	75.6	82.1
Baichuan-32B	91.3	88.5	89.9	89.5	88.2	89.4	88.8	78.2	69.3	74.5
DeepSeek-R1-Distill-Qwen-32B	96.1	96.3	97.4	96.5	96.0	96.1	92.4	84.1	76.6	82.4
DeepSeek-R1-Distill-Llama-70B	97.0	96.2	96.5	97.0	95.8	96.4	95.9	88.7	85.9	87.8

Table 1. Pre-training results on QA tasks.

Figure 6. Performance of ExpressMind with RL Alignment.

Figure 7. The Reasoning Time of the RL Alignment.

Figure 8. An Example of Incident Response Strategy Generation.

Figure 9. The Ablation Experiment of RAG.

Panel of Figure 10 — Figure 10. Results of Traffic Incident Detection.

Panel of Figure 11 — Figure 11. Results of Traffic Incident Detection.

Model	LLM	BLEU-4	ROUGE-L	CIDEr	BERTScore
MoTIF	LLaMA2-7B	82.59	87.45	69.95	88.43
VideoLLaMA3	Qwen 2.5-7B	79.63	85.32	67.42	85.33
MiniCPM-V 4.5	LLaMA 3-8B	81.98	87.13	69.76	88.04
InternVL3.5	Qwen 3-38B	84.53	88.94	71.67	88.97
Qwen3-VL	Qwen 3-32B	84.85	89.30	72.96	89.18
ExpressMind-VL	ExpressMind-14B	85.24	89.25	73.36	89.28

Table 2. Automated evaluation of quantitative results.

Panel of Figure 12 — Figure 12. Ablation Results of VPA.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Pre-training Hyperparameters
Parameter Name	Value
Model Architecture	Qwen-14B
DeepSpeed Strategy	ZeRO Stage 3
Precision	bfloat16
Max Sequence Length	8192
Optimizer	AdamW

Appendix

Table 3. Hyperparameter Settings for Pre-training and GRPO Algorithm

Figure 13. The System Prompt of GPT-Score.

Figure 14. The System Prompt of SFT.

Stage Index ( $k$ )	Stage Name	Control Token ( $S_{k}$ )
1	Perception	[Incident Description]
2	Analysis	[Causal Inference]
3	Decision	[Response Strategy Formulation]
4	Reflection	[Strategy Evaluation]

Appendix

Table 4. Definition of Reasoning Stage Delimiters

Stage	Representative Keywords (Translated)
$S_{1}$ : Perception	Multi-vehicle pileup, Hazardous chemical leakage, Occupying emergency lane, Visibility range, Traffic volume saturation, Fire spreading
$S_{2}$ : Analysis	Secondary accident risk, Chain reaction, Brake failure, Fatigue driving, Lane capacity reduction, Danger radius
$S_{3}$ : Decision	Remote diversion, Upstream interception, Green wave control, Air-ground coordination, Break-bulk transport, Gating control
$S_{4}$ : Evaluation	Residual congestion, Rescue efficiency, Public sentiment monitoring, Secondary damage assessment

Appendix

Table 5. Examples of Expert Vocabulary

\mathcal{V}_{k}

for Each Stage

Figure 15. Example of Textual Knowledge QA.

Figure 16. Example of the Incident CoT Structured Data.

Figure 17. Visualization of the Traffic Incident VQA Data.

Figure 18. Application of EpxressMind-VL.

Intelligent Transportation Systems

Intelligent Transportation Systems apply sensing, communication, AI, and optimization to make transport networks safer and more efficient.

See all 299 papers

Apr 29, 2026

TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation

Large language models (LLMs) and multimodal large models (MLLMs) are increasingly used for transportation tasks such as regulation question answering, traffic management support, engineering review, and autonomous-driving scene reasoning. Yet transportation workflows are rule-intensive, computation-intensive, safety-critical, and inherently multimodal. Existing general benchmarks provide limited evidence of whether a model can apply regulations correctly, perform verifiable engineering calculations, or interpret traffic scenes reliably, while the small number of public transportation benchmarks remain narrow in scope and rarely support fine-grained diagnosis across text, images, and point-cloud data. To address this gap, we present TRIP-Evaluate, an open multimodal benchmark for large models in transportation. The benchmark organizes 837 items using a role-task-knowledge taxonomy that covers vehicle, traffic-management, traveler, and planning-and-design functions. Each item is annotated with capability, modality, and difficulty labels, enabling diagnosis from overall accuracy down to specific failure modes. The current release includes 596 text items, 198 image items, and 43 point-cloud items. TRIP-Evaluate also standardizes item construction, quality control, prompting, decoding, and scoring to improve cross-model comparability. Results on a diverse panel of models show that text-based performance is improving, but substantial weaknesses remain in multi-step engineering calculation, rule-constrained reasoning, multimodal scene understanding, and point-cloud understanding. Overall, TRIP-Evaluate provides a reproducible, diagnosable, and engineering-aligned evaluation baseline for model selection, regression testing, and safer deployment in transportation applications.

Han Gong, Zhen Zhou, Yunyang Shi +4

School of Transportation Southeast University Nanjing, Jiangsu 211189 · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, Jiangsu 214122 · Department of Civil and Environmental Engineering Hong Kong Polytechnic University Kowloon, Hong Kong

LLM Evaluation Intelligent Transportation Systems Multimodal Model Evaluation Point Cloud Understanding

Ask PDF

Apr 11, 2026

AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Traffic Accident Detection (TAD) and Traffic Accident Understanding (TAU). However, existing studies mainly focus on describing and interpreting accident videos, leaving room for deeper causal reasoning and integration of legal knowledge. Traffic Accident Responsibility Allocation (TARA) is a more challenging task that requires multi-step reasoning grounded in traffic regulations. To address this, we introduce AITP (Artificial Intelligence Traffic Police), a multimodal large language model for responsibility reasoning and allocation. AITP enhances reasoning via a Multimodal Chain-of-Thought (MCoT) mechanism and integrates legal knowledge through Retrieval-Augmented Generation (RAG). We further present DecaTARA, a decathlon-style benchmark unifying ten interrelated traffic accident reasoning tasks with 67,941 annotated videos and 195,821 question-answer pairs. Extensive experiments show that AITP achieves state-of-the-art performance across responsibility allocation, TAD, and TAU tasks, establishing a new paradigm for reasoning-driven multimodal traffic analysis.

Zijin Zhou, Songan Zhang

Global Institute of Future Technology, Shanghai Jiao Tong University

Multimodal Large Language Models

Ask PDF

Jun 14, 2026

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics. However, existing traffic-oriented multimodal benchmarks largely emphasize passive visual recognition or isolated video understanding, offering limited support for evaluating structure-aware traffic reasoning under controlled conditions. We introduce OmniTraffic, a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view--BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human--model gap, with the most pronounced failures in topology-grounded and spatio-temporal reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific multimodal reasoning. Beyond a fixed dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.

Maonan Wang, Zhengyan Huang, Kemou Jiang +13

The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3

Autonomous Driving Datasets Spatiotemporal Reasoning Autonomous Driving Perception Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

Abstract

Intelligent Transportation Systems

Explore similar work

TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation

AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning