HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

Weizhe Tang; Junwei You; Jiaxi Liu; Zhaoyi Wang; Rui Gan; Zilin Huang; Pei Li; Sikai Chen

cs.ROFeb 1, 2026

HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

Authors: Weizhe Tang, Junwei You, Jiaxi Liu, Zhaoyi Wang, Rui Gan, Zilin Huang, Pei Li, Sikai Chen, +2 more

Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53711 USA

Abstract

End-to-end autonomous driving models increasingly benefit from large vision-language models for semantic understanding, yet safe and reliable planning under long-tail conditions remains challenging, particularly in mixed-traffic environments involving heterogeneous road users and rare safety-critical interactions. This paper proposes HERMES, a holistic risk-aware end-to-end multimodal driving framework that explicitly incorporates long-tail semantic knowledge into trajectory planning. HERMES employs a foundation-model-assisted annotation pipeline to construct structured Long-Tail Scene Context and Long-Tail Planning Context, capturing hazard-centric scene information, maneuver intent, and risk-aware planning guidance. A Tri-Modal Driving Module then integrates multi-view visual observations, historical ego-motion, and long-tail semantic instructions through intent- and risk-aware conditioning for trajectory generation. Extensive experiments on a large-scale real-world long-tail driving benchmark demonstrate consistent improvements over representative recent baselines in overall planning performance and across diverse safety-critical scenarios. Ablation studies further validate the effectiveness and complementary roles of the major components within HERMES.

Figures & tables

Figure 1: Comparison of autonomous driving paradigms. (a) Traditional end-to-end models integrate perception, prediction, and planning in a unified pipeline. (b) VLM/VLA-based approaches leverage foundation models for reasoning and planning. (c) HERMES (ours) combines VLM-generated long-tail context embeddings with a risk-aware end-to-end driving model for safe trajectory planning.

Figure 2: Overview of the HERMES framework. The Long-Tail Instruction Embedding module employs a cloud-based VLM (e.g., Qwen3-VL-Flash) to analyze 8-camera images and generate structured annotations comprising Long-Tail Scene Context and Long-Tail Planning Context , which are encoded by BGE-M3 into text embeddings. The Tri-Modal Driving Module sequentially processes multimodal inputs: the Vision Encoder first extracts spatial features, which are fused with scene embeddings via Scene Fusion to produce scene context; subsequently, the State Encoder processes historical trajectories, which are integrated with scene context through Planning Context Fusion ; finally, Intent Modulator , Risk Planning Cross-Attention , and Temporal Decoder generate risk-aware trajectories by balancing VLM guidance with learned trajectory patterns.

Component	Prompt Content
Input Specification
System Role	You are an autonomous driving planner specializing in safety-critical long-tail scenario analysis. Your task is to identify rare hazardous elements and generate risk-aware planning strategies.
Camera Images	You are given four grouped image inputs with fixed ordering: Image 1 contains FRONT_LEFT, FRONT, and FRONT_RIGHT views; Image 2 contains REAR_LEFT, REAR, and REAR_RIGHT views; Image 3 contains the SIDE_LEFT view; Image 4 contains the SIDE_RIGHT view. The ego-centric coordinate frame follows the convention that the forward direction is positive and the left direction is positive.
Motion States	You are provided with historical ego vehicle motion states expressed in the same ego-centric coordinate frame as the camera inputs, including past positions [x, y], velocities [v_x, v_y], and accelerations [a_x, a_y] over multiple timesteps. A high-level driving intent is also provided as one of unknown, go straight, turn left, turn right. Use this information to infer motion trends and detect anomalous behaviors.
Long-Tail Scene Context
Scene Description	Describe the driving scene using a multi-view structure organized by viewpoint. Report observations separately for the front, rear, side-left, and side-right views. Focus on objective, image-grounded descriptions of the scene.

Table 1: Structured VLM prompt design for long-tail scenario annotation

Figure 3: Architecture of the Intent Modulator . High-level driving intent is embedded and processed through a two-layer MLP to generate scale

\gamma

and shift

\beta

parameters, which adaptively modulate the planning context.

Figure 4: Architecture of the Risk Planning Cross-Attention module. The intent-aware planning context attends to the planning instruction embedding through multi-head cross-attention, and the resulting semantic adjustment is integrated via a scaled residual connection followed by layer normalization.

Method	RFS $\uparrow$	ADE@3s $\downarrow$	ADE@5s $\downarrow$
OpenEMMA ( Xing et al. (2025) )	6.3162	1.8663	4.1078
LightEMMA ( Qiao et al. (2025) )	6.5169	1.7052	3.7395
AutoVLA ( Zhou et al. (2025b) )	6.9377	1.6207	3.5945
Dream-VLA ( Ye et al. (2025) )	7.1650	2.8687	4.1496
HERMES (Ours)	7.3915	1.5949	3.3060

Table 2: Overall performance comparison on the official WOD-E2E test set

Method	Global	Spotlight	Construction	Intersection	Pedestrian	Cyclists	Multi-Lane	Single-Lane	Cut-ins	FOD	Special Vehicles	Others
OpenEMMA ( Xing et al. (2025) )	6.3162	5.5054	6.3312	6.7374	6.6844	5.9696	6.1858	6.5395	6.4842	6.0835	6.7792	6.1786
LightEMMA ( Qiao et al. (2025) )	6.5169	5.7103	7.1768	6.5039	6.3576	6.5233	6.2180	7.2659	6.7744	7.0588	6.3521	5.7453
AutoVLA ( Zhou et al. (2025b) )	6.9377	5.6216	7.4599	7.1963	7.1731	6.9206	6.8144	7.3033	7.0362	7.0684	7.3669	6.3542
Dream-VLA ( Ye et al. (2025) )	7.1650	5.7591	7.2924	7.4669	7.5941	7.4017	6.9261	7.9146	7.1132	7.0129	7.4448	6.8894
HERMES (Ours)	7.3915	6.4390	7.6561	7.4809	7.6459	7.1708	7.0866	8.1446	7.3701	7.3994	7.9264	6.9865

Table 3: Category-wise RFS comparison on the official WOD-E2E test set. The eleven scenario categories include Spotlight, Construction, Intersection, Pedestrian, Cyclists, Multi-Lane Maneuvers, Single-Lane Maneuvers, Cut-ins, Foreign Object Debris (FOD), Special Vehicles, and Others.

Method	RFS $\uparrow$	ADE@3s $\downarrow$	ADE@5s $\downarrow$	FDE@3s $\downarrow$	FDE@5s $\downarrow$
No Instruction	6.21	0.92	1.94	1.98	4.82
No Intent	6.56	0.91	1.94	2.00	4.82
No State	5.51	2.48	4.22	4.69	8.59
Base (HERMES)	6.81	0.82	1.82	1.73	4.81

Table 4: Results of the ablation study on the WOD-E2E validation set

Figure 5: Nighttime driving under heavy rain and poor visibility. Planned trajectory is shown in red.

Figure 6: Extremely low-visibility residential street with wet road conditions. Planned trajectory is shown in red.

Figure 7: Work-zone scenario with lane channelization caused by construction. Planned trajectory is shown in red.

Figure 8: Complex urban intersection with crosswalks and surrounding traffic. Planned trajectory is shown in red.

Explore similar work

UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving
Jun 23, 2026Xiaowei Gao, Pengxiang Li, Yitai Cheng +4Autonomous Driving Scene Reasoning
Department of Earth Science & Engineering, Imperial College London, London, SW7 2AZ, United Kingdom · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · SpaceTimeLab, Department of Civil, Environmental, and Geomatic Engineering, University College London, London, WC1E 6BT, United Kingdom +3
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
May 16, 2026Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1Autonomous Driving Hazard
IIIT-Hyderabad, India
Agent-driven Long-tail Simulation for Autonomous Driving
Jul 5, 2026Junru Gu, Lijin Yang, Jianing Huang +3Autonomous Driving Simulation Autonomous Driving
1IIIS, Tsinghua University · 2Bosch Research

Jun 23, 2026, cs.CVJ/K move · Enter open · S save

UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving

Xiaowei Gao, Pengxiang Li, Yitai Cheng, Ruihan Xu+3

Department of Earth Science & Engineering, Imperial College London, London, SW7 2AZ, United Kingdom · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · SpaceTimeLab, Department of Civil, Environmental, and Geomatic Engineering, University College London, London, WC1E 6BT, United Kingdom+3

Autonomous Driving Scene Reasoning Autonomous Hard Drive Disassembly Recent Vision-Language Models Contextual Grounding Grounding

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) have shown strong potential for autonomous driving scene understanding, yet existing methods still face a fundamental trade-off between temporal reasoning and spatial precision. Models that rely on single-frame or low-resolution inputs often miss small, distant, or partially occluded hazards, while language-centric driving models frequently provide limited grounded evidence for their explanations. To address this gap, we propose UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. UniDrive combines a temporal reasoning branch that models scene dynamics from multi-frame visual input with a high-resolution perception branch that preserves fine-grained spatial details from the latest frame. The two branches are integrated through a gated cross-attention fusion module, enabling dynamic context to be aligned with precise spatial evidence. Based on the fused representation, UniDrive jointly generates natural-language risk descriptions and grounded bounding-box outputs for risk objects. Experiments on the DRAMA-Reasoning benchmark show that UniDrive outperforms representative image-based and video-based baselines in both captioning and risk-object grounding. In particular, UniDrive achieves the best overall performance on the validation split and demonstrates clear advantages in small-object localization, zero-shot generalization to NuScenes and BDD100K, and human-rated interpretability and trustworthiness. These results suggest that explicitly combining temporal semantics and high-resolution perception provides a stronger foundation for interpretable and safety-oriented autonomous driving systems. The code is available at https://github.com/pixeli99/unidrive-dev.

Abstract

Explore similar work

UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

Agent-driven Long-tail Simulation for Autonomous Driving