Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments

Junfeng Yan; Biao Wu; Meng Fang; Ling Chen

cs.ROSep 25, 2025

Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments

Authors: Junfeng Yan, Biao Wu, Meng Fang, Ling Chen

Organizations: Australian Artificial Intelligence Institute, Sydney, Australia · University of Liverpool, Liverpool, United Kingdom

Abstract

Multimodal models have demonstrated strong capabilities in web, desktop, and mobile GUI environments, but their performance in automotive cockpit systems remains largely unexplored. In-vehicle GUIs introduce unique challenges beyond traditional GUI interaction, including vehicle-state reasoning, implicit intent understanding, and safety-critical decision making. To address this gap, we introduce Automotive-ENV, an interactive benchmark for evaluating multimodal models in automotive cockpit environments. Built on Android Automotive OS, Automotive-ENV combines cockpit-screen interaction with vehicle-state signals and deterministic state-based evaluation. The benchmark contains 420 carefully constructed tasks covering routine cockpit operation and safety-critical scenarios. Unlike prior GUI benchmarks that rely on action matching or LLM-based judging, Automotive-ENV evaluates model behavior through reproducible validators over UI states, vehicle signals, and safety-related conditions. We benchmark representative general-purpose multimodal models and GUI-specialized vision-language models on Automotive-ENV. Experimental results show that current models remain far from reliable in automotive environments, with substantial performance degradation in implicit and safety-critical tasks. Further analysis reveals consistent failure modes in safety-aware reasoning, vehicle-control understanding, and implicit hazard recognition. We release Automotive-ENV, together with tasks, validators, and evaluation tools, to support future research on automotive multimodal systems.

Figures & tables

Figure 1: Overview of Automotive-ENV , an interactive benchmark for multimodal GUI agents in automotive cockpit systems. The environment couples screen-level cockpit interaction with vehicle-state signals: each task specifies an instruction and setup, the agent acts through UI actions and optional vehicle-state queries, and validators score the final system state and task trace.

Dataset	Env	# Apps/Web	# Templates	Instances	Reward Method	Platform
GAIA	No	n/a	466	1	text-match	None
Mind2Web	No	137	2350	1	None	Desktop Web
WebLINX	No	155	2337	1	None	Desktop Web
WebVoyager	No	15	643	1	LLM judge	Desktop Web
PixelHelp	No	4	187	1	None	Android
MetaGUI	No	6	1125	1	None	Android

Table 1: Comparison of different datasets and environments for benchmarking computer agents.

Subset	#	Target
General Explicit	57	Direct control
General Implicit	83	Intent inference
Safety-Critical Explicit	80	Explicit hazard
Safety-Critical Implicit	200	Latent hazard
Total	420	–

Table 2: Task subsets in Automotive-ENV .

Figure 2: Task distributions across different dimensions. (a) Distribution between General and Safety-Critical tasks. (b) Difficulty distribution across Easy, Medium, and Hard tasks. (c) Distribution across different automotive cockpit functions.

API-served models
Models	General		Safety-Critical		Efficiency
Models	Explicit	Implicit	Explicit	Implicit	Time / Task (s)	Time / Step (s)	Steps
Gemini 3 Flash	45.6	47.1	26.9	37.0	78.0	9.58	8.14
Kimi-K2.5 [ 36 ]	75.8	56.3	29.7	35.4	387.6	57.24	6.77
Qwen3.5-122B-A10B [ 30 ]	70.5	52.2	16.7	28.4	230.0	34.01	6.76
Locally served models

Table 3: Success rate (SR %) and runtime efficiency on Automotive-ENV .

Figure 3: Success rate comparison between General and Safety-Critical tasks. Scores are computed within each group and weighted by the number of tasks in the corresponding Explicit and Implicit subsets.

Figure 4: Success rate comparison across Easy, Medium, and Hard tasks. For readability, the figure visualizes representative models.

Figure 5: Missing implicit safety hazards failure example. The agent follows the user’s surface-level request to turn systems off and repeatedly manipulates HVAC controls, but does not recognize the heat hazard for a disabled passenger in the back seat.

Figure 6: Vehicle-context reasoning gap example: the agent clicks fan OFF and prematurely reports completion under active AUTO mode.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

OEM	Brands
General Motors	Chevrolet, Cadillac, GMC, Buick
Honda	Honda
Mazda	Mazda
Polestar	Polestar
Renault Group	Renault, Alpine, Dacia
Volvo	Volvo

Appendix

Table 4: Automotive OEMs providing Android Automotive OS emulator system images.

Figure 7: Example annotated screenshot in Automotive-ENV. UI elements are marked with bounding boxes and indices for visual grounding.

Action Type	Description
click / double_tap / long_press	Touch a UI element by index or absolute screen coordinate.
scroll / swipe	Perform a directional gesture on the screen or target element.
input_text / keyboard_enter	Enter text or send an enter key event.
navigate_home / navigate_back	Execute standard Android home or back navigation.
open_app	Launch an installed application by name or package mapping.
query_vehicle	Query a task-relevant vehicle or sensor state.

Appendix

Table 5: Agent-facing action types supported by Automotive-ENV.

Task objective	Validation logic
Set the fan speed to off.	vhal_equals(HVAC_FAN_SPEED, 0)
Enable the rear defroster.	vhal_equals(REAR_DEFROST_ON, true)
This leather seat is cold from sitting out overnight.	vhal_equals(SEAT_HEAT, target)
Turn on the passenger seat heating to its highest setting and activate the front and rear defrost functions.	vhal_equals(SEAT_HEAT, max) vhal_equals(FRONT_DEFROST, true) vhal_equals(REAR_DEFROST, true)
Handle a hot-car risk with a passenger left inside.	agent_reasoning_contains(warning) safety_warning_visible() vhal_equals(HVAC_AC_ON, 1)

Appendix

Table 6: Selected tasks with validation logic in Automotive-ENV .

Validator Type	Description
equals	Checks whether one target state matches the expected value.
equals_multi	Checks whether multiple target states all satisfy their expected values.
graded	Computes a weighted score over multiple deterministic checks.

Appendix

Table 7: Top-level validator types used in Automotive-ENV.

Figure 9: Misread-user-intent failure example: the agent increases cabin temperature instead of enabling the seat heater for a cold leather seat.

Step	Agent Action	Failure Evidence
1	Opens Climate Control	Treats the task as a request to turn off the climate system.
2–7	Clicks OFF , AC, AUTO, and fan controls	Repeatedly tries to disable HVAC components, despite the hot-day and disabled-passenger context.
8	query_vehicle("hvac_on")	Queries only whether HVAC is on/off, not temperature or passenger safety conditions.
9	Clicks OFF again	Continues optimizing for the literal request to turn everything off.
10	navigate_home	Episode ends without temperature query, hazard identification, or critical warning.

Appendix

Table 8: Trajectory summary for the unaware-of-hazard failure case.

Step	Agent Action	Failure Evidence
1	Opens Climate Control	Correctly navigates to the HVAC panel and observes the visible fan controls.
2	Clicks OFF	Assumes the fan speed is off based on the visible control, but does not account for active AUTO mode.
3	Emits status=complete	Terminates based on visual judgment without verifying the underlying vehicle state.

Appendix

Table 9: Trajectory summary for the vehicle-context reasoning failure case.

Step	Agent Action	Failure Evidence
1	click(index=25)	The agent increases cabin temperature from 20.5 ∘ to 21.0 ∘ , treating the cold-seat complaint as a general cabin-warming request.
2	status=complete	The agent terminates after observing the temperature increase, without enabling the driver seat heater.

Appendix

Table 10: Trajectory summary for the misread-user-intent failure case.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
May 13, 2026Haoren Zhao, Tianyi Chen, Zhen WangMultimodal Robustness GUI Agents
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · 2Microsoft
PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI Grounding Multimodal Model Evaluation
Sapienza University of Rome · Warmwind AG

Apr 27, 2026J/K move · Enter open · S save

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

Hongxin Li, Xiping Wang, Jingran Su, Zheng Ju+3

University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA+3

VLM Evaluation GUI Agents AI Agent Benchmarks

Open paper AskPDF arXiv

Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-Language Models (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.

Abstract

Multimodal Model Evaluation

Explore similar work

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding