Defining an Evaluation Method for External Human-Machine Interfaces

Jose Gonzalez-Belmonte; Jaerock Kwon

cs.ROApr 14, 2026

Defining an Evaluation Method for External Human-Machine Interfaces

Authors: Jose Gonzalez-Belmonte, Jaerock Kwon

Organizations: Electrical and Computer Engineering, University of Michigan-Dearborn, 4901 Evergreen Rd, Dearborn, 48128, Michigan, United States of America. · Math and Computer Science, Lawrence Technological University, 21000 W Ten Mile Rd, Southfield, 48076, Michigan, United States of America.

Abstract

As the number of fatalities involving Autonomous Vehicles increase, the need for a universal method of communicating between vehicles and other agents on the road has also increased. Over the past decade, numerous proposals of external Human-Machine Interfaces (eHMIs) have been brought forward with the purpose of bridging this communication gap, with none yet to be determined as the ideal one. This work proposes a universal evaluation method conformed of 223 questions to objectively evaluate and compare different proposals and arrive at a conclusion. The questionnaire is divided into 7 categories that evaluate different aspects of any given proposal that uses eHMIs: ease of standardization, cost effectiveness, accessibility, ease of understanding, multifacetedness in communication, positioning, and readability. In order to test the method it was used on four existing proposals, plus a baseline using only kinematic motions, in order to both exemplify the application of the evaluation method and offer a baseline score for future comparison. The result of this testing suggests that the ideal method of machine-human communication is a combination of intentionally-designed vehicle kinematics and distributed well-placed text-based displays, but it also reveals knowledge gaps in the readability of eHMIs and the speed at which different observers may learn their meaning. This paper proposes future work related to these uncertainties, along with future testing with the proposed method.

Figures & tables

Figure 1: Diagram illustrating the configurations used to determine visibility.

Figure 2: Diagram illustrating the external parts of the SUV from the simulation software, as defined by this section of the questionnaire.

Figure 3: Results showing how many times a Grid Point was recorded, across all scenarios where the vehicle was an SUV driving North, on the left side of the virtual environment, and the observer was looking North, towards the front of the vehicle.

Vehicle Part	Configurations where any Grid Point on the vehicle part was recorded in at least 50% of its permutations
Back Bumper.	II,X
Back Central Light	II
Back Fenders	II,IV,VII,VIII
Back Low Reflector	X
Back Plate	II,X
Back Window Rails	II,IV,VII,VIII

Table 1: Visibility of different external vehicle elements. Each row indicates the configurations from Fig. 1 in which any Grid Point on the given vehicle part was captured in at least 50% of its permutations.

Figure 4: The baseline example with no eHMIs.

Figure 5: Version of frontal braking lights being evaluated.

Figure 6: Example of the Knight Rider display mid-animation.

Figure 7: Example of the smiling display, taken from a promotional video of the Smiling Car concept [ 55 ] .

Figure 8: Example of a text display on the front bumper of a vehicle.

NO EHMI: ELEMENT 1 (VEHICLE CHASSIS)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	0	0	0	0	0	0	0	0	0
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
0	0	0	0	0	0	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	0	1	0	0	0	3

Table 2: Standardization Questionnaire Results - Vehicle Kinematics

FRONT BRAKING LIGHTS: ELEMENT 1 (LIGHTS)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	1	0	0	0	0	0	0	0	0
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
1	2	0	0	1	0	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	0	0	0	0	0	0

Table 3: Standardization Questionnaire Results - Frontal Braking Lights

KNIGHT RIDER DISPLAY: ELEMENT 1 (DISPLAY)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	1	0	1	1	0	1	0	0	2
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
1	0	0	0	1	0	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	0	1.5	0	0	0	0

Table 4: Standardization Questionnaire Results - Knight Rider Screen Display

KNIGHT RIDER DISPLAY: ELEMENT 2 (LIGHTS)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	1	0	0	0	0	0	0	0	0
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
1	0	0	0	1	0	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	0	1.5	0	0	0	1

Table 5: Standardization Questionnaire Results - Knight Rider LED Lights

BUMPER SMILE DISPLAY: ELEMENT 1 (DISPLAY)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	1	0	1	1	0	1	0	0	2
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
1	0	0	0	1	0	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	0	0	0	0	0	0

Table 6: Standardization Questionnaire Results - Bumper Smile Display

BUMPER TEXT DISPLAY: ELEMENT 1 (DISPLAY)
S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
0	1	4	1	1	0	1	0	0	0
S11	S12	S13	S14	S15	S16	S17	S18	S19	S20
0	0	0	0	0	2	0	0	0	0
S21	S22	S23	S24	S25	S26	S27
0	1	0	0	0	0	0

Table 7: Standardization Questionnaire Results - Bumper Text Display

STANDARDIZATION RESULTS
Baseline	31
	No eHMI	FBL	KRD	BSD	BTD
Penalty Total (All Elements)	4.00	5.00	15.00	8.00	11.00
TOTAL	10.00	9.63	5.93	8.52	7.41

Table 8: Standardization Questionnaire Results - Standardization Results

NO EHMI RESULTS
CE1_NoeHMI	0.00
CE2_NoeHMI	0.00
CE3_NoeHMI	0.00
CE4_NoeHMI	0.00
CE5_NoeHMI	0.00

Table 9: Cost Effectiveness Questionnaire Results - No eHMI

TRAIL LIGHTS RESULTS
CE1_BrakeLight	39.68
CE2_BrakeLight	34.13
CE3_BrakeLight	45.50
CE4_BrakeLight	53.94
CE5_BrakeLight	53.94

Table 10: Cost Effectiveness Questionnaire Results - Tail Lights

LED DISPLAYS RESULTS
CE1_LEDDisplay	4,606.90
CE2_LEDDisplay	0.00
CE3_LEDDisplay	0.00
CE4_LEDDisplay	2,303.45
CE5_LEDDisplay	27.59

Table 11: Cost Effectiveness Questionnaire Results - LED Display

LED LIGHT ARRAY RESULTS
CE1_LEDLights	1.13
CE2_LEDLights	0.8475
CE3_LEDLights	1.13
CE4_LEDLights	0.01
CE5_LEDLights	0.03

Table 12: Cost Effectiveness Questionnaire Results - LED Light Array

COST EFFECTIVENESS RESULTS
	No eHMI	FBL	KRD	BSD	BTD
CE1	0.00	39.68	4,608.03	4,606.90	4,606.90
CE2	0.00	34.13	0.85	0.00	0.00
CE3	0.00	45.50	1.13	0.00	0.00
CE4	0.00	53.94	2,303.46	2,303.45	2,303.45
CE5	0.00	53.94	27.62	27.59	27.59

Table 13: Cost Effectiveness Questionnaire Results

ACCESSIBILITY RESULTS
	No eHMI	FBL	KRD	BSD	BTD
A1	0	0	1	1	0
A2	0	0	0	0	1
A3	0	0	0	0	0
A4	0	0	0	0	0
A5	0	1	0	0	0

Table 14: Accessibility Questionnaire Results

EASE OF UNDERSTANDING RESULTS
	No eHMI	FBL	KRD	BSD	BTD
EU1	0	0	0	0	0
EU2	65	74	79	74	76
EU3	0	0	0	0	0
EU4	0	0	0	0	0
EU5	0	0	0	0	0

Table 15: Ease of Understanding Questionnaire Results

CONSTANT COMMUNICATION RESULTS
	No eHMI	FBL	KRD	BSD	BTD
CC1	1	0	0	0	1
CC2	0	0	0	0	0
CC3	1	1	0	0	0
CC4	0	0	0	0	0
CC5	0	0	1	1	1

Table 16: Constant Communication Questionnaire Results

POSITIONING RESULTS
	No eHMI	FBL	KRD	BSD	BTD
P1	1	1	1	1	1
P2	0	0	0	0	0
P3	0	0	0	0	0
P4	0	0	0	0	0
P5	2	0	0	0	0

Table 17: Positioning Questionnaire Results

READABILITY RESULTS
	No eHMI	FBL	KRD	BSD	BTD
R1	0	0	1	1	1
R2	0	0	1	1	1
R3	0	0	1	1	1
R4	0	0	0	0	0
R5	0	0	0	0	0

Table 18: Readability Questionnaire Results

FINAL RESULTS
	No eHMI	FBL	KRD	BSD	BTD
Standarization	10.00	9.63	5.93	8.52	7.41
Cost Effectiveness	10.00	9.95	8.56	8.56	8.56
Accessibility	3.56	4.25	4.79	4.79	6.30
Ease of Understanding	0.81	0.93	0.99	0.93	0.95
Constant Communication	0.83	0.42	1.25	1.25	1.67

Table 19: Evaluation Results

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

Apr 13, 2026

Using Unwrapped Full Color Space Recording to Measure the Exposedness of Vehicle Exterior Parts for External Human Machine Interfaces

One of the concerns with autonomous vehicles is their ability to communicate their intent to other road users, specially pedestrians, in order to prevent accidents. External Human-Machine Interfaces (eHMIs) are the proposed solution to this issue, through the introduction of electronic devices on the exterior of a vehicle that communicate when the vehicle is planning on slowing down or yielding. This paper uses the technique of unwrapping the faces of a mesh onto a texture where every pixel is a unique color, as well as a series of animated simulations made and ran in the Unity game engine, to measure how many times is each point on a 2015 Ford F-150 King Ranch is unobstructed to a pedestrian attempting to cross the road at a four-way intersection. By cross-referencing the results with a color-coded map of the labeled parts on the exterior of the vehicle, it was concluded that while the bumper, grill, and hood were the parts of the vehicle visible to the crossing pedestrian most often, the existence of other vehicles on the same lane that might obstruct the view of these makes them insufficient. The study recommends instead a distributive approach to eHMIs by using both the windshield and frontal fenders as simultaneous placements for these devices.

Jose Gonzalez-Belmonte, Jaerock Kwon

Electrical and Computer Engineering, University of Michigan-Dearborn, Dearborn, United States of America

Autonomous Driving Autonomous Driving Safety Evaluation

Ask PDF

Jun 22, 2026

When Robots Rate Their Own Interactions: Engagement Validity and the Strangeness Failure

Human-robot interaction (HRI) evaluation relies almost exclusively on human-completed questionnaires, leaving the robot's perspective unexamined. We propose an inverted evaluation, in which LLM-powered robots complete the same standardized instruments from their own perspective, and test whether these ratings agree with human ground truth. In Study~~1, five LLMs completed HRI-CUES, Godspeed, and RoSAS questionnaires for 25~~interactions (

N = 1{,}522

evaluations) from the HRI-CUES dataset. LLMs achieved moderate-to-strong agreement on engagement dimensions (satisfaction

r

up to

.65

and enjoyment

r

up to

.72

) with excellent test-retest reliability (ICC

\geq .82

), but systematically inverted the comfort/strangeness dimension (

r = -.44

-.67

, all

p < .05

), conflating engagement with comfort. In Study~~2, a Nao robot running Claude~~Sonnet~4.5 replicated these patterns in live interactions (

N = 4

), including real-time turn-by-turn assessment. The strangeness failure persisted across five models, synthetic controls, and embodied deployment for two participants. We argue that current LLM-based robots lack access to the internal affective states needed to assess constructs like strangeness, and that inverted evaluation requires supplementary modalities (e.g., physiological signals, gaze, proxemics) to move beyond behavioral proxies. These findings establish boundary conditions for using LLMs as interaction evaluators in HRI.

Victor Lockwood, Hasan Mahmud, Mohammad Javad Khojasteh +2

MS in AI Program Rochester Institute of Technology Rochester, New York, USA · School of Communication Rochester Institute of Technology Rochester, New York, USA · Electrical and Microelectronic Engineering Rochester Institute of Technology Rochester, New York, USA

LLM Agent Evaluation Human-Robot Interaction LLM Agent Reliability

Ask PDF

Jun 23, 2026

It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces

Artificial intelligence (AI) can enhance what people who use augmentative and alternative communication (AAC) are able to do with their systems. However, evaluating AI-powered AAC interfaces can be difficult. People are intersectional beings and current evaluation metrics can struggle to capture the multifaceted and nuanced desires people may have for their AAC. We explore the complicated nature of six AAC problem spaces, explore how AI might be used in these spaces, and suggest more robust methods of evaluation that take the intersectional nuances of people into account. We also discuss broader issues that arise across these problem spaces and how they could be addressed using our proposed evaluation methods.

Blade Frisch, Will Wade, Dylan Gaines +4

Michigan Technological University Houghton, Michigan, USA · Smartbox Assistive Technology Ltd Bristol, UK · Kennesaw State University Marietta, Georgia, USA +2

Human-Centered AI

Ask PDF

Abstract

Autonomous Driving

Explore similar work

Using Unwrapped Full Color Space Recording to Measure the Exposedness of Vehicle Exterior Parts for External Human Machine Interfaces

When Robots Rate Their Own Interactions: Engagement Validity and the Strangeness Failure

It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces