MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation

Takumi Taki; Masato Kobayashi; Yuki Uranishi

cs.ROMar 31, 2026

MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation

Authors: Takumi Taki, Masato Kobayashi, Yuki Uranishi

Organizations: Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka · Graduate School of Maritime Sciences, Kobe University

Abstract

Autonomous mobile robots operating in human-shared indoor environments often require paths that reflect human spatial intentions, such as avoiding interference with pedestrian flow or maintaining comfortable clearance. However, conventional path planners primarily optimize geometric costs and provide limited support for explicit route specification by human operators. This paper presents MRReP, a Mixed Reality-based interface that enables users to draw a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures. The drawn HRP is integrated into the robot navigation stack through a custom Hand-drawn Reference Path Planner, which converts the user-specified point sequence into a global path for autonomous navigation. We evaluated MRReP in a within-subject experiment against a conventional 2D baseline interface. The results demonstrated that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment. These findings suggest that direct path specification in MR is an effective approach for incorporating human spatial intention into mobile robot navigation. Additional material is available at https://mertcookimg.github.io/mrrep

Figures & tables

Fig. 1 : Overview of MRReP. (a) A user draws a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures and an MR-HMD. (b) The HRP is sent to the robot navigation system, enabling autonomous navigation along the specified path.

	C1	C2	C3	C4	MRReP (Ours)
Path specification accuracy	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Path following accuracy	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Spatial intuitiveness	$\times$	$\times$	$\times$	$\times$	$\checkmark$
No fixed sensor required	$\checkmark$	$-$	$\times$	$\checkmark$	$\checkmark$

TABLE I : Comparison of selected path-specification interfaces.

Fig. 2 : Example of an HRP and its corresponding global path. (a) HRP overlaid on the floor in HoloLens 2; the yellow pin indicates the goal. (b) Generated global path and costmap.

Fig. 3 : System architecture and data flow. The drawn HRP is stored in the HoloLens 2 database, transmitted to ROS 2 via the SEND function, converted into a global path and goal pose, and then used for robot navigation in Navigation2.

Fig. 4 : Menu panel. (a) Main menu in the initial state. (b) Path operation menu with selectable operation modes.

Fig. 5 : ADD operation. (a) A floor cursor appears when the user extends the hand. (b) Maintaining a pinch gesture draws an HRP as a sequence of waypoints. (c) Releasing the pinch completes the path and places a goal pin at the endpoint.

Fig. 8 : Experimental environment. (a) Stage A: straight path. (b) Stage B: piecewise linear path with multiple 45-degree turns.

Fig. 9 : Target paths used in the practice and main tasks. (a) Tape layout in the experimental environment. (b) Corresponding paths on the ROS 2 costmap. T-shaped markers indicate the goal positions.

Fig. 10 : Task procedure. (i) Path drawing. (ii) Path submission. (iii) Robot navigation.

	Median [IQR]		$p$ -value
	2D	MR
Stage A	97.0% [12.4]	100% [0.137]	0.0712
Stage B	65.7% [19.4]	100% [6.17]	0.00121

TABLE II : Percentage of HRP within the GT.

Panel of Figure 13 — Fig. 11 : Evaluation of HRP coverage within the GT.

Fig. 12 : Pixel-wise comparison between the GT and drawn region. (i) Ground Truth. (ii) Example of drawn region on the grid map. (iii) Corresponding pixel classification. TP is shown in black, FP in green, FN in red, and TN in white.

Panel of Figure 15 — Fig. 13 : HRP accuracy by stage.

Stage		Accuracy		Precision		Recall		Specificity		F1-Score
		2D	MR	2D	MR	2D	MR	2D	MR	2D	MR
A	$M$ † [%]	98.7	99.4	71.6	84.9	65.5	91.6	99.4	99.6	70.3	88.1
	$p$	$2.14\times 10^{-3}$		0.0335		$6.10\times 10^{-5}$		0.105		$4.27\times 10^{-4}$
B	$M$ [%]	96.8	98.9	52.9	83.6	59.0	83.7	98.1	99.4	56.5	83.4
	$p$	$9.16\times 10^{-5}$		$9.16\times 10^{-5}$		$9.16\times 10^{-5}$		$9.16\times 10^{-5}$		$9.16\times 10^{-5}$
† $M$ : Median. Reported values are medians across participants.

TABLE III : HRP accuracy metrics.

Panel of Figure 17 — Fig. 14 : Drawing operations and task completion time.

	Median [IQR]		$p$ -value
	2D	MR
Stage A	29.6 [27.9]	53.0 [33.4]	0.0290
Stage B	47.2 [61.6]	69.3 [58.1]	0.323

TABLE IV : Task completion time [sec].

Panel of Figure 19 — Fig. 15 : Global paths and robot trajectories by stage and condition.

Metric	Median [IQR]		Effect size	$p$ -value
	2D	MR
SUS	51.3 [23.1]	75.0 [16.3]	0.711	0.00586
NASA-TLX	61.5 [22.1]	47.7 [21.3]	0.621	0.0110

TABLE V : Subjective evaluation results.

Panel of Figure 21 — Fig. 16 : Subjective evaluation scores.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 532 papers

MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model
Mar 4, 2026Eduardo Iglesius, Masato Kobayashi, Yuki UranishiRobot Navigation Human-Robot Interaction
The University of Osaka · Kobe University
SPADE: Sketch-guided Path Planning Augmented with Diffusion Experts
Jun 2, 2026Charbel Abi Hana, Tatiana Ghantous, Mikael Khalil +1Data Augmentation Robot Navigation
IDEALworks GmbH · IMT Atlantique · IDEALworks GmbH & Saint Joseph University of Beirut
ERUPT: An Open Toolkit for Interfacing with Robot Motion Planners in Extended Reality
Oct 2, 2025Isaac Ngui, Courtney McBeth, André Santos +6Robotics Human-Robot Interaction
Parasol Lab, Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, IL 61820 USA · Insper, São Paulo, Brazil · Department of Computer Science at Instituto Tecnológico Autónomo de México (ITAM), Mexico City, Mexico

Mar 4, 2026J/K move · Enter open · S save

MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model

Eduardo Iglesius, Masato Kobayashi, Yuki Uranishi

The University of Osaka · Kobe University

Robot Navigation Human-Robot Interaction

Open paper AskPDF arXiv

Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: https://mertcookimg.github.io/mrpos

Abstract

Robot Navigation

Explore similar work

MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model

SPADE: Sketch-guided Path Planning Augmented with Diffusion Experts

ERUPT: An Open Toolkit for Interfacing with Robot Motion Planners in Extended Reality