Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: https://mertcookimg.github.io/mrpos
Figures & tables
Fig. 1 : MRPoS Workflow . The workflow is divided into 2 phases, which are MR-beacon Generation and Robot Navigation. (1-a, 1-b) shows the MR-beacon generation by using spatial pointer and voice. (1-c) shows the final state after the generation. (2-a, 2-b) shows the robot navigation towards the created MR-beacon. (2-c) shows the final state.
Fig. 2 : System Design Diagram. Our system consists of four main modules: the HoloLens 2, serving as the user interface to acquire, analyze, and visualize MR-beacons; ROS 2, acting as a bridge between the HoloLens 2 and the robot; a Model Server to process user voice data; and the robot itself. Components highlighted in green represent our novel contributions.
Gesture-Based MR
Language-Centric
MRPoS (Hybrid)
Natural Language Interaction
✗
✓
✓
Visual Intent Transparency
✓
✗
✓
Precise Spatial Grounding
✓
✗
✓
Low Workload
✗
✓
✓
TABLE I : Qualitative design-space summary of robotic navigation interface paradigms.
Fig. 3 : Hand Menu. (a) The primary layout of the hand menu. (b) The Beacon submenu, accessed via the main menu.
Fig. 4 : Add Function (Multiple Objects). (a) Initial empty environment. (b) User positioning the first MR-beacon candidate. (c) User specifying orientation toward the water bottle via speech by specifying the name. (d, e) User repeating the same process for the coffee machine and flower pot simultaneously. (f) Final state with three generated MR-beacons.
Fig. 5 : Edit Function. (a) Initial state with an MR-beacon. (b) User selecting the MR-beacon via spatial pointing. (c) User relocating it and specifying a new orientation (flower pot) via speech. (d) Final state of the modified MR-beacon.
Fig. 6 : Go Function. (a) Initial state with an MR-beacon. (b) User pointing to the target MR-beacon and issuing the “Go” command. (c) Robot navigating to the target. (d) Successful arrival at the destination.
Fig. 7 : Delete Function. (a) Initial state with an MR-beacon. (b) Pointing to select the target MR-beacon. (c) Issuing the “Delete” command to remove the MR-beacon. (d) Final state with the MR-beacon removed.
Fig. 8 : MR-label . (a) illustrates the components of the MR-label, which consists of a name panel to display the location’s name and a name stand to mark its position on the floor. (b) demonstrates the deployment of MR-labels in a real-world
Fig. 9 : Database. The MR-label database logs the ID, position (PO), and name (NM). The MR-beacons database logs the ID, position (PO), and rotation (RO).
Fig. 10 : Experiment Environment . (a) shows the figure of the real-world experiment environment. (b) shows the figure of the experiment environment from 2D map SLAM.
Fig. 11 : Overall Experiment Flow comparing MRNaB and MRPoS. Upper image shows the experiment flow while the lower image shows the main experiment flow. Both groups access the hand menu to generate the initial objects. Next, MRNaB employs hand gestures to instantiate and select MR-beacon. In contrast, MRPoS utilizes speech and spatial pointing to create and select MR-beacon. Once selected, robot starts the navigation.
Fig. 12 : Time and Action Count Result . Statistical difference on Stage 1 & 2 for time and on Stage 2 for action.
Time [sec]
Action Count
Stage
MRNaB
MRPoS
MRNaB
MRPoS
Stage 1
17.21
5.37
1.38
1.12
Stage 2
42.92
17.88
3.44
1.69
TABLE II : Time and Action Count Results
Fig. 13 : Location and Rotation Error per MR-beacon Result .
Location Error [m]
Rotation Error [ ∘ ]
Stage
MRNaB
MRPoS
MRNaB
MRPoS
Stage 1
0.084
0.080
4.70
8.27
Stage 2
0.049
0.059
2.98
3.83
TABLE III : Location and Rotation Error
Fig. 14 : SUS and NASA-TLX Result . Statistical difference can be found for NASA-TLX
Metric
MRNaB
MRPoS
p -value
Effect Size
SUS Score ↑
70.16
76.72
0.064
0.464
NASA-TLX ↓
35.27
21.67
0.013
0.620
TABLE IV : Statistical Analysis of Subjective Metrics (SUS and NASA-TLX)
Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka · Graduate School of Maritime Sciences, Kobe University