cs.ROSep 25, 2026

Comparative Evaluation of an XR Pen-based Control Interface for Semi-Autonomous Mobile Robot Navigation in Service Environments

Authors: Alicia Torc, Carl Tornberg, Eric Piette, Renaud Ronsse, Benoit Macq, Gustavo Alfonso Garcia Ricardez, Lotfi El Hafi, Tadahiro Taniguchi

Organizations: Universit´e catholique de Louvain (UCLouvain); 1 Place de l’Universit´e, Louvain-la-Neuve 1348, Belgium. · Ritsumeikan University; 1-1-1 Noji-Higashi, Kusatsu, Shiga 525-8577, Japan. · Kyoto University; Yoshida-Honmachi, Sakyo, Kyoto 606-8501, Japan.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Service robots remain difficult to deploy in domestic environments, partly because fully autonomous operation is not yet reliable in unpredictable surroundings, and partly because conventional control methods remain inaccessible to novice users. Extended Reality (XR) enables operators to visualize robot information overlaid onto the real world and to interact with augmented elements. Yet, common XR control methods, such as motion controllers and hand gestures, are still perceived as unintuitive. This paper presents a control interface that uses a commercial XR pen to command a semi-autonomous mobile robot in Augmented Reality (AR): the operator points at a position in the room, selects it, and drags an augmented arrow to set the desired orientation of the robot at this destination. Two additional interfaces, based on the XR motion controllers and hand gestures, were developed within the same framework. To assess the performance and users' perception of these interfaces, and of the XR pen in particular, a study with 10 participants compared four control methods, i.e., the XR pen, the XR motion controllers, hand gestures, and a computer-based baseline RViz, in navigation tasks performed in a home-like environment. Results show that the XR pen significantly outperforms the other methods in task selection time with the most consistent selections, and that the XR motion controllers obtain the best perceived workload and usability scores, ahead of the computer-based baseline, supporting XR-based control as an intuitive alternative for novice users. However, technical limitations in the integration of the recently released XR pen currently hold back its user experience.

Figures & tables

Fig. 1: System overview of the proposed approach. Each colored box groups the components running on one device, and the four control methods are numbered (1) to (4) throughout the paper. The XR inputs (2)–(4) are tracked by the headset running the Unity-based XR application, which sends the selected navigation goals to a semi-autonomous mobile robot over the network, while (1) the computer-based RViz baseline runs on the laptop.
Fig. 2: Integration of the proposed approach, numbered as in Fig. 1 : the scripts of the XR application onboard the headset detect the selection inputs of the tracked devices, shown with the corresponding operator’s point of view, and publish the navigation goals through a ROS TCP connection to a node on the laptop, which relays them to the navigation topic of the robot and also runs (1) the RViz baseline.
Fig. 3: Operator’s point of view through the XR headset for (2) the XR motion controllers, (3) the XR pen, and (4) hand gestures, from left to right. A ray is cast from the connected input toward the ground, and an augmented arrow marks the selected navigation goal and orientation.
Fig. 4: Experimental setup. Top: the simulated daily living environment with the three target locations, the operator seat, and the initial pose of the robot. Bottom: the four compared control methods, namely (1) the computer-based RViz baseline together with the XR-based (2) XR motion controllers, (3) XR pen, and (4) hand gestures.
Fig. 5: Overview of all the trial types performed by each participant: for each of the four control methods (1) to (4), one stationary trial (S) and one dynamic trial (M) for each of the three target locations, with the NASA-TLX and SUS questionnaires completed after each control method.
Metric(1) RViz (baseline)(2) XR motion controllers(3) XR pen(4) Hand gesturespp -value
Qualitative results: user experience
NASA Task Load Index (per dimension 0–20, raw workload 0–100)
Mental demand5.653.357.007.65.2392
Physical demand2.652.555.803.95.1440
Temporal demand4.904.456.307.10.5045
Performance7.558.4510.409.65.7089
TABLE I: Qualitative (NASA-TLX and SUS) and quantitative (task efficiency) results for the four compared control methods, numbered as in Fig. 1 , with the pp -values of the single-factor ANOVA analyses. In each row, the best value is in bold and the worst underlined, a lower value being better for every metric except the SUS score. In the last column, pp -values below the .05.05 significance level are in bold.
More in this topic

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 587 papers

Explore similar work

CardsList
Jul 27, 2026

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming

Extended Reality (XR) is increasingly used in human-robot interaction to communicate robot intent, planned motion, reachability, and state. We argue that XR should also be understood as a mediation layer for situated human control in human-robot teaming. Situated human control denotes the human collaborator's ability to understand, shape, authorize, and interrupt robot action within the concrete physical, social, and temporal context in which that action unfolds. We ground this perspective in scenarios from robot-assisted bedside nursing, multi-arm supervisory control, and collaborative assembly under divided attention. Across these scenarios, robot autonomy must remain inspectable and adjustable as people move, goals change, sensing is incomplete, control roles shift, and plans become invalid. We identify four mediation functions connecting human intent and robot autonomy, robot plans and human judgment, levels of shared control, and team roles, handover, and recovery. Building on these functions, we derive six design dimensions: joint action possibilities, socio-physical constraints, uncertainty and plan validity, multimodal control and correction, roles, handover, and accountability, and anticipatory recovery. The paper outlines a research agenda for XR systems that make robot autonomy more actionable and accountable in dynamic shared environments.

Jens Grubert, John Dudley, Eyal Ofek +1

Coburg University of Applied Sciences and Arts · University of Cambridge · University of Birmingham

Human-Robot CollaborationHuman-Robot Interaction
AskPDF
Mar 26, 2026

RHINO-AR: An Augmented Reality Exhibit for Teaching Mobile Robotics Concepts in Museums

Understanding autonomous mobile robots is challenging for people without robotics expertise, because key processes such as sensing, mapping, and planning are invisible. We present RHINO-AR, an interactive Augmented Reality (AR) museum exhibit that reintroduces the historical mobile robot RHINO into its original exhibition environment at the Deutsches Museum Bonn. To make the autonomous behavior that once made RHINO historically significant accessible again, our previous work, RHINO-VR, reconstructed the robot and its environment in a fully virtual museum scene. While engaging, this fully virtual presentation separated visitors from the real exhibition space and from the physical robot on display. RHINO-AR closes this reality gap by placing a virtual reconstruction of RHINO directly into the real museum space. Implemented on a Magic Leap 2 headset using Unity, our system combines real-time environment meshing with interactive visualizations of LiDAR sensing, traversability, and path planning to make otherwise invisible robotics processes understandable to non-expert visitors. We evaluated RHINO-AR in a two-day museum study with 22 participants, assessing usability, technical performance, satisfaction, conceptual understanding, and preference compared to RHINO-VR. The results indicate that RHINO-AR was well received, conveyed the targeted navigation concepts to a non-expert audience, and was generally preferred over the VR exhibit due to its stronger physical grounding and increased realism.

Nils Dengler, Tim Graf, Leif Van Holland +3

Humanoid Robots Lab, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence and the Center for Robotics, Bonn, Germany

Augmented RealityMobile RoboticsRobot NavigationRoboticsRobotics Education
AskPDF
Mar 4, 2026

MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model

Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: https://mertcookimg.github.io/mrpos

Eduardo Iglesius, Masato Kobayashi, Yuki Uranishi

The University of Osaka · Kobe University

Robot NavigationHuman-Robot Interaction
AskPDF