Mixed reality (MR) allows users to inspect planned robot motion within their physical surroundings, but displaying trajectories from robotics software requires converting their joint configurations and timing into a format the MR application can play back. This paper presents ROS2MR, an extension of the MR-Robotics LAB platform that converts YAML-serialized ROS 2 joint trajectories, including MoveIt motion plans, into the platform's existing JSON playback format. The converter validates the input, maps arm joints by name, and resamples the trajectory at a nominal rate of 30 Hz, using cubic Hermite interpolation when joint velocities are supplied and linear interpolation otherwise, while binary gripper commands are sampled separately using a zero-order hold. A Python export helper combines sequences of motion plans and gripper states into a single trajectory and checks positional agreement at segment boundaries. The MR application was also adapted for Apple Vision Pro in addition to Meta Quest 3. A technical evaluation showed that the converter preserved the normalized duration and endpoint configurations for dense, sparse, and irregularly timed inputs. With supplied velocities, the position RMSE was approximately 2.91x10^-5 degrees for all sampling patterns; without them, sparse input increased it to 0.85 degrees, and close positional agreement did not guarantee accurate velocity profiles. A pick-and-place sequence retained its open--closed--open gripper order, although off-grid transitions were shifted to the output sampling grid. The resulting file-based pathway enables inspection of ROS-generated trajectories in MR without a live ROS connection.
Figures & tables
Figure 1: Trajectory conversion workflow in MR-Robotics LAB. The ROS 2 pathway adds a Python export helper and timing-aware trajectory processing, while both input pathways share JSON generation, cloud storage, and mixed-reality playback.
Figure 2: Web interface for trajectory conversion in MR-Robotics LAB [ Berrezueta-Guzman et al., 2026 ] .
Model
Class
Payload (kg)
Reach (mm)
KUKA
KR3 R540
Industrial
3
541
KR6 R900 sixx
Industrial
6
901.5
KR16 R1610-2
Industrial
16
1612
KR20 R1810-2
Industrial
20
1813
KR 120 R2500 pro
Industrial
120
2496
Table 2: Robot models available in MR-Robotics LAB.
Figure 3: Joint 6 position and velocity for an eight-second synthetic cubic trajectory sampled at t={0,2,4,6,8} s. Hermite interpolation with supplied velocities overlaps the reference at this scale; position-only conversion uses linear interpolation and finite differences (FD). Circles mark source positions and supplied velocities.
Input
Velocities
RMSE
Maximum
Dense
Yes
2.91×10−5
5.33×10−5
Sparse
Yes
2.91×10−5
5.33×10−5
Irregular
Yes
2.91×10−5
5.33×10−5
Dense
No
9.20×10−5
3.29×10−4
Sparse
No
0.8536
2.4243
Irregular
No
0.4293
1.4111
Table 3: Position errors for the controlled cubic trajectories. Errors are reported in degrees; supplied velocities are indicated by Yes or No.
Quantity
Velocities
RMSE
Maximum
Position
Yes
2.84×10−5
5.00×10−5
Position
No
7.06×10−4
0.002566
Velocity
Yes
2.58×10−5
5.00×10−5
Velocity
No
0.05249
1.0682
Table 4: Errors for the paired UR5e spline test trajectories. Position errors are in degrees and velocity errors in degrees per second.
Teaching robotics relies on screen-based simulation, showing robot motion in an abstract coordinate frame rather than at real scale in the learner's own space, while access to physical hardware is limited by cost, safety, and scheduling constraints. We present MR-Robotics LAB, a mixed-reality (MR) platform that replays MATLAB-generated robot trajectories at real scale within the learner's physical environment. A browser-based service validates a MATLAB workspace file (.mat), normalizes units, and publishes a versioned JSON trajectory; a Unity application on a Meta Quest 3 then reproduces the authored joint configurations under position control and replays them at the declared frame rate within a physics-enabled scene that supports collision detection and end-effector grasping. A formative single-group evaluation with engineering students found that participants reported low setup effort (M = 4.67 on a 5-point scale) and perceived support for workspace understanding from multi-viewpoint inspection (M = 4.56), and 83% of participants affirmed their willingness to use the platform in an introductory robotics course. The evaluation instrument records only perceived outcomes, without counterbalancing or a learning measure, so no comparative advantage over desktop simulation is claimed. The contribution is a reusable simulation-to-MR trajectory pathway and design guidance for hardware-free robot visualization in engineering education.
Santiago Berrezueta-Guzman, Habiba-Loai Khalil, Andrei Koshelev +2
Technical University of Munich, Heilbronn, Germany
Autonomous mobile robots operating in human-shared indoor environments often require paths that reflect human spatial intentions, such as avoiding interference with pedestrian flow or maintaining comfortable clearance. However, conventional path planners primarily optimize geometric costs and provide limited support for explicit route specification by human operators. This paper presents MRReP, a Mixed Reality-based interface that enables users to draw a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures. The drawn HRP is integrated into the robot navigation stack through a custom Hand-drawn Reference Path Planner, which converts the user-specified point sequence into a global path for autonomous navigation. We evaluated MRReP in a within-subject experiment against a conventional 2D baseline interface. The results demonstrated that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment. These findings suggest that direct path specification in MR is an effective approach for incorporating human spatial intention into mobile robot navigation. Additional material is available at https://mertcookimg.github.io/mrrep
Takumi Taki, Masato Kobayashi, Yuki Uranishi
Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka · Graduate School of Maritime Sciences, Kobe University
Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: https://mertcookimg.github.io/mrpos