Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

Carl Tornberg; Alicia Torck; Lotfi El Hafi; Tadahiro Taniguchi

cs.ROSep 25, 2026

Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

Authors: Carl Tornberg, Alicia Torck, Lotfi El Hafi, Tadahiro Taniguchi

Organizations: Ritsumeikan University, 1-1-1 Noji-Higashi, Kusatsu, Shiga 525-8577, Japan · Université catholique de Louvain (UCLouvain), 1 Place de l’Université, Louvain-la-Neuve 1348, Belgium · Kyoto University, Yoshida-Honmachi, Sakyo, Kyoto 606-8501, Japan

Abstract

As populations in developed countries age and labor shortages intensify, Cybernetic Avatars (CAs) are proposed to extend human capabilities through robotic embodiments, requiring effective Human-Robot Interaction (HRI) frameworks. Extended Reality (XR), an umbrella term for Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR), offers such interfaces, but prior research typically fixes the XR modality without evaluating its effect on task outcomes. This study examines whether the XR modality impacts HRI performance and whether an adaptive interface adjusting the level of virtuality along the Reality-Virtuality Continuum (RVC) at runtime improves it. A custom XR application interfaced with a mobile manipulator supports immersive control and runtime modality switching. In a within-participant multi-room pick-and-place experiment comparing fixed AR, AV, and VR with dynamic RVC through task metrics, the NASA-TLX, and the System Usability Scale (SUS), this study demonstrates that 1) the fixed reality modality affects HRI results, and 2) dynamically changing the modality along the RVC improves them. AR yielded significantly lower mental demand, effort, and frustration than AV and VR, while the dynamic RVC condition achieved the highest throughput and lowest workload, highlighting the value of adaptive XR interfaces for human-robot symbiosis. The implementation is available at https://github.com/CarlTornberg/XR-HRI.

Figures & tables

Figure 1: Overview of the proposed approach: an XR HMD operator’s level of reality can be adjusted along the RVC to teleoperate robots to perform tasks.

Title	Description	XR Modality
[ 50 ]	“Augmented reality (AR) interfaces promise to address these challenges, as AR enables us to design expressive visual feedback without many of the constraints of physical reality.”	AR
[ 30 ]	“Augmented Reality (AR) has become a popular multidisciplinary research field over the last decades. […] In robotics, AR acts as a new medium for interaction and information exchange with autonomous systems increasing the efficiency of the Human-Robot Interaction (HRI). The most common definition of AR by Azuma [ 2 ] states that in AR “3D virtual objects are integrated into a 3D real environment in real-time” ”	AR/XR
[ 61 ]	“AR creates a perceptual layer to achieve the complementarity of reality and the environment. Under the premise of AR technology, Mixed Reality (MR) is a further development, using advanced computer technology, image processing technology, and human-computer interaction technology to generate a visual environment with the characteristics of virtual and real integration, in which virtual and real objects coexist and user can interact with them in realtime [ 48 ] ”	MR
[ 58 ]	“VAM interfaces allow users to see 3D virtual imagery in a virtual space or contextually embedded within their environment. […] can also be used hands-free when in the form of a head-mounted display (HMD) that allows for more fluid and natural interactions” “AR visualization techniques offer significant promise in HRI due to their capability to effortlessly communicate robot intent and ” “VR techniques, however, provide unique opportunities for safe, flexible, and novel environments”	VAM/XR
[ 3 ]	“Although the use of AR allows for enhancing HRC through the addition of virtual interfaces, cues, etc., its efficacy first needs to be tested. Such testing of AR interface can be easier to do in VR, where the virtual interfaces can be emulated in a range of scenarios as described above, and thereby would be tried against several options. This, in turn, results in an agile development of solutions that are not limited to a specific laboratory/experimental context.”	VR

Table 1: Summary of XR modalities reasoning by surveys in HRI.

Title	Description	XR Modality
[ 11 ]	“In VR-based HRI, a virtual environment (VE) aided by the necessary sensors provides the operator with an immersive sense of his/her presence at the real location undertaking the tasks” “AR can assist the users to interact intuitively with the virtual objects and spatial information for task planning within the actual working environment”	AR
[ 2 ]	“Recently, Virtual Reality (VR) has been explored as a potential method to simplify demonstration collection process [ 14 ] , [ 62 ] , but such approaches involve additional efforts such as creating realistic VR environments, leading to further complications. We are inspired by an alternative approach, suggested by [ 5 ] , to use Augmented Reality (AR) to enable more natural collection of demonstrations.”	AR
[ 27 ]	“enabling the operator to reschedule tasks on the fly, and how users prefer to coordinate and collaborate with robots. To address these, we propose an Augmented Reality interface”	AR
[ 38 ]	“this paper presents an Augmented Reality-assisted robot programming system […] Augmented Reality (AR) has been widely applied to develop applications to provide guidance and/or electronic user manuals to assist users in manufacturing processes”	AR
[ 57 ]	“explore an alternative design space: using Augmented Reality […] is inspired by prior research envisioning that Augmented Reality head-mounted displays (ARHMDs) might one-day support intuitive human-robot communication for collocated users”	AR
[ 41 ]	“provide digital information for increased situational awareness, […] Using AR to display information, such as robot state, progress and even intent, will enhance understanding, grounding, and thus collaboration.”	AR

Table 2: Summary of AR reasoning in HRI papers.

Title	Description	XR Modality
[ 47 ]	“there are several examples of usage of Augmented Reality (AR) and Mixed Reality (MR) in industry”	AR/MR
[ 40 ]	“human-robot interaction (HRI) requires intuitive and efficient interfaces. One of the best ways to implement this in HRI is augmented and Mixed Reality, according to [ 15 ] .”	AR/MR
[ 23 ]	“MR-headsets […] a more recently developed type of interface, which allows for a bigger variety of interactions.”	MR
[ 49 ]	“MR has been explored as a tool for effective communication between humans and robots […] hold the potential to make the robot’s internal state intuitively understandable to human collaborators by projecting its planned actions as holograms.”	MR
[ 29 ]	“Unlike Augmented Reality (AR), which is limited to the overlay of digital visual content onto the observed scene, MR creates immersive experiences by bringing interactive and spatially contextualized holograms into the real world.”	MR
[ 54 ]	“the key benefits of using MR is the freedom of moving between several points of view to resolve visual ambiguities from 3D to 2D projection”	MR

Table 3: Summary of MR reasoning in HRI papers.

Title	Description	XR Modality
[ 26 ]	“Although the use of AR allows for enhancing HRC through the addition of virtual interfaces, cues, etc., its efficacy first needs to be tested. Such testing of AR interface can be easier to do in VR, where the virtual interfaces can be emulated in a range of scenarios as described above, and thereby would be tried against several options. This, in turn, results in an agile development of solutions that are not limited to a specific laboratory/experimental context.”	VR
[ 21 ]	“[…] leveraging Virtual Reality (VR) technology, which can deliver immersive user experiences with intuitive controls”	VR
[ 60 ]	“It is not always trivial to decide which technology is better when it comes to HRI applications. Despite this, there is a lack of studies that compare the effectiveness of VAM and screen-based interfaces for HRI.” “We chose VR because it provides an immersive remote interaction with the robot.”	VR
[ 45 ]	“Virtual Reality to provide demonstrations to the robotic system in a safe and simplified manner”	VR
[ 39 ]	Missing explanation. “To this end, we performed an experiment in Virtual Reality (VR) where a simulated robot offers tools with different degrees of visual occlusion [of the object handed over].”	VR
[ 19 ]	“Virtual reality has become an increasingly accepted solution in recent years to deal with these limitations and problems.”	VR

Table 4: Summary of VR reasoning in HRI papers.

Figure 2: Milgram’s Reality-Virtuality Continuum, a linear spectrum spanning from a fully real environment to a fully immersive virtual environment.

Figure 3: Implementation diagram. The XR device and robot communicate wirelessly via TCP, and the robot’s egocentric and exocentric data is visualized along the RVC. The user can control the robot’s navigation and manipulation, as well as the level of virtual content along the RVC using the XR motion controllers.

Figure 13: The XR motion controller’s button mapping, where the right hand controls the robot, and the left hand controls the level of reality along the RVC.

Figure 20: Overview of the preliminary experiment’s system and controller types [ 53 ] , in which the participant semi-autonomously navigated the robot to three predefined target locations, each corresponding to a position where the robot could feasibly perform a simulated grasp.

Figure 21: Scene overview of the experiment, showing the simulated service environment with the three locations between which participants transport the two objects.

Figure 22: Mean Time Navigating (in seconds) with 95% confidence intervals across conditions. No statistically significant differences were found between conditions.

Cases	Sum of Squares	df	Mean Square	F	p
Time Navigating [s]	$5199.066$	$3$	$1733.022$	$1.217$	$0.323$
Residuals	$38460.168$	$27$	$1424.451$

Table 5: Within-Subjects Analysis of Variance for Time Spent Navigating Across Conditions

Time Navigating [s]	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$214.106$	$35.848$	$11.336$	$0.167$
AV	$10$	$206.549$	$36.248$	$11.463$	$0.175$
VR	$10$	$187.368$	$56.075$	$17.732$	$0.299$
RVC	$10$	$216.334$	$29.030$	$9.180$	$0.134$

Table 6: Descriptive Statistics for Time Spent Navigating Across Conditions

Figure 23: Mean Time Manipulating (in seconds) with 95% confidence intervals across conditions. No statistically significant differences were found between conditions.

Cases	Sum of Squares	df	Mean Square	F	p
Time Manipulating [s]	$6640.947$	$3$	$2213.649$	$1.493$	$0.239$
Residuals	$40025.749$	$27$	$1482.435$

Table 7: Within-Subjects Analysis of Variance for Time Spent Manipulating Across Conditions

Time Manipulating [s]	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$377.974$	$34.631$	$10.951$	$0.092$
AV	$10$	$393.451$	$36.248$	$11.463$	$0.092$
VR	$10$	$411.901$	$55.562$	$17.570$	$0.135$
RVC	$10$	$383.666$	$29.030$	$9.180$	$0.076$

Table 8: Descriptive Statistics for Time Spent Manipulating Across Conditions

Figure 24: Mean Objects Grasped with 95% confidence intervals across conditions. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Objects Grasped	$13.875$	$3$	$4.625$	$4.325$	$0.013$
Residuals	$28.875$	$27$	$1.069$

Table 9: Within-Subjects Analysis of Variance for Objects Grasped Across Conditions

Objects Grasped	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$3.300$	$1.252$	$0.396$	$0.379$
AV	$10$	$2.400$	$0.966$	$0.306$	$0.403$
VR	$10$	$2.200$	$1.398$	$0.442$	$0.636$
RVC	$10$	$3.600$	$1.075$	$0.340$	$0.299$

Table 10: Descriptive Statistics for Objects Grasped Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$0.900$	$0.526$	$9$	$1.711$	$0.364$
	VR	$1.100$	$0.433$	$9$	$2.538$	$0.127$
	RVC	$-0.300$	$0.423$	$9$	$-0.709$	$0.992$
AV	VR	$0.200$	$0.573$	$9$	$0.349$	$0.992$
	RVC	$-1.200$	$0.389$	$9$	$-3.087$	$0.065$
VR	RVC	$-1.400$	$0.400$	$9$	$-3.500$	$0.040$

Table 11: Post Hoc Comparisons for Objects Grasped Across Conditions

Figure 25: Mean Objects Placed with 95% confidence intervals across conditions. A statistically significant main effect was found, though no pairwise differences reached significance.

Cases	Sum of Squares	df	Mean Square	F	p
Objects Placed	$10.100$	$3$	$3.367$	$3.579$	$0.027$
Residuals	$25.400$	$27$	$0.941$

Table 12: Within-Subjects Analysis of Variance for Objects Placed Across Conditions

Objects Placed	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$2.800$	$1.135$	$0.359$	$0.405$
AV	$10$	$1.700$	$0.949$	$0.300$	$0.558$
VR	$10$	$2.300$	$1.059$	$0.335$	$0.461$
RVC	$10$	$3.000$	$1.054$	$0.333$	$0.351$

Table 13: Descriptive Statistics for Objects Placed Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$1.100$	$0.407$	$9$	$2.703$	$0.121$
	VR	$0.500$	$0.500$	$9$	$1.000$	$0.687$
	RVC	$-0.200$	$0.359$	$9$	$-0.557$	$0.687$
AV	VR	$-0.600$	$0.452$	$9$	$-1.327$	$0.652$
	RVC	$-1.300$	$0.448$	$9$	$-2.899$	$0.106$
VR	RVC	$-0.700$	$0.423$	$9$	$-1.655$	$0.529$

Table 14: Post Hoc Comparisons for Objects Placed Across Conditions

Figure 26: Mean Objects Placed on Incorrect Face with 95% confidence intervals across conditions. No statistically significant main effect was found.

Cases	Sum of Squares	df	Mean Square	F	p
Objects Placed on Incorrect Face	$1.300$	$3$	$0.433$	$1.519$	$0.232$
Residuals	$7.700$	$27$	$0.285$

Table 15: Within-Subjects Analysis of Variance for Objects Placed on Incorrect Face Across Conditions

Objects Placed on Incorrect Face	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$0.300$	$0.675$	$0.213$	$2.250$
AV	$10$	$0.500$	$0.707$	$0.224$	$1.414$
VR	$10$	$0.000$	$0.000$	$0.000$	NaN
RVC	$10$	$0.200$	$0.422$	$0.133$	$2.108$

Table 16: Descriptive Statistics for Objects Placed on Incorrect Face Across Conditions

Figure 27: Mean Objects Dropped with 95% confidence intervals across conditions. No statistically significant main effect was found.

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Objects Dropped	Greenhouse-Geisser	$0.900$	$2.022$	$0.445$	$0.669$	$0.526$
Residuals	Greenhouse-Geisser	$12.100$	$18.198$	$0.665$

Table 17: Within-Subjects Analysis of Variance for Objects Dropped Across Conditions

Objects Dropped	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$0.500$	$0.707$	$0.224$	$1.414$
AV	$10$	$0.500$	$0.707$	$0.224$	$1.414$
VR	$10$	$0.800$	$1.135$	$0.359$	$1.419$
RVC	$10$	$0.400$	$0.516$	$0.163$	$1.291$

Table 18: Descriptive Statistics for Objects Dropped Across Conditions

Figure 28: Mean Collisions with 95% confidence intervals across conditions. No statistically significant main effect was found.

Cases	Sum of Squares	df	Mean Square	F	p
Collisions	$2.275$	$3$	$0.758$	$1.367$	$0.274$
Residuals	$14.975$	$27$	$0.555$

Table 19: Within-Subjects Analysis of Variance for Collisions Across Conditions

Collisions	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$0.700$	$0.483$	$0.153$	$0.690$
AV	$10$	$1.000$	$1.247$	$0.394$	$1.247$
VR	$10$	$1.100$	$1.101$	$0.348$	$1.000$
RVC	$10$	$0.500$	$0.707$	$0.224$	$1.414$

Table 20: Descriptive Statistics for Collisions Across Conditions

Figure 29: NASA-TLX with 95% confidence intervals across conditions. A statistically significant main effect was found, and pairwise differences are denoted by

*

Figure 30: System Usability Scale (SUS) scores across XR conditions.

Condition	AR	AV	VR	RVC
Score	$67.8$	$38.8$	$36.0$	$68.0$

Table 21: System Usability Scale Scores Across Conditions

Figure 31: Mean Time in Modality with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Time in Modality [s]	Greenhouse-Geisser	$1.793\times 10^{+6}$	$1.202$	$1.491\times 10^{+6}$	$90.125$	$<$ .001
Residuals	Greenhouse-Geisser	$179043.752$	$10.819$	$16548.988$

Table 22: Within-Subjects Analysis of Variance for Time in Modality Across Modalities

Time in Modality [s]	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$25.473$	$37.754$	$11.939$	$1.482$
AR	$10$	$497.530$	$114.134$	$36.092$	$0.229$
AR+	$10$	$52.105$	$65.756$	$20.794$	$1.262$
AV	$10$	$18.906$	$32.765$	$10.361$	$1.733$
VR	$10$	$5.986$	$6.668$	$2.109$	$1.114$

Table 23: Descriptive Statistics for Time in Modality Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-472.056$	$44.357$	$9$	$-10.642$	$<$ .001
	AR+	$-26.632$	$20.568$	$9$	$-1.295$	$0.683$
	AV	$6.568$	$13.414$	$9$	$0.490$	$0.683$
	VR	$19.487$	$11.468$	$9$	$1.699$	$0.494$
AR	AR+	$445.425$	$55.931$	$9$	$7.964$	$<$ .001
	AV	$478.624$	$45.113$	$9$	$10.610$	$<$ .001

Table 24: Post Hoc Comparisons for Time in Modality Across Modalities

Figure 32: Mean Time Navigating with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Time Navigating	Greenhouse-Geisser	$130435.193$	$1.493$	$87377.560$	$15.914$	$<$ .001
Residuals	Greenhouse-Geisser	$73767.332$	$13.435$	$5490.687$

Table 25: Within-Subjects Analysis of Variance for Time Navigating Across Modalities

Time Navigating	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$19.808$	$33.656$	$10.643$	$1.699$
AR	$10$	$143.108$	$65.788$	$20.804$	$0.460$
AR+	$10$	$37.150$	$48.145$	$15.225$	$1.296$
AV	$10$	$11.245$	$23.391$	$7.397$	$2.080$
VR	$10$	$5.028$	$6.247$	$1.975$	$1.242$

Table 26: Descriptive Statistics for Time Navigating Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-123.300$	$27.279$	$9$	$-4.520$	$0.012$
	AR+	$-17.342$	$16.253$	$9$	$-1.067$	$0.941$
	AV	$8.563$	$13.592$	$9$	$0.630$	$0.941$
	VR	$14.780$	$10.780$	$9$	$1.371$	$0.814$
AR	AR+	$105.958$	$34.893$	$9$	$3.037$	$0.099$
	AV	$131.863$	$25.660$	$9$	$5.139$	$0.006$

Table 27: Post Hoc Comparisons for Time Navigating Across Modalities

Figure 33: Mean Time Manipulating with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Time Manipulating	Greenhouse-Geisser	$138674.410$	$1.192$	$116316.682$	$19.529$	$<$ .001
Residuals	Greenhouse-Geisser	$63908.829$	$10.730$	$5956.128$

Table 28: Within-Subjects Analysis of Variance for Time Manipulating Across Modalities

Time Manipulating	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$5.661$	$9.909$	$3.133$	$1.750$
AR	$10$	$143.108$	$65.788$	$20.804$	$0.460$
AR+	$10$	$37.150$	$48.145$	$15.225$	$1.296$
AV	$10$	$11.245$	$23.391$	$7.397$	$2.080$
VR	$10$	$5.028$	$6.247$	$1.975$	$1.242$

Table 29: Descriptive Statistics for Time Manipulating Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-137.447$	$23.163$	$9$	$-5.934$	$0.002$
	AR+	$-31.489$	$12.728$	$9$	$-2.474$	$0.212$
	AV	$-5.585$	$7.197$	$9$	$-0.776$	$1.000$
	VR	$0.632$	$3.350$	$9$	$0.189$	$1.000$
AR	AR+	$105.958$	$34.893$	$9$	$3.037$	$0.099$
	AV	$131.863$	$25.660$	$9$	$5.139$	$0.005$

Table 30: Post Hoc Comparisons for Time Manipulating Across Modalities

Figure 34: Mean Objects Grasped with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Objects Grasped	$114.080$	$4$	$28.520$	$95.776$	$<$ .001
Residuals	$10.720$	$36$	$0.298$

Table 31: Within-Subjects Analysis of Variance for Objects Grasped Across Modalities

Objects Grasped	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$0.000$	$0.000$	$0.000$	NaN
AR	$10$	$3.800$	$1.135$	$0.359$	$0.299$
AR+	$10$	$0.000$	$0.000$	$0.000$	NaN
AV	$10$	$0.100$	$0.316$	$0.100$	$3.162$
VR	$10$	$0.000$	$0.000$	$0.000$	NaN

Table 32: Descriptive Statistics for Objects Grasped Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-3.800$	$0.359$	$9$	$-10.585$	$<$ .001
	AR+	$0.000$	$0.000$	$9$	NaN	NaN
	AV	$-0.100$	$0.100$	$9$	$-1.000$	$1.000$
	VR	$0.000$	$0.000$	$9$	NaN	NaN
AR	AR+	$3.800$	$0.359$	$9$	$10.585$	$<$ .001
	AV	$3.700$	$0.423$	$9$	$8.748$	$<$ .001

Table 33: Post Hoc Comparisons for Objects Grasped Across Modalities

Figure 35: Mean Objects Placed with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Objects Placed	$63.920$	$4$	$15.980$	$57.071$	$<$ .001
Residuals	$10.080$	$36$	$0.280$

Table 34: Within-Subjects Analysis of Variance for Objects Placed Across Modalities

Objects Placed	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$0.100$	$0.316$	$0.100$	$3.162$
AR	$10$	$2.900$	$0.994$	$0.314$	$0.343$
AR+	$10$	$0.100$	$0.316$	$0.100$	$3.162$
AV	$10$	$0.100$	$0.316$	$0.100$	$3.162$
VR	$10$	$0.000$	$0.000$	$0.000$	NaN

Table 35: Descriptive Statistics for Objects Placed Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-2.800$	$0.359$	$9$	$-7.799$	$<$ .001
	AR+	$0.000$	$0.000$	$9$	NaN	NaN
	AV	$-2.776\times 10^{-17}$	$0.149$	$9$	$-1.862\times 10^{-16}$	$1.000$
	VR	$0.100$	$0.100$	$9$	$1.000$	$1.000$
AR	AR+	$2.800$	$0.359$	$9$	$7.799$	$<$ .001
	AV	$2.800$	$0.359$	$9$	$7.799$	$<$ .001

Table 36: Post Hoc Comparisons for Objects Placed Across Modalities

Figure 36: Mean Objects Placed on Incorrect Face with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Objects Placed on Incorrect Face	$0.320$	$4$	$0.080$	$2.250$	$0.083$
Residuals	$1.280$	$36$	$0.036$

Table 37: Within-Subjects Analysis of Variance for Objects Placed on Incorrect Face Across Modalities

Objects Placed on Incorrect Face	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$0.000$	$0.000$	$0.000$	NaN
AR	$10$	$0.200$	$0.422$	$0.133$	$2.108$
AR+	$10$	$0.000$	$0.000$	$0.000$	NaN
AV	$10$	$0.000$	$0.000$	$0.000$	NaN
VR	$10$	$0.000$	$0.000$	$0.000$	NaN

Table 38: Descriptive Statistics for Objects Placed on Incorrect Face Across Modalities

Figure 37: Mean Objects Dropped with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Objects Dropped	$2.000$	$4$	$0.500$	$9.000$	$<$ .001
Residuals	$2.000$	$36$	$0.056$

Table 39: Within-Subjects Analysis of Variance for Objects Dropped Across Modalities

Objects Dropped	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$0.000$	$0.000$	$0.000$	NaN
AR	$10$	$0.500$	$0.527$	$0.167$	$1.054$
AR+	$10$	$0.000$	$0.000$	$0.000$	NaN
AV	$10$	$0.000$	$0.000$	$0.000$	NaN
VR	$10$	$0.000$	$0.000$	$0.000$	NaN

Table 40: Descriptive Statistics for Objects Dropped Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-0.500$	$0.167$	$9$	$-3.000$	$0.060$
	AR+	$0.000$	$0.000$	$9$	NaN	NaN
	AV	$0.000$	$0.000$	$9$	NaN	NaN
	VR	$0.000$	$0.000$	$9$	NaN	NaN
AR	AR+	$0.500$	$0.167$	$9$	$3.000$	$0.060$
	AV	$0.500$	$0.167$	$9$	$3.000$	$0.060$

Table 41: Post Hoc Comparisons for Objects Dropped Across Modalities

Figure 38: Mean Collisions with 95% confidence intervals across modalities. A statistically significant main effect was found, and pairwise differences are denoted by

*

Cases	Sum of Squares	df	Mean Square	F	p
Collisions in RVC	$2.000$	$4$	$0.500$	$5.000$	$0.003$
Residuals	$3.600$	$36$	$0.100$

Table 42: Within-Subjects Analysis of Variance for Collisions Across Modalities

Collisions in RVC	N	Mean	SD	SE	Coefficient of variation
Reality	$10$	$0.000$	$0.000$	$0.000$	NaN
AR	$10$	$0.500$	$0.707$	$0.224$	$1.414$
AR+	$10$	$0.000$	$0.000$	$0.000$	NaN
AV	$10$	$0.000$	$0.000$	$0.000$	NaN
VR	$10$	$0.000$	$0.000$	$0.000$	NaN

Table 43: Descriptive Statistics for Collisions Across Modalities

		Mean Difference	SE	df	t	p holm
Reality	AR	$-0.500$	$0.224$	$9$	$-2.236$	$0.209$
	AR+	$0.000$	$0.000$	$9$	NaN	NaN
	AV	$0.000$	$0.000$	$9$	NaN	NaN
	VR	$0.000$	$0.000$	$9$	NaN	NaN
AR	AR+	$0.500$	$0.224$	$9$	$2.236$	$0.209$
	AV	$0.500$	$0.224$	$9$	$2.236$	$0.209$

Table 44: Post Hoc Comparisons for Collisions Across Modalities

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Mental Demand	Greenhouse-Geisser	$516.875$	$1.579$	$327.355$	$18.881$	$<$ .001
Residuals	Greenhouse-Geisser	$246.375$	$14.210$	$17.338$

Appendix

Table 45: Within-Subjects Analysis of Variance for the NASA-TLX Mental Demand Across Conditions

Mental Demand	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$7.700$	$4.990$	$1.578$	$0.648$
AV	$10$	$14.100$	$4.909$	$1.552$	$0.348$
VR	$10$	$14.800$	$2.936$	$0.929$	$0.198$
RVC	$10$	$6.900$	$4.067$	$1.286$	$0.589$

Appendix

Table 46: Descriptive Statistics for NASA-TLX Mental Demand Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$-6.400$	$1.863$	$9$	$-3.435$	$0.022$
	VR	$-7.100$	$1.362$	$9$	$-5.214$	$0.003$
	RVC	$0.800$	$0.512$	$9$	$1.562$	$0.305$
AV	VR	$-0.700$	$1.221$	$9$	$-0.573$	$0.580$
	RVC	$7.200$	$1.625$	$9$	$4.431$	$0.007$
VR	RVC	$7.900$	$1.110$	$9$	$7.117$	$<$ .001

Appendix

Table 47: Post Hoc Comparisons for NASA-TLX Mental Demand Across Conditions

Cases	Sum of Squares	df	Mean Square	F	p
Physical Demand	$80.900$	$3$	$26.967$	$5.311$	$0.005$
Residuals	$137.100$	$27$	$5.078$

Appendix

Table 48: Within-Subjects Analysis of Variance for the NASA-TLX Physical Demand Across Conditions

Physical Demand	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$3.800$	$1.751$	$0.554$	$0.461$
AV	$10$	$7.000$	$3.621$	$1.145$	$0.517$
VR	$10$	$6.900$	$3.381$	$1.069$	$0.490$
RVC	$10$	$4.500$	$1.650$	$0.522$	$0.367$

Appendix

Table 49: Descriptive Statistics for NASA-TLX Physical Demand Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$-3.200$	$1.162$	$9$	$-2.753$	$0.134$
	VR	$-3.100$	$1.224$	$9$	$-2.532$	$0.161$
	RVC	$-0.700$	$0.517$	$9$	$-1.353$	$0.418$
AV	VR	$0.100$	$0.752$	$9$	$0.133$	$0.897$
	RVC	$2.500$	$1.035$	$9$	$2.414$	$0.161$
VR	RVC	$2.400$	$1.157$	$9$	$2.075$	$0.203$

Appendix

Table 50: Post Hoc Comparisons for NASA-TLX Physical Demand Across Conditions

Cases	Sum of Squares	df	Mean Square	F	p
Temporal Demand	$6.200$	$3$	$2.067$	$0.351$	$0.788$
Residuals	$158.800$	$27$	$5.881$

Appendix

Table 51: Within-Subjects Analysis of Variance for the NASA-TLX Temporal Demand Across Conditions

Temporal Demand	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$8.400$	$4.452$	$1.408$	$0.530$
AV	$10$	$7.700$	$2.869$	$0.907$	$0.373$
VR	$10$	$7.300$	$3.433$	$1.086$	$0.470$
RVC	$10$	$7.800$	$3.994$	$1.263$	$0.512$

Appendix

Table 52: Descriptive Statistics for NASA-TLX Temporal Demand Across Conditions

Cases	Sum of Squares	df	Mean Square	F	p
Performance	$55.475$	$3$	$18.492$	$0.892$	$0.458$
Residuals	$559.775$	$27$	$20.732$

Appendix

Table 53: Within-Subjects Analysis of Variance for the NASA-TLX Performance Across Conditions

Performance	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$7.600$	$4.427$	$1.400$	$0.583$
AV	$10$	$8.700$	$4.945$	$1.564$	$0.568$
VR	$10$	$10.700$	$5.229$	$1.654$	$0.489$
RVC	$10$	$8.100$	$6.315$	$1.997$	$0.780$

Appendix

Table 54: Descriptive Statistics for NASA-TLX Performance Across Conditions

Cases	Sum of Squares	df	Mean Square	F	p
Effort	$313.675$	$3$	$104.558$	$11.833$	$<$ .001
Residuals	$238.575$	$27$	$8.836$

Appendix

Table 55: Within-Subjects Analysis of Variance for the NASA-TLX Effort Across Conditions

Effort	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$7.100$	$4.254$	$1.345$	$0.599$
AV	$10$	$11.800$	$4.662$	$1.474$	$0.395$
VR	$10$	$11.500$	$4.089$	$1.293$	$0.356$
RVC	$10$	$5.300$	$3.713$	$1.174$	$0.701$

Appendix

Table 56: Descriptive Statistics for NASA-TLX Effort Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$-4.700$	$1.155$	$9$	$-4.069$	$0.011$
	VR	$-4.400$	$1.600$	$9$	$-2.750$	$0.067$
	RVC	$1.800$	$1.041$	$9$	$1.728$	$0.236$
AV	VR	$0.300$	$1.680$	$9$	$0.179$	$0.862$
	RVC	$6.500$	$1.088$	$9$	$5.975$	$0.001$
VR	RVC	$6.200$	$1.272$	$9$	$4.875$	$0.004$

Appendix

Table 57: Post Hoc Comparisons for NASA-TLX Effort Across Conditions

Cases	Sphericity Correction	Sum of Squares	df	Mean Square	F	p
Frustration	Greenhouse-Geisser	$648.200$	$1.741$	$372.407$	$12.702$	$<$ .001
Residuals	Greenhouse-Geisser	$459.300$	$15.665$	$29.320$

Appendix

Table 58: Within-Subjects Analysis of Variance for the NASA-TLX Frustration Across Conditions

Frustration	N	Mean	SD	SE	Coefficient of variation
AR	$10$	$5.900$	$5.109$	$1.616$	$0.866$
AV	$10$	$9.900$	$3.872$	$1.224$	$0.391$
VR	$10$	$14.000$	$4.422$	$1.398$	$0.316$
RVC	$10$	$3.400$	$2.066$	$0.653$	$0.608$

Appendix

Table 59: Descriptive Statistics for NASA-TLX Frustration Across Conditions

		Mean Difference	SE	df	t	p holm
AR	AV	$-4.000$	$2.671$	$9$	$-1.498$	$0.337$
	VR	$-8.100$	$1.670$	$9$	$-4.851$	$0.004$
	RVC	$2.500$	$1.845$	$9$	$1.355$	$0.337$
AV	VR	$-4.100$	$1.683$	$9$	$-2.436$	$0.113$
	RVC	$6.500$	$1.310$	$9$	$4.961$	$0.004$
VR	RVC	$10.600$	$1.593$	$9$	$6.654$	$<$ .001

Appendix

Table 60: Post Hoc Comparisons for NASA-TLX Frustration Across Conditions

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming
Jul 27, 2026Jens Grubert, John Dudley, Eyal Ofek +1Human-Robot Collaboration Human-Robot Interaction
Coburg University of Applied Sciences and Arts · University of Cambridge · University of Birmingham
Validating Virtual Reality for Studying Multimodal Human-Robot Interaction in Socially Aware Robot Navigation
Jul 10, 2026Hariharan Arunachalam, Phani Teja Singamaneni, Rachid AlamiVirtual Reality Social Robot Navigation
LAAS-CNRS, Universite de Toulouse, France · Inria Nancy, France
Examining the Difference in Human Behavior Between Virtual and Real-World Human-Robot Teaming
Sep 16, 2026Sean Dallas, Absalat Getachew, Motaz AbuHijleh +4Virtual Reality Human-Robot Collaboration
Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, United States · College of Innovation & Technology, University of Michigan-Flint, Flint, MI 48502, United States · U.S. Army DEVCOM Ground Vehicle Systems Center (GVSC), United States

Jul 27, 2026J/K move · Enter open · S save

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming

Jens Grubert, John Dudley, Eyal Ofek, Per Ola Kristensson

Coburg University of Applied Sciences and Arts · University of Cambridge · University of Birmingham

Human-Robot Collaboration Human-Robot Interaction

Open paper AskPDF arXiv

Extended Reality (XR) is increasingly used in human-robot interaction to communicate robot intent, planned motion, reachability, and state. We argue that XR should also be understood as a mediation layer for situated human control in human-robot teaming. Situated human control denotes the human collaborator's ability to understand, shape, authorize, and interrupt robot action within the concrete physical, social, and temporal context in which that action unfolds. We ground this perspective in scenarios from robot-assisted bedside nursing, multi-arm supervisory control, and collaborative assembly under divided attention. Across these scenarios, robot autonomy must remain inspectable and adjustable as people move, goals change, sensing is incomplete, control roles shift, and plans become invalid. We identify four mediation functions connecting human intent and robot autonomy, robot plans and human judgment, levels of shared control, and team roles, handover, and recovery. Building on these functions, we derive six design dimensions: joint action possibilities, socio-physical constraints, uncertainty and plan validity, multimodal control and correction, roles, handover, and accountability, and anticipatory recovery. The paper outlines a research agenda for XR systems that make robot autonomy more actionable and accountable in dynamic shared environments.

Abstract

Human-Robot Interaction

Explore similar work

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming

Validating Virtual Reality for Studying Multimodal Human-Robot Interaction in Socially Aware Robot Navigation

Examining the Difference in Human Behavior Between Virtual and Real-World Human-Robot Teaming