cs.HCMar 16, 2026

The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces

Authors: Mohammad Dastgheib, Fatemeh Pourmahdian

Organizations: University of California, Riverside · Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Extended Reality (XR) interfaces impose both ergonomic and cognitive demands, yet current systems often force a binary choice between hand-based input, which can produce fatigue, and gaze-based input, which is vulnerable to the Midas Touch problem and precision limitations. We introduce the xr-adaptive-modality-2025 platform, a web-based open-source framework for studying whether modality-specific adaptive interventions can improve XR-relevant pointing performance and reduce workload relative to static unimodal interaction. The platform combines physiologically informed gaze simulation, an ISO 9241-9 multidirectional tapping task, and two modality-specific adaptive interventions: gaze declutter and hand target-width inflation. We evaluated the system in a 2 x 2 x 2 within-subjects design manipulating Modality (Hand vs. Gaze), UI Mode (Static vs. Adaptive), and Pressure (Yes vs. No). Results from N=69 participants show that hand yielded higher throughput than gaze (5.17 vs. 4.73 bits/s), lower error (1.8% vs. 19.1%), and lower NASA-TLX workload. Crucially, error profiles differed sharply by modality: gaze errors were predominantly slips (99.2%), whereas hand errors were predominantly misses (95.7%), consistent with the Midas Touch account. Of the two adaptive interventions, only gaze declutter executed in this dataset; it modestly reduced timeouts but not slips. Hand width inflation was not evaluable due to a UI integration bug. These findings reveal modality-specific failure modes with direct implications for adaptive policy design, and establish the platform as a reproducible infrastructure for future studies.

Figures & tables

Figure 1: Psychophysically-grounded gaze proxy pipeline. Panel A (left): The three-stage simulation: raw mouse input is differentiated to estimate angular velocity; velocity above 120 deg/s triggers saccadic suppression (cursor freezes, then jumps to new position); velocity below 30 deg/s triggers the fixation transform (Gaussian jitter σ≈\sigma\approx 0.5 mm + first-order lag smoothing). Panel B (center): Output signal examples—fixation mode (B1) produces continuous spatial noise consistent with microsaccade statistics; saccade mode (B2) produces a cursor freeze and ballistic jump, reproducing the perceptual blind phase of real saccades. Panel C (right): The selection model, with a dwell-confirm tolerance ring (target radius + 10 px) that accommodates fixation jitter while keeping sensitivity well-defined.
Figure 2: Task UI overview. ISO 9241-9 multi-directional tapping task: participants select highlighted targets on a central canvas. A side HUD shows modality and block-level feedback.
MetricHandGaze
Throughput (bits/s)5.17 [5.06, 5.27]4.73 [4.58, 4.88]
Error Rate (%)1.77 [1.22, 2.32]19.09 [17.58, 20.59]
Movement Time (s)1.09 [1.07, 1.11]1.19 [1.14, 1.23]
Table 1: Primary performance metrics by modality. Values are mean [95% CI]. Hand produced higher throughput and lower error rate than gaze.
Figure 3: Primary performance by modality. (A) Throughput (bits/s): hand produced higher throughput than gaze. (B) Error rate (%): gaze showed substantially higher error rate than hand. Error bars show 95% CI.
Error TypeHandGaze
Slip (accidental activation)0%99.2%
Miss (target not acquired)95.7%0%
Timeout4.3%0.8%
Table 2: Distribution of error types by modality. Gaze errors are predominantly slips; hand errors are predominantly misses.
Figure 4: Error type composition by modality. Gaze errors are predominantly slips (accidental activations); hand errors are predominantly misses.
NASA-TLX SubscaleHandGaze
Mental Demand33.8 [30.1, 37.4]45.1 [41.4, 48.9]
Physical Demand33.2 [29.4, 37.0]41.1 [37.1, 45.0]
Temporal Demand38.9 [35.3, 42.5]46.5 [43.1, 50.0]
Performance54.3 [48.6, 59.9]52.7 [48.4, 57.0]
Effort38.3 [34.4, 42.3]47.1 [43.3, 51.0]
Frustration31.4 [27.5, 35.4]43.6 [39.6, 47.7]
Table 3: NASA-TLX subscale means [95% CI] by modality. Higher = more workload.
Figure 5: NASA-TLX overall workload (0–100) by modality. Gaze imposed higher subjective workload than hand. Error bars show 95% CI.
Figure 6: NASA-TLX subscales by modality. Point-range plot comparing Hand and Gaze on each subscale (Mental, Physical, Temporal, Performance, Effort, Frustration). Gaze shows higher scores on most dimensions; Physical Demand and Frustration show the largest modality differences. Error bars show 95% CI.
Conditiont0t_{0} (latent) [95% HDI]Drift BaseID Slope [95% HDI]Pressure [95% HDI]
Hand – Static-2.85 [-3.46, -2.24]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Hand – Adaptive-3.01 [-3.64, -2.42]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Gaze – Static-1.41 [-1.87, -1.00]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Gaze – Adaptive-0.97 [-1.39, -0.57]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Table 4: LBA parameter estimates by modality and UI mode. Point estimates with 95% highest-density intervals. t0t_{0} (latent): verification-related offset on the model’s latent scale; values are not in milliseconds. More rightward (less negative) values indicate longer verification-phase duration under the fitted parameterization. ID slope: effect of difficulty on drift (negative = harder trials reduce drift). Pressure slope: effect of time pressure on threshold. Drift base and threshold intercept shared across conditions.
Figure 7: Non-decision time ( t0t_{0} ) by condition. Each row shows one condition; the point is the posterior mean and the horizontal line is the 95% highest-density interval. t0t_{0} is shown on the model’s latent scale, not in raw milliseconds. More rightward (less negative) values correspond to a larger verification-related offset under the fitted parameterization. Gaze conditions (top two rows) show longer verification-related offsets than hand conditions (bottom two rows), consistent with the empirical verification-phase RT and the Midas Touch account.
Figure 8: Empirical verification-phase RT by condition. Mean verification time (ms) from first target entry to selection, with 95% CI. The condition ordering in this figure matches the latent t0t_{0} ordering in Figure 7 , supporting the interpretation that gaze conditions impose a longer verification-phase burden than hand conditions.
ModalityDominant Failure ModeRecommended Adaptation
GazeSlips (unintended activation)Declutter for distraction-driven hesitation; intent disambiguation (dwell, confirmation) for core Midas Touch
HandMisses (spatial targeting)Width inflation / target expansion for acquisition difficulty
Table 5: Design implications: modality-specific failure modes and recommended adaptive interventions.
ConditionSlope (s/bit)R2R^{2}
Hand – Static0.1550.54
Hand – Adaptive0.1460.54
Gaze – Static0.1790.35
Gaze – Adaptive0.190.25
Table 6: Fitts’ Law regression (MT ~ IDeID_{e} ) by condition. Slope = movement-time cost per bit (smaller = higher efficiency); R2R^{2} = proportion of variance explained.
Figure 9: Fitts’ Law validation. Movement time vs. effective Index of Difficulty by modality. Linear fits with 95% CI bands.
Figure 10: MCMC trace plots for key LBA parameters. Left panels: posterior densities; right panels: sampling chains. Drift rate slope (ID) and threshold slope (pressure) show unimodal posteriors and good mixing; non-decision time reflects the four condition-specific estimates.

Explore similar work

CardsList
Mar 19, 2026

SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.

Vasco Xu, Brian Chen, Eric J. Gonzalez +4

Vasco Xu and Henry Hoffmann are with the University of Chicago. · Brian Chen is with Northwestern University. · The remaining co-authors are with Google. +1

Gesture RecognitionMultimodal Sensor FusionHuman-Object InteractionWearable Sensing
AskPDF
Sep 25, 2026

Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

As populations in developed countries age and labor shortages intensify, Cybernetic Avatars (CAs) are proposed to extend human capabilities through robotic embodiments, requiring effective Human-Robot Interaction (HRI) frameworks. Extended Reality (XR), an umbrella term for Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR), offers such interfaces, but prior research typically fixes the XR modality without evaluating its effect on task outcomes. This study examines whether the XR modality impacts HRI performance and whether an adaptive interface adjusting the level of virtuality along the Reality-Virtuality Continuum (RVC) at runtime improves it. A custom XR application interfaced with a mobile manipulator supports immersive control and runtime modality switching. In a within-participant multi-room pick-and-place experiment comparing fixed AR, AV, and VR with dynamic RVC through task metrics, the NASA-TLX, and the System Usability Scale (SUS), this study demonstrates that 1) the fixed reality modality affects HRI results, and 2) dynamically changing the modality along the RVC improves them. AR yielded significantly lower mental demand, effort, and frustration than AV and VR, while the dynamic RVC condition achieved the highest throughput and lowest workload, highlighting the value of adaptive XR interfaces for human-robot symbiosis. The implementation is available at https://github.com/CarlTornberg/XR-HRI.

Carl Tornberg, Alicia Torck, Lotfi El Hafi +1

Ritsumeikan University, 1-1-1 Noji-Higashi, Kusatsu, Shiga 525-8577, Japan · Université catholique de Louvain (UCLouvain), 1 Place de l’Université, Louvain-la-Neuve 1348, Belgium · Kyoto University, Yoshida-Honmachi, Sakyo, Kyoto 606-8501, Japan

Human-Robot InteractionVirtual RealityRobot Teleoperation
AskPDF
Apr 6, 2026

ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality

We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html

Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu +3

King Abdullah University of Science and Technology (KAUST), Saudi Arabia.

Human-Object Interaction
AskPDF