cs.HCMar 16, 2026

The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces

Authors: Mohammad Dastgheib, Fatemeh Pourmahdian

Organizations: University of California, Riverside · Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Extended Reality (XR) interfaces impose both ergonomic and cognitive demands, yet current systems often force a binary choice between hand-based input, which can produce fatigue, and gaze-based input, which is vulnerable to the Midas Touch problem and precision limitations. We introduce the xr-adaptive-modality-2025 platform, a web-based open-source framework for studying whether modality-specific adaptive interventions can improve XR-relevant pointing performance and reduce workload relative to static unimodal interaction. The platform combines physiologically informed gaze simulation, an ISO 9241-9 multidirectional tapping task, and two modality-specific adaptive interventions: gaze declutter and hand target-width inflation. We evaluated the system in a 2 x 2 x 2 within-subjects design manipulating Modality (Hand vs. Gaze), UI Mode (Static vs. Adaptive), and Pressure (Yes vs. No). Results from N=69 participants show that hand yielded higher throughput than gaze (5.17 vs. 4.73 bits/s), lower error (1.8% vs. 19.1%), and lower NASA-TLX workload. Crucially, error profiles differed sharply by modality: gaze errors were predominantly slips (99.2%), whereas hand errors were predominantly misses (95.7%), consistent with the Midas Touch account. Of the two adaptive interventions, only gaze declutter executed in this dataset; it modestly reduced timeouts but not slips. Hand width inflation was not evaluable due to a UI integration bug. These findings reveal modality-specific failure modes with direct implications for adaptive policy design, and establish the platform as a reproducible infrastructure for future studies.

Figures & tables

Figure 1: Psychophysically-grounded gaze proxy pipeline. Panel A (left): The three-stage simulation: raw mouse input is differentiated to estimate angular velocity; velocity above 120 deg/s triggers saccadic suppression (cursor freezes, then jumps to new position); velocity below 30 deg/s triggers the fixation transform (Gaussian jitter σ≈\sigma\approx 0.5 mm + first-order lag smoothing). Panel B (center): Output signal examples—fixation mode (B1) produces continuous spatial noise consistent with microsaccade statistics; saccade mode (B2) produces a cursor freeze and ballistic jump, reproducing the perceptual blind phase of real saccades. Panel C (right): The selection model, with a dwell-confirm tolerance ring (target radius + 10 px) that accommodates fixation jitter while keeping sensitivity well-defined.
Figure 2: Task UI overview. ISO 9241-9 multi-directional tapping task: participants select highlighted targets on a central canvas. A side HUD shows modality and block-level feedback.
MetricHandGaze
Throughput (bits/s)5.17 [5.06, 5.27]4.73 [4.58, 4.88]
Error Rate (%)1.77 [1.22, 2.32]19.09 [17.58, 20.59]
Movement Time (s)1.09 [1.07, 1.11]1.19 [1.14, 1.23]
Table 1: Primary performance metrics by modality. Values are mean [95% CI]. Hand produced higher throughput and lower error rate than gaze.
Figure 3: Primary performance by modality. (A) Throughput (bits/s): hand produced higher throughput than gaze. (B) Error rate (%): gaze showed substantially higher error rate than hand. Error bars show 95% CI.
Error TypeHandGaze
Slip (accidental activation)0%99.2%
Miss (target not acquired)95.7%0%
Timeout4.3%0.8%
Table 2: Distribution of error types by modality. Gaze errors are predominantly slips; hand errors are predominantly misses.
Figure 4: Error type composition by modality. Gaze errors are predominantly slips (accidental activations); hand errors are predominantly misses.
NASA-TLX SubscaleHandGaze
Mental Demand33.8 [30.1, 37.4]45.1 [41.4, 48.9]
Physical Demand33.2 [29.4, 37.0]41.1 [37.1, 45.0]
Temporal Demand38.9 [35.3, 42.5]46.5 [43.1, 50.0]
Performance54.3 [48.6, 59.9]52.7 [48.4, 57.0]
Effort38.3 [34.4, 42.3]47.1 [43.3, 51.0]
Frustration31.4 [27.5, 35.4]43.6 [39.6, 47.7]
Table 3: NASA-TLX subscale means [95% CI] by modality. Higher = more workload.
Figure 5: NASA-TLX overall workload (0–100) by modality. Gaze imposed higher subjective workload than hand. Error bars show 95% CI.
Figure 6: NASA-TLX subscales by modality. Point-range plot comparing Hand and Gaze on each subscale (Mental, Physical, Temporal, Performance, Effort, Frustration). Gaze shows higher scores on most dimensions; Physical Demand and Frustration show the largest modality differences. Error bars show 95% CI.
Conditiont0t_{0} (latent) [95% HDI]Drift BaseID Slope [95% HDI]Pressure [95% HDI]
Hand – Static-2.85 [-3.46, -2.24]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Hand – Adaptive-3.01 [-3.64, -2.42]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Gaze – Static-1.41 [-1.87, -1.00]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Gaze – Adaptive-0.97 [-1.39, -0.57]5.03-0.93 [-0.95, -0.92]0.06 [0.03, 0.09]
Table 4: LBA parameter estimates by modality and UI mode. Point estimates with 95% highest-density intervals. t0t_{0} (latent): verification-related offset on the model’s latent scale; values are not in milliseconds. More rightward (less negative) values indicate longer verification-phase duration under the fitted parameterization. ID slope: effect of difficulty on drift (negative = harder trials reduce drift). Pressure slope: effect of time pressure on threshold. Drift base and threshold intercept shared across conditions.
Figure 7: Non-decision time ( t0t_{0} ) by condition. Each row shows one condition; the point is the posterior mean and the horizontal line is the 95% highest-density interval. t0t_{0} is shown on the model’s latent scale, not in raw milliseconds. More rightward (less negative) values correspond to a larger verification-related offset under the fitted parameterization. Gaze conditions (top two rows) show longer verification-related offsets than hand conditions (bottom two rows), consistent with the empirical verification-phase RT and the Midas Touch account.
Figure 8: Empirical verification-phase RT by condition. Mean verification time (ms) from first target entry to selection, with 95% CI. The condition ordering in this figure matches the latent t0t_{0} ordering in Figure 7 , supporting the interpretation that gaze conditions impose a longer verification-phase burden than hand conditions.
ModalityDominant Failure ModeRecommended Adaptation
GazeSlips (unintended activation)Declutter for distraction-driven hesitation; intent disambiguation (dwell, confirmation) for core Midas Touch
HandMisses (spatial targeting)Width inflation / target expansion for acquisition difficulty
Table 5: Design implications: modality-specific failure modes and recommended adaptive interventions.
ConditionSlope (s/bit)R2R^{2}
Hand – Static0.1550.54
Hand – Adaptive0.1460.54
Gaze – Static0.1790.35
Gaze – Adaptive0.190.25
Table 6: Fitts’ Law regression (MT ~ IDeID_{e} ) by condition. Slope = movement-time cost per bit (smaller = higher efficiency); R2R^{2} = proportion of variance explained.
Figure 9: Fitts’ Law validation. Movement time vs. effective Index of Difficulty by modality. Linear fits with 95% CI bands.
Figure 10: MCMC trace plots for key LBA parameters. Left panels: posterior densities; right panels: sampling chains. Drift rate slope (ID) and threshold slope (pressure) show unimodal posteriors and good mixing; non-decision time reflects the four condition-specific estimates.

Explore similar work

CardsList

  1. SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

    Mar 19, 2026Vasco Xu, Brian Chen, Eric J. Gonzalez +4Gesture RecognitionMultimodal Sensor Fusion

    Vasco Xu and Henry Hoffmann are with the University of Chicago. · Brian Chen is with Northwestern University. · The remaining co-authors are with Google. +1

  2. Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

    Sep 25, 2026Carl Tornberg, Alicia Torck, Lotfi El Hafi +1Human-Robot InteractionVirtual Reality

    Ritsumeikan University, 1-1-1 Noji-Higashi, Kusatsu, Shiga 525-8577, Japan · Université catholique de Louvain (UCLouvain), 1 Place de l’Université, Louvain-la-Neuve 1348, Belgium · Kyoto University, Yoshida-Honmachi, Sakyo, Kyoto 606-8501, Japan

  3. ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality

    Apr 6, 2026Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu +3Human-Object Interaction

    King Abdullah University of Science and Technology (KAUST), Saudi Arabia.