cs.CVMar 19, 2026

SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

Authors: Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, Karan Ahuja

Organizations: Vasco Xu and Henry Hoffmann are with the University of Chicago. · Brian Chen is with Northwestern University. · The remaining co-authors are with Google. · Karan Ahuja is with Northwestern University and Google.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.

Figures & tables

Figure 1: Accurate surface-based interactions are challenging due to errors in hand tracking and surface plane estimation. (a) There is an offset between the virtual and real hand leading to imprecise inputs: (b) False negatives, where the user is touching a surface but the system thinks it is not and (c) False positives, where the user is not touching the surface but the system thinks it is.
Figure 2: Our data collection setup consists of a Meta Quest 3 headset to capture hand tracking data, a Google Pixel Watch 3 to record IMU data, and Samsung Galaxy Tablet S9 to obtain ground truth annotations for surface contact and gesture events. Participants perform data collection in two conditions — sitting at a desk and standing near a wall — to emulate real-world interaction scenarios.
Figure 3: The SurfaceXR gesture set: single tap, double tap, swipe left, swipe right, pinch in, pinch out, swipe up and swipe down.
Figure 4: (a) Per-gesture recognition latency, (b) Contact state detection onset and offset latency distributions.
Figure 5: Summary of gesture recognition and contact state detection accuracy across different input modalities: IMU-only, Hand-only, and the multimodal model (SurfaceXR), with and without surface plane registration.
Figure 6: Normalized confusion matrix for gesture prediction (in %) using SurfaceXR multimodal model, aggregated across all users.
Figure 7: Drawing application in XR with color selection and tap to clear canvas.
Figure 8: The system offers enough precision for text entry while its comprehensive gesture detection functions as a complete trackpad — providing cursor control, scroll swipes, and double tap selection.
Figure 9: IMU alone can be used to support AR application on glasses.
More in this topic

Gesture Recognition

Gesture recognition maps body, hand, or muscle movements to communicative or control actions using cameras and wearable or wireless sensors.

See all 56 papers

Explore similar work

CardsList

  1. Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G

    Jun 22, 2026Nabeel Nisar Bhat, Javad Sameri, Rreze Halili +3Integrated Sensing and CommunicationSurface Electromyography

    IDLab, University of Antwerp - imec, Antwerp, Belgium · IDLab, Ghent University - imec, Ghent, Belgium · Light and Lighting Technology Lab, KU Leuven, Ghent, Belgium

  2. The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces

    Mar 16, 2026Mohammad Dastgheib, Fatemeh Pourmahdian

    University of California, Riverside · Independent Researcher

  3. Real-Time Hand Gesture Recognition for OpenXR Using Transformer-Based Machine Learning

    Sep 21, 2026Salar Rezayani, Russell ButlerHuman-Computer InteractionHuman Activity Recognition

    University of Waterloo · Bishop’s University