Feel My Focus: Gaze-Triggered Digital Twin Creation for Remote Collaboration
Authors: Shih-Yu Ma, Suibi Che-Chuan Weng, Ming-Yuan Pai, Tom Yeh, Ellen Yi-Luen Do
Organizations: Department of Computer Science, University of Colorado Boulder, Boulder, CO, USA · ATLAS Institute, University of Colorado Boulder, Boulder, CO, USA · Department of Music, National Taiwan Normal University, Taipei, Taiwan
We present Feel My Focus, a gaze-triggered mixed reality (MR) demonstration that enables rapid creation and sharing of lightweight 3D digital twins during remote collaboration. Unlike systems that use gaze primarily as a referential pointer, Feel My Focus treats sustained visual attention as a generative action: a fixation on a physical object initiates single-image capture, attention-guided segmentation, and one-shot 3D reconstruction, allowing a remote partner to inspect and discuss the object in situ. The system integrates gaze-tracking glasses, modular segmentation and reconstruction services, and an MR client for runtime streaming and interaction. This design prioritizes responsiveness and experiential continuity over photometric fidelity. We report observations from a public demonstration, highlighting design trade-offs, failure modes, and user control considerations for deploying gaze-triggered generative systems in real-world collaborative settings.
Figures & tables
Fig. 1: Overview of Feel My Focus . (A) The Capture User’s gaze highlights an attended object, which is detected and segmented. (B) The Capture User wearing gaze-tracking glasses focuses on a real-world item. (C) The MR User receives the reconstructed object in an immersive environment. (D) The digital twin appears in the MR scene for inspection and interaction.
Fig. 2: Pipeline of Feel My Focus . A sustained gaze fixation from the Capture User is captured through gaze-tracking glasses and processed into a heatmap and image. The SAM/YOLO service segments the attended object and assigns a label, which is then reconstructed into a textured 3D mesh by Real3D. The resulting digital twin is transmitted to the MR User for immersive inspection and interaction.
Fig. 3: Representative demo outputs of Feel My Focus. (A) Gaze-guided capture with attention heatmap and detection box for a laptop, and (B) the resulting digital twin streamed to the MR client. (C–D) Corresponding capture and reconstructed digital twin for a handheld remote control. These examples illustrate how sustained gaze instantiates shareable 3D proxies for live mixed reality interaction.