Feel My Focus: Gaze-Triggered Digital Twin Creation for Remote Collaboration
Authors: Shih-Yu Ma, Suibi Che-Chuan Weng, Ming-Yuan Pai, Tom Yeh, Ellen Yi-Luen Do
Organizations: Department of Computer Science, University of Colorado Boulder, Boulder, CO, USA · ATLAS Institute, University of Colorado Boulder, Boulder, CO, USA · Department of Music, National Taiwan Normal University, Taipei, Taiwan
We present Feel My Focus, a gaze-triggered mixed reality (MR) demonstration that enables rapid creation and sharing of lightweight 3D digital twins during remote collaboration. Unlike systems that use gaze primarily as a referential pointer, Feel My Focus treats sustained visual attention as a generative action: a fixation on a physical object initiates single-image capture, attention-guided segmentation, and one-shot 3D reconstruction, allowing a remote partner to inspect and discuss the object in situ. The system integrates gaze-tracking glasses, modular segmentation and reconstruction services, and an MR client for runtime streaming and interaction. This design prioritizes responsiveness and experiential continuity over photometric fidelity. We report observations from a public demonstration, highlighting design trade-offs, failure modes, and user control considerations for deploying gaze-triggered generative systems in real-world collaborative settings.
Figures & tables
Fig. 1: Overview of Feel My Focus . (A) The Capture User’s gaze highlights an attended object, which is detected and segmented. (B) The Capture User wearing gaze-tracking glasses focuses on a real-world item. (C) The MR User receives the reconstructed object in an immersive environment. (D) The digital twin appears in the MR scene for inspection and interaction.
Fig. 2: Pipeline of Feel My Focus . A sustained gaze fixation from the Capture User is captured through gaze-tracking glasses and processed into a heatmap and image. The SAM/YOLO service segments the attended object and assigns a label, which is then reconstructed into a textured 3D mesh by Real3D. The resulting digital twin is transmitted to the MR User for immersive inspection and interaction.
Fig. 3: Representative demo outputs of Feel My Focus. (A) Gaze-guided capture with attention heatmap and detection box for a laptop, and (B) the resulting digital twin streamed to the MR client. (C–D) Corresponding capture and reconstructed digital twin for a handheld remote control. These examples illustrate how sustained gaze instantiates shareable 3D proxies for live mixed reality interaction.
Building high-fidelity digital twins of articulated objects from visual data remains a central challenge. Existing approaches depend on multi-view captures of the object in discrete, static states, which severely constrains their real-world scalability. In this paper, we introduce Articulat3D, a novel framework that constructs such digital twins from casually captured monocular videos by jointly enforcing explicit 3D geometric and motion constraints. We first propose Motion Prior-Driven Initialization, which leverages 3D point tracks to exploit the low-dimensional structure of articulated motion. By modeling scene dynamics with a compact set of motion bases, we facilitate soft decomposition of the scene into multiple rigidly moving groups. Building on this initialization, we introduce Geometric and Motion Constraints Refinement, which enforces physically plausible articulation through learnable kinematic primitives parameterized by a joint axis, a pivot point, and per-frame motion scalars, yielding reconstructions that are both geometrically accurate and temporally coherent. Extensive experiments demonstrate that Articulat3D achieves state-of-the-art performance on synthetic benchmarks and real-world casually captured monocular videos, significantly advancing the feasibility of digital twin creation under uncontrolled real-world conditions. Our project page is available at https://maxwell-zhao.github.io/Articulat3D/.
Lijun Guo, Haoyu Zhao, Xingyue Zhao +5
Wuhan University · Hong Kong Embodied AI Lab · The Chinese University of Hong Kong +3
Accurate and safe robotic manipulation under dynamic and visually occluded conditions remains a core challenge in real-world deployment. We introduce SyncTwin, a novel digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware robotic manipulation in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states and uses stored assets to complete partially observed geometry. The synchronized twin enables motion planners to compute collision-aware robot trajectories in simulation, with execution on the real robot closing the real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves manipulation performance and motion safety, demonstrating the effectiveness of digital twin synchronization for real-world robotic execution.
Ruopeng Huang, Boyu Yang, Wenlong Gui +3
University of California, Riverside · University of Southern California
We propose Int3DNet, a scene-aware network that predicts 3D intention areas directly from scene geometry and head-hand motion cues, enabling robust human intention prediction without explicit object-level perception. In Mixed Reality (MR), intention prediction is critical as it enables the system to anticipate user actions and respond proactively, reducing interaction delays and ensuring seamless user experiences. Our method employs a cross attention fusion of sparse motion cues and scene point clouds, offering a novel approach that directly interprets the user's spatial intention within the scene. We evaluated Int3DNet on MoGaze and CIRCLE datasets, which are public datasets for full-body human-scene interactions, showing consistent performance across time horizons of up to 1500 ms and outperforming the baselines, even in diverse and unseen scenes. Moreover, we demonstrate the usability of proposed method through a demonstration of efficient visual question answering (VQA) based on intention areas. Int3DNet provides reliable 3D intention areas derived from head-hand motion and scene geometry, thus enabling seamless interaction between humans and MR systems through proactive processing of intention areas.