GLANCE: Gaze-Led Attention Network for Compressed Edge-inference
Authors: Neeraj Solanki, Hong Ding, Sepehr Tabrizchi, Ali Shafiee Sarvestani, Shaahin Angizi, David Z. Pan, Arman Roohi
Organizations: Neeraj Solanki, Hong Ding, Sepehr Tabrizchi, Ali Shafiee Sarvestani, and Arman Roohi are with the Department of Electrical and Computer Engineering, University of Illinois Chicago, Chicago IL, USA. · S. Angizi is with the Department of Electrical and Computer Engineering, New Jersey Institute of Technology (NJIT), Newark, NJ, USA · D. Z. Pan is with the Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA.
Real-time object detection in AR/VR systems faces critical computational constraints, requiring sub-10,ms latency within tight power budgets. Inspired by biological foveal vision, we propose a two-stage pipeline that combines differentiable weightless neural networks for ultra-efficient gaze estimation with attention-guided region-of-interest object detection. Our approach eliminates arithmetic-intensive operations by performing gaze tracking through memory lookups rather than multiply-accumulate computations, achieving an angular error of 8.32∘ with only 393 MACs and 2.2 KiB of memory per frame. Gaze predictions guide selective object detection on attended regions, reducing computational burden by 40-50% and energy consumption by 65%. Deployed on the Arduino Nano 33 BLE, our system achieves 48.1% mAP on COCO (51.8% on attended objects) while maintaining sub-10,ms latency, meeting stringent AR/VR requirements by improving the communication time by ×177. Compared to the global YOLOv12n baseline, which achieves 39.2%, 63.4%, and 83.1% accuracy for small, MEDium, and LARGE objects, respectively, the ROI-based method yields 51.3%, 72.1%, and 88.1% under the same settings. This work shows that memory-centric architectures with explicit attention modeling offer better efficiency and accuracy for resource-constrained wearable platforms than uniform processing.
Figures & tables
Fig. 1 : (a) GLANCE overview: DWN-based gaze estimation drives union-of-ROIs attention; a periodic/conditional policy ships a single mosaic crop to the detector. (b) MCU/host dataflow. The host forms a per-frame spatial union to produce one union-of-ROIs mosaic and maintains a temporal state Ut to trigger YOLO under a periodic/budget policy.
Fig. 2 : Proposed DWN architecture.
Fig. 3 : (a) ROI scale from gaze uncertainty. ROI side length as a function of hit probability p , with S(p)=2ep and e=46.7 px as the projected uncertainty for an 8.3∘ gaze error. (b) ROI selection from gaze uncertainty. Visualization of the 2D gaze uncertainty field ( μ=8.3∘ , e≈46.7 px) with circular ROIs for p={0.5,0.7,0.9} mapped to frame sizes of (48 2 , 64 2 , 80 2 px).
Fig. 4 : GLANCE end-to-end system delay across varying ROI sizes: (a) 48 × 48, (b) 64 × 64, and (c) 80 × 80, measured on the Arduino Nano 33 BLE (microcontroller) and NVIDIA RTX 3090 (host), w.r.t. the required processing steps 1 – 7 .
Object Size
Count Number
Acc (ROI Count N)
1
2
3
4
5
10
15
20
25
30
40
50
Global
small
48
29.55
37.96
42.67
45.62
47.94
49.06
51.34
50.72
50.24
51.26
50.67
49.25
64
34.68
42.50
48.46
48.92
45.16
49.32
51.02
52.17
49.74
48.69
50.94
48.41
42.05
80
41.09
43.52
46.46
46.42
45.16
48.49
49.35
50.31
48.69
48.63
50.38
48.81
MEDium
48
53.29
59.17
61.35
64.44
66.10
72.11 ∗
72.79
72.67
71.99
72.84
72.66
71.99
64
65.69
67.47
71.59
71.63
71.93
71.82
71.62
72.31
72.89
72.14
72.11
71.90
67.16
TABLE II : GLANCE detection accuracy (%) across ROI counts (K) for different object sizes and ROI patch sizes.
Fig. 5 : (a) Memory footprint and (b) energy improvement of GLANCE vs. the conventional approach at the MCU end.
Fig. 6 : Comparative analysis of ROI-based detection accuracy under increasing ROI counts (up to 30) for ROI sizes of (a) 48 px, (b) 64 px, and (c) 80 px, illustrating performance convergence for different object sizes.
Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.
We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder. Instead of forming and processing full-resolution images, our approach leverages a microlens array with a co-designed binary chromium mask to perform spatially multiplexed optical encoding, producing a compact set of measurements sufficient for gaze estimation. By integrating sensing and feature extraction in the optical domain, the proposed system eliminates the need for high-bandwidth image readout and substantially reduces computational overhead. The encoded measurements are captured by a 4 x 4 phototransistor array and mapped to gaze direction using a lightweight neural network. Our proof-of-concept prototype enables an end-to-end sensing-to-inference latency of 3.4 ms, outperforming published research systems. We demonstrate the effectiveness of our approach on both simulated and real-world data, achieving competitive gaze estimation accuracy while significantly reducing end-to-end latency compared to conventional camera-based pipelines. The proposed system establishes a new operating point in the accuracy-latency-compute trade-off for latency- and resource-constrained gaze tracking. This work highlights the potential of task-driven optical sensing for ultra-low-latency, computationally efficient human-computer interaction systems.
Yidan Zheng, Matheus Souza, Kaizhang Kang +3
King Abdullah University of Science and Technology (KAUST)
Eye tracking (ET) plays a critical role in augmented and virtual reality applications. However, rapidly deploying high-accuracy, on-device gaze estimation for new products remains challenging because hardware configurations (e.g., camera placement, camera pose, and illumination) often change across device generations. Visual foundation models (VFMs) excel on natural-image benchmarks and offer a promising path to rapid training and deployment; yet, we find that off-the-shelf VFMs still struggle to reach high accuracy on specialized near-eye infrared images. To close this gap, we introduce DistillGaze, a framework that distills a VFM using labeled synthetic data and unlabeled real data for rapid, high-accuracy on-device gaze estimation. DistillGaze proceeds in two stages. First, we adapt a VFM into a domain-specialized teacher using synthetic gaze labels and unlabeled real images. Synthetic data provide scalable, high-quality gaze supervision, while unlabeled real data bridges the synthetic-to-real domain gap. Second, we train an on-device student from both teacher guidance and self-training. Evaluated on a large-scale crowd-sourced dataset spanning more than 2,000 participants, DistillGaze reduces median gaze error by 58.6% relative to synthetic-only baselines while maintaining a lightweight 256K-parameter model suitable for real-time on-device deployment. More broadly, DistillGaze offers an efficient path to training and deploying ET models that adapt to hardware changes, and a recipe for combining synthetic supervision with unlabeled real data in on-device regression tasks.
Gaze estimation is instrumental in modern virtual reality (VR) systems. Despite significant progress in remote-camera gaze estimation, VR gaze research remains constrained by data scarcity, particularly the lack of large-scale, accurately labeled datasets captured with the off-axis camera configurations typical of modern headsets. Gaze annotation is difficult since fixation on intended targets cannot be guaranteed. To address these challenges, we introduce VRGaze, the first large-scale off-axis gaze estimation dataset for VR, comprising 2.1 million near-eye infrared images collected from 68 participants. We further propose GazeShift, an attention-guided unsupervised framework for learning gaze representations without labeled data. Unlike prior redirection-based methods that rely on multi-view or 3D geometry, GazeShift is tailored to near-eye imagery, achieving effective gaze-appearance disentanglement in a compact, real-time model. GazeShift embeddings can be optionally adapted to individual users via lightweight few-shot calibration, achieving a 1.84° mean error on VRGaze. On the remote-camera MPIIGaze dataset, the model achieves a 7.15° person-agnostic error, doing so with 10x fewer parameters and 35x fewer FLOPs than baseline methods. Deployed natively on a VR headset GPU, inference takes only 5 ms. Combined with demonstrated robustness to illumination changes, these results highlight GazeShift as a label-efficient, real-time solution for VR gaze tracking. Project code and the VRGaze dataset are released at https://github.com/gazeshift3/gazeshift
Gil Shapira, Ishay Goldin, Evgeny Artyomov +3
Samsung Semiconductor Israel R&D Center (SIRC) · Samsung Electronics · Bar-Ilan University