Hand-object interactions (HOIs) can be captured by smartglasses to learn how people manipulate objects in daily life. Automatic logging of these interactions from an egocentric perspective provides time-stamped evidence of daily actions for applications like memory assistance. However, continuous RGB capture and wireless transmission exceed the energy and bandwidth budgets of smartglasses. We present EventHOI, an event-driven sensing and compression system that couples selective RGB capture with HOI-aware encoding to enable egocentric logging over Bluetooth Low Energy (BLE) while preserving downstream HOI logging performance. Our key insight is to use low-power event sensing both to determine when RGB capture is needed and to identify where visual detail must be preserved. A lightweight IMU-based egomotion cancellation pipeline suppresses egomotion-induced events so that residual HOI motion can guide RGB triggering and spatial compression. We build a smartglasses prototype and conduct an end-to-end evaluation of EventHOI on a 50-hour egocentric benchmark with 97 subjects and a 20-hour real-world dataset we collect from 10 participants, together covering a broad range of daily-life scenarios and environments. Compared to RGB-only baselines, EventHOI reduces always-on backend power consumption by 5.8-10.4x and uplink bandwidth by 2.9x, with minimal HOI recognition accuracy loss. The resulting RGB uplink of less than 130 kbps fits within commodity BLE bandwidth budgets, making stable BLE video streaming feasible for egocentric HOI logging. Demo video and project website: https://eventhoi.github.io
Figures & tables
Figure 1. EventHOI uses low-power event sensing to (1) detect and selectively trigger the high-power RGB camera and (2) compress RGB uplink for HOI understanding.
Figure 2. Comparisons of sensors in baseline methods used to reduce RGB camera usage.
Workload
Power (W)
Battery Life (hrs)
Use Cases and Examples
Radio
Sensor
Off
Off
0.1
7.5
No video capability, no stream, on-device only
BLE
Off
0.15 ( 1.5× )
5.0 ( 67% )
Low-resource modality stream (image, audio, etc.)
Off
RGB
0.7 ( 7× )
1.1 ( 15% )
No real-time video stream
WiFi
RGB
1.1 ( 11× )
0.7 ( 9% )
Video stream (Ray-Ban Meta ( Meta, 2025 ) , Project Aria ( Engel and others, 2023 ) )
Table 1. Power consumption and expected battery life. Red indicates comparisons to the first-row idle state.
Figure 3. Preliminary study on VLM-based HOI understanding for frame rate and uplink bandwidth.
Figure 4. (b) Asynchronous event voxels are aggregated to (c) an event frame (red for positive and blue for negative events). (d) Egomotion-induced and HOI-induced events. (e) Background lighting and bystander distractors after egomotion cancellation.
Figure 5. EventHOI system overview.
Figure 6. Frame-to-frame egomotion fields u (red arrows) derived from the rotation matrix R .
Figure 7. Example of distractor filters to residual optical flow. Boxes in the left figure indicate the verdicts of filters.
Figure 8. EventHOI prototype for data collection.
Figure 9. Nymeria ( Ma and others, 2025 ) sequence selection for evaluation, daily time-use survey results ( U.S. Bureau of Labor Statistics, 2025 ) (excluding sleeping time), and the recording time of our real-world testbed (§ 7 ).
Figure 10. Examples of HOI recognition and localization tasks for memory logging and retrieval.
Baselines
EventHOI Ablations
Uniform
YOLO ( Khanam and Hussain, 2024 )
Hands23 ( Cheng et al., 2023 )
Vanilla EVS
Detector only
EventHOI
Reduce RGB frames?
No
Yes ∗
No
No
Yes
Yes
Can compress frames?
No
No
Yes
No
No
Yes
On-device feasible?
Yes
Yes
No
Yes
Yes
Yes
Table 2. Comparison of baselines and EventHOI ablations.
Figure 11. Top-30 EventHOI (0.5 FPS effective frame rate) improvements over uniform sampling (0.5 FPS) on hand motions (blue) and object recognition (green). Red is the average.
Figure 12. Effective FPS, uplink bandwidth, and accuracy for baselines and EventHOI ablations on the Nymeria dataset ( Ma and others, 2025 ) .
Figure 13. EventHOI recognition and localization accuracy.
Figure 14. Ablation studies on HOI-to-Noise threshold values for detector and minimal frame size for encoder.
Figure 15. Effective FPS, uplink bandwidth, and accuracy on the real-world testbed by participants (P1–10) and scenarios.
Figure 16. Failure in flow aliasing on repetitive textures.
Figure 17. Power consumption breakdown.
Figure 18. EventHOI real-world examples. The plot shows the RGB sampling rate and the icons indicate different HOIs.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 19. Optical flows (red arrows) from (a) v2e-simulated and (b) real event frames while the user holds a water bottle.
Location
Leisure
Working
Eating
Cooking
Housework
Recreation
Total
Home1
2.5 (P1, P2, P6)
2.6 (P1, P2)
0.8 (P1, P2, P6)
1.6 (P1, P6)
1.6 (P1, P2, P6)
2.2 (P1, P2, P6)
11.3
Home2
–
3.4 (P4)
0.3 (P4)
0.8 (P4)
–
–
4.5
Workspace1
0.3 (P5)
0.5 (P3)
0.5 (P1)
–
–
–
1.3
Workspace2
0.3 (P9)
0.7 (P3, P10)
–
–
–
0.8 (P8, P9)
1.7
Office
0.4 (P7)
0.7 (P10)
–
–
–
–
1.1
Total
3.5
7.8
1.6
2.4
1.6
2.9
19.8
Appendix
Table 3. Recording hours and participants (P1–10) of the real-world testbed per location and scenario.
Figure 20. Recording hours of the real-world testbed across the time of day.
Figure 21. Examples of HOIs in the real-world testbed.
Smart eyewear enables unobtrusive, context-aware interaction through multimodal sensors and on-device intelligence, but is severely limited by power, memory, and compute constraints in a compact form factor. Open-hardware platforms supporting event-based vision and embedded ML at this scale are rare. This work introduces an open-source smart glasses platform for rapid prototyping of novel sensors and algorithms. Its modular design uses a flexible FPC interposer to support both event-based and frame-based cameras without full PCB redesign. A hardware-software co-designed power management system combines a configurable PMIC with event-driven wake-up via an nRF5340 coordinator, keeping the GAP9 RISC-V SoC powered down between inferences. The prototype achieves up to 11.5 hours of continuous on-device ML from a 200 mAh battery. As a demonstration, an egocentric hand gesture recognition pipeline was evaluated on the LynX dataset using polarity-separated event histograms from a Prophesee GENX320 camera. R(2+1)D achieved the best cross-subject accuracy of 83.94% (macro F1 = 0.781) under leave-two-subjects-out validation, with 78.3 ms end-to-end inference latency on the GAP9. Temporal augmentation and removal of ambiguous classes provided the largest gains (+8.9 pp). All hardware designs, firmware, and models are released open source.
Pietro Bonazzi, Julian Moosmann, Ahmet Celik +2
Department of Information Technology and Electrical Engineering, ETH Zürich, 8092 Zürich, Switzerland
Modeling human-object interactions (HOI) from an egocentric perspective is a critical yet challenging task, particularly when relying on sparse signals from wearable devices like smart glasses and watches. We present ECHO, the first unified framework to jointly recover human pose, object motion, and contact dynamics solely from head and wrist tracking. To tackle the underconstrained nature of this problem, we introduce a novel tri-variate diffusion process with independent noise schedules that models the mutual dependencies between the human, object, and interaction modalities. This formulation allows ECHO to operate with flexible input configurations, making it robust to intermittent tracking and capable of leveraging partial observations. Crucially, it enables training on a combination of large-scale human motion datasets and smaller HOI collections, learning strong priors while capturing interaction nuances. Furthermore, we employ a smooth inpainting inference mechanism that enables the generation of temporally consistent interactions for arbitrarily long sequences. Extensive evaluations demonstrate that ECHO achieves state-of-the-art performance, significantly outperforming existing methods lacking such flexibility. The project page is available at https://ptrvilya.github.io/echo/.
Ilya A. Petrov, Vladimir Guzov, Riccardo Marin +5
University of Tübingen, Germany · Tübingen AI Center, Germany · Technical University of Munich, Germany +3
Existing hand detection algorithms work on images and the detection rate is restricted by the frame rate of the camera. In hand detection applications for moving robotic systems, conventional cameras cause motion blur, especially in darker lighting conditions. We can leverage the use of event-based cameras which possess a high dynamic range, high temporal resolution, and low power consumption. Recent work has shown that using a stereo setup of an event-based and a frame-based camera improves detection accuracy and the bandwidth-latency tradeoff. The main bottleneck in using event-based cameras in object detection and recognition tasks is a relatively low amount of training data. In this work, we propose a methodology and an exemplary synthetic event-based hand dataset from an egocentric, first-person view perspective. The data is synthesized from the existing RGB Egohands dataset with the v2e toolbox. Parameters of the v2e toolbox are varied to provide versions of the dataset with different lighting conditions and scales. Ground truth detections are generated with a fine-tuned YOLOv8 model which is applied to the RGB images in the Egohands dataset and interpolated on the high-temporal resolution events. We use the multi-modal dataset to perform hand detection with existing object detection algorithms which use a multi-modal setup of event and RGB cameras and demonstrate performance comparable to the state-of-the-art.
Bharghav Kota, Yulia Sandamirskaya
Zurich University of Applied Sciences, Wädenswil, Switzerland · Zurich University of Applied Sciences Wädenswil, Switzerland