VisionClaw: Always-On AI Agents through Smart Glasses

Xiaoan Liu; DaeHo Lee; Eric J Gonzalez; Mar Gonzalez-Franco; Ryo Suzuki

cs.HCApr 3, 2026

VisionClaw: Always-On AI Agents through Smart Glasses

Authors: Xiaoan Liu, DaeHo Lee, Eric J Gonzalez, Mar Gonzalez-Franco, Ryo Suzuki

Organizations: University of Colorado Boulder, Boulder, Colorado, USA · Gwangju Institute of Science and Technology, Gwangju, Republic of Korea · Google, Seattle, Washington, USA

Abstract

We present VisionClaw, an always-on wearable AI agent that integrates live egocentric perception with agentic task execution. Running on Meta Ray-Ban smart glasses, VisionClaw continuously perceives real-world context and enables in-situ, speech-driven action initiation and delegation via OpenClaw AI agents. Therefore, users can directly execute tasks through the smart glasses, such as adding real-world objects to an Amazon cart, generating notes from physical documents, receiving meeting briefings on the go, creating events from posters, or controlling IoT devices. We evaluate VisionClaw through a controlled laboratory study (N=12) and a longitudinal deployment study (N=5). Results show that integrating perception and execution enables faster task completion and reduces interaction overhead compared to non-always-on and non-agent baselines. Beyond performance gains, deployment findings reveal a shift in interaction: tasks are initiated opportunistically during ongoing activities, and execution is increasingly delegated rather than manually controlled. These results suggest a new paradigm for wearable AI agents, where perception and action are continuously coupled to support situated, hands-free interaction.

Figures & tables

Figure 1. VisionClaw integrates always-on egocentric perception with agentic task execution on smart glasses. A user holding a product says “Can you check the reviews and price for this? If they look good, add it to my Amazon cart.” The system 1) identifies the product through visual perception, 2) autonomously executes a multi-step browser workflow to find and add the item on Amazon, and 3) confirms completion through spoken feedback—all without the user touching a screen. A person wearing smart glasses in a retail store holds an Aesop hand wash bottle. Three overlaid panels show the system workflow: visual perception identifying the product, agentic execution adding it to an Amazon cart, and spoken confirmation of the completed action.

Figure 2. System architecture of VisionClaw. The wearable device layer captures audio and video from Meta Ray-Ban smart glasses via the DAT SDK and streams them through an phone app. These always-on streams are sent to the Gemini Live API over a persistent WebSocket connection. Gemini processes the multimodal input and either responds with spoken audio or issues tool calls routed to OpenClaw for execution via dual HTTP and WebSocket channels. A system architecture diagram with three color-coded containers arranged left to right. The green container labeled Wearable Device contains Meta Ray-Ban and iPhone App boxes side by side, connected by DAT SDK. The blue container labeled Multimodal AI contains a Gemini Live box with capture\_photo and execute tool tags. The red container labeled Agentic Execution contains an OpenClaw box and six category boxes: Retrieve, Shop, Communicate, Control, Save, and Recall. Horizontal arrows show audio and video streams flowing from the wearable device to Gemini, tool calls and results between Gemini and OpenClaw, and a horizontal blue arrow at the bottom showing spoken audio response returning to the user. — Figure 2. System architecture of VisionClaw. The wearable device layer captures audio and video from Meta Ray-Ban smart glasses via the DAT SDK and streams them through an phone app. These always-on streams are sent to the Gemini Live API over a persistent WebSocket connection. Gemini processes the multimodal input and either responds with spoken audio or issues tool calls routed to OpenClaw for execution via dual HTTP and WebSocket channels. A system architecture diagram with three color-coded containers arranged left to right. The green container labeled Wearable Device contains Meta Ray-Ban and iPhone App boxes side by side, connected by DAT SDK. The blue container labeled Multimodal AI contains a Gemini Live box with capture_photo and execute tool tags. The red container labeled Agentic Execution contains an OpenClaw box and six category boxes: Retrieve, Shop, Communicate, Control, Save, and Recall. Horizontal arrows show audio and video streams flowing from the wearable device to Gemini, tool calls and results between Gemini and OpenClaw, and a horizontal blue arrow at the bottom showing spoken audio response returning to the user.

Condition	Visual Perception	Agentic Execution
Always-On Only	✓
Agent Only		✓
Always-On and Agent	✓	✓

Table 1. Capabilities supported in each condition. A table comparing three experimental conditions—Always-On Only, Agent Only, and Always-On and Agent—across two capabilities: Visual Perception and Agentic Execution. Always-On Only includes Visual Perception but not Agentic Execution, Agent Only includes Agentic Execution but not Visual Perception, and Always-On and Agent includes both capabilities.

Figure 3. Overview of the four tasks used in the study A sequence of example tasks used in the study. From left to right: (1) a physical receipt is converted into a structured grocery memo on a laptop (note taking), (2) a printed academic paper is used to compose an email draft (email composition), (3) a book is identified and compared on an online shopping interface (product lookup), and (4) a handwritten note instructing “Turn on the light” corresponds to a smart home control interface (device control). The figure illustrates how tasks are grounded in real-world physical artifacts and mapped to digital actions.

Figure 4. Task completion time. Asterisks next to labels indicate significance from Friedman tests, and bracketed asterisks indicate significance from Wilcoxon signed-rank tests. (* p < .05, ** p < .01.) Box plots showing task completion time (in seconds) across four tasks—Note Taking, Email Composition, Product Lookup, and Device Control—for three conditions: Always-On + Agent, Agent Only, and Always-On Only. Boxes indicate median and interquartile range, with individual data points overlaid. Significant differences are observed in Note Taking, Email Composition, and Device Control, where Always-On + Agent generally achieves faster completion times compared to the other conditions, particularly outperforming Always-On Only.

Figure 5. NASA-TLX. Asterisks next to labels indicate significance from Friedman tests, and bracketed asterisks indicate significance from Wilcoxon signed-rank tests. Lower scores indicate better performance. (* p < .05, ** p < .01) Box plots of NASA-TLX workload scores (0–20 scale) across six dimensions: Mental Demand, Physical Demand, Temporal Demand, Effort, Frustration, and Performance, for three conditions (Always-On + Agent, Agent Only, Always-On Only). Lower scores indicate lower workload (except Performance, where lower indicates better perceived performance). Significant reductions in Mental Demand, Temporal Demand, and Frustration are observed for Always-On + Agent compared to the other conditions, indicating reduced cognitive load.

Figure 6. Self-authored questionnaire. Asterisks next to labels indicate significance from Friedman tests, and bracketed asterisks indicate significance from Wilcoxon signed-rank tests. (* p < .05, ** p < .01). Diverging stacked bar charts showing percentage distributions of participant responses (7-point Likert scale) for six subjective measures: Perceived Control, Reliability, Trust, Ease of Use, Usefulness, and Confidence, across three conditions. Responses range from Strongly Disagree (left) to Strongly Agree (right). Always-On + Agent shows higher positive responses for Perceived Control and Usefulness, with statistically significant differences indicated, while other measures show similar trends without consistent significance.

Figure 7. Representative use cases from the deployment study, one per category. Each scenario shows a participant wearing smart glasses issuing a voice command grounded in their physical context, with the agent executing the corresponding digital action autonomously. A 3-by-2 grid of six over-the-shoulder photographs showing a person wearing smart glasses in different everyday scenarios. Top row: pointing at a research poster in a hallway, operating a kitchen appliance, eating a meal in a cafeteria. Bottom row: looking at a menu board in a coffee shop, browsing headphones in an electronics store, reaching toward a light switch at home. Each photo has a white speech bubble with the user's voice command.

Figure 8. Usage log of the deployment study. An interactive version of this visualization can be seen at the following link. Data has been anonymized with private or confidential information removed. https://deployment-study-vis.vercel.app A multi-panel scatter plot showing interactions over time across six categories: Communicate, Retrieve, Save, Recall, Shop, and Control. The figure is arranged as six panels in two rows, with three panels per row, each labeled with a category. In each panel, the horizontal axis represents time in days from Day 1 to Day 50, and the vertical axis represents time of day, with labeled ticks at 6:00, 12:00, and 18:00. Each data point is shown as a circle representing an interaction. The size of each circle varies, with larger circles indicating a higher number of interactions. A legend indicates increasing circle size corresponding to increasing interaction counts. In most panels, interactions are sparse in earlier days and more frequent in later days, with many points clustered toward the right side of the plots. The distribution of points varies across categories, with some categories showing denser clusters and larger circles than others.

Figure 9. Findings on interactions observed in the longitudinal deployment study, illustrating four recurring patterns: multi-turn conversation, opportunistic capture and recall, screenless interaction, and interaction that evolves with personal data. An illustrated figure showing four interaction patterns observed in a longitudinal deployment study. The figure is divided into four panels. The first panel shows a user in a car asking about a past group meeting, representing multi-turn conversation that builds across multiple queries. The second panel shows a user reading and asking to save a quote, representing capturing information during an ongoing activity. The third panel shows a user walking outdoors while reflecting on system behavior, representing screenless interaction during everyday movement. The fourth panel shows a user recalling a past visit at a location, representing interaction based on previously accumulated personal data.

Figure 10. Future research directions for always-on agentic interaction. Three sketch illustrations in a row. Left: a diverse group of people wearing smart glasses. Center: smart glasses with a lightbulb icon indicating proactive suggestions. Right: a person looking at an object with AR overlay information.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11. A taxonomy of use cases observed in the deployment study. The figure organizes interactions into six categories—communicate, retrieve, save, recall, shop, and control—each illustrated with four representative everyday scenarios. The figure presents six categories of use cases: Communicate, Retrieve, Save, Recall, Shop, and Control, each illustrated with example scenarios. Communicate includes checking email, sending messages, and sharing content. Retrieve includes querying news, transportation information, media details, and troubleshooting resources. Save includes capturing and storing information such as receipts, contextual details, and conversations. Recall includes retrieving previously stored personal information such as past activities, shopping lists, and receipts. Shop includes finding products, checking reviews, restocking items, and adding items to a cart. Control includes executing actions such as controlling devices, editing documents, adjusting settings, and uploading content. Each category is represented with multiple illustrated scenes showing users interacting with the system in everyday contexts.

Task	Metric	Always-On + Agent	Agent Only	Always-On Only
Note Taking	Completion time (s)	$\mathbf{102.20}\pm 40.48$	$127.72\pm 46.60$	$149.29\pm 53.30$
	Difficulty (1–7)	$\mathbf{2.33}\pm 1.07$	$2.92\pm 1.83$	$3.67\pm 1.83$
	Success rate	$0.583\pm 0.515$	$\mathbf{1.000}\pm 0.000$	$\mathbf{1.000}\pm 0.000$
Email Composition	Completion time (s)	$\bf{105.74}\pm 31.72$	$131.11\pm 60.89$	$216.42\pm 88.20$
	Difficulty (1–7)	$\mathbf{2.50}\pm 1.45$	$2.83\pm 1.34$	$4.25\pm 1.14$
	Success rate	$0.833\pm 0.389$	$0.667\pm 0.492$	$\mathbf{1.000}\pm 0.000$

Appendix

Table 2. Task-level descriptive statistics (

M\pm SD

; participant-level means). A table reporting task-level descriptive statistics across four tasks, Note Taking, Email Composition, Product Lookup, and Device Control, and three conditions: Always-On + Agent, Agent Only, and Always-On Only. For each task, the table reports completion time in seconds, difficulty on a 1 to 7 scale, and success rate. All values are presented as mean and standard deviation for each condition.

Task	Metric	Friedman $\chi^{2}$	$p$	A vs B	A vs C	B vs C
Note Taking	Completion time	7.1667	$\mathbf{0.0278}$	0.4072	0.0630	0.4072
	Difficulty	6.0000	$\mathbf{0.0498}$	0.6797	$\mathbf{0.0234}$	0.6797
	Success rate	3.1250	0.2096	1.0000	0.1875	0.1875
Email Composition	Completion time	10.1667	$\mathbf{0.0062}$	$\mathbf{0.0322}$	$\mathbf{0.0029}$	0.5186
	Difficulty	11.7917	$\mathbf{0.0028}$	$\mathbf{0.0156}$	$\mathbf{0.0029}$	0.7109
	Success rate	1.5000	0.4724	0.3750	1.0000	1.0000

Appendix

Table 3. Task-level inferential results (Friedman

\chi^{2}

and Holm-corrected Wilcoxon pairwise tests,

n=12

). (A : Always-On Only, B : Agent Only, C : Always-On + Agent) A table presenting task-level inferential statistics across four tasks, Note Taking, Email Composition, Product Lookup, and Device Control, and three conditions: Always-On Only, Agent Only, and Always-On + Agent. For each task and metric, including completion time, difficulty, and success rate, the table reports the Friedman test statistic, the corresponding p-value, and pairwise comparison p-values using Holm-corrected Wilcoxon tests between each pair of conditions.

NASA-TLX (0–20)
Measure	Always-On + Agent	Agent Only	Always-On Only
Mental Demand	$\mathbf{2.83}\pm 2.76$	$4.50\pm 3.80$	$4.83\pm 3.04$
Physical Demand	$2.08\pm 3.58$	$\mathbf{1.25}\pm 1.36$	$2.25\pm 1.66$
Temporal Demand	$2.83\pm 3.97$	$\mathbf{2.50}\pm 3.15$	$3.83\pm 4.43$
Effort	$\mathbf{2.58}\pm 4.01$	$3.83\pm 3.07$	$4.00\pm 2.70$
Frustration	$\mathbf{2.58}\pm 2.43$	$2.83\pm 3.41$	$5.17\pm 4.53$

Appendix

Table 4. Subjective ratings by condition (participant-level,

n=12

), reported as

M\pm SD

. A table showing subjective ratings across three conditions: Always-On + Agent, Agent Only, and Always-On Only. The table includes NASA-TLX workload measures—mental demand, physical demand, temporal demand, effort, frustration, and performance—reported on a 0 to 20 scale, as well as self-authored measures—perceived control, reliability, trust, ease of use, usefulness, and confidence—reported on a 1 to 7 scale. All values are presented as mean and standard deviation.

NASA-TLX (0–20)
Measure	Friedman $\chi^{2}$	$p$	A vs B	A vs C	B vs C
Mental Demand	9.8000	$\mathbf{0.0074}$	0.1879	0.0511	0.1879
Physical Demand	4.4706	0.1070	0.0967	0.6882	0.6882
Temporal Demand	6.4138	$\mathbf{0.0405}$	0.0947	0.0828	0.4581
Effort	5.7727	0.0558	0.8219	0.3275	0.3275
Frustration	6.0000	$\mathbf{0.0498}$	0.3616	$\mathbf{0.0307}$	0.8780

Appendix

Table 5. Subjective ratings inferential results: Friedman

\chi^{2}

and Holm-corrected Wilcoxon post-hoc

p

values (

n=12

). (A : Always-On Only, B : Agent Only, C : Always-On + Agent) A table presenting inferential statistics for subjective ratings across three conditions: Always-On Only, Agent Only, and Always-On + Agent. For each NASA-TLX and self-authored measure, the table reports the Friedman test statistic, the corresponding p-value, and pairwise comparison p-values using Holm-corrected Wilcoxon tests between each pair of conditions.

Explore similar work

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI
Mar 1, 2026Sicheng Yang, Yukai Huang, Weitong Cai +8Web Agents Tool-Using Vision-Language Agents
Shenzhen International Graduate School Tsinghua University Shenzhen, China · Independent Researcher London, United Kingdom · Queen Mary University of London London, United Kingdom +2
AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization
Sep 17, 2026Xu Yuan, Yi Wang, Zhuohang Jiang +8Wearable Sensing Agentic AI
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
Jul 14, 2026Yunxin Li, Jinchao Li, Shibo Su +7Mobile GUI Automation GUI Agents
Lychee Team∗, Harbin Institute of Technology, Shenzhen

Mar 1, 2026J/K move · Enter open · S save

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI

Sicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun+7

Shenzhen International Graduate School Tsinghua University Shenzhen, China · Independent Researcher London, United Kingdom · Queen Mary University of London London, United Kingdom+2

Web Agents Tool-Using Vision-Language Agents Long-Horizon LLM Agents Egocentric Video QA

Open paper AskPDF arXiv

What if accessing the web did not require a screen, a stable desk, or even free hands? For people navigating crowded cities, living with low vision, or experiencing cognitive overload, smart glasses coupled with AI agents could turn the web into an always-on assistive layer over daily life. We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools. An egocentric reasoning core combines Temporal Chain-of-Thought with Hierarchical Context Compression to support long-horizon question answering and decision support over continuous first-person video, far beyond a single model's context window. Additionally, a lightweight multimodal intent layer maps noisy speech and gaze into structured commands. We further implement and evaluate a cloud-native WebRTC pipeline integrating streaming speech, video, and control messages into a unified channel for smart glasses and browsers. In parallel, we deploy an on-premise WebSocket baseline, exposing concrete trade-offs between local inference and cloud offloading in terms of latency, mobility, and resource use. Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance, and a human-in-the-loop study on smart glasses shows higher task completion and user satisfaction than leading commercial baselines. Taken together, these results indicate that web-connected egocentric co-pilots can be a practical path toward more accessible, context-aware assistance in everyday life. By grounding operation in web-native communication primitives and modular, auditable tool use, Egocentric Co-Pilot offers a concrete blueprint for assistive, always-on web agents that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.

Abstract

Explore similar work

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI

AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill