cs.HCMar 1, 2026

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI

Authors: Sicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun, Fengyi Fang, You He, Yiqiao Xie, Jiankang Deng, +3 more

Organizations: Shenzhen International Graduate School Tsinghua University Shenzhen, China · Independent Researcher London, United Kingdom · Queen Mary University of London London, United Kingdom · Imperial College London London, United Kingdom · University Of Surrey Guildford, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

What if accessing the web did not require a screen, a stable desk, or even free hands? For people navigating crowded cities, living with low vision, or experiencing cognitive overload, smart glasses coupled with AI agents could turn the web into an always-on assistive layer over daily life. We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools. An egocentric reasoning core combines Temporal Chain-of-Thought with Hierarchical Context Compression to support long-horizon question answering and decision support over continuous first-person video, far beyond a single model's context window. Additionally, a lightweight multimodal intent layer maps noisy speech and gaze into structured commands. We further implement and evaluate a cloud-native WebRTC pipeline integrating streaming speech, video, and control messages into a unified channel for smart glasses and browsers. In parallel, we deploy an on-premise WebSocket baseline, exposing concrete trade-offs between local inference and cloud offloading in terms of latency, mobility, and resource use. Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance, and a human-in-the-loop study on smart glasses shows higher task completion and user satisfaction than leading commercial baselines. Taken together, these results indicate that web-connected egocentric co-pilots can be a practical path toward more accessible, context-aware assistance in everyday life. By grounding operation in web-native communication primitives and modular, auditable tool use, Egocentric Co-Pilot offers a concrete blueprint for assistive, always-on web agents that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.

Figures & tables

Figure 1. Overview of Egocentric Co-Pilot vs. Monolithic MLLM. Top: Monolithic MLLMs struggle with specialized reasoning task (e.g., a strategy board game), often providing evasive answers. Bottom: Our framework uses an LLM orchestrator that leverages a toolbox of specialized neuro-symbolic modules. It successfully interprets the user’s request and invokes the perception modules to generate board state, which is then solved by a dedicated game engine, leading to a precise and actionable solution. Egocentric Co-Pilot pipeline contrasted with a monolithic model.
Figure 2. Our Reasoning Core pipeline. It integrates Temporal Chain-of-Thought (T-CoT) for short-term analysis and Hierarchical Context Compression (HCC) for long-term memory. The figure illustrates the T-CoT path where our fine-tuned MLLM processes a temporally bounded query.
Figure 3. On-device architecture for real-time multimodal interaction. Audio and video are parallel-processed and multiplexed into a bidirectional channel between smart glasses and the cloud backend. We use WebRTC ( Sredojev et al., 2015 ) (H.264 video, Opus audio, data channel), with a custom WebSocket variant as an on-premise baseline.
DatasetModelAccuracy (%)
Egolife ( Yang et al., 2025a )LLaVA-OV ( Li et al., 2025a )30.8 *
GPT-4o ( Hurst et al., 2024 )36.2 *
Gemini-1.5-Pro ( Reid et al., 2024 )36.9 *
Qwen2.5 VL ( Bai et al., 2025 )38.1
Ours40.9
HD-EPIC ( Perrett et al., 2025 )VideoLlama 2 ( Cheng et al., 2024b )27.4 *
Table 1. Comparison against state-of-the-art methods on the Egolife and HD-EPIC benchmarks. Results marked with an asterisk (*) are reported in the original papers; all other results are from our reproductions using official code.
DatasetVariantAccuracy (%)
Egolife ( Yang et al., 2025a )Ours40.9
w/o Fine-tuning39.2 (-1.7)
w/o T-CoT39.5 (-1.4)
w/o HCC38.9 (-2.0)
w/o Transcript40.1 (-0.8)
HD-EPIC ( Perrett et al., 2025 )Ours46.23
Table 2. Ablation study of our Reasoning Core on Egolife and HD-EPIC. We report the full model (“Ours”) and variants where a single component is removed; the performance drop is shown in parentheses.
Figure 4. Core capabilities of the LLM-orchestrated framework. Our system interprets multimodal user intent and dynamically composes neuro-symbolic tools via the MCP protocol. (a) Foundational Tool Use: a simple query triggers a VLM for object recognition and an external API call. (b) Structured Task Management: natural language is translated into a structured API call for a native device application. (c) Complex Neuro-Symbolic Reasoning: the board-game co-pilot integrates a vision tool (neuro), a deterministic game engine (symbolic), and an LLM for semantic explanation. (d) Spatiotemporal Memory: the system resolves a deictic reference (“this”) by visually tracking an object through occlusion and recalling it from memory.
Figure 5. Subjective evaluation of Egocentric Co-Pilot against commercial smart-glasses devices and a human baseline. Bars show mean 5-point Likert ratings (higher is better); asterisks (*) denote devices whose default interaction pattern deviates from continuous conversational AI.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6. Example interaction logs shown to participants. Each column corresponds to a different system’s response to the same user query. By evaluating pre-recorded logs instead of live interactions, we avoid confounding AI quality with hardware, network, or UI differences.

Explore similar work

CardsList
Apr 3, 2026

VisionClaw: Always-On AI Agents through Smart Glasses

We present VisionClaw, an always-on wearable AI agent that integrates live egocentric perception with agentic task execution. Running on Meta Ray-Ban smart glasses, VisionClaw continuously perceives real-world context and enables in-situ, speech-driven action initiation and delegation via OpenClaw AI agents. Therefore, users can directly execute tasks through the smart glasses, such as adding real-world objects to an Amazon cart, generating notes from physical documents, receiving meeting briefings on the go, creating events from posters, or controlling IoT devices. We evaluate VisionClaw through a controlled laboratory study (N=12) and a longitudinal deployment study (N=5). Results show that integrating perception and execution enables faster task completion and reduces interaction overhead compared to non-always-on and non-agent baselines. Beyond performance gains, deployment findings reveal a shift in interaction: tasks are initiated opportunistically during ongoing activities, and execution is increasingly delegated rather than manually controlled. These results suggest a new paradigm for wearable AI agents, where perception and action are continuously coupled to support situated, hands-free interaction.

Xiaoan Liu, DaeHo Lee, Eric J Gonzalez +2

University of Colorado Boulder, Boulder, Colorado, USA · Gwangju Institute of Science and Technology, Gwangju, Republic of Korea · Google, Seattle, Washington, USA

Multimodal AgentsTool-Using Agents
AskPDF
Sep 17, 2026

AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization

Recent advances in artificial intelligence (AI) are reshaping smart glasses from egocentric capture and display devices into platforms for wearable intelligence. Smart glasses increasingly serve as wearable AI systems that connect first-person observation with real-time assistance under strict form-factor constraints. We frame this transition through the lens of AI smart glasses and define them as a system-level concept in which egocentric sensing, resource-aware computing, intelligent reasoning, multimodal interaction, and real-world application constraints are co-designed for personalized assistance in the physical world. To systematically study this perspective, we organize the survey around four connected dimensions. First, we examine the hardware foundation that bounds sensing, computation, feedback delivery, and sustained deployment. Second, we study wearable intelligence, where egocentric signals are transformed into perceptual, contextual, and agentic capabilities. Third, we discuss interaction design, through which users request, receive, correct, and regulate assistance during ongoing activity. Fourth, we analyze application scenarios across healthcare, accessibility, situated learning, daily life assistance, cultural tourism, and industrial support, showing how domain requirements reshape system design and evaluation. We further identify five cross-cutting research challenges for future AI smart glasses: next-generation hardware, trustworthy egocentric intelligence, lifelong personalized memory, proactive intelligence, and embodied foundation models. By centering smart glasses as wearable-intelligence platforms, this survey provides a unified framework for organizing technologies, applications, and open challenges in this emerging area.

Xu Yuan, Yi Wang, Zhuohang Jiang +8

The Hong Kong Polytechnic University · The Chinese University of Hong Kong

Wearable SensingAgentic AI
AskPDF
Mar 23, 2026

Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos

Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.

Shoubin Yu, Lei Shu, Antoine Yang +6

Google DeepMind · UNC Chapel Hill

Agent BenchmarksWeb Agent BenchmarksAI Agent EvaluationEgocentric Video Understanding
AskPDF