cs.AIFeb 16, 2026

EmbeWebAgent: Embedding Web Agents into Any Customized UI

Authors: Chenyang Ma, Clyde Fare, Matthew Wilson, Dave Braines

Organizations: IBM Research Europe, UK · University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Most web agents operate at the human interface level, observing screenshots or raw DOM trees without application-level access, which limits robustness and action expressiveness. In enterprise settings, however, explicit control of both the frontend and backend is available. We present EmbeWebAgent, a framework for embedding agents directly into existing UIs using lightweight frontend hooks (curated ARIA and URL-based observations, and a per-page function registry exposed via a WebSocket) and a reusable backend workflow that performs reasoning and takes actions. EmbeWebAgent is stack-agnostic (e.g., React or Angular), supports mixed-granularity actions ranging from GUI primitives to higher-level composites, and orchestrates navigation, manipulation, and domain-specific analytics via MCP tools. Our demo shows minimal retrofitting effort and robust multi-step behaviors grounded in a live UI setting. Live Demo: https://youtu.be/Cy06Ljee1JQ

Figures & tables

Figure 1: (a) Interface-level agent (screenshots/DOM trees, simulated clicks) vs. (b) Embedded agent (EmbeWebAgent) with ARIA labels as observations and explicit UI actions.
Figure 2: EmbeWebAgent pipeline. The frontend shim exposes curated ARIA observations and a per-page function registry via a WebSocket. The backend maintains session state, filters actions according to the current page, and coordinates a multi-agent workflow to infer navigation and manipulation actions, as well as invoke domain tools (via MCP or web APIs).
Figure 3: One page of the chemistry analysis UI used in our demo.
Figure 4: Testing interface. Integration tests evaluate action correctness and latency under simulated frontend-backend interaction.
More in this topic

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

Explore similar work

CardsList

  1. Web Agents Should Use Typed Actions Instead of Click-Based Browsing

    Feb 19, 2026Linxi Jiang, Rui Xi, Zhijie Liu +3Web AgentsLLM Agent Reliability

    The Ohio State University · University of British Columbia · Microsoft Research

  2. Back to Basics: A Skill-Centric Approach to Generalist Web Agents

    Oct 26, 2026Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui +2Web AgentsLLM Agent Skill Learning

    University of Utah · Islamic University of Technology

  3. MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

    Apr 9, 2026Tanmay Gupta, Piper Wolters, Zixian Ma +13Web AgentsLLM Agent Training

    Allen Institute for AI · University of Washington · UNC-Chapel Hill