HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

Zijian Cai; Haowei Liu; Kailai Chen; Lin Peng; Cong Wan; Ziqi Zhang; Chunfeng Yuan; Bing Li

cs.AIOct 12, 2026

HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

Authors: Zijian Cai, Haowei Liu, Kailai Chen, Lin Peng, Cong Wan, Ziqi Zhang, Chunfeng Yuan, Bing Li,

Organizations: Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, CASIA · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · Alibaba Group ·

Abstract

Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluated separately. To bridge this gap, we introduce HyWe, a framework that unifies both capabilities within one model and trajectory. Its two modes use dedicated prompts and tool sets to sustain sub-goal execution; explicit switching selects the needed capability, and shared history lets each mode build on the other's findings. To provide demonstrations of these behaviors, we develop HyWeCollector, a fully automatic pipeline that uses task prototypes, four switching patterns, and multi-agent synthesis to produce verified hybrid trajectories. We train HyWe-35B on these demonstrations to unify efficient information gathering and browser interaction in a single 35B web agent. To evaluate hybrid task completion, we introduce HyWeBench: 100 human-written tasks on websites distinct from training, with human-written sub-goal checklists and an outcome-based evaluation pipeline. Hybrid training enables HyWe-35B to achieve 47.0% success on this benchmark, improving on the untrained base model by 19.0 percentage points and outperforming existing search-based and operation-based agents. Even before training, the same base model solves 28.0% of tasks with HyWe, versus 11.0% with all tools under one prompt. Beyond hybrid tasks, HyWe-35B achieves 59.0% and 88.3% success on Online-Mind2Web and WebVoyager, outperforming existing open-source baselines and cutting action cost per successful task by 48.3% and 51.9% compared with Fara-7B.

Figures & tables

Figure 1: One real-world hybrid task from HyWeBench , three agents; key steps are taken from their actual trajectories. (a) A search agent finds the verse but cannot play the video. (b) An operation agent gets lost in complex web navigation and reaches the 100-step limit. (c) HyWe-35B finds the verse with search tools, switches to operation tools, and plays the video in 9 steps.

Figure 2: Overview of the HyWe framework. Mode-specific prompts and tools guide sub-goal execution; the switch action lets the model move between search-based and operation-based modes as needed, and shared history carries evidence forward (Section 3.1 ). The HyWeCollector data pipeline supplies task prototypes, switching patterns, and website/entity bindings to a multi-agent framework for task synthesis, execution, control, and review (Section 3.2 ). Colors mark segments: consecutive steps in one mode, separated by mode switches. Prompts and tool calls are abbreviated.

Frontier LLMs
Model / system	Size	Online-Mind2Web			WebVoyager
Model / system	Size	SR $\uparrow$	Steps/SR $\downarrow$	Switch	SR $\uparrow$	Steps/SR $\downarrow$	Switch
GPT-5.5	–	74.7	–	–	85.1	–	–
GPT-5 (SoM)	–	57.7	–	–	90.6	–	–
o3 (SoM)	–	55.4	–	–	79.3	–	–
Claude-Opus-4.6	–	–	–	–	88.0	–	–

Table 1: Success and efficiency on Online-Mind2Web and WebVoyager. SR: task success rate (%); Steps/SR: total actions (including failures) per success in our re-runs, using separate success labels from the SR column; Switch: tasks with a mode switch (%). Bold/underline: best/second-best SR and Steps/SR among open-source and HyWe rows. Small values: changes from base (rates in percentage points); green/red/gray: improvement/decline/switch-rate changes.

\uparrow

\downarrow

: higher/lower is better; –: unreported.

System	Size	Mode	Success $\uparrow$	Info. $\uparrow$	Op. $\uparrow$	Sw.	%Sw.
Search-based agents
XYZ-Aquila-mini ( XYZ Agentic Team, 2026 )	35B-A3B	$\mathcal{S}$	22.0	71.2	26.7	–	–
Agents-A1 ( Bai et al., 2026 )	35B-A3B	$\mathcal{S}$	13.0	70.9	23.1	–	–
Operation-based agents
GUI-Owl-1.5 ( Xu et al., 2026 )	32B	$\mathcal{O}$	7.0	34.3	20.2	–	–
Fara-7B ( Awadallah et al., 2025 )	7B	$\mathcal{O}$	13.0	37.5	35.2	–	–

Table 2: Performance on HyWeBench .

\mathcal{S}

\mathcal{O}

denote search-based/ operation-based modes. Success: tasks with all checklist items achieved (%); Info. and Op.: achieved information and operation checklist items (%), respectively. Sw.: mean switches per task; %Sw.: tasks with at least one switch (%). Bold/underlined values mark the best/second-best scores;

\uparrow

: higher is better; –: unavailable or inapplicable.

(a) Framework design
System	Mode	Succ. $\uparrow$	Info. $\uparrow$	Op. $\uparrow$
Base model, naive merging	$\mathcal{S}{+}\mathcal{O}$	11.0	36.9	15.8
Base model, HyWe	$\mathcal{S}{+}\mathcal{O}$	28.0	74.8	43.7
(b) Modes within HyWe
Base model	$\mathcal{S}$	7.0	73.8	15.8
Base model	$\mathcal{O}$	26.0	57.6	47.8

Table 3: Framework and mode ablations on HyWeBench (all

100

tasks). Mode: available tools (

\mathcal{S}

search-based,

\mathcal{O}

operation-based). Succ.: tasks with all sub-goals achieved (%); Info./Op.: completion rates of information/operation sub-goals (%). Bold: best base-model scores within each panel; HyWe-35B is a training reference.

Frontier LLMs
Model / system	Size	HLE $\uparrow$	DS-2510 $\uparrow$	BC-en $\uparrow$
GPT-5.6 Sol	–	58.0	–	90.4
Claude Fable 5	–	64.5	–	88.0
Qwen3.8-Max	2.4T	56.2	–	–
Kimi-K3	2.8T	56.0	–	91.2
Kimi-K2.6	1T	54.0	–	83.2

Table 4: Deep search performance (accuracy, %; higher is better). HLE: Humanity’s Last Exam; DS-2510: xbench-DeepSearch-2510; BC-en: BrowseComp-EN. Superscript

t

marks text-only HLE results; others use the full set; –: unreported.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Case study on one HyWeBench task with seven sub-goals. Rows 1–3 use the same base model with search-based tools only, operation-based tools only, or both tool sets under one prompt (naive merging); row 4 is HyWe-35B . Each block of a strip is one step, and all strips share one step scale. Dashed lines link key steps to their tool calls (bottom) and results (top). Texts and screenshots are taken from the evaluation runs; quotations are shortened.

Figure 4: Case study on a WebVoyager task, showing switching pattern $\mathrm{A}_{1}$ ( $\mathcal{O}\!\to\!\mathcal{S}$ for efficiency). The rows compare GUI-Owl-32B, the base model under HyWe , and HyWe-35B . The layout follows Figure 3 , and all strips share one step scale. The GUI-Owl-32B screenshots come from its raw rollout.

Figure 5: Case study on an Online-Mind2Web task, showing switching patterns $\mathrm{A}_{1}$ and $\mathrm{A}_{2}$ ( $\mathcal{S}\!\to\!\mathcal{O}$ after a search attempt falls short). The rows and layout are the same as in Figure 4 .

Figure 6: Pilot study on

100

InSTA tasks with search-based (S) and operation-based (O) runs. (a) Success overlap; circle and intersection areas are proportional to success counts. (b,c) Cumulative success rates for information-lookup and interaction-oriented tasks, classified from task descriptions and reference steps. Each denominator includes all tasks in its stratum, including failures. The five uncertain tasks appear only in (a). The curves use matching linear axes and count successful runs completed within the indicated number of steps; they summarize fixed-budget runs, not separate evaluations rerun at each budget.

Task demand	Tasks	Search-based successes	Operation-based successes
Information lookup	$52$	$43$ ( $82.7\%$ )	$35$ ( $67.3\%$ )
Interaction-oriented	$43$	$15$ ( $34.9\%$ )	$24$ ( $55.8\%$ )
Uncertain	$5$	$2$ ( $40.0\%$ )	$3$ ( $60.0\%$ )
All tasks	$100$	$60$ ( $60.0\%$ )	$62$ ( $62.0\%$ )

Appendix

Table 5: Pilot results by task requirement. Successes are counts, with success rates in parentheses. Each rate uses all tasks in its row as the denominator, including failed runs.

Benchmark	$n$	Protocol
Online-Mind2Web ( Xue et al., 2025 )	300	adapted WebJudge; o4-mini
WebVoyager ( He et al., 2024 )	643	adapted auto-evaluator; GPT-4o
HyWeBench	100	human-written tasks and info/op checklists; at least one op sub-goal per task
HLE ( Phan et al., 2025 )	2,500	reference-answer judge
xbench-DeepSearch-2510 ( Chen et al., 2025 )	100	full 2510 release; reference-answer judge
BrowseComp-EN ( Wei et al., 2025 )	1,266	reference-answer judge

Appendix

Table 6: Benchmark sizes and scoring protocols.

n

is the full benchmark size.

Pattern	Start	Working modes	Difficulty	Switching motivation
$\mathrm{A}_{1}$	$\mathcal{O}$	$\mathcal{S}$	standard (1 site)	The opening decision corrects from $\mathcal{O}$ to $\mathcal{S}$ before any browser work: the task is plainly search-answerable.
$\mathrm{A}_{2}$	$\mathcal{S}$	$\mathcal{S},\mathcal{O}$	standard (1 site)	A bounded, good-faith $\mathcal{S}$ attempt (two materially different exact queries plus one result inspection), then fall back to $\mathcal{O}$ once structural insufficiency is established.
$\mathrm{A}_{3}$	$\mathcal{S}$	$\mathcal{S},\mathcal{O}$	easy / medium / hard (1 / 2 / 3 sites)	$\mathcal{S}$ produces an entity, constraint, identifier, threshold, or shortlist that $\mathcal{O}$ requires.
$\mathrm{A}_{4}$	$\mathcal{O}$	$\mathcal{O},\mathcal{S}$	easy / medium / hard (1 / 2 / 3 sites)	$\mathcal{O}$ surfaces an entity, offer, item, metric, or status that binds the subsequent $\mathcal{S}$ question.

Appendix

Table 7: Switching patterns and difficulty levels.

\mathcal{S}

\mathcal{O}

denote search-based/operation-based modes. Start is the initial mode; Working modes lists the modes used to perform the task.

Judgement	Input	Consumption
Unified audit (two stages)	rollout, contract, final answer; screenshots for $\mathcal{O}$ phases	per-phase progress account; leakage steps $\to$ rewrite; task-success and route verdicts $\to$ filter
Leakage rewrite	leakage steps and their reasoning	reasoning replaced in place under fact/decision/coherence constraints, then re-verified
Residue detection	controller’s feedback log (which message reached which step)	second rewrite pass; trajectories with pervasive leakage dropped
Site fidelity ( $\mathrm{A}_{1}$ )	final values, observations	drop if the answer was not obtained from the named site (judged value by value)
Switch rationale ( $\mathrm{A}_{1}/\mathrm{A}_{2}$ )	rewritten reasoning at the switch step	drop if the switch was not triggered by the pattern’s intended condition
Answer coverage	answer schema, all observations	drop if a required field is unobtainable; otherwise merge fields obtained in earlier phases (numeric values re-verified against observations)

Appendix

Table 8: Trajectory review checks.

Setting	Value
Backbone	Qwen3.6-35B-A3B ( Qwen Team, 2026 )
Trainable parameters	language model; vision encoder and visual projector frozen
Objective	next-token cross-entropy
Sample unit	one segment (context rendered by the harness; targets = the segment’s assistant turns)
Context length	131,072 tokens (folded to the 64K watermark when building samples)
Reasoning tokens	retained and supervised

Appendix

Table 9: Training configuration.

Tool	Arguments	Function
Search-based tools ( $\mathcal{S}$ )
search	query (string or list)	Search the web; return ranked snippets with titles, URLs, sources, and dates.
scholar	query (string or list)	Search academic sources; return publication details, citation counts, and PDF links.
visit	url (string or list), goal , parse_type	Extract goal-relevant text from pages or PDFs; defaults to the current URL and does not change browser state.
Operation-based tools ( $\mathcal{O}$ )
goto / new_tab	url	Open a URL in the current or a new browser tab.

Appendix

Table 10: HyWe tool functions and arguments, grouped by mode. Bold arguments are required; all other arguments are optional.

Parameter	Value
Viewport	$1280\times 720$
Coordinate grid	$0$ – $1000$ (converted to pixels by the runtime)
Step budget $T_{\max}$ per attempt (operation-based)	100
Max output tokens per turn	8,192
Temperature / top- $p$ / top- $k$	1.0 / 0.95 / 20
Repetition penalty	1.05

Appendix

Table 11: HyWe runtime parameters for the base model and HyWe-35B in Table 1 .

Prototype	Interaction it captures	out	in
flight_live_fare	Live flight fare and itinerary: run a date-, route-, cabin-, and stop-constrained flight search and read the currently displayed itinerary and fare	live_offer	entity , candidate_set , location , date_window , constraint_set , identifier
stay_availability	Live lodging availability: apply destination, dates, party, amenity, and budget constraints and inspect the currently available properties	ranked_item	entity , candidate_set , location , date_window , constraint_set
tour_slot	Tour or experience slot: select an experience, date, time, language, and group size and read the available slot and price	live_offer	entity , candidate_set , location , date_window , constraint_set
car_rental_quote	Car-rental quote: enter pickup and return details plus vehicle constraints and read the available rental and full quote	quote	entity , location , date_window , constraint_set , identifier
event_ticket	Event ticket inventory: locate an event and inspect live date, section, quantity, and price inventory	live_offer	entity , candidate_set , location , date_window , constraint_set , identifier
insurance_quote	Insurance quote wizard: complete a profile-, asset-, location-, and coverage-specific quote flow and read the generated estimate	quote	entity , constraint_set , identifier , threshold , classification

Appendix

Table 12: Operation-based task prototypes

P_{\mathcal{O}}

(

48

total). out : the prototype’s output type; in : the search-based output types it accepts as inputs.

Prototype	Information need it captures	out	role
organization_background	Organization identity and background: establish who operates the relevant site or entity, where it is based, and the key dated background fact needed by the user	fact_set	consumer
entity_by_constraints	Entity identification by public constraints: identify the named entity that satisfies the public, stable constraints in the scenario	entity	producer
candidate_shortlist	Public candidate shortlist: build a short candidate set satisfying stable public criteria before live comparison	candidate_set	producer
nearest_relevant_location	Resolve a relevant location: identify the location, hub, venue, district, or waypoint that best satisfies the stated geographic constraint	location	producer
official_date_window	Resolve an official date window: determine the authoritative event, season, deadline, or eligibility dates needed to parameterize the live task	date_window	producer
requirement_derivation	Derive requirement set: derive the stable policy, technical, accessibility, dietary, or eligibility requirements that the live result must satisfy	constraint_set	producer

Appendix

Table 13: Search-based task prototypes

P_{\mathcal{S}}

(

36

total). out : the prototype’s output type. A producer supplies an operation-based input; a consumer uses an operation-based output; cross-site bridges serve both roles.

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

Web$hell: A Unified Interface for Agentic Web Access
Oct 9, 2026Sunghwan Kim, Deogyong Kim, Sangam Lee +2Web Agents LLM Agent Harnesses
Yonsei University
Back to Basics: A Skill-Centric Approach to Generalist Web Agents
Oct 26, 2026Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui +2Web Agents LLM Agent Skill Learning
University of Utah · Islamic University of Technology
MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
Apr 9, 2026Tanmay Gupta, Piper Wolters, Zixian Ma +13Web Agents LLM Agent Training
Allen Institute for AI · University of Washington · UNC-Chapel Hill

Oct 9, 2026J/K move · Enter open · S save

Web$hell: A Unified Interface for Agentic Web Access

Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon+1

Yonsei University

Web Agents LLM Agent Harnesses Long-Horizon LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are becoming direct users of the open web. As they take on increasingly complex and information-intensive tasks, effective web access is becoming a core capability for agentic problem solving. Existing approaches have advanced agentic web access through different access models, including search-augmented agents, browser-based agents, and, more recently, filesystem-based agents. Nevertheless, as users expect agents to complete long-horizon, end-to-end tasks, they increasingly need to conduct broad information gathering, perform browser actions, and revisit acquired evidence within a single trajectory. Yet our preliminary analyses reveal that existing access models struggle to meet these demands. They primarily support either information gathering or browser interaction, while combining their capabilities can involve substantial costs or performance trade-offs. To this end, we present Web$hell, an interface that unifies web access under a single access model. Using a common schema defined by WebContract, WebCompiler compiles live webpages into addressable information and action objects. These objects are maintained in WebCache and handled through a shared set of operations, providing agents with consistent, operable, and persistent web access. Through extensive experiments across web search, web navigation, and deep research benchmarks, we show that Web$hell achieves strong task performance at lower cost. Our results further indicate the importance of interface design for effective web access, with Web$hell opening new possibilities for more capable and efficient LLM agents on the web.

Abstract

Web Agents

Explore similar work

Web$hell: A Unified Interface for Agentic Web Access

Back to Basics: A Skill-Centric Approach to Generalist Web Agents

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web