HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks
Organizations: Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, CASIA · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · Alibaba Group ·
Abstract
Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluated separately. To bridge this gap, we introduce HyWe, a framework that unifies both capabilities within one model and trajectory. Its two modes use dedicated prompts and tool sets to sustain sub-goal execution; explicit switching selects the needed capability, and shared history lets each mode build on the other's findings. To provide demonstrations of these behaviors, we develop HyWeCollector, a fully automatic pipeline that uses task prototypes, four switching patterns, and multi-agent synthesis to produce verified hybrid trajectories. We train HyWe-35B on these demonstrations to unify efficient information gathering and browser interaction in a single 35B web agent. To evaluate hybrid task completion, we introduce HyWeBench: 100 human-written tasks on websites distinct from training, with human-written sub-goal checklists and an outcome-based evaluation pipeline. Hybrid training enables HyWe-35B to achieve 47.0% success on this benchmark, improving on the untrained base model by 19.0 percentage points and outperforming existing search-based and operation-based agents. Even before training, the same base model solves 28.0% of tasks with HyWe, versus 11.0% with all tools under one prompt. Beyond hybrid tasks, HyWe-35B achieves 59.0% and 88.3% success on Online-Mind2Web and WebVoyager, outperforming existing open-source baselines and cutting action cost per successful task by 48.3% and 51.9% compared with Fara-7B.
Figures & tables
| Model / system | Size | Online-Mind2Web | WebVoyager | ||||
|---|---|---|---|---|---|---|---|
| SR | Steps/SR | Switch | SR | Steps/SR | Switch | ||
| Frontier LLMs | |||||||
| GPT-5.5 | – | 74.7 | – | – | 85.1 | – | – |
| GPT-5 (SoM) | – | 57.7 | – | – | 90.6 | – | – |
| o3 (SoM) | – | 55.4 | – | – | 79.3 | – | – |
| Claude-Opus-4.6 | – | – | – | – | 88.0 | – | – |
| System | Size | Mode | Success | Info. | Op. | Sw. | %Sw. |
| Search-based agents | |||||||
| XYZ-Aquila-mini ( XYZ Agentic Team, 2026 ) | 35B-A3B | 22.0 | 71.2 | 26.7 | – | – | |
| Agents-A1 ( Bai et al., 2026 ) | 35B-A3B | 13.0 | 70.9 | 23.1 | – | – | |
| Operation-based agents | |||||||
| GUI-Owl-1.5 ( Xu et al., 2026 ) | 32B | 7.0 | 34.3 | 20.2 | – | – | |
| Fara-7B ( Awadallah et al., 2025 ) | 7B | 13.0 | 37.5 | 35.2 | – | – | |
| System | Mode | Succ. | Info. | Op. |
|---|---|---|---|---|
| (a) Framework design | ||||
| Base model, naive merging | 11.0 | 36.9 | 15.8 | |
| Base model, HyWe | 28.0 | 74.8 | 43.7 | |
| (b) Modes within HyWe | ||||
| Base model | 7.0 | 73.8 | 15.8 | |
| Base model | 26.0 | 57.6 | 47.8 | |
| Model / system | Size | HLE | DS-2510 | BC-en |
|---|---|---|---|---|
| Frontier LLMs | ||||
| GPT-5.6 Sol | – | 58.0 | – | 90.4 |
| Claude Fable 5 | – | 64.5 | – | 88.0 |
| Qwen3.8-Max | 2.4T | 56.2 | – | – |
| Kimi-K3 | 2.8T | 56.0 | – | 91.2 |
| Kimi-K2.6 | 1T | 54.0 | – | 83.2 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Task demand | Tasks | Search-based successes | Operation-based successes |
|---|---|---|---|
| Information lookup | ( ) | ( ) | |
| Interaction-oriented | ( ) | ( ) | |
| Uncertain | ( ) | ( ) | |
| All tasks | ( ) | ( ) |
| Benchmark | Protocol | |
|---|---|---|
| Online-Mind2Web ( Xue et al., 2025 ) | 300 | adapted WebJudge; o4-mini |
| WebVoyager ( He et al., 2024 ) | 643 | adapted auto-evaluator; GPT-4o |
| HyWeBench | 100 | human-written tasks and info/op checklists; at least one op sub-goal per task |
| HLE ( Phan et al., 2025 ) | 2,500 | reference-answer judge |
| xbench-DeepSearch-2510 ( Chen et al., 2025 ) | 100 | full 2510 release; reference-answer judge |
| BrowseComp-EN ( Wei et al., 2025 ) | 1,266 | reference-answer judge |
| Pattern | Start | Working modes | Difficulty | Switching motivation |
|---|---|---|---|---|
| standard (1 site) | The opening decision corrects from to before any browser work: the task is plainly search-answerable. | |||
| standard (1 site) | A bounded, good-faith attempt (two materially different exact queries plus one result inspection), then fall back to once structural insufficiency is established. | |||
| easy / medium / hard (1 / 2 / 3 sites) | produces an entity, constraint, identifier, threshold, or shortlist that requires. | |||
| easy / medium / hard (1 / 2 / 3 sites) | surfaces an entity, offer, item, metric, or status that binds the subsequent question. |
| Judgement | Input | Consumption |
|---|---|---|
| Unified audit (two stages) | rollout, contract, final answer; screenshots for phases | per-phase progress account; leakage steps rewrite; task-success and route verdicts filter |
| Leakage rewrite | leakage steps and their reasoning | reasoning replaced in place under fact/decision/coherence constraints, then re-verified |
| Residue detection | controller’s feedback log (which message reached which step) | second rewrite pass; trajectories with pervasive leakage dropped |
| Site fidelity ( ) | final values, observations | drop if the answer was not obtained from the named site (judged value by value) |
| Switch rationale ( ) | rewritten reasoning at the switch step | drop if the switch was not triggered by the pattern’s intended condition |
| Answer coverage | answer schema, all observations | drop if a required field is unobtainable; otherwise merge fields obtained in earlier phases (numeric values re-verified against observations) |
| Setting | Value |
|---|---|
| Backbone | Qwen3.6-35B-A3B ( Qwen Team, 2026 ) |
| Trainable parameters | language model; vision encoder and visual projector frozen |
| Objective | next-token cross-entropy |
| Sample unit | one segment (context rendered by the harness; targets = the segment’s assistant turns) |
| Context length | 131,072 tokens (folded to the 64K watermark when building samples) |
| Reasoning tokens | retained and supervised |
| Tool | Arguments | Function |
| Search-based tools ( ) | ||
| search | query (string or list) | Search the web; return ranked snippets with titles, URLs, sources, and dates. |
| scholar | query (string or list) | Search academic sources; return publication details, citation counts, and PDF links. |
| visit | url (string or list), goal , parse_type | Extract goal-relevant text from pages or PDFs; defaults to the current URL and does not change browser state. |
| Operation-based tools ( ) | ||
| goto / new_tab | url | Open a URL in the current or a new browser tab. |
| Parameter | Value |
|---|---|
| Viewport | |
| Coordinate grid | – (converted to pixels by the runtime) |
| Step budget per attempt (operation-based) | 100 |
| Max output tokens per turn | 8,192 |
| Temperature / top- / top- | 1.0 / 0.95 / 20 |
| Repetition penalty | 1.05 |
| Prototype | Interaction it captures | out | in |
|---|---|---|---|
| flight_live_fare | Live flight fare and itinerary: run a date-, route-, cabin-, and stop-constrained flight search and read the currently displayed itinerary and fare | live_offer | entity , candidate_set , location , date_window , constraint_set , identifier |
| stay_availability | Live lodging availability: apply destination, dates, party, amenity, and budget constraints and inspect the currently available properties | ranked_item | entity , candidate_set , location , date_window , constraint_set |
| tour_slot | Tour or experience slot: select an experience, date, time, language, and group size and read the available slot and price | live_offer | entity , candidate_set , location , date_window , constraint_set |
| car_rental_quote | Car-rental quote: enter pickup and return details plus vehicle constraints and read the available rental and full quote | quote | entity , location , date_window , constraint_set , identifier |
| event_ticket | Event ticket inventory: locate an event and inspect live date, section, quantity, and price inventory | live_offer | entity , candidate_set , location , date_window , constraint_set , identifier |
| insurance_quote | Insurance quote wizard: complete a profile-, asset-, location-, and coverage-specific quote flow and read the generated estimate | quote | entity , constraint_set , identifier , threshold , classification |
| Prototype | Information need it captures | out | role |
|---|---|---|---|
| organization_background | Organization identity and background: establish who operates the relevant site or entity, where it is based, and the key dated background fact needed by the user | fact_set | consumer |
| entity_by_constraints | Entity identification by public constraints: identify the named entity that satisfies the public, stable constraints in the scenario | entity | producer |
| candidate_shortlist | Public candidate shortlist: build a short candidate set satisfying stable public criteria before live comparison | candidate_set | producer |
| nearest_relevant_location | Resolve a relevant location: identify the location, hub, venue, district, or waypoint that best satisfies the stated geographic constraint | location | producer |
| official_date_window | Resolve an official date window: determine the authoritative event, season, deadline, or eligibility dates needed to parameterize the live task | date_window | producer |
| requirement_derivation | Derive requirement set: derive the stable policy, technical, accessibility, dietary, or eligibility requirements that the live result must satisfy | constraint_set | producer |
Web Agents
Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.
See all 174 papers