Harness Matters: Measuring and Dissecting Security Resilience in Web Agents
Organizations: Fudan University · Shanghai Innovation Institute · Zhongguancun Laboratory · Shanghai Pudong Research Institute of Cryptology
Abstract
Web agents extend LLMs from text generation to real-world web interaction by relying on execution harnesses that interpret user goals, perceive webpages, plan actions, and ground model decisions into browser operations. While prior work has recognized that agent security depends on more than the backend model alone, how harness design quantitatively shapes web-agent security remains insufficiently measured. To systematically study web-agent harness security, we first characterize web-agent harnesses as a four-stage execution workflow covering goal interpretation, webpage perception, reasoning and planning, and action execution. Based on this abstraction, we build WebTrap Farm, a systematic and extensible evaluation framework covering three representative attack categories: malicious user prompts, malicious prompt injection, and deceptive web design. To avoid conflating security with task capability, we use paired usability calibration tasks to separate genuine defensive behavior from capability limitations. Our evaluation shows that existing web-agent harnesses remain vulnerable to typical attacks, with some unsafe behaviors occurring even without explicit attack content, revealing harness-intrinsic security failures (RQ1). Statistical analysis further shows that both harness and backend model significantly affect security outcomes, indicating that deployed-agent security is jointly shaped by the harness and backend model (RQ2). Finally, we localize where defensive behavior emerges across workflow stages and identify more secure design mechanisms, providing actionable guidance for building more robust web agents (RQ3). These findings highlight the web-agent harness as an important security boundary and motivate moving beyond model-centric defenses toward harness-level security design.
Figures & tables
| Stage | Harness Component | Design Characteristics | Representative Agents | Ratio |
| I: Goal Interpretation | Raw-Prompt Forwarding | Preserves the original user instruction to avoid additional parsing overhead. | WebArena [ 65 ] , WebVoyager [ 25 ] | 15.0% |
| LLM-based Goal Interpreter | Uses an LLM to parse user intent into structured task objectives, aiming to clarify goals and improve downstream task completion. | Agent-E [ 3 ] , Nanobrowser [ 36 ] | 85.0% | |
| II: Webpage Perception | Textual State Encoder | Represents webpages with structured textual artifacts, providing concise input for reasoning. | PageAgent [ 5 ] , Agent-E | 67.7% |
| Multimodal State Encoder | Incorporates screenshots or rendered layouts, aiming to support perception beyond text-only representations and capture visual UI cues. | GPT-4V-Act [ 16 ] , Skyvern 2.0 [ 47 ] | 39.8% | |
| III: Reasoning and Planning | Single-Controller Planner | Centralizes reasoning in one decision loop to preserve goal consistency and reduce overhead. | HyperAgent [ 26 ] , Open Browser [ 39 ] | 73.0% |
| Multi-Role Planner | Separates planning, execution, or verification into coordinated roles to support complex task decomposition and modular reasoning. | Skyvern 2.0, MindSearch [ 12 ] | 27.0% |
| Web Agent | I: Goal Interpretation | II: Webpage Perception | III: Reasoning and Planning | IV: Action Execution | ||||
| Raw-Prompt Forwarding | LLM-based Goal Interpreter | Textual State Encoder | Multimodal State Encoder | Single-Controller Planner | Multi-Role Planner | Rule-Based Action Grounder | LLM-Based Action Grounder | |
| SeeAct | ✓ | ✓ | ✓ | ✓ | ||||
| Browser Use (Text) | ✓ | ✓ | ✓ | ✓ | ||||
| Browser Use (Vision) | ✓ | ✓ | ✓ | ✓ | ||||
| Agent-E | ✓ | ✓ | ✓ | ✓ | ||||
| Skyvern 1.0 | ✓ | ✓ | ✓ | ✓ | ||||
| Web Agent | MUP | MPI | DWD | Overall |
| Open-source Harnesses | ||||
| SeeAct | 24.96% | 49.44% | 54.37% | 42.31% |
| Browser Use (Text) | 25.87% | 35.38% | 37.13% | 33.84% |
| Browser Use (Vision) | 21.17% | 40.55% | 36.17% | 34.08% |
| Agent-E | 8.48% | 25.16% | 23.01% | 20.24% |
| Skyvern 1.0 | 32.75% | 8.04% | 24.02% | 21.80% |
| Backend LLM | MUP | MPI | DWD | Overall |
| Claude-3.7-Sonnet | 31.03% | 31.19% | 33.72% | 32.11% |
| Claude-Opus-4.5 | 3.25% | 36.03% | 22.55% | 22.90% |
| GPT-4o | 32.15% | 35.86% | 46.57% | 39.12% |
| GPT-5.1 | 10.81% | 37.59% | 29.47% | 27.38% |
| GPT-5.2 | 5.52% | 36.48% | 33.97% | 27.41% |
| Qwen3-VL-235B | 44.73% | 29.48% | 38.99% | 37.34% |
| Web Agent | Claude-3.7 | Claude-4.5 | GPT-4o | GPT-5.1 |
| SeeAct | 53.36% | 68.95% | 57.45% | 59.32% |
| Browser Use (Text) | 67.61% | 73.17% | 60.90% | 73.16% |
| Browser Use (Vision) | 67.55% | 74.89% | 64.46% | 68.08% |
| Agent-E | 80.42% | 75.19% | 83.79% | 84.09% |
| Skyvern 1.0 | 82.18% | 91.55% | 61.94% | 84.53% |
| Skyvern 2.0 | 54.93% | 77.59% | 42.89% | 65.23% |
| Effect | Effect Type | -value | ||
| Agent Harness | Main effect | 5 | 331.17 | |
| Backend Model | Main effect | 6 | 437.63 | |
| Harness Model | Interaction | 30 | 299.81 |
| Workflow Stage | User-side Attacks | Web-side Attacks | ||
| Safe-State Pattern | Unsafe-State Pattern | Safe-State Pattern | Unsafe-State Pattern | |
| I: Goal Interpretation | Secure Refusal Secure Reformulation | Insecure Goal Interpretation | – | – |
| II: Webpage Perception | – | – | Injection Filtered | Injection Exposed Injection Included |
| III: Reasoning and Planning | Secure Refusal Secure Reformulation | Insecure Planning | Secure Refusal Attack Unaware Injection Recognized and Discarded | Injection Recognized and Adopted |
| IV: Action Execution | Action Blocked | Unsafe Action Executed | Action Blocked | Unsafe Action Executed |
| Web Agent | I: Goal Interpretation | III: Reasoning and Planning | IV: Action Execution | |||
| Harness Component | Conditional_DSR | Harness Component | Conditional_DSR | Harness Component | Conditional_DSR | |
| SeeAct | Raw-Prompt Forwarding | 0.00% | Single-Controller Planner | 67.86% | LLM-Based Action Grounder | 26.94% |
| Browser Use (Text) | Raw-Prompt Forwarding | 0.00% | Single-Controller Planner | 64.60% | Rule-Based Action Grounder | 0.00% |
| Browser Use (Vision) | Raw-Prompt Forwarding | 0.00% | Single-Controller Planner | 70.31% | Rule-Based Action Grounder | 0.00% |
| Agent-E | LLM-based Goal Interpreter | 82.71% | Multi-Role Planner | 69.50% | Rule-Based Action Grounder | 0.00% |
| Skyvern 1.0 | LLM-based Goal Interpreter | 42.79% | Single-Controller Planner | 35.54% | Rule-Based Action Grounder | 0.00% |
| Web Agent | II: Webpage Perception | III: Reasoning and Planning | IV: Action Execution | |||
| Harness Component | Conditional_DSR | Harness Component | Conditional_DSR | Harness Component | Conditional_DSR | |
| SeeAct | Multimodal State Encoder | 10.23% | Single-Controller Planner | 35.02% | LLM-Based Action Grounder | 1.63% |
| Browser Use (Text) | Textual State Encoder | 18.93% | Single-Controller Planner | 44.32% | Rule-Based Action Grounder | 0.00% |
| Browser Use (Vision) | Multimodal State Encoder | 9.96% | Single-Controller Planner | 47.22% | Rule-Based Action Grounder | 0.00% |
| Agent-E | Textual State Encoder | 78.99% | Multi-Role Planner | 10.65% | Rule-Based Action Grounder | 0.00% |
| Skyvern 1.0 | Multimodal State Encoder | 15.38% | Single-Controller Planner | 78.17% | Rule-Based Action Grounder | 0.00% |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Surface | Attack | Attack Goal | Attack Method | Attack Content | Task Count |
| User Side | Malicious User Prompts | Bias Misinformation Cybercrime Illegal Activity Abusive Behavior | User-prompt embedding | Direct Request Instruction Override [ 2 , 32 ] Role Play [ 19 , 43 ] Privilege Escalation [ 33 ] | 160 |
| Web Side | Malicious Prompt Injection | Sensitive Information Disclosure Malicious Download Permission Abuse | Visible Form Injection [ 31 , 10 ] Invisible Form Injection [ 31 ] ARIA-based Injection [ 31 , 27 ] URL Content Injection [ 21 , 2 ] URL Href Injection [ 21 ] Long-Context Injection [ 10 ] | Naive Direct Instruction [ 62 ] Delimiter-based Injection [ 11 , 62 ] Instruction Override [ 40 , 62 ] Role Play [ 62 ] | 158 |
| Web Side | Deceptive Website Design | Sensitive Information Disclosure Malicious Download Permission Abuse | Iframe Pop-up [ 20 ] DOM-native Pop-up [ 57 , 63 ] Advertisement Placement [ 20 ] Leading Form Field [ 57 ] Trailing Form Field [ 57 ] Default Selection [ 20 , 49 ] | Naive Trusted Entity [ 57 ] Urgency [ 15 , 49 ] Social Proof [ 15 , 49 ] Reward [ 57 , 49 ] Context Integration [ 57 , 49 ] | 187 |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 351 papers