cs.CROct 19, 2026

AutoRedClaw: Automated Red-Teaming for Claw-Style Agents

Authors: Hao Li, Tongyan Hu, Weidi Luo, Shanghao Shi, Xiaogeng Liu, Aiden Suh, Yulong Cao, Ning Zhang, 

Organizations: Washington University in St. Louis · Johns Hopkins University · National University of Singapore · University of Georgia · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Agent systems are shifting from task-scoped tool-use agents to persistent, personalized assistants such as OpenClaw. These claw-style agents maintain configuration files (e.g., AGENTS.md), long-term memory, and reusable skills. Once poisoned, this persistent state is repeatedly reloaded into the agent's context, creating persistent threats across tasks and sessions. Existing security evaluation systems struggle to evaluate these complex claw-style systems since they rely heavily on manually constructed test cases and environments, which limits their scale and coverage; in addition, these manually designed cases quickly become outdated as agents evolve, and their evaluation settings leave a gap with how claw-style agents are deployed in practice. To address this, we introduce AutoRedClaw, an automated red-teaming framework that generates end-to-end attacks together with the environments needed to execute them. AutoRedClaw combines a strict quality-checking protocol with execution-based validation to ensure the correctness, realism, and effectiveness of generated samples, and adopts task-driven environment construction to keep costs low. To achieve broad coverage, it organizes generation along risk categories, attack strategies, attack stages, and attack surfaces, and uses experience-guided exploration to steer generation toward under-explored regions. AutoRedClaw achieves an effective attack rate above 30% at less than $0.10 per successful sample. From its outputs, we construct RedClaw-Bench, a 330-case benchmark that yields substantially higher attack success rates (ASRs) than AgentDojo, AgentDyn, and DTap. Across seven harnesses and four frontier models, most agent systems exceed 50% ASR, showing that our red-teaming pipeline provides a strong evaluation signal even for the most advanced agent systems. Our code and benchmark are released at https://github.com/SaFo-Lab/AutoRedClaw.

Figures & tables

Figure 1: Agent systems transfer from tool-use agents to persistent personalized agents.
Figure 2: Overview of the AutoRedClaw pipeline. AutoRedClaw generates task and attack metadata, materializes the required environments, and validates the resulting cases to construct a red-teaming dataset. Harness adapters enable evaluation of diverse claw-style agents using the generated dataset.
Figure 3: Left: Collected Successful Attacks versus Generated Samples. Right: Coverage of Risk Category ×\times Strategy Combinations versus Generated Samples.
 GPT-5.4Claude-Sonnet-4.5Gemini-3.1-proDeepSeek V4 ProAverage
HarnessUUASRUUASRUUASRUUASRUUASR
Inject: establishment of persistent poisoning
OpenClaw76.064.776.062.082.754.779.381.378.565.7
Nanobot78.752.773.356.786.769.382.078.080.264.2
NemoClaw68.737.376.060.0--77.371.374.056.2
SeClaw39.39.350.06.066.016.756.020.052.813.0
Table 1: Utility ( UU ) and attack success rate (ASR) on RedClaw-Bench (%). “-” indicates that the results are inapplicable. NemoClaw–Gemini results are unavailable because the Gemini compatibility plugin causes OpenClaw’s SSRF checks to block the internal inference endpoint. For the two coding agents, we evaluate their natively supported model providers.
Figure 4: Vulnerability profiles of five agent harnesses across dimensions. Each curve reports the ASR (%) averaged across GPT-5.4, Claude-Sonnet-4.5, and Gemini-3.1-Pro.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ScenarioTask Examples
EmailSend email or update mail client settings.
Contact ManagementImport, export, or register contacts in an application.
Employee ManagementUpdate employee profiles, directories, or leave requests.
Support TicketsSubmit service requests and retrieve ticket records.
Bug ReportsReport login timeouts or a stalled sign-in page.
Project ManagementPublish project updates and team status reports.
Appendix
Table 2: The 25 functional scenarios covered by RedClaw-Bench, with representative benign tasks.
Figure 5: Distribution of RedClaw-Bench across Risk Categories and Attack Strategies.
Figure 6: The comparison of existing benchmarks.
 GPT-5.4Claude-Sonnet-4.5Gemini-3.1-proDeepSeek V4 Pro
SurfaceUUASRUUASRUUASRUUASR
OpenClaw
AGENTS.md76.770.080.070.083.356.776.786.7
MEMORY.md86.770.090.063.390.053.390.073.3
USER.md76.766.790.073.386.756.790.083.3
SOUL.md73.376.776.780.076.756.783.386.7
Appendix
Table 3: Detailed Inject results (establishment of persistent poisoning) on the frozen benchmark (%). Each evaluated surface contains 30 cases. UU denotes utility under attack. Dashes indicate unavailable or inapplicable results.
 GPT-5.4Claude-Sonnet-4.5Gemini-3.1-proDeepSeek V4 Pro
SurfaceUUASRUUASRUUASRUUASR
OpenClaw
AGENTS.md53.360.083.386.776.786.786.796.7
MEMORY.md50.043.376.770.076.780.076.793.3
USER.md66.756.796.746.793.366.793.390.0
SOUL.md16.720.063.370.063.380.053.373.3
Appendix
Table 4: Detailed Trigger results (downstream attacker actions) on the frozen benchmark (%). Each evaluated surface contains 30 cases. UU denotes utility under attack. Dashes indicate unavailable or inapplicable results.
HarnessModelUUASR
CodeXGPT-5.470.023.3
Claude CodeClaude-Sonnet-4.536.716.7
Appendix
Table 5: Utility ( UU ) and ASR (%) of the two coding agents on the standard IPI cases of RedClaw-Bench.
Figure 7: An example skill injection task generated by AutoRedClaw.
Figure 8: Eight example application environments generated by AutoRedClaw.
More in this topic

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 510 papers

Explore similar work

CardsList
Mar 19, 2026

ClawTrap: A MITM-Based Red-Teaming Framework for Real-World OpenClaw Security Evaluation

Autonomous web agents such as OpenClaw are rapidly moving into high-impact real-world workflows, but their security robustness under live network threats remains insufficiently evaluated. Existing benchmarks mainly focus on static sandbox settings and content-level prompt attacks, which leaves a practical gap for network-layer security testing. In this paper, we present ClawTrap, a MITM-based red-teaming framework for real-world OpenClaw security evaluation. ClawTrap supports diverse and customizable attack forms, including Static HTML Replacement, Iframe Popup Injection, and Dynamic Content Modification, and provides a reproducible pipeline for rule-driven interception, transformation, and auditing. This design lays the foundation for future research to construct richer, customizable MITM attacks and to perform systematic security testing across agent frameworks and model backbones. Our empirical study shows clear model stratification: weaker models are more likely to trust tampered observations and produce unsafe outputs, while stronger models demonstrate better anomaly attribution and safer fallback strategies. These findings indicate that reliable OpenClaw security evaluation should explicitly incorporate dynamic real-world MITM conditions rather than relying only on static sandbox protocols.

Haochen Zhao, Shaoyang Cui

School of Computing National University of Singapore Singapore · Department of Psychological and Cognitive Sciences Tsinghua University Beijing, China

AI Agent SecurityWeb AgentsAI Agent Security BenchmarksAutomated Red Teaming
AskPDF
Jun 29, 2026

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services. They take on system-level responsibilities -- installing packages, maintaining state, scheduling subtasks, and mediating I/O -- making security failures far more severe than in other agents. Yet existing benchmarks focus on model responses and tool calls, leaving cross-component failure modes largely unmeasured. We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role, whose Skills resemble user-installed applications, and whose Plugins resemble loadable extensions with runtime privileges. Each component has a classical counterpart whose protection mechanisms -- refined over decades of cybersecurity research -- are absent on the agent side. From this perspective, we develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces (Skill Supply-Chain Integrity, Persistent State Exploitation, Cross-Boundary Data Flow, and Indirect Prompt Injection), executed in containerized replicas of real agent platforms with canary-marked credentials and evaluated via automated taint tracking across nine output channels. We evaluate three platforms (OpenClaw, NemoClaw, SeClaw) and five frontier LLMs. The highest attack success rate reaches 70%; malicious Plugins succeed in 100% of cases regardless of the LLM. SeClaw cuts GPT-5.4's attack success rate from 70% to 22%, partly through utility-security tradeoffs rather than active defenses, while Claude-Opus-4.6 already sits near a 22% floor on every platform. These results expose the inadequacy of current defenses and suggest directions for future hardening. Code and data: https://github.com/sunblaze-ucb/SafeClawArena.

Peizhi Niu, Wenjie Qu, Shangding Gu +14

UIUC · NUS · UC Berkeley +8

AI Agent SecurityLLM Agent SecurityAI Agent Security BenchmarksIndirect Prompt InjectionAgent Skill Security Auditing
AskPDF
Apr 3, 2026

A Systematic Security Evaluation of OpenClaw and Its Variants

Tool-augmented AI agents substantially extend the practical capabilities of large language models, but they also introduce security risks that cannot be identified through model-only evaluation. In this paper, we present a systematic security assessment of six representative OpenClaw-series agent frameworks, namely OpenClaw, AutoClaw, QClaw, KimiClaw, MaxClaw, and ArkClaw, under multiple backbone models. To support this study, we construct a benchmark of 205 test cases covering representative attack behaviors across the full agent execution lifecycle, enabling unified evaluation of risk exposure at both the framework and model levels. Our results show that all evaluated agents exhibit substantial security vulnerabilities, and that agentized systems are significantly riskier than their underlying models used in isolation. In particular, reconnaissance and discovery behaviors emerge as the most common weaknesses, while different frameworks expose distinct high-risk profiles, including credential leakage, lateral movement, privilege escalation, and resource development. These findings indicate that the security of modern agent systems is shaped not only by the safety properties of the backbone model, but also by the coupling among model capability, tool use, multi-step planning, and runtime orchestration. We further show that once an agent is granted execution capability and persistent runtime context, weaknesses arising in early stages can be amplified into concrete system-level failures. Overall, our study highlights the need to move beyond prompt-level safeguards toward lifecycle-wide security governance for intelligent agent frameworks.

Yuhang Wang, Haichang Gao, Zhenxing Niu +4

Xidian University · Data Science & Artificial Intelligence Research Institute, China Unicom

AI Agent SecurityLLM Agent SecurityAI Agent Security Benchmarks
AskPDF