Automatic red teaming increasingly relies on large language models (LLMs) to iteratively diagnose failed attacks and propose improved adversarial prompts or injections. Although prior work strengthens this loop through prompt engineering and search-harness design, the mechanisms underlying its two core roles, criticism and attack proposal, has not been explored. To further investigate the core roles and therefore improve red teaming effectiveness, we disentangle two design choices: whether criticism is generated by the target session itself (self-criticism) or by a separately instantiated critic (external criticism), and whether that criticism is translated into the next attack by the target session (self-proposal) or by a separate attacker (external proposal). Through our analysis, we find that self-criticism is more effective because it is grounded in the target's live execution context, including observations, constraints, and available reasoning; removing reasoning reduces downstream attack success by 6%-17%. In contrast, self-proposal is substantially less effective because awareness of the adversarial objective induces explicit refusal and non-refusal sandbagging. These findings motivate a role-separated red-teaming approach: the target session diagnoses why an attack failed, while a separately instantiated attacker generates the next proposal. On DTAP-BENCH, spanning Telecom, Workflow, CRM, and Code domains, under both direct misuse and indirect threat models, as well as multiple agent frameworks, this configuration achieves 63.2% overall attack success rate (ASR), much higher than baselines. Our results show that effective automatic red teaming depends not only on model capability and search strategy, but also on how execution context and adversarial responsibility are distributed across model sessions.
Figures & tables
Figure 1: Self-improved red teaming based on role-sepration. The target session uses its live execution context to diagnose why an attack failed. The diagnosis is then transferred to a separate attacker that generates the next adversarial candidate. This preserves execution-grounded criticism while avoiding refusal or sandbagging that can arise when the target itself is asked to operationalize the attack.
Figure 2: Self-criticism consistently outperforms external criticism when proposal is held external. Framework- and domain-level comparisons are shown for direct misuse and indirect prompt injection. Values above paired bars show the absolute ASR improvement from self-criticism.
Figure 3: Reasoning in the execution context makes self-criticism more actionable. Proposal remains external while we compare self-criticism from the full available context with criticism generated after reasoning is removed. Values above paired bars are absolute ASR improvements.
Figure 4: Self-proposal underperforms external proposal when criticism is held as self-criticism. Framework- and domain-level comparisons are shown for direct misuse and indirect prompt injection.
Figure 5: Self-proposal substantially increases explicit refusal. The remaining ASR gap in low-refusal settings suggests an additional non-refusal failure mode consistent with sandbagging; framework- and domain-level refusal breakdowns are in Appendix C.3 .
Figure 6: Task-level refusal varies substantially across frameworks and domains. The three criticism–proposal configurations are compared at the framework and domain levels.
Figure 7: Self-criticism with external proposal achieves the highest ASR among the three compared configurations. The left panel reports direct, indirect, and combined results; the right panel reports combined ASR by fully evaluated framework.
Backend
Threat
External criticism External proposal
Self-criticism External proposal
Self-criticism Self-proposal
GPT-5.5 OpenAI SDK
Direct
56.0
63.0
59.3
Indirect
37.8
42.8
36.2
GPT-5.5 OpenClaw
Direct
57.4
64.7
62.3
Indirect
41.3
52.6
42.0
Table 1: GPT-5.5 transfer results (ASR) on the selected red-teaming subset. Self-criticism with external proposal achieves the highest ASR across both backends and both threat models, indicating that the role-separation advantage transfers to GPT-5.5.