cs.AIMar 21, 2026

Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions

Authors: Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis

Organizations: Columbia University · dAI Team, Ethereum Foundation · University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fixed library of prompt attacks to adaptive strategies that steer agents toward unfavorable outcomes. We propose profit-driven red teaming, a stress-testing protocol that replaces handcrafted attacks with a learned opponent trained to maximize its profit using only scalar outcome feedback. The protocol requires no LLM-as-judge scoring, attack labels, or attack taxonomy, and is designed for structured settings with auditable outcomes. We instantiate it in a lean arena of four canonical economic interactions, which provide a controlled testbed for adaptive exploitability. In controlled experiments, agents that appear strong against static baselines become consistently exploitable under profit-optimized pressure, and the learned opponent discovers probing, anchoring, and deceptive commitments without explicit instruction. We then distill exploit episodes into concise prompt rules for the agent, which make most previously observed failures ineffective and substantially improve target performance. These results suggest that profit-driven red-team data can provide a practical route to improving robustness in structured agent settings with auditable outcomes.

Figures & tables

TargetsAs_{A} (Baseline)sAs_{A} (Red-teamed)ΔsA\Delta s_{A}95% CIpp -value
GPT-OSS-120B20.565.00+44.50[65.0, 65.0]9.5×10−309.5\times 10^{-30}
Qwen3-32B21.064.45+43.45[63.8, 65.1]3.3×10−243.3\times 10^{-24}
MiniMax-M2.521.557.85+36.35[57.54, 58.16]5.3×10−235.3\times 10^{-23}
GLM-4.622.055.25+33.25[50.6, 59.9]7.3×10−137.3\times 10^{-13}
Kimi-K221.2542.40+21.15[40.99, 43.81]2.5×10−242.5\times 10^{-24}
GPT-5.220.2539.10+18.85[25.28, 52.92]5.7×10−35.7\times 10^{-3}
Table 1: Ultimatum bargaining . Mean attacker surplus before and after profit-driven red teaming. The total resource is R=100R=100 and the two players have reservation values (cA,cB)=(30,30)(c_{A},c_{B})=(30,30) . ΔsA\Delta s_{A} is the increase in mean attacker surplus under red-teamed play. The 95% confidence interval is for ΔsA\Delta s_{A} , and the one-sided pp -value tests whether red-teamed surplus exceeds baseline surplus.
Figure 1: Attacker surplus before and after profit-driven red teaming across the four games and target models. In each setting, the optimized attacker consistently achieves higher surplus than the baseline attacker.
Figure 2: Strategy clusters discovered by the profit-optimized opponent across four games. Each point represents an episode snippet, colored by its post hoc cluster label after semantic grouping.
TargetsDs_{D} (Unhardened)sDs_{D} (Hardened)ΔsD\Delta s_{D}95% CIpp -value
GPT-OSS-120B-25.007.00+32.00[29.40, 34.60]5.2×10−165.2\times 10^{-16}
Qwen3-32B-24.455.00+29.45[29.45, 29.45]1.7×10−2981.7\times 10^{-298}
Kimi-K2-2.405.70+8.10[6.13, 10.07]7.6×10−87.6\times 10^{-8}
MiniMax-M2.5-17.851.55+19.40[19.10, 19.70]1.5×10−291.5\times 10^{-29}
GLM-4.6-15.254.10+19.35[18.63, 20.07]2.6×10−222.6\times 10^{-22}
GPT-5.2-13.109.30+22.40[20.78, 24.02]5.7×10−175.7\times 10^{-17}
Table 2: Ultimatum bargaining. Mean target surplus before and after attack distillation. ΔsD\Delta s_{D} is the increase in mean target surplus after hardening. The 95% confidence interval is for ΔsD\Delta s_{D} , and the one-sided pp -value tests whether hardened surplus exceeds unhardened surplus.
Figure 3: Target surplus before and after attack distillation across the four games and target models.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Short nameOpenRouter identifier
GPT-OSS-120Bopenai/gpt-oss-120b:nitro
Qwen3-32Bqwen/qwen3-32b:nitro
Kimi-K2moonshotai/kimi-k2-0905:nitro
MiniMax-M2.5minimax/minimax-m2.5:nitro
GLM-4.6z-ai/glm-4.6:nitro
GPT-5.2openai/gpt-5.2
Appendix
Table 3: Model short names used in tables and corresponding OpenRouter identifiers
GameTargetsAs_{A} (Baseline)sAs_{A} (Red-teamed)ΔsA\Delta s_{A}95% CIpp -value
First-price auctionGPT-OSS-120B12.229.15+16.95[27.58, 30.72]6.8×10−96.8\times 10^{-9}
 Qwen3-32B14.4920.49+6.00[15.19, 25.79]2.3×10−22.3\times 10^{-2}
 MiniMax-M2.56.824.05+17.25[20.75, 27.35]4.8×10−94.8\times 10^{-9}
 GLM-4.69.1527.60+18.45[25.71, 29.49]2.9×10−122.9\times 10^{-12}
 Kimi-K24.4527.35+22.90[24.56, 30.14]2.4×10−142.4\times 10^{-14}
 GPT-5.27.6216.80+9.18[11.52, 22.08]3.0×10−33.0\times 10^{-3}
Appendix
Table 4: Additional games. Mean attacker surplus before and after profit-driven red teaming across first-price auction, bilateral trade, and provision-point. ΔsA\Delta s_{A} is the increase in mean attacker surplus under red-teamed play. The 95% confidence interval is for ΔsA\Delta s_{A} , and the one-sided pp -value tests whether red-teamed surplus exceeds baseline surplus.
More in this topic

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Explore similar work

CardsList
May 5, 2026

Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours

AI systems are entering critical domains like healthcare, finance, and defense, yet remain vulnerable to adversarial attacks. While AI red teaming is a primary defense, current approaches force operators into manual, library-specific workflows. Operators spend weeks hand-crafting workflows - assembling attacks, transforms, and scorers. When results fall short, workflows must be rebuilt. As a result, operators spend more time constructing workflows than probing targets for security and safety vulnerabilities. We introduce an AI red teaming agent built on the open-source Dreadnode SDK. The agent creates workflows grounded in 45+ adversarial attacks, 450+ transforms, and 130+ scorers. Operators can probe multi-agent systems, multilingual, and multimodal targets, focusing on what to probe rather than how to implement it. We make three contributions: 1. Agentic interface. Operators describe goals in natural language via the Dreadnode TUI (Terminal User Interface). The agent handles attack selection, transform composition, execution, and reporting, letting operators focus on red teaming. Weeks compress to hours. 2. Unified framework. A single framework for probing traditional ML models (adversarial examples) and generative AI systems (jailbreaks), removing the need for separate libraries. 3. Llama Scout case study. We red team Meta Llama Scout and achieve an 85% attack success rate with severity up to 1.0, using zero human-developed code

Raja Sekhar Rao Dheekonda, Will Pearce, Nick Landers

1Dreadnode, USA

Adversarial Attacks on LLMsLLM Red TeamingAI Agent SecurityAutomated Red Teaming
AskPDF
Sep 23, 2026

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

Automated red teaming often replays a fixed set of prompts, which measures known risks but cannot learn from failures found during testing. We present CART (Closed-Loop Adaptive Red Teaming), a framework that uses each result to guide what it tests next. CART begins with broad risk coverage, follows weaknesses that emerge, keeps new probes diverse, and records the evidence and source of every finding. It separates the Challenger that creates tests, the Target being tested, which may be a text-only model or a bounded tool-using agent, and the Judge that evaluates the results, allowing these roles to be studied independently. Across three evaluation families (Frontier, JAH, and Agentic), CART discovers more failures and higher average risk than static seed replay for every Target with an available baseline. The gains extend to tool-mediated agent tests, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise. These results describe what the test policies discover, not how often failures occur in real deployments. We also find that Challenger-Judge choices affect the evidence uncovered, highlighting the need for role separation and independent review. Overall, CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses.

Dongdong Zhang, Tengchao Lv, Yilin Jia +9

Microsoft Research

Automated Red TeamingLanguage Model Safety EvaluationAI Agent Auditing
AskPDF
Aug 27, 2026

RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution

LLM-based agents are increasingly deployed in product-level execution harnesses, where jailbreaks can trigger harmful tool use and persistent state changes, creating greater risks than unsafe text generation alone. Existing automatic red-teaming methods often rely on fixed attacks, while recent agentic attackers coordinate multiple jailbreak tools and show stronger potential through trajectory-based retrieval. However, such retrieval can reuse misleading experiences due to retrieval bias and unclear tool credit, and full trajectories add context overhead while reducing interpretability. We propose RedEvoAgent, a black-box red-teaming agent that distills cross-case attack trajectories into a concise, human-readable attack skill. The attack skill adaptively evolves through tool-effectiveness profiling and Deciding-Tool Attribution for skill updates, and a validation ratchet that retains only updates improving validation performance. Experiments on multiple benchmarks, target models, and target execution harnesses show that RedEvoAgent outperforms fixed and agentic baselines, improves tool efficiency, and transfers across attacker models and target execution harnesses.

Junjie Zhang, Hui Liu, Kecheng Chen +3

City University of Hong Kong · Shenzhen MSU-BIT University

Automated Red TeamingLLM Agent SecurityLLM Jailbreak AttacksLLM Agent Skill LearningSelf-Evolving Agents
AskPDF