cs.AIMar 21, 2026

Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions

Authors: Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis

Organizations: Columbia University · dAI Team, Ethereum Foundation · University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fixed library of prompt attacks to adaptive strategies that steer agents toward unfavorable outcomes. We propose profit-driven red teaming, a stress-testing protocol that replaces handcrafted attacks with a learned opponent trained to maximize its profit using only scalar outcome feedback. The protocol requires no LLM-as-judge scoring, attack labels, or attack taxonomy, and is designed for structured settings with auditable outcomes. We instantiate it in a lean arena of four canonical economic interactions, which provide a controlled testbed for adaptive exploitability. In controlled experiments, agents that appear strong against static baselines become consistently exploitable under profit-optimized pressure, and the learned opponent discovers probing, anchoring, and deceptive commitments without explicit instruction. We then distill exploit episodes into concise prompt rules for the agent, which make most previously observed failures ineffective and substantially improve target performance. These results suggest that profit-driven red-team data can provide a practical route to improving robustness in structured agent settings with auditable outcomes.

Figures & tables

TargetsAs_{A} (Baseline)sAs_{A} (Red-teamed)ΔsA\Delta s_{A}95% CIpp -value
GPT-OSS-120B20.565.00+44.50[65.0, 65.0]9.5×10−309.5\times 10^{-30}
Qwen3-32B21.064.45+43.45[63.8, 65.1]3.3×10−243.3\times 10^{-24}
MiniMax-M2.521.557.85+36.35[57.54, 58.16]5.3×10−235.3\times 10^{-23}
GLM-4.622.055.25+33.25[50.6, 59.9]7.3×10−137.3\times 10^{-13}
Kimi-K221.2542.40+21.15[40.99, 43.81]2.5×10−242.5\times 10^{-24}
GPT-5.220.2539.10+18.85[25.28, 52.92]5.7×10−35.7\times 10^{-3}
Table 1: Ultimatum bargaining . Mean attacker surplus before and after profit-driven red teaming. The total resource is R=100R=100 and the two players have reservation values (cA,cB)=(30,30)(c_{A},c_{B})=(30,30) . ΔsA\Delta s_{A} is the increase in mean attacker surplus under red-teamed play. The 95% confidence interval is for ΔsA\Delta s_{A} , and the one-sided pp -value tests whether red-teamed surplus exceeds baseline surplus.
Figure 1: Attacker surplus before and after profit-driven red teaming across the four games and target models. In each setting, the optimized attacker consistently achieves higher surplus than the baseline attacker.
Figure 2: Strategy clusters discovered by the profit-optimized opponent across four games. Each point represents an episode snippet, colored by its post hoc cluster label after semantic grouping.
TargetsDs_{D} (Unhardened)sDs_{D} (Hardened)ΔsD\Delta s_{D}95% CIpp -value
GPT-OSS-120B-25.007.00+32.00[29.40, 34.60]5.2×10−165.2\times 10^{-16}
Qwen3-32B-24.455.00+29.45[29.45, 29.45]1.7×10−2981.7\times 10^{-298}
Kimi-K2-2.405.70+8.10[6.13, 10.07]7.6×10−87.6\times 10^{-8}
MiniMax-M2.5-17.851.55+19.40[19.10, 19.70]1.5×10−291.5\times 10^{-29}
GLM-4.6-15.254.10+19.35[18.63, 20.07]2.6×10−222.6\times 10^{-22}
GPT-5.2-13.109.30+22.40[20.78, 24.02]5.7×10−175.7\times 10^{-17}
Table 2: Ultimatum bargaining. Mean target surplus before and after attack distillation. ΔsD\Delta s_{D} is the increase in mean target surplus after hardening. The 95% confidence interval is for ΔsD\Delta s_{D} , and the one-sided pp -value tests whether hardened surplus exceeds unhardened surplus.
Figure 3: Target surplus before and after attack distillation across the four games and target models.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Short nameOpenRouter identifier
GPT-OSS-120Bopenai/gpt-oss-120b:nitro
Qwen3-32Bqwen/qwen3-32b:nitro
Kimi-K2moonshotai/kimi-k2-0905:nitro
MiniMax-M2.5minimax/minimax-m2.5:nitro
GLM-4.6z-ai/glm-4.6:nitro
GPT-5.2openai/gpt-5.2
Appendix
Table 3: Model short names used in tables and corresponding OpenRouter identifiers
GameTargetsAs_{A} (Baseline)sAs_{A} (Red-teamed)ΔsA\Delta s_{A}95% CIpp -value
First-price auctionGPT-OSS-120B12.229.15+16.95[27.58, 30.72]6.8×10−96.8\times 10^{-9}
 Qwen3-32B14.4920.49+6.00[15.19, 25.79]2.3×10−22.3\times 10^{-2}
 MiniMax-M2.56.824.05+17.25[20.75, 27.35]4.8×10−94.8\times 10^{-9}
 GLM-4.69.1527.60+18.45[25.71, 29.49]2.9×10−122.9\times 10^{-12}
 Kimi-K24.4527.35+22.90[24.56, 30.14]2.4×10−142.4\times 10^{-14}
 GPT-5.27.6216.80+9.18[11.52, 22.08]3.0×10−33.0\times 10^{-3}
Appendix
Table 4: Additional games. Mean attacker surplus before and after profit-driven red teaming across first-price auction, bilateral trade, and provision-point. ΔsA\Delta s_{A} is the increase in mean attacker surplus under red-teamed play. The 95% confidence interval is for ΔsA\Delta s_{A} , and the one-sided pp -value tests whether red-teamed surplus exceeds baseline surplus.
More in this topic

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Explore similar work

CardsList

  1. Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours

    May 5, 2026Raja Sekhar Rao Dheekonda, Will Pearce, Nick LandersAdversarial Attacks on LLMsLLM Red Teaming

    1Dreadnode, USA

  2. CART: Closed-Loop Adaptive Red Teaming for Large Language Models

    Sep 23, 2026Dongdong Zhang, Tengchao Lv, Yilin Jia +9Automated Red TeamingLanguage Model Safety Evaluation

    Microsoft Research

  3. RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution

    Aug 27, 2026Junjie Zhang, Hui Liu, Kecheng Chen +3Automated Red TeamingLLM Agent Security

    City University of Hong Kong · Shenzhen MSU-BIT University