cs.CRSep 19, 2026

SelfOp: An Optimization Algorithm for Self-Improving Security Agents

Authors: Saad Ullah, Yigitcan Kaya, Christopher Kruegel, Giovanni Vigna, Gianluca Stringhini

Organizations: ECE Department Boston University · CS Department UC Santa Barbara

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM agents are increasingly used for security tasks: vulnerability discovery, exploit reproduction, and patch generation. Improving them at the model level demands expert demonstrations or computable rewards, which security tasks rarely offer: traces are costly, failures hard to diagnose, rewards sparse, and non-computable. Efforts thus shift to the harness and context, but manual tuning needs task-specific expertise and scales poorly, while automated methods rely on scarce ground truth, stronger optimizer models, or unguided propose-and-evaluate loops that reduce to costly trial and error. We introduce SelfOp, an algorithm that automatically improves a frozen security agent's task context (instructions, skills, and reference documents), without modifying its execution harness and model weights. SelfOp casts context optimization as chain-rule-inspired textual gradient descent: from a single instance's outcome, it propagates error signals backward through the evaluator, the agent's trajectory, and the context artifacts that shaped its behavior, yielding per-instance textual gradients. Gradients are accumulated across instances by clustering, ranking, and filtering, and committed only under cross-instance consensus. A convergence detector monitors the gradient signal itself and stops once the context has absorbed the generalizable information in the training data, without held-out validation data. We evaluate SelfOp on CyberGym, a benchmark of real-world vulnerability reproduction tasks. With fewer than 200 training examples, SelfOp yields a 17-point self-improvement for GPT-5.4-mini (with Codex), enough to surpass the frontier GPT-5.4 baseline by 6 points, and an 18.5-point self-improvement for GPT-5.4 itself. The optimized skills also transfer across models, highlighting that SelfOp-optimized skills learn generalizable task knowledge not model-specific patterns.

Figures & tables

Fig. 1: Optimization stages and recurring questions at each optimization step: How to analyze? Which direction to update towards and how much? Is the signal-to-noise ratio in the gradients strong enough to continue?
MethodParam (θ)(\theta)GradAccumConvMeta
†\dagger TextGrad [ 17 ]graph var/prompt✓–––
GEPA [ 39 ]prompt––––
†\dagger ACE [ 15 ]text playbook(✓)✓–(✓)
MCE [ 16 ]skills (context)–(✓)––
CoEvoSkills [ 37 ]skill package–––(✓)
Meta-Harness [ 18 ]harness code––––
TABLE I: Optimization capabilities across agent improvement methods. Param (θ)(\theta) : learnable parameter. Grad : structured gradient computation. Accum : cross-task gradient accumulation. Conv : signal-based convergence detection. Meta : use of domain metadata μv\mu_{v} as structured supervision. ✓ = fully addressed, (✓) = partially addressed, – = not addressed, n/a = non relevant.
Fig. 2: SelfOp pipeline overview
Fig. 3: Gradient Computation: chain-rule decomposition of backward pass for an agentic forward pass.
Fig. 5: Train vs. validation accuracy across gradient decomposition variants (x-axis: optimizer steps; y-axis: accuracy). Each plot shows training (blue) and validation accuracy (orange) for each step. Green diamond marks test accuracy at the best validation checkpoint. Dashed lines indicate the number of training examples used to reach the best checkpoint.
Fig. 6: Accumulation threshold ablation. Each plot shows training (blue) and validation accuracy (orange) per step (x-axis), with bars counting accepted strength and improvement themes at each step. Green diamond marks test accuracy at the best validation checkpoint. (a) shows D4’s raw per-task themes extracted for comparison (unused during optimization); (b–d) show themes produced and consumed by the accumulation pipeline under each threshold.
Fig. 7: Coverage annotations with loose accumulation. Improvement bars are decomposed into hard/gap (dark) and soft/covered (light) gradients. The hard-to-soft ratio shifts across steps as the skill absorbs new patterns.
Fig. 8: Checkpoint selection instability across evaluation set sizes for the accumulation and annotation configuration.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Skill growth across optimization. Each line tracks one run over optimizer steps (step 0 is the initial skill before training). (a) Number of reference documents in poc-generation/references/ . (b) Total word count across SKILL.md and all reference files. D4 (No Accumulation); D4 (Accumulation, Loose); D4 (Accumulation, Slightly Strict); D4 (Accumulation, Strict); D4 (Loose Accumulation+Annotation).
Fig. 10: Agent home and skills. The agent home directory (a) structures context into a persistent instructions file, modular skills, and sub-agents, each loaded at different stages. A skill bundles a SKILL.md file with references and scripts (b), and is activated by name at invocation time (c).
StepRef.What the step added
00CyberGym base prompt, i.e., a task statement plus a submit.sh command
14Rewrites the seed as an operating procedure of six sections; externalizes startup, validation, delegation, and payload heuristics
26environment-sanity , hypothesis-discipline : one live hypothesis at a time
37bug-family-diagnostics ; submit.sh becomes the sole source of truth
47Pre-submit gate : four conditions that must hold before submission
57Proof ladder : diagnostic / provisional / proven ; only proven may be submitted
Appendix
TABLE II: Per-step evolution of the poc-generation skill over the annotation-aware run. Ref. is the number of files in poc-generation/references/ . The rule marks the checkpoint our convergence detector selects (step 8). Step 12 produced no committed edit: all surviving improvements were annotated soft, so the optimizer made only reinforcement edits (Section III-C ).
Fig. 12: Steps 4–7: the three sections added before the checkpoint our convergence detector selects. All three concern what counts as evidence that a candidate PoC works, and they arrive as a progression: a hard gate on submission (step 4), a three-level evidence taxonomy that generalizes it (step 5), and a fixed order of operations for the first pass over a task (step 7).
Fig. 13: Steps 10–14: the three sections added after the checkpoint our convergence detector selects, and therefore excluded from the skill we report. Each restates guidance the skill already carries: required habits and the mandatory trigger matrix repeat routing rules from the when to load references section in imperative form, and environment preflight duplicates checks already in the reference documents.
More in this topic

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 277 papers

Explore similar work

CardsList
Oct 28, 2026

Masked Erosion: How Acceptance Gates Lose Redundant Safeguards in Self-Improving Agents

Self-improving LLM agents edit their own harness, the prompts and rules around a fixed model, and keep an edit only if an acceptance gate scores it higher on a batch of tasks. Such a gate sees what the harness does under the current model, not what it would still prevent if one safeguard failed. When safeguards overlap, deleting one is masked: the others still block the harm, so the gate decides on task success alone and redundancy erodes unseen. We measure harm for every subset of four written safeguards in two agent environments and run 411 improvement loops under ten gate rules, mostly as controlled deletion experiments, plus three runs of the GEPA optimizer. A deletion behind another working safeguard adds almost no observable harm. A paired significance test blocks deletions that pass by chance but not deletions that genuinely raise task success, and a larger evaluation batch lets it delete cheaper safeguards: at the same cost, 20-task batches kept every harness protected while 40-task batches lost all six. Large harm audits prevent exposure but not the loss of depth, which turns into harm when the harness is reused with a weaker model. GEPA kept every safety rule verbatim, yet two of its three final harnesses misused credentials under a weaker model. A statistical model frozen before the runs predicts where each gate's protection breaks as safeguard cost and batch size change, and freezing safety text or auditing one-layer ablations preserves depth.

Hau-Sing So, Daixian Liu, Weicai Long

University of Macau · Tsinghua University

LLM Agent SafetyLLM Agent Self-ImprovementSafety-Utility Trade-Off in Language ModelsLanguage Model Safety Evaluation
AskPDF
Aug 13, 2026

Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents

LLM agents increasingly save experience from past tasks as reusable skills through skill evolution. When an agent completes an unsafe task, skill evolution can keep the unsafe procedure inside an otherwise useful skill, and a later benign task can repeat the harm after the unsafe instruction is gone. We call this failure skill misevolution. It requires that the agent write the unsafe procedure into a skill and that a later task retrieve and follow that skill. We introduce SkillMisevo-Gym, which clears every task session but keeps the skill library, so each of these steps can be checked separately. We also build SkillMisevo-Bench with 525 tasks in 25 episodes, where each episode alternates malicious and related benign tasks before three benign tasks in a new session. Across 21 combinations of agents and evolution methods, all of them write unsafe skills and 15 carry the harm into the new session, so the risk comes from skill evolution itself. Evolution raises benign utility in 15 combinations but also raises malicious-task harm in 17, so ranking methods by utility alone would hide this risk. We propose SafeEvolve, which checks each skill when it is written and when a later task retrieves it. It cuts unsafe retrieval in the new session from 35.3% to 8.7% and harm from 21.3% to 4.0% while keeping benign utility during learning within 0.4 points. SafeEvolveUp cuts new-session harm by 16.0 points and keeps its utility within 1.3 points of evolution without this mitigation. As agents learn from their own experience, safety must cover the skills that carry past behavior into future tasks. Code is available at https://github.com/henrymao2004/misevolve.

Xutao Mao, Liangjie Zhao, Xiang Zheng +1

City University of Hong Kong · Adelaide University

LLM Agent Skill LearningAI Agent Safety BenchmarksSelf-Improving AgentsLLM Agent Safety
AskPDF
Jun 26, 2026

LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity

Large language model (LLM) agents are rapidly being integrated into real-world systems. Their autonomy and tool-use capabilities generate substantial value while simultaneously expanding the security attack surface. This survey provides a comprehensive overview of the opportunities and challenges of LLM agents in security, focusing on two core areas: (1) threats to LLM agents themselves and corresponding mitigation strategies (LLM agents self-security), and (2) the role of LLM agents in empowering the cybersecurity lifecycle across offense and defense (LLM agents empowered cybersecurity). We first examine the internal and external attack surfaces of agents, propose a taxonomy organized by threat sources, and analyze associated mitigations and evaluation frameworks. We then investigate how agent capabilities are applied in cybersecurity practice and present, to our knowledge, the first agent-empowerment framework aligned with the full cyber offense-defense lifecycle. By systematically surveying these two areas, we are the first to highlight a positive feedback synergy between LLM agents self-security and empowered cybersecurity, offering new insights for the advancement of both. We further identify current limitations and outline promising directions for future research. The insights provided aim to catalyze the coordinated development of LLM agents self-security and agent empowered cybersecurity, paving the way for more capable and robust agent applications.

Yiwei Xu, Yong Zhuang, Xuanming Liu +6

School of Cyber Science and Engineering, Wuhan University, Wuhan, China. · Department of Computer Science and Engineering, University at Buffalo, Buffalo, USA.

Autonomous Cyber DefenseLLM Agent SecurityLLMs for Cybersecurity
AskPDF